Transcript for "Slop or Not: Pangram's Max Spero on Authenticity in an AI World": Hello everyone welcome. Thank you so much for joining us today. Happy Thursday end of end of August. We're already here for those of you who have never joined our webinars before, my name is Shelby Bridges. I am the head of user success here at Quoted. This webinar is a part of our ongoing Quoted Webinar Series, where our goal is to really give you a look and insights into what's happening across the media and PR industry as ever changing as it is. So our goal is to not just provide theory, but really real data around the conversations with the goal of equipping you with ideas, tools, anything that you can integrate into your day to day work. So the great news is for everyone that this is not a quoted sales pitch at the end of the day. I will be very candid that I think I am maybe the most nervous about this webinar so far. And I think that's because today's topic has become increasingly difficult to ignore, but also feels very personal for people when we start talking about AI likelihood, the use of AI. Philosophically, it feels like people think maybe we're all against AI. And I and I really wanna have that conversation. I want us to be able to open up this floor for QA and be able to talk about all of these concerns really directly. So that's where I feel like, you know, day to day we get grilled pretty heavily here at Quoted. And so that's why we thought let's bring in our guest Max to be able to talk about this. As some of you may know when deciding to integrate an AI detection tool into our Quoted platform, The goal is really not to accuse anybody of being a bot. It's really to give our players on both sides of the network level playing field to see what your content is looking like before you're submitting it to a journalist and vice versa. And what do we feel comfortable using in stories when we're talking with journalists as well? So the relationship, with AI journalist sources, it's all ever changing. I recently worked with Suman Bactaria on an article with Fast Company, and I think he summarized this so well where he was saying, AI detection tools themselves are not a verdict. They're just a signal. And so that's really what I want you all to keep in mind today as we're going through this and we're having these conversations. The final judgment is not an accusation. It's not a judgment call on your character. It's helping everybody make smarter decisions, really understand who it is that they're engaging with. So these, these tools as we're going to discuss are one of a few different ways that journalists are ultimately going to decide whether or not they feel comfortable working with you or working with that story or expert for, or expert source for a story. So for today's structure, of our call a little bit different than normal. What we did was we wanted to take a look at three different angles for every angle we've pulled data from journalists that we surveyed across the board. We pulled data from the quoted backside of what we see for analytics, how we see this engaging in our network. And then we also have Max here to answer those really hard hitting questions as it relates and ties into the PanGram integration. So first we're going to look at the scale of the problem. How often are journalists actually encounter encountering those AI generated pitches? Second, we're gonna get a little bit more uncomfortable and we're gonna talk about who's actually driving it and what is driving it. What's that largest audience using AI in pitch submissions. Lastly, we're going to get into the instinct gap, perhaps maybe the most interesting part as that target is going to be ever moving as we all know as AI and everything continues to adapt and change. So each section, as I mentioned, has real survey data, has real data from quoted in the analytics that we've studied. And then of course, Max, he's going to be our go to expert on PanGram itself. So before we get started, of course, you guys know I love a good housekeeping. And so before we get started today, our audience a little bit different. We've expanded our audience today. So we have our normal PR agencies representatives in here. We do have experts and sources who are representing themselves within the quoted platform or maybe you're not quoted users at all. That is totally okay. And then lastly, we also have media users who are joining us today. We felt that this topic was really inclusive across all parties. And so we've got a little bit of everybody and that's where we wanna open it up, make sure that we have plenty of time for that q and a at the end. So as we're going throughout this conversation, we have questions already geared in specific for Max. So some of those questions might come up, but also feel free to pop those into that Q and A tab that you see in the browser page. If you see a question you really wanna make sure that we get to or that you also had a question on, feel free to upvote those. We do prioritize those at the end of the calls so that we can get to the most sought after questions first. Lastly, we shouldn't have to say this, but please keep the chat respectful, of course, when chatting with one another, but also please try to avoid any kind of product service advertising, anything like that. And we'll be asked this a 100 times in the chat. I know from here, but yes, this is being recorded. Yes, we will share the recording afterwards so you guys can all, suspect to see that soon. Alright. Now without further ado, I want to introduce our guest today, which is Max Sparrow. Today we are joined by Max. He is the familiar with. He's a seasoned machine learning engineer and the founder of Pangram. Before founding Pangram and I'm getting a little bit of back noise there on you, Max. There we go. Okay. Before founding PanGram, Max worked on autonomous vehicles at Niro where he led their learning effort, or had their active learning effort. He also has a long history of deploying successful machine learning products at companies including Google, Two Sigma, and Yelp. He holds a bachelor's in theoretical computer science and a master's in artificial intelligence from Stanford university. So Max, thank you so much for joining us. Thanks so much for having me. Yes. Anything, Excited to hear. we need to be worried about or are you worried about or you feel good? You're going to get grilled before we get. started. No. Okay. Yeah. It's exciting. Well, Okay. I'm I'm at home because, office Internet was a little bit spotty yesterday, but so it should be fine on that front. Yeah. Yeah. Great. Awesome. Well, let me I'm gonna jibber jabber a little bit more here, and let's just get started with our first section, which is going to be the scale of our problem. Right? So how big is this actually? So when we look at our survey data, as I mentioned again, we surveyed more than 500 journalists within the network to understand how much they're experiencing AI pitches, their concern around AI pitches, and how it's impacting their scope of work. So, the first thing that jumped out to us here is 80% of journalists we surveyed said they suspect AI generate generated pitches at least a few times a month. More than half of those said or or sorry. Almost half of those, 47% said that they're seeing them daily or weekly. So this isn't necessarily something that's happening once in a while. It's significant portion of journalists, routine, what they're what they're seeing in their inbox, something that they're regularly having to mitigate. As it relates to the quoted side of things, when we look at what we're seeing internally inside of quoted, we found a couple of other interesting data points. So, some of you might know this. We launched the integration with our partner, Pangram, here in last December. Since then, we've analyzed more than 54,000 pitches. And across the platform, we've seen an AI likelihood of 33%. Now I do wanna be really careful with that number. That is not saying 33 of our pitches are AI. I wanna make sure that everybody's clear and understanding that. We are saying that 33% of or sorry. We're not saying 33% of pitches are definitely AI generated. It's It's telling us that we're seeing a meaningful amount of writing. So 33% of the writing itself is giving characteristics associated with AI generated content. So we have the journalist perspective now. We have some quoted perspective of what we see, especially with this integration. So now I wanna jump into these questions with you, Max. So let's start kind of reeling it all the way back. Obviously, PanGram exists because this problem exists. But I'm curious about kind of your origin story. So what was the moment that made you decide PanGram needed to exist? What problem were you actually trying to solve? Yeah. I mean, so early on, like, chat GPT era, like, I was seeing a lot of AI generated content. I was like I felt like myself, I couldn't tell if this was AI generated or not. And so I thought, like, this is just gonna be, like, a huge problem if, like, anybody with, access to AI can just produce, like, infinite content that's indistinguishable from from human. And ends up that's not really the case. Like, today, a lot of people, they they can tell whether something's AI generated or not just by, like, looking at it and, like, having that intuition. But I still think Paragums is really valuable for those who haven't built up that intuition, or or are operating on a larger scale. Yeah. So do you think that it was primarily about, like, people trying to deceive or is it kind of more complicated than that? I think it's more complicated than that. I I think there are, like, aspects where, like, yeah, you can be a bad actor, for example, like, in Russian like like, a news outlet trying to push, like, misinformation with just, like, like, flooding the Internet with, like, AI generated news stories. That's, like, one side of things where it's, like, very obviously deception. But I think there's the other side, which is just, like, yeah, more of, like, some like, different people are using AI to to different degrees, and we want to kind of understand, like, what is the degree of of AI use and, like, how can we how can different people, like, set their own policies around that? So, like, an educator might. have a very different policy than, like, the New York Times, for example. Yeah. What do you think was the biggest problem starting out or the biggest hurdle? I think it was there were many. I think, like, accuracy early on was, like, still a little shaky. Obviously, like like, like, when we were starting in 2023, like, the state of AI detectors were, like so there's a whole bunch of things that, like, don't really work that well. And so, like, what are we gonna do here? So I think even at the start when we were able to, like, surpass kind of, like, the existing state of the art, it still was like it went from, like, 1% false positive rate to, like, one and a thousand false positive rate, which I think still was, like, a little too high for our own comfort. So I think that was that was a big hurdle, like, making. the model really accurate. It it took us a while. It took us, honestly, years to get there. Yeah. Amazing. Okay. Well, let's jump into the second segment here and then talk about what's actually driving it. And so, obviously, we've acknowledged that this is happening at a very meaningful scale, especially for our journalists and our media users. So once you know that AI generated or AI assisted content is showing up regularly, the next question kind of becomes, well, who's creating it? Because it's very easy to have a conversation about AI slop as if it's one sort of homogeneous problem or does our survey suggest that concerns are a little more specific than that. Right? When we asked journalists what concerned them the most, they said that 63% was their concern that it was generic, low call it, low quality content. That's just simply wasting their time. Of course the second bullet point here in the second highest was the, their concerns around, like, trust erosion and and fake credentials. So when we look at this, it's not necessarily journalists being, again, philosophically opposed to AI and the use of AI. It's the concern really seems to boil down to something much more practical when we look at it from our standpoint and that's I'm already overwhelmed with pitches and now AI makes it easier to send more generic pitches. You're making my job even harder. You're giving more to sift through. That's not quality and not good content. So when we look at that from the quoted standpoint, we kind of looked at it from a different angle here and we wanted to see the usage by profile type. And so, when we compare self representing experts in the network versus PR and communications professionals, what we found was that the AI likelihood scores are on average 6.9% higher than a PR communications professional. So again, I want to pause there. This is not telling us that every self representing expert is using AI and PR professionals are not, of course, but it does suggest there are differences in the way that AI assisted writing is showing up across different groups of individuals. So that's where I think this conversation really gets a lot more interesting. This is where we start to get those questions that people kind of don't understand how these tools work, really trying to understand, whether or not AI exists in the writing and what's triggering something to reflect as AI. So, that's where we want to jump into. Our next question here with Max is sometimes writing gets flagged as AI generated when it's not AI. Some, some writing can evade detection that is AI. Can you talk about this disconnect and what's driving it? Yeah. For sure. So I think the, like, AI versus human writing is really more of, like, a spectrum than a full than a binary. So, like, for example, if I write out, like, a nice pitch myself and then I ask AI to help just, like, clean up my grammar and make it better, the goal of Pangram is to say that this is human written even though it is technically text that came out of a AI model. And kind of on the other side of things, if I, like, write, you know, five words or 10 words and then say, like, hey. Turn this into, like, a full on, like, email or or document. Like, that is something that we very much want to classify as AI as close to a 100% of the time as possible. So with that said, Pangram is a probabilistic model, so it's not always accurate. Our research and other third party research as well has found that Panagram has a false positive rate of about one in ten thousand. That means about one in ten thousand times when something is fully human written, Panagram will come back and say that there's AI in it. Or vice versa, Paragum has a false negative rate of around one in three hundred, one in five hundred. That means if there's something that's AI generated, Paragum might say it's not fully AI generated. And so I think this is a error rate that we just need to like, we are always working to improve. But because it's a probabilistic system, it will never be zero, and we need to design our processes keeping that in mind. And I assume on on the I mean, I don't wanna get too technical here, and I'm sure most people listening don't. But from a just curiosity perspective, I assume there's a certain, like, algorithm or level where it's like I know we talk about, like, natural language processing as part of it where it's like, sure that that sentence may be grammatically correct, but it almost reads like AI because it is almost too grammatically correct where I know that most humans wouldn't in fact write that way. So we see that all the time where people are saying things like, I'm actually dumbing down my sentences so that I stop getting flagged for AI. Yeah. I mean, I, like, wouldn't ask or suggest people to do do that. I think, Right. Yeah. like, I I I think that's that's kind of crazy. But I also do think, like, sometimes it's better if, like sometimes people, I feel like, they don't realize how much ChatGPT changes, their own words when they ask AI to polish it or make it better or, you know, more descriptive. Like, oftentimes, like, what I want like, show show me the prompt. Show me, like, the raw thoughts that came from you, not the, like, polished, chat GPT output, which is going to be more generic. It's gonna have less of the of an original voice there. Yeah. Yeah. Okay. I guess and one other question. Do you think that we're moving towards a point where, like, AI generated writing will become, like, essentially indistinguishable from human writing, or do you think there's always gonna be some kind of signal or detection that it will be able to, like, easily identify? I think there's always some signal. Like, these Frontier language models, they're very smart. They're very capable, but they still have certain opinions. They're very opinionated in their token choice. They're very, they they still have idiosyncrasies, that make it pretty, clear that, you know, when you're looking at text, it came from Quad or ChatGPT. And. going forward in not that long, there are going to be watermarks on all of the major frontier models, which is just gonna make this in, the the problem of detectability even more easy on our side, but I think we also have higher ambitions. We want to tell you the degree of human versus AI input. We don't wanna just say, like, you know, this is I think, ideally, we should be able to say, like, this like, even, like, this prompt had had this much effort put into it versus this was a low effort prompt, and and we're not there yet. But that's sort of, like, future looking where we want to be. Yeah. Okay. Maybe I kind of asked this in, in a sort of way, but beyond plagiarism detection, image detection, AI assisted flags, like what are the next kind of trends? I know you've touched on this just a little bit, but what else is the Pangram team watching in that space? Yeah. I mean, I think the the big things we're watching are watermarks. This is gonna be a really important component of the detection system going forward. And then, like, really just getting granular within the AI assistant scale, saying, like, this looks like it was light copy edits versus this looks like it was fully rewritten, and AI added more detail. Yeah. Okay. Amazing. All right. Well, let's jump into this third section here, which is, as I mentioned, kind of more of our instinct gap, which is probably a moving target as certain AI models get better, as we see personas, images, things like that improve over time. So there's how are journalists actually identifying fakes? How are they identifying fake content? How are they identifying fake individuals? Candidly, it quoted. We're seeing more and more fake individuals versus content. And obviously, that's what we're more concerned about on our end. So journalists, you know, they are doing this quite regularly and you'll see that what what our journalists in the survey here said that they're really relying on their gut instinct. And I think that shows you that they've all been doing this for a long time. They develop these instincts. They know what a bad pitch sounds like. They know when something feels generic. They know when language doesn't sound like a real person. So 72% of our survey respondents did say that they are actually still relying more so on their gut instinct and those red flags versus 23% said that they actually use and fully rely on like AI detection tools. And I think that's what creates that instinct gap. Where we talked about at the very beginning, this is more of a signal, and it's not a verdict. And that's really where journalists are relying primarily on their own judgment, but also still using tools to how to better understand who I'm working with is real or not. What we see and quoted is also quite interesting. So the number of pitches that we analyzed, was from December and it was about 54,000 pitches as again. So, AI of course is getting better. The writing is getting better. The models are changing, but I also think people are adopting AI in their workflow more on a regular basis day to day. I think it's become much more available. People are using it in ways that they don't even quite understand that they're using an AI model such as Grammarly. We see that all of the time. So, the number of pitches, what we did analyze was that 54,000. And what we did see from December, on average, we were seeing that our pitches were averaging about a 28% AI likelihood where that has jumped up to about a 54% AI likelihood. So, or sorry, 46% AI likelihood. I think I have that wrong there. But so a big jump there. Now I don't want this to mean that like AI has increased by 18 percentage point. I don't think that's a direct measurement of AI usage overall or saying that these things are AI, but I do think that that volume has increased because like I said, I think there's so many tools that people are utilizing that might be impacting with how their writing and their work actually comes across. And I do think that it has become more and more available. My mom told me the other day she used chat GPT to look up a recipe and I was so impressed that she even knew what chat GPT was. So I think it is becoming more of a norm across people's day to day workflows and lives of actually using these LLMs. And we'll continue to probably see that knowledge or lack of knowledge in some some of this case where people are using these LLMs on a more regular basis and not understanding how that impacts how the reading and content comes across. So, kicking it back over to you, Max. What does it really take to keep an AI detector accurate as these LLMs, one, become more popular as I mentioned, but also they keep changing. I think there was this morning, I got an email that Claude had shifted to a new model. So how do you how are you guys able to keep up with that? And how do you how do you manage that when it's seemingly ever changing on that side of things? Yeah. I mean, the state of AI is moving so quickly. So a lot of what we're doing is continuing data collection with the newest cloud models, the newest chat to the team models, the newest Chinese open source models, just generating text and prose generated by these models to understand what this landscape is looking like, make sure we can detect it correctly. So so what it really looks like in practice is every, like, three to six weeks, we're pulling new data from the latest language models and retraining a new pangram model. Well, sounds like a lot of work. Yeah. It's it's part of, yeah, part of the challenge is, like, building systems to make it easy because if if we're doing it, yeah, the the more manual steps we have, the harder it is. So the more we can automate this, like, data collection by using the AI APIs, the better Pangram will get over time. Yeah, certainly. I wanted to talk about, I know we sort of touched on false positives and we'll, we'll get into the QA section here as well. Cause I see that there's questions coming in. But, and maybe this is just my personal own greed here wanting to know for a few weeks, it seems like every time I logged into LinkedIn, it was somebody sharing pangram or some other AI detection tool with the declaration of independence or Jane Austin and people being like, see, even our founding fathers were bots or whatever. And, you know, trying to prove that these AI detection tools don't work that well or that they're inaccurate. And so, I think, you know, from my standpoint, I'm thinking like, well, this is a content that you still didn't write. So in my mind, it is accurate because it is flagging it. Like you're not Jane Austen. You're not the founding father, but maybe I'm wrong and thinking of it in that manner. Could you talk a little bit about that? Cause I think that was like, you know, I've, I'm sure you saw it on your LinkedIn as well. I still, continue to see some, but for a couple of weeks, that was all I was seeing. And it was kind of rage bait at. some point. Yeah. Yeah. Pangram is not one of those AI detectors that says the declaration of independence or the constitution or the bible is AI generated. Pangram says it's a 100% human written because, first off, it is. It would be incorrect to say that it's not. But second off, it's because we use different, methods for detecting AI. There's other AI detectors that are potentially less accurate that use a method of, like, how surprising is this text? And then they say, unsurprising text is AI, which a lot of AI texts are unsurprising, but also, like, the declaration of independence is is also unsurprising because, it's it's been memorized, and it's it's known by all the language models. Yeah. I'm curious. I can kind of address some of these, messages in the chat, if. And we have a couple is a the QA. time. Yeah. We have some QA ones here as well. I mean, we'll pop these up. Yes. Perfect. Thank you. So, how accurate is the program? I wrote a pitch without any form of AI and it said it was 84% AI AI and 16%. Yeah. human. I mean, this this happens on occasion. Again, the pan gram pan gram's false positive rate is about one in ten thousand, so that means this could happen especially with many tens of thousands of users unquoted every single day. But the goal is to make it less more accurate over time. How do you prevent your tool from classifying well written content as AI? For professional writers, this can result in false positives from AI detectors. Paragram actually does really well on well written content. So we have benchmarked on tons and tons of content, both from famous authors in the past and well written authors today. And we see that the one in ten thousand, that, false positive rate still holds up even on well written, articulate, edited writing. It's that's not enough to trigger the AI detector. It really needs to be something from chat GPT or Claude that is, like, using a specific word choice or phrasing or sentence from an AI language model. Do you, do you find, and sorry, Liz, if you were going to share one, if we were popping more questions up, but I, it kind of sparked another question in my mind. Do you find that there are certain tools or like common tools that people are maybe using day to day that will trigger things like, for example, what we, we deal with on our support chat is oftentimes our first thing is like, well, are you using Grammarly or something like that? That does have a certain level of AI that allows you to like rewrite your sentence structure. And it does write the sentences in a way that does give them a more generic, kind of rewrite and people don't realize that that also kind of makes them look like AI. Or do you find tools like that or things that you regularly see people using and not understanding how it could be impacting how their content appears? Yeah. Yeah. I mean, I think the big one is, like, people using AI tools like ChatChippy and Grammarly. I or ChatChippy and Claude. I think Grammarly, like, really kind of depends. Grammarly has two modes, the, like, grammar correction mode, which Pangram doesn't catch, or the one that, like, will make it more detailed, improve it, Yeah. and and that will get flagged as AI by PanGram. So I see people here who say, like, yeah. Like, they have, like, clients who use cloud skills to, you know, produce, quotes or text. And, yes, that will be classified by PanGram as AI because it you're using a Claude skill that's that's sort of that is the goal of PanGram, and that's PanGram working correctly. Yeah. But, also, this is indicative of if PanGram can tell, then likely the person on the other side can tell, and that's how, what you're sending is going to be. It will be perceived as likely or potentially AI generated to to many people. Yeah. We certainly find on our end, like, the we see people that will say, like, I well, I I did use AI to write this, but, like, I I they're all my original thoughts. And I I guess, like, our thought is, yes. We understand that. We know that everybody is very busy. Right? But, unfortunately, on on the receiving end, I can't tell your intention behind the content that you submitted and a journalist or a media user can't either. And so even though they might be your original ideas, you still had it written in the form of an AI model or an AI tool. And so that does start to read the same. So it might be new ideas, new unique perspectives, but it still reads a certain way. And I think that's where it's like one of those flags where it's like, I'm not sure I feel comfortable using this or not. And so we see that so often where people are like, well, I'm saving time. So I plugged it in to rewrite it and it's like, I get it. I understand why you're doing it. I know that we're all busy. But unfortunately that, that intention, while you might have the best intentions behind it, it does tend to end up appearing as the very generically written or very GPT esque written, content that a journalist kind of has that gut instinct to scroll right past. How how does Tangram check its false positive and false negative rate? And how do I guess, how would you keep that up to date over time as well? Mhmm. Yeah. So for both of these, we use, Internet scale data from for the human text, it's from before 2022. So we're looking at over a million human documents, at a large scale over all types of text from blog posts to recipes, essays, emails, kind of like the whole, like, any type of writing that you can imagine, we are checking with Pangram. And then what we're doing is we're literally just measuring, like, how many of these human texts does Tangram classify as AI. So it's about one in 10,000, which is, you know, 0.01%. But over the course of, 10,000,000 documents, that's gonna be, a 100 or a thousand or something like that. Yeah. And then how do we check our false negative rate? What we do is we generate a bunch of AI content, that is similar in topic or style to the original human content, and then we measure how many of these that we get right versus wrong. And, typically, it looks like we get above 99 right, but, again, not a 100%. There's sometimes sometimes we miss things. Sometimes it's too short. Sometimes, there's not enough AI signal in the text for us to pick up. So then I assume you probably have to continue to adapt that, like, AI generated text between different models and across different providers as well as they all continue to change as regularly as. they do. Yeah. So so we have, like, a kind of, like, false negative rate measure for every single AI model. Like, Gemini three, we get, like, 98%. Cloud, 4.5, we. get, like, 99.3%, something like that. Interesting. Okay. Awesome. Do we have another one here, Liz? Perfect. When is the tipping point and where is there so much AI generated content in the world that even human written writing starts resembling it just because that's the style norm? Kind of a philosophical question there. a good question. Yeah. I mean, I I think this is a question of, like, how much is language going to homogenize over the the coming decades. And I think to some degree, we've seen a little bit of language homogenization with just, like, globalization, you know, like, American English and, like, Hollywood English kind of being dominant in the media to the point where, like, other people in other parts of the world kind of pick up the, like, American dialects. So I think it's definitely, like, a thing that happens, on the scale of years or decades. With that said, I think that what we're looking at is a little bit different than, like, population level styles or dialects. I think we're looking to see, like, do the, like, specific word choices, or, you know, style like, stylistic choices or sentence structures line up with AI consistently over a full document. So even if somebody will write a little bit more like AI, they maybe use Delve that they didn't use before. On average, what we've seen is it's not enough to trip an AI detector like Pangram up significantly. We we we see no evidence of, like, increased error rates here. But, obviously, this is something that we're tracking because we I I do believe very strongly that language is shifting, and we need to be able to catch that and measure it. Yeah. I recently saw a study, that was it was talking about, non native English speakers and and looking at AI detection tools and the the rate that they get flagged versus, like, native English speakers if we were just looking at it across the board. Could you talk a little bit more about that and kind of how there would be a higher likelihood of that being flagged and looking like AI? Oh, definitely. Yeah. So there's there's a number of, English as a second language benchmarks for non native English speakers. And so on I think there's about 26,000 total texts from these non native English speakers. And then on these benchmarks, Pangram gets about three of them incorrect. So this is around the same, error rate of about one in 10,000 that we see on the broader Internet. Got it. Okay. Okay. I have a question more for quoted. Uh-oh. We're in the hot seat. Are you guys finding that reporters are trusting the pan gram score or even if your pitch says 100% human, are reporters still suspicious? Yeah. I think we, I mean, they're not still suspicious of everyone. I think I think PanGram is one domino out of a variety of things that they look at. And then if that's what the when we're looking at that survey, especially when chatting with journalists, there are many different things across the board that start to raise some red flags or green flags for journalists. And I think that there is with the world that we live in right now, and I already mentioned it. It's like, as it's really difficult to tell what businesses are even real or what personas are even real. So, maybe this is my shameless plug. So thanks for teeing me up, Sarah. This is the time to talk about your quoted profiles and everybody will laugh at me because you know, I'm always talking about your quoted profiles, but build them out, make them human. It's the same as your LinkedIn, right? Like how can I show up front that I am a real human? I have that media experience. I have that background because if you're in quoted, you're pitching a completely blank empty profile that's going to be one of the first barriers like how can I quickly determine that this person if I'm scrolling through a bunch of pitches, we all know journalists inboxes are crazy? How How can I quickly go in and determine this person is real? I feel comfortable working with them. Then I'm looking at the content. I'm looking at anything else that might give me a sense that I don't or I do feel comfortable working with this company. So or this brand or this person, whatever that might be. So I think Pangram itself is, like I said, I think it's one aspect. I don't think any journalist that we have in our platform is saying 100%. I'm not using these people because of the Pangram score. I think it's just something that they use in conjunction with all the other details. That's also a big piece of why we integrated it into both sides, we said. Hey, we want you to know how you reflect and how you might look to a journalist on the flip side. We wanna give you that level playing field. We want you to know. Hey Hey your profile. Let's make it stand out. Let's make sure that you look like a real human and we're having really good engagement but also like if you are maybe reading like AI, let's understand why and let's dig into that or let's stop using Grammarly at a really high level to help improve our pitch experience or our pitch success rate as well. Okay. Some expert sources slash clients will try to use AI to make their statements. It can be difficult to explain why they shouldn't do this even though it seems obvious to me. Any tips for navigating this? Some people seem eager to outsource their expertise to AI. Yeah. I mean, I I kind of, like, see this in whole bunch of different, like, scenarios. I I think sometimes it's, like, a difficult conversation to say, like, the reason, like, you're valued is, like, it's is you. It's your expertise, not how good you prompt ChatGPT. Right? So, Mhmm. like, no matter how well you prompt ChatGPT, like, it's not going to be the same as getting the words directly out of your mouth. Not to mention that things that come out of an AI like chat g p t, like, it's going to be more sanitized. It's gonna sound less like your original voice and more like this mono voice that from AI where, you know, so many things from AI just sound very similar, and people can immediately tell just by reading it. Yeah. I totally agree. I I think so often what we see on on our end is we have a journalist writing on a topic that they don't know everything on. They're not the expert in that space. And so when they come in, they're hoping for somebody to come in and break down that topic for them. And so any of us could go out to an LLM and search a topic and get the answer in the five bullet points. And so when you go into your pitches and it looks like, Oh, these are the same five bullet points that I just Googled or searched in Gemini or whatever myself, I'm then looking for the pitch that looks like that unique perspective. So they're almost breaking it down for me a bit more right like if I'm talking to my accountant, I constantly have to tell him like hey, you need to speak to me like I'm a fifth grader like I need you to break this down. I don't live day to day in tax law and all of these other areas. So explain it to me in words that I can understand. And that's exactly what these journalists are trying to do for their audience as well. We don't want to read an accounting book when we're writing a story on finance. We wanna read a story that's really relatable and that translates well to the audience. And so the more like AI can't give that personal experience, they can't tell stories. It can't give that perspective. I mean, it can, but you can usually tell if it's a fake kind of made up perspective as well. But I think those are the pitches that we often see standing out. We've also had people that tell us in the past like, oh yeah, we didn't use AI at all. But when we compare it to all of the other pitches in the inbox, the pitches have the same exact four bullet points and every single person in the inbox had those same bullet points. They're like, Oh yeah, when you're remodeling a bathroom at low cost, you wanna focus on these four things and it's like not one person had a unique perspective there. That's really interesting. Sounds a little bit like an echo chamber. Okay. Totally got one more. agree. Yeah. Couple more coming up here. Okay. Can you explain the difference between the pangram model used on the pangram site versus the quoted platform? The same responses entered into each place can result in different scores. Yeah. So so this one's probably my bad. So we upgraded the Pangram model last month, and then, yeah. We we upgraded the program model last month. But for a pure brief period in time, the quoted platform was on the old version of the program model, program three while we're on program four. I believe it's either addressed now, or if not, it will be in the next few days. So they should be, running on the same model very shortly. Thanks for taking that heat for us, Max. Yeah. Yeah. My bad. My bad. No, no, that's okay. I mean, I think that's, I mean, that's gotta be common though, right? Like we're, we're gonna continue to see that as these models continue to change and grow and adapt. Like I said, I logged in today. Claude's on a new model this morning or Claude has updates this morning. So. I assume on your side, that is more updates for your team and everything you're doing on your end. Yeah. Okay. Do you use AI generated script to find if the pitch is AI generated? I'm not sure I understand the question. Like, we we're not asking chat GPT if the pitch is AI generated. We're we're training our own. It's somewhere between, like, an algorithm and a model is maybe the best way to understand it. Like, the same way that a Waymo, when it's driving around, if it sees a stop sign, it knows that there's a stop sign there. That's like the same kind of algorithm that Pangram is using to tell you if text is AI generated or not. Awesome. Okay. Got a couple more here. More job postings for writers want candidates who make an who make AI text appear more human generated. Will writers who take such jobs be setting themselves up for extinction? Very My interesting. it's, yes. I I think that's sort of like a yeah. I feel like that's not really a great place for someone to be as a writer to be taking AI text and humanizing it. I mean, like, I definitely, like, see the appeal, from a business standpoint of we want to go faster. It's probably gonna be faster to start with an AI draft, and humanize it than to, start from scratch and produce something original. But I think what we're gonna end up seeing is even if you're, like, taking the AI text and rewriting it in your own words so it comes back human, it still is going to, like, have the same four bullet points and still have a very, like, generic story behind it, which is the really, important thing that a lot of people don't really consider when they're using AI. Yeah. And I know you and I kind of talked about this a little bit, but the there's also a threshold for publications and what they're willing to accept for AI, AI usage. Some publications don't care if there's any percentage of AI likelihood. Some will say 0%. And so I think that's, there's no silver bullet on our end, of course, to say like, no, everything needs to be zero, everything needs to be under 25, everything needs to be under 35%, whatever. That's going to probably continue to change and adapt. And I assume, like you mentioned with the release of watermarks and, and all these other different tools that will also adjust where I think maybe some of these publications will go back, and say that, like, we're not willing to accept AI generated content now that we have a better ability to tell exactly what's human written. Why is yeah. I I think something else that's kind of important as well is, like, there is so much low effort AI content that it, like, really if even if something's like, like, more, like, legitimate back and forth like human AI collaboration, it's just going to look the same as the low effort Yeah. of AI content unless it stands out in some other way. So it's not just the Pangram score being 100% human that would make something. stand out. It would also be something like just being special and out of distribution, not what an AI would normally write Yeah. by itself. Yeah, certainly. Sorry, Liz, you can put that one back on. I was asking my own questions again. I'm jumping the queue. No. Yeah. I was kinda trying to answer that one, anyway. Okay. I rewrote something myself from a Claude draft and it still showed as AI. Why? Oh, interesting. This can happen on occasion if yeah. I I think this can happen when, parts and, like, phrases from the claw draft end up getting, like, reused in the rewrite. Panagram will still see these as, like, signs of AI writing. It's, like, like, full of phrases or sentences that are used in the, like, new text. I think this would be similar as well, when watermarks come online. If you reuse too much of the original claw draft, then even the watermark could trigger even if you. rewrote some of the text. Awesome. You mentioned that watermarks are going to be a big deal. What's the conventional wisdom answer and what's the unexpected, surprising possible answer? I think, yeah, I I I think conventional wisdom for watermarks is, you know, AI is just going to be more detectable than ever, and we're going to, like, have high certainty that things that AI was used in certain cases and high certainty that there's no AI used. But maybe the, like, the less conventional answer is that watermarks are not particularly good at understanding the degree of AI use, like, whether this was, like, AI for polish versus fully AI generated. So, well, I think kind of remains to be seen, like, how popular watermarks versus model based detection end up being. I think as Pangram, we're going to try to incorporate watermarks so we can give you the full story. Like, not only is there cloud watermark here, but we believe that this was light polish and not, like, fully AI generated or something like that. Interesting. Okay. Are you communicating with journalists about the purpose and intent of AI checkers so they understand that a certain percentage score doesn't doesn't automatically mean the content is bad or should be. dismissed? I think that one's probably geared more towards the quoted team. But yeah, I mean, we, it's all up to, you know, we allow every journalist to have their own processes, procedures. Of course, we're doing our best to try to educate the masses across the board. Right? So same, same with our goal of let's give this tool to both sides because what we found when we started asking journalists about a year ago, what they were using to detect for AI, they'd be saying like, hey, these are a 100% AI. And we're like, well, what tools are you using? And so that's what led us to our partner here at Pangram to say, let's at least put the one that we know to be the most accurate and that is doing more work in this industry than than anybody else. And let's just give everybody that same experience across the board to give the level playing field. I don't know that it's so much even on the journalists sometimes. I think it's more so on the editor in the publications, where they ultimately have those rules and those guidelines that say, we will not accept anything that remotely reflects as AI generated, just because we are being overly cautious and we're, we're concerned about that and the validity or vice versa. We have some where they're like, Hey, the response is perfect. It's everything we need. It might show us a 100% AI, but we just really liked the commentary and everything that was provided. So I think that's, there's several factors that go into that, but yes, of course at Quota, we're doing our best to kind of educate like these webinars to say, this is what these tools are reading. This is why we've provided them. And everybody should still proceed at their own risk, whether or not you want to work with people and really get that human perspective, on, or your own gut instinct, I guess, on whether or not you feel comfortable utilizing that content. My human written pitches keep getting flagged as AI generated. What are you what what are your best tips to preventing this in the future? What can I change in my writing? I think I would need to know more context on, like, why these pitches are getting flagged as AI generated. Are we, like, starting from a AI draft, or are we, like, using AI for talking points? Or are we or is there AI polished at the end? Or I mean, I've never seen this before, but potentially somebody who talks exactly like chatty petite. But but I feel like that's probably less likely. Sarah, if if you're willing for it, you know, shoot us an email, at info at Pangram, and I'm happy to dive in and help you debug this a little bit. It's so it's the poor death of the, the em dash. I know everybody's, everybody hates the em dash now and it's I I still use that dash. I I'm just. gonna go and see it. I'm. I'm a defender. Okay. Why doesn't Pangram distinguish between fully AI generated writing and genuinely human written content that's simply been polished with tools like Grammarly or AI assisted detection or dictation. Some detectors already make that distinction and it seems especially important for people who use AI to clean up their own thoughts rather than generate the content for them. Great question. So PanGram does have classifications that are AI assisted or AI polished separate from fully AI generated. I am not a 100% sure. I I'm pretty sure that this is implemented and quoted, but I'll I will go, like, double check afterwards. But I think the important thing is, like, this is the future we're moving towards, and we are getting better at this over time as well. I think this is really important to be able to distinguish between fully AI generated and AI assisted in a less 100% way. Yeah. Yeah. And I don't think that part is integrated into Quota yet. So I know it's, Okay. it's something that we're, we can be working towards. out to Kevin. Okay. Okay. And maybe our last question here or do we have one more? I see that. Okay. With so many AI detection tools using different methodologies, do you think one, maybe, PanGram will eventually emerge as the industry standard? Yes. And I think it's gonna be. Well, you're you're asking the. Yeah. Yeah. So, That course, seems gonna be a little bit biased. she was setting you up there. Yeah. Yeah. Well well, I appreciate the question, Samantha. I think, yes. I think some of the, like, advantages of there being an industry standard is there's no sort of, like, disagreement of, like, oh, you know, this, like, zero GPT said it was AI, but Pangrep said it was human. So, like, what is it really? I think the best thing to do is we have third party benchmarks. Other people go in and say, this is the most accurate AI detector today, and we're able to sort of just, like, agree on that. And then other people can have more confidence in using a tool like this without worrying about is it correct or not. So I that's the future I see us going. I'm hoping for that at least. But, also, in order to get there, we really do need to earn the trust of everybody and be the most accurate, at all stages of the AI assistant scale. So we're really working to earn people's trust and get there. Yeah. Well, thank you. I appreciate it. It's, we we certainly like I said, here at Quoted, we obviously we found PanGram to be the best and staying ahead of the curve as well because it's obviously ever changing. For those of you, I mean, PanGram for us is obviously like an integrated third party tool. We do partner with these guys and we were so happy to have it integrated into the quoted experience. You can also always sign up for your own PanGram account. I find it very interesting. It does give you more analytics, more detail. It breaks down if you were putting your pitches and you wanted them analyzed further, there is a lot more detail in there that's not in that aspect of the quoted experience. So I welcome all of you to do that. Sign up, shameless plug for, for, PanGram, of course. So Max, any final closing thoughts, anything else that we didn't cover you wanted to be sure we touched on? I think that's it. Thanks so much for having me on. It's really, exciting to get to talk to you all, and I hope I've helped clear up some, help clarify a lot about how Pangram works. Yes. Yeah, certainly. Yeah, no, we, we're all very aware that the concern is real. The usage patterns we're seeing are very, very telling. We hope that this is helpful for everyone in how they integrate this into their day to day strategies. Maybe some of you believers. I know we get a lot of, disgruntled individuals sometimes. So hopefully giving you some insights into how to individuals sometimes. So hopefully giving you some insights into how to better utilize these tools and functions in your experience. So most of all, we're so grateful for not only joining us today. Thank you so much, but also the work you're doing in this space to continue to build a tool, to help educate us all in this really challenging kind of ever changing world. So, World's, moving so fast. yes, yes. And then lastly, thanks to everybody else who joined us today. And as always the engagement from you all, you're such a wonderful community and we're so grateful for all of you, coming in and listening to us jibber jabber each month. So we will circulate this again as a recording along with any other additional resources we talked about today. But thank you again and we will see you all next month. Thanks,