UPDATE: Today, July 29, 2026, Pangram updated its model to Pangram 4.0. All of the below tests were run on model 3.3.2, as can be seen in the screenshots. To me, this presents another complication into the Pangram scan issue. Something that scans human one day may scan AI the next, or vice-versa and every result in between. My experience with technology and software tells that we should not assume each update will be an improvement, and that some may introduce flaws.
In fact, today I ran a piece of test text through the Pangram tool on Substack, on Pangram’s website (as a cut-and-paste and then as a Word file), and through its Google Doc Plug-In. Pangram rated the text—which was identical in each scan—different for each, ranging from 91% Human down to 55% Human. The only difference in the text was where it was scanned and, in the instance of the web tool, whether it was scanned as a file or as pasted text. That is a level of inconsistency that should not exist in this tool, which is being offered as reliable proof of human versus AI authorship.
Anyway, here is the original article.
I ran a small adversarial test on Pangram.
I wrote a few short paragraphs, about 200 - 300 words total over the course of edits. The idea was to put together something unmistakably human. Then I ran it through the new Pangram Substack feature and got, unsurprisingly, a 100% human rating. Because it was.
After that, I wanted to see how hard it would be to edit my way to a 100% AI rating. Essentially, to feel around for the edges of Pangram’s human versus AI verdict. None of it was written by AI. I just wanted to attack the concept and see how easy or hard it was to game. Once I did that, I reversed each change, word by word, to see precisely where Pangram flipped the AI/human rating. I tracked all of it in Google Docs so I have the entire revision history, which is posted in screenshots below. As I outline below, I had to strip the passage of most of what resembled my actual creative voice.
So, in this essay, I take you through that reversal, which I think is the most interesting part of the experiment.
Keep in mind, I’m not making an argument that Pangram is good or bad, or that Substack was right or wrong to add it. Instead, my goal is simple: to explore the boundaries between Pangram’s 100% Human, AI-Assisted, and 100% AI ratings. The hope here is to open a discussion based on something other than the fear and anger currently flowing through the Substack community. Mostly, though, I’m interested in laying out the process rather than deciding what anyone else should think.
I would also be interested to hear what the Substack Team, Chris Best, Max Spero or anyone at Pangram thinks about the process or results.
So, let’s get to it.
The Experiment: A Simple Human-Written Passage
As mentioned above, I wrote a 226-word passage in Google Docs. I normally write in Word, but Google Docs preserves a detailed revision history, which matters for this experiment because, whatever Pangram eventually concluded, I would have a timestamped record of the edits over time as they relate to the Pangram result.
Below is a screenshot of my control passage. It’s a short, meta discussion of the experiment written specifically for this experiment.
From there, I began editing in the hopes of fooling Pangram into giving me a 100% AI rating. To get there I revised the passage toward formal and away from conversational. I replaced familiar words with abstract alternatives, added qualifications, and removed some of the phrasing that sounded, at least to me, like me. Words like version became iteration, also became likewise, and initially became originally. I added words such as deliberately, interpretation, exploration, practical, and revision process. I took some of the life out of it—though to be fair, the initial writing was pretty boring and lifeless to begin with.
Some of those edits made the passage subjectively worse, at least according to me, which was fine. Humans have been producing stiff and abstract writing for a long time without any help from robots. I wasn’t trying to make something good, I was trying to see whether I could move the detector’s result by changing the texture of my own writing.
Eventually, Pangram classified the passage as 100% AI generated. I’ll save you from the step-by-step process in getting there, but if anyone wants it, I have it and will gladly append it to the end of this essay. Just ask. (EDIT: Someone did ask, so I added the forward pass video to the end of this essay, with a short description of what it is above it).
Anyway, here is the first version of the passage where I scanned and triggered a 100% AI result.
That result was false, obviously. The passage wasn’t AI generated, partly AI generated, or even revised with AI assistance. I deliberately pushed the detector toward a false positive. I don’t think anyone—including Pangram—believes the detector is foolproof, so that’s not the point here. But it does establish that, with around an hour and a half of work, I was able to engineer a false result. In the future, knowing what I do now, I could probably get there faster, and some people may get there without any effort at all.
Anyway, once I reached that point, I started walking backward. And that’s where the experiment is more interesting.
Reversing the Edits
I saved the version Pangram classified as AI and began reversing the changes one at a time. After each edit, I scanned the passage in the detector again. I wanted to see whether the path back would be gradual or offer some way to predict what Pangram was detecting. Frankly, I imagined I would see a sliding scale of 74% AI, 52% AI, 24% AI Assisted, etc.
First, I changed the word shift back to the phrase changed also. But that didn’t change the Pangram rating.
Reversing the next edit by changing iteration back to version moved the classification from AI to AI-assisted.
That made sense to me. I removed a more formal word and replaced it with an ordinary one. But then I changed likewise back to also, and Pangram moved the passage from AI-assisted back to AI.
At first, I was confused. Likewise is the type of word people point to when compiling lists of supposed AI tells (NOTE: Medium authors, please stop writing these articles. They are almost all so horrible and also probably completely wrong). Yet replacing it with the plainer and more conversational “also” moved the passage back into Pangram’s AI category.
Next, restoring the word initially produced no change in the rating.
Removing the word deliberately moved the passage back to AI-assisted, but again, it was 100% or nothing. No sliding scale.
Removing the word interpretation did nothing as well. (NOTE: I added the word “and,” between style and interpretation, but it is not shown in this screenshot. It also did nothing).
Changing exploration to exercise did nothing.
Replacing such as with like did nothing as well.
Removing practical did nothing.
Then, I removed the word revision. And then, with no sliding scale indicating anything less than 100% in any category, Pangram rated the passage 100% Human.
Now, I assume the result had little to do with the word revision. By then, I assume, the passage as a whole was moving closer to one of the detector’s “human” boundaries. And, for some reason, losing the word revision made the difference. Some small change was eventually going to push it across, and this happened to be that change.
I was hoping to see something more gradual, but that didn’t happen. That single change was a marker for where Pangram’s visible rating label flipped. There is nothing about that single change that explains the flip by itself, and I imagine if I had made the backward, step-by-step revisions in some other order, a different word may have been the trigger point.
What Changed in the Writing
The individual word substitutions are the easiest changes to follow, because I can show them to you in screenshots. But those changes don’t fully describe what happened during the first forward revision from 100% Human to 100% AI. By the time Pangram classified the passage as 100% AI, I changed the overall texture and patterns in the passage considerably in ways that are not immediately apparent.
Below is a deeper dive into the patterns and, if I’m using the word properly, stylometry, of the passage as (a) originally written, (b) at the first 100% AI rating, and then (c) at the first point it was rated as 100% Human on the reverse revision pass.
I think you’ll figure out the table below, but one term, nominalizations, may need explanation. A nominalization is a noun formed from another kind of word, often a verb. For example, revise becomes revision or interpret becomes interpretation. Nominalizations can be useful, but they also make writing abstract. When you nominalize a word, you turn an active, concrete action into a description of an action. And like passive voice, it’s tiring to read. It’s like talking about doing something instead of just doing it.
For example, instead of saying “I revised the passage,” the later version talks about the “revision process.” That helped make the passage more formal, put it closer to an academic register, and it reads less immediate or personal. It also is something AI writing tends to do. My best guess is that most AI models have been trained on millions of pages of crappy corporate writing as well as stilted academic and legal writing. Creative works are the minority of written content in the world. Though I could easily be wrong.
Anyway, here’s what I measured, and you’ll notice a significant increase in my use of nominalizations on the way to the AI rating.
My average sentence grew from about fifteen words to twenty. More importantly, the variation among those sentences collapsed. The original passage had a living, uneven cadence, with short sentences beside much longer ones. By the time Pangram called it AI, the sentences approached a much narrower range. The prose became longer and more uniform at the same time.
I also edited the vocabulary toward heavier words. Words of three or more syllables rose from 8 percent of the passage to about 30 percent. As mentioned above, nominalizations increased sharply. The first person also disappeared, though I’m not sure how that mattered. In the original passage, I referred to myself eleven times. In the AI-labeled version, I cut that out entirely. Passive voice appeared, I reversed contractions, and the general tone shifted away from conversational toward something more formal.
The original, I think, sounded like the written version of a person explaining, orally, what he intended to try. The later version sounded like a high school science paper. It’s an impersonal description of a procedure, its purpose, and how results would be interpreted. The writer gradually left the writing. Or, maybe more telling, the voice did. Which may be an important note for anyone using AI as a tool for creation.
Clearly, though, this is guesswork on my part. It’s my best estimate of what an AI voice sounds like. There are endless ways I could have gotten there. The numbers describe changes in the passage and offer some ideas, but they don’t give me access to the Pangram model’s reasoning. Still, they show that my forward edits did more than insert a handful of stereotypically artificial words. I made the prose more abstract—which I think is the most important pattern—less personal, and far more rhythmically regular before it reached the 100% AI result.
The reverse pass, however, shows something different.
Once the passage passed, in reverse, Pangram’s “AI” boundary and was once more rated human, the measurements I noted barely changed. Average sentence length moved from 20.4 words to 20.1. Sentence-length variation remained at roughly .25 or .26 throughout. Syllables per word drifted from 2.03 to 1.99, while the share of polysyllabic words moved from about 30 percent to 29 percent.
During that same sequence, Pangram moved from AI to AI-assisted, back to AI, then eventually to Human.
The final version Pangram called human still didn’t resemble the texture or feel of my original passage. To me, the voice was neutered and the sentence cadence was still fairly even. The phrasing remained abstract and the first person was still missing. On every measure I looked at, the final human-labeled version was closer to the passage Pangram called AI than the original passage Pangram called Human. To my eye, it reads closer to AI as well.
I think the reverse of the word likewise to also is interesting too, because it defies the idea that there is a list of suspicious “AI words.” When I returned to the plainer also, Pangram moved the result toward AI, not away from it. That cuts against the importance of an AI word checklist.
So, while the factors I measured in the forward edits describe how much the prose changed, the measurements in the reversal of those same edits show how little they mattered to get to the absolute boundary between an AI and human rating. Once the passage was close to a threshold, a small edit produced a categorical difference from Pangram even though the writing remained almost identical by the measures I was able to calculate and had little resemblance to the original control passage.
Of course, I may be missing some important measurements. If you other patterns in the passages, I’d love to know about them. It would only help us understand the boundaries better.
What Pangram’s Result Actually Means
Pangram does not claim to identify AI writing by counting long words, measuring sentence-length variation, or checking the passage against a list of supposedly suspicious terms. It says its model analyzes patterns and relationships across the submitted text.
The technical explanation involves tokens, embeddings, neural networks, and high-dimensional space. I’ve read, and I think I understand it, but frankly, the intricacies of it are above my pay grade. But I think I understand enough. Essentially, Pangram breaks a written passage into small pieces, converts the text into numerical representations, and then analyzes the pattern of each segment across the text. It then generates a numerical score based on the pattern and uses that score to rate the writing based on how closely it resembles patterns it learned from millions of human and AI generated works.
That helps explain why changing likewise to also could move the passage toward AI rather than away from it. Pangram did not decide that one word belonged to people and the other to machines. The substitution changed a larger pattern, just enough, and the model responded to that pattern in a way I couldn’t see, because Pangram doesn’t allow users to see the pattern scores.
Under Pangram’s published thresholds, a score at or below .25 is human, a score at or above .75 is AI, and the space between is divided into Lightly AI-assisted and Moderately AI-assisted. A passage sitting at the edge of any of those boundaries may need only a small change to be labeled human or AI.
What I Think I Found
To me, Pangram seems good at what it does. It’s analyzing patterns based on millions of passages. But that means that if your writing is scored at .25 in Pangram’s system, it will deem your passage Human, though a single edit could push you past that boundary.
Now, every classification system needs boundaries. That, by itself, is not a defect. At some point, the model has to decide which label to display. The problem is that Pangram does not show Substack readers the underlying score or scoring system or tell them how close the writing came to the boundary. Instead, it gives very broad categories that, in the real world, are being used as verdicts instead of statistical probabilities.
Even when you do get the sliding scale—i.e., 30% AI / 70% human—that is just Pangram applying this same score system to blocks of text within the essay. And to Pangram, a finding of 10 percent AI-assisted may be meaningfully different from 100 percent AI generated. But to a reader already suspicious of AI, both collapse into the same conclusion: the author used AI. And that, I think, is where we are encountering the current “witch hunt” and “scarlet letter” problems.
Because, from there, human nature does the rest. Pangram’s suspicion of AI use, no matter how statistically probable is still just a suspicion. There will, naturally, be people writing at the boundaries of Pangram’s categories, just as there will be models or prompts at those edges as well. Small edits to either will result in false results.
Pangram claims a fantastic 1 in 10,000 false positives, but I don’t trust that number. It feels like Crest telling me 9 out of 10 dentists prefer their toothpaste. Though, I’ll give them this: it’s better documented than most. The method was peer-reviewed at ICLR 2026, and independent researchers at Chicago Booth found essentially zero false positives on medium-to-long text. So the number isn’t pure snake oil. But it holds up on long writing and seems to slip on short passages, which is exactly what Substack is aiming this tool at when it allows it to be used on Notes and comments.
So, there’s nothing independently establishing the 1 in 10,000 figure for every kind of writing. Pangram’s own information shows false-positive rates vary for different types of writing and that its model, at least the one tested in this essay, is intended for long-form writing made in complete sentences. I guess that means no poetry? Anyway, the Chicago researchers also found commercial detectors became less accurate when used on passages under fifty words. That’s shorter than anything Substack currently permits users to scan, so the study doesn’t prove Pangram is unreliable on a 101-word Note or comment. Still, Substack is inviting readers to use a a tool intended for long-form prose on fairly short and informal writing in the form of Notes and comments. That seems worth watching.
Also, I don’t know what Pangram did to determine its distribution of human scores, but I feel I can assume most writers don’t land comfortably near the zero or .25 score. The distribution is probably more like a bell curve with the bulk of “human” samples in the middle. But wherever the cutoffs are placed, some entirely human writing will still fall close to the edges. That’s unavoidable. And some styles may be more vulnerable than others, such as formal writing, academic writing, legal writing, or technical manuals. My guess is that writing with longer sentences, fewer first-person references, more abstraction, or highly regular structure is more likely to be flagged.
And there is another a larger complication developing around the edges of this issue. As AI generated language becomes more common, people read more of it. Some of its rhythms, transitions, structures, and vocabulary worm their way into human writing whether the writer uses AI or not. People learn language partly by exposure. We absorb the writing around us.
That could make the distinction Pangram is trying to draw less stable over time. Human writing may begin to resemble AI writing because AI was trained on human writing and then poured back into the culture at enormous scale. The patterns will continue influencing one another and the detection boundaries will be less stable.
Ultimately, despite a lot of concerns and reservations, Pangram seems to be pretty good at what it does, or at least how it frames its own claims. But I’m not certain I trust the way Pangram presents its results to the world, nor do I trust it will hold up in the long term. I think the system needs more transparency. I think it should allow you to see the entire underlying score. Show how close a tested passage sits to the nearest boundary and explain its labels as predictions and probabilities rather than certainties or verdicts.
As for Substack, it appears to want something like a Good Housekeeping seal: a quick, readable signal that lets people feel confident about what they’re reading. And I get that. The actual Pangram feature, however, produces something far messier—a statistical estimate built from patterns, thresholds, and uncertainty.
Hiding that mess makes the feature easier to use.
But it also makes the results easier to misuse.
Anyway, that’s my experiment. I’d be interested to know your thoughts on the results or method. Or if you’ve done your own test, what did you see? This is a situation where we need more information, not less, and we are likely going to have to develop it ourselves.
—Charlie
NOTE: Because tsunimee asked, here is the full, time-stamped, forward editing pass. I screen recorded the Google Docs revision history, step by step. I cut off the right bar, because it reveals my name and, as many of you know, I write under a pen name. Anyway, the video starts with the blank page, me writing the initial passage and then, me editing toward an AI score. When I do that, I inserted a title at the top of the passage stating “EDITS TO MAKE THIS READ AS AI NOW.” I do not include the backwards pass here, because the screen shots above show that.
You can subscribe to Charlie Finch’s Substack for free. There is also an optional paid tier, which works more like a donation to support the writing. But nothing is paywalled. I appreciate your reads, restacks, and comments. Thanks.
"It feels like Crest telling me 9 out of 10 dentists prefer their toothpaste" <-- haha. This made me chuckle. Thank you for the experiment. Loved it! ♥ Please can I register interest to see the forward pass (human -> ai)? Did you really do it slowly?
I'm conducting some empirical experiments, and the auto-analysis capability of Claude Opus 5 is truly impressive. It distinguishes what AI is and what it is not. It recognizes typical AI-generated sentences and recommends how and where to modify your text to make it sound entirely human. Pangram is a toy 🧸 if confronted with the full power of Claude, and I am not even bothered to use Fable 5 because it is too expensive. Anyway, I believe that at the moment human content is still superior to AI content.