How AI Text Detectors Actually Work
"AI detector" covers at least three unrelated technologies with wildly different accuracy. Knowing which one you are facing tells you how much to trust its answer — and whether anything you do to your text will change it.
1. Watermark detectors — reliable, and rarely available
A watermark detector tests for a signal that the generating model deliberately inserted. It needs the secret key used at generation time. Given the key, it recomputes which tokens fell on the model's green list and runs a z-test on the count.
This is the only category that approaches proof. A z-score of 4 corresponds to a false-positive rate on the order of one in ten thousand, and the maths is transparent rather than a black box.
The catch is availability. Only the key holder can run it. In practice that means the model provider, and only for their own model's output — so an institution cannot simply download a watermark detector and check submissions.
2. Perplexity classifiers — available, and much weaker
This is what most commercial "AI detectors" are. They take a language model and ask: how surprising is this text to me?
Perplexity measures how well a model predicts the words that actually appear. Burstiness measures the variance of that surprise across a document. The premise is that machine text is unusually smooth — it hovers near the most probable next word, so perplexity is low and steady, while human writing lurches between the obvious and the unexpected.
The premise is roughly true on average and unreliable for any individual document, because a great deal of legitimate human writing is also smooth and predictable:
- Non-native English. Writers working in a second language use more common words and more regular constructions. This is the single largest source of false accusations, and it is a documented, repeatable bias — a 2023 Stanford study found detectors flagged over half of TOEFL essays by non-native speakers as AI-generated while barely misclassifying essays by native speakers.
- Technical and legal writing, where formulaic phrasing is the professional standard.
- Edited prose. Copy-editing removes idiosyncrasy. Polished writing looks more machine-like than a first draft.
- Short texts. Under a couple of hundred words there is not enough signal for any statistical claim.
OpenAI withdrew its own AI Text Classifier in 2023, citing low accuracy. That is worth weighing: the organisation with the best possible access to the models could not make this approach work reliably.
3. Stylometry — comparative, and narrow
Stylometry compares a document against a known corpus of the same author's writing: sentence rhythm, function-word frequencies, punctuation habits. It answers "does this match how this person usually writes?" rather than "was this machine-generated?"
It can be genuinely informative when a good baseline exists — a lecturer with a term's worth of a student's work has real signal. It says nothing without one, and a change in register (a formal report versus casual email) reads as a change in authorship.
What this means for editing
Because these mechanisms differ, edits affect them differently:
| Change | Watermark detector | Perplexity classifier |
|---|---|---|
| Swap a few content words | Measurable effect — each swap re-rolls two token positions | Little effect — distribution barely moves |
| Restructure sentences | Large effect | Moderate effect |
| Vary sentence length | Some effect | Large effect — burstiness is the metric |
| Add a personal anecdote | Effect on that passage | Large effect |
| Fix typos | Small effect | Can increase the score |
Note the last row. Cleaning up a document makes it more regular, which makes it look more machine-written to a perplexity classifier. People sometimes edit their way into a worse score.
The targeting problem
If watermark strength is a count over token positions, the signal is not spread evenly. Some passages carry a disproportionate share, and a passage with an above-average green-token rate contributes more to the document's total than one sitting at chance.
That gives a principled way to edit: score each segment separately, rank them, and rewrite only the few that carry the weight. You get most of the statistical effect for a small fraction of the changes — which matters, because every change is a risk to your meaning.
This is exactly what our tool does: it scores every segment, isolates the three to five heaviest, generates several candidate rewrites of each, re-scores them, and keeps only the versions that measurably lower the score. Rewrites that fail to help are discarded rather than applied.
How to read a score you are given
- Ask which kind of detector produced it. "98% AI" from a perplexity classifier is a guess with a known bias; a watermark z-score of 6 is near-certainty.
- Check the length. Verdicts on short passages are not statistically meaningful, whatever confidence they display.
- Treat a single number as an opening question. Drafts, version history and process evidence outweigh any classifier output — and hold up far better in a dispute.
Sources
- Liang et al., GPT Detectors Are Biased Against Non-Native English Writers (2023).
- Kirchenbauer et al., A Watermark for Large Language Models (2023).