AI Detector vs Watermark Detector: Not the Same Test
These two get used as synonyms constantly, and they are not remotely the same thing. One is a guess based on how text reads. The other is a keyed statistical test on how the text was generated. Confusing them leads people to trust the wrong number.
Watermark detector: a keyed test
A watermark detector asks a narrow, well-posed question: is this token sequence consistent with what the generator would have produced using key K? It needs K. Given K, it recomputes the generator's biased choices across the document and runs a significance test on the result.
When it fires on a long enough sample, it is strong evidence — the false-positive rate is a computable number rather than an impression. When it does not fire, it tells you very little, because most text in the world was not produced by that generator with that key.
Its limitation is availability. Only the key holder can run it. Anthropic has announced watermarking for Claude and says a detection API is coming, but until that ships, no third party — FreeAIText included — can run Claude's test.
AI detector: a stylistic guess
What most people mean by "AI detector" is a classifier that reads the text and estimates how machine-like it looks. Typically it measures perplexity (how predictable the words are to a reference model) and burstiness (how much that predictability varies).
No key is involved, no cooperation from the generator is needed, and it works on any text — which is exactly why it is unreliable. The premise, that machine text is smoother than human text, holds on average and fails constantly on individuals. Non-native English writing, technical and legal prose, and heavily copy-edited text all read as "smooth", and get flagged. A 2023 Stanford study found detectors flagged over half of TOEFL essays by non-native speakers while barely misclassifying native speakers.
Side by side
| Watermark detector | AI detector (classifier) | |
|---|---|---|
| Question answered | Was this generated by model X under key K? | Does this read like machine text? |
| Needs the vendor's key | Yes | No |
| Publicly runnable | No — not for Claude today | Yes, many vendors |
| False-positive rate | Computable, very low on long samples | High and unevenly distributed |
| Meaning of a positive | Strong evidence of that generator's involvement | Weak stylistic guess |
| Meaning of a negative | Almost nothing | Almost nothing |
| Affected by light editing | Somewhat — each edit disturbs token positions | Barely |
| Affected by tidying up | Little | Can make the score worse |
What a positive result does and does not mean
A watermark hit means the generator was probably involved in producing the text. It does not establish authorship. A person can write an argument, ask a model to rephrase it, and end up with watermarked tokens carrying entirely human thinking. A person can also paste model output verbatim. The statistic cannot tell those apart — see what a Claude watermark does and does not prove.
A classifier hit means the text reads smoothly. That is all it means.
What a negative result does not mean
Neither tool can certify human authorship. A watermark detector returns nothing for output from any model that was not watermarking, for text produced before a watermark rolled out, and for samples too short to test. A classifier returns nothing for machine text that has been edited, or that simply happens to be irregular.
"No watermark detected" is not "written by a human". It is "this particular signal was not found".
Where FreeAIText sits
Neither category, and we would rather be precise about it. FreeAIText's current engine is a local research estimator: it reproduces the published green-list z-test in your browser using a key that is not any vendor's, so it measures statistical watermark pressure. It is not Anthropic's detector and it is not a style classifier.
If and when Anthropic publishes its detection API, real verification becomes a configuration change on our side, and every score will be labelled with which provider produced it. The current state is on the API status page.
How to read a score someone hands you
- Establish the category. Keyed watermark test, style classifier, or local estimate? They are not interchangeable.
- Check the sample length. Short passages cannot support a statistical claim from any of them.
- Ask what a negative would have looked like. A test that almost never returns negative is not evidence.
- Weigh process evidence higher. Drafts and version history beat any single number, and hold up better under scrutiny.
Sources
- Anthropic, Watermarking Claude's text output.
- Liang et al., GPT Detectors Are Biased Against Non-Native English Writers (2023).
- Kirchenbauer et al., A Watermark for Large Language Models (2023).