AI detectors are less accurate than their marketing suggests. They do best on long, unedited text produced straight from a chatbot. They do worst on short passages, edited AI text, and writing by non-native English speakers, which they wrongly flag as AI at high rates. A detector score is a signal worth investigating, not proof.
What detectors actually measure
Most AI detectors are classifiers: models trained on examples of human and machine writing that estimate how likely a new passage is to be machine-generated. Many rely on statistical signals, such as how predictable each word is (often called perplexity) and how much that predictability varies from sentence to sentence (burstiness). AI text tends to be smoother and more uniform than human writing. But plenty of human writing is smooth and uniform too: formal essays, technical documentation, and anything written to a template.
What the research found
- OpenAI withdrew its own detector. OpenAI released an AI text classifier in January 2023 and withdrew it in July 2023, citing its low accuracy. By OpenAI’s own figures, it correctly identified only 26% of AI-written text and wrongly labelled 9% of human text as AI.
- Fourteen tools, none reliable. A 2023 study led by Debora Weber-Wulff, published in the International Journal for Educational Integrity, tested 14 detection tools and concluded that none was accurate or reliable. All scored below 80% accuracy, and they did worse on AI text that had been paraphrased or manually edited.
- Bias against non-native writers. A 2023 Stanford study by Liang and colleagues, published in Patterns, ran TOEFL essays by non-native English speakers through seven detectors. On average, 61% were flagged as AI-generated, and almost all of them were flagged by at least one detector. Essays by US eighth-graders were classified far more accurately.
- Newer studies agree. Larger studies since then have found the same pattern: false-positive rates on human writing vary enormously from one detector to the next, from almost zero to nearly every sample.
Why accuracy varies so much
| Factor | Effect on accuracy |
|---|---|
| Text length | Short passages (a paragraph or two) give the detector too little to work with |
| Editing | AI text that has been rewritten by a person, or by another tool, is much harder to detect |
| Writing style | Formal, simple or formulaic human writing is more likely to be flagged |
| Writer’s first language | Non-native English writing is flagged more often |
| Model age | Detectors trained on older models can lag behind newer ones |
What vendors claim
Detector companies publish accuracy figures, often above 95%. These usually come from their own test sets, which may not resemble the text you are checking. Turnitin, for example, has said its document-level false-positive rate is below 1% for documents with more than 20% AI writing, and it deliberately does not display low scores because they are unreliable. Even a 1% false-positive rate means many wrongly flagged students across a large university. Independent tests rarely reproduce the headline numbers.
How to use a detector responsibly
- Treat the score as a prompt to look closer, not as a finding.
- Check more than one tool. If they disagree, that tells you something.
- Look at the process, not just the product. Drafts, version history and a short conversation about the work are better evidence than any score.
- Be especially careful with non-native writers and with short or highly structured texts.
If your own writing gets flagged
Keep your drafts and your document version history, since they are the strongest evidence that you wrote something yourself. If you write in a second language, say so; the research above is on your side. Some writers also use an editing tool such as Walter Writes AI to check how their text reads and to vary its rhythm before submitting. Whatever you use, make sure the final text says what you mean in your own voice.
The short version: AI detectors can be useful for spotting text that deserves a second look. They are not accurate enough to settle the question on their own.