On 26 August 2026 we ran 496 documents of genuine human writing through the deployed detector. One of them was accused. It came from the informal register of the corpus — 109 pieces of ordinary non-spam email from the public Enron archive, written between 1999 and 2002. Whoever wrote it did not use a language model, because there was no language model to use. Our engine said AI anyway.
That is the whole of our measured false-accusation rate: one document in 496, 0.2%. The run is recorded in evidence/DETECTION-STATUS.md and the corpus is evidence/human-corpus.jsonl. The threshold has since moved from 0.90 to 0.85 and the count is unchanged — still one in 496.
The other error is the opposite one: AI text that gets through. On an 18-document set of AI-written passages, the current configuration catches 12 and misses 6. Eighteen documents, all from a single generator. That is the weakest number in the file and it is the first thing we intend to improve.
Both are errors. They are not the same size, and no averaging makes them comparable. A missed AI essay is a grade that should have been different. A wrongly accused student is a disciplinary hearing, a transcript note, an appeal they have to fund and argue, and — because appeals turn on credibility — a conversation in which the burden has quietly moved onto them. One error costs an institution some accuracy in its grading. The other costs one person, who did nothing, a great deal.
The asymmetry is a design constraint, not a slogan
Saying a detector should favour caution is easy. Showing where that preference sits in the code is the only way to tell whether it exists. Three places it does in ours.
- Aggregation is the mean over windows, never the maximum. A long document is cut into up to four windows of 512 tokens (a token is roughly three-quarters of a word), spread evenly across it rather than taken from the front, and those scores are averaged. Taking the highest instead would make false accusations rise with document length: twenty windows of ordinary human prose will eventually throw up one high score by chance. The cost is real: a short AI passage buried in a long human document gets diluted, and we miss it. That is the tolerable side of the trade, because a miss is something a customer can go and find other evidence about.
- There is a hundred-word floor. Below it the engine does not score at all. Detectors of this family flag genuine human writing on short samples, where there is not enough signal and the number is noise wearing the costume of a measurement. The floor is a declared judgement call. The engine's older guard was ten words, which happily scored a tweet with total confidence.
- The firing threshold is 0.85, not the 0.50 the model's own card suggests. A threshold is the probability at which the engine stops describing and starts accusing.
That last one is a trade, so here it is as one. Sweeping the threshold across the 496 human documents and the 18 AI ones gives:
- 0.50 — catches 16 of 18 AI documents, accuses 5 of 496 human ones (1.0%)
- 0.70 — catches 13 of 18, accuses 3 of 496 (0.6%)
- 0.85, the current setting — catches 12 of 18, accuses 1 of 496 (0.2%)
- 0.90 — catches 11 of 18, accuses 1 of 496 (0.2%)
- 0.95 — catches 11 of 18, accuses 0 of 496 (0.0%)
We moved from 0.90 to 0.85 because 0.90 was strictly dominated: the same false-positive rate, less detection, no safety bought. Below 0.85 the trade stops being free and starts being a policy choice about how many wrong accusations are acceptable. We have not made that choice, and we are not going to make it quietly.
Note what this list cannot do. It cannot tell you 0.85 is correct, because the threshold has never been fitted to a corpus held out from the ones above. It is declared policy, which is precisely why we publish no accuracy figure for this product. A number chosen by hand and then measured on the data that informed the choice is not a calibration.
One way to score beautifully on false positives is a trap. The statistical engine we replaced measured about 0.6% — on 342 of the 496 documents, because the service was shedding a third of requests at the time — lower than you would expect from a cruder instrument. It also scored 0 of 3 genuinely AI-written passages as AI. An engine that fires on almost nothing misfires on almost nothing. That low rate was not a virtue; it was a symptom.
The population we cannot measure is the population most at risk
Our corpus has three registers: 231 academic abstracts from arXiv, 156 news articles from 2007 to 2015, and the 109 Enron emails. Every document predates 2022, deliberately, because if the human side of the corpus contains AI-written or AI-polished text then the false-positive rate comes out silently too low — and a reassuring number is worse than no number.
What it does not contain is any writing by non-native English speakers. That gap matters more than everything above it.
Liang, Yuksekgonul, Mao, Wu and Zou tested seven GPT detectors on 91 TOEFL essays written by non-native English speakers (Patterns, 2023; arXiv:2304.02819). The average false-positive rate across those detectors was 61.22%. All seven unanimously flagged 18 of the 91 essays as AI-written; 89 of the 91 were flagged by at least one. The same detectors were near-perfect on 88 essays by US eighth-graders from the Hewlett Foundation's ASAP dataset.
The follow-up experiment should worry anyone running a disciplinary process. When the authors prompted a language model to rewrite those essays with more sophisticated language, the average false-positive rate fell from 61.22% to 11.77%. The detectors were substantially reading fluency: simpler vocabulary and plainer sentences looked like machine output. That is a bias with a direction, and it points at the students with the least idiomatic English and, usually, the least standing to appeal.
We have not measured our own rate on that population, so we do not know ours. The slice is missing for licensing reasons, not oversight: JFLEG is CC BY-NC-SA and therefore non-commercial, the TOEFL sets on HuggingFace carry no licence at all, and Lang-8 is unavailable. Closing the gap means commissioning writing with an explicit assignment of rights — a backlog item with a cost attached.
Until then, one sentence has to be carried alongside the 0.2%: it is a rate measured on three registers of native-register English, and it does not transfer to a population the corpus does not contain. Anyone quoting it at a student whose first language is not English is quoting a number about somebody else.
What our engine refuses to say
The text lane has exactly four outcomes: scored, too_short, skipped, error. Only scored carries a number. The other three carry nothing, because there is nowhere honest to put a figure nobody measured — and a module-level check raises if a score ever appears without the scored status, so a fabricated one cannot leave even by accident. If the model could not run, you get a sentence saying so, not a confident-looking 12%.
The verdict vocabulary is shorter still: ai, or uncertain. There is no human verdict for text and no code path that can produce one. This is not modesty. A low score is also what paraphrased AI text looks like, and lightly edited AI text, and AI text written in a plain register. Nothing in a probability computed over token statistics is provenance of who typed the characters, and provenance is what a human verdict requires. Only a verified capture manifest cryptographically bound to those exact bytes returns human anywhere in this product, and that path exists for images and video, not text.
Which means, said plainly: if a detector has accused you and you bring your document to us hoping to be cleared, we cannot clear you. We can decline to accuse you. Those are different things and we are not going to blur them, even though the blurred version would sell better.
Two more limits, same breath. We have run no adversarial measurement: paraphrasing is known to defeat detectors of this family and we have not quantified by how much. And the classifier's published quality figures — 0.9475 AUROC across seven datasets it had not seen — are the model author's numbers on the author's benchmarks. No study of it on our corpus exists, so none of them is ours to publish.
If a detector has accused you
An interval is the useful thing to ask for.
A confidence interval states how far the instrument's answer could move without anything about your document changing. A score of 72 spanning 40 to 100, and a score of 72 spanning 70 to 74, are completely different claims, and only one is worth acting on. Our text lane reports a score, a low bound, a high bound and a label. The width starts at a floor of 8 points that never reaches zero — one instrument, no measured error rate on our corpus — then adds half the spread between the document's own window scores, then a penalty proportional to how much of the document went unread; the whole width is capped at 40 points.
Those widths are declared policy too; no coverage study backs them. The interval states how much we distrust our own instrument. It is not a measured 95% coverage band, and calling it one would be the thing we are warning you about.
A single number with no interval is not an answer. It is a point on a scale whose width nobody has told you.
Seven things to ask for, in writing, from whoever is holding the score:
- What threshold produced this accusation, and was it fitted to a corpus held out from the one it was measured on, or chosen by hand?
- What is the tool's false-positive rate, on what corpus, of what size, collected when? "Under 1%" with no denominator is not a rate.
- Does that corpus contain writing like mine? Ask specifically whether it includes non-native English, and cite Liang et al. if the answer is no.
- How much of my document was actually read? Windowed detectors sample long documents. Ask what fraction of the tokens was scored.
- What interval accompanies this score, and what does its width mean? If there is no interval, ask why the tool does not publish one.
- Can this tool ever output "human"? If it cannot, then a low score is not a clearance, a high score is the only statement it is capable of making, and the score is evidence in one direction only.
- What non-detector evidence exists? Drafts, version history, revision timestamps, an oral defence of the argument. A detector score is one signal; a document's history is a different kind of evidence, and the kind that survives cross-examination.
The one document we accused is still in evidence/human-corpus.jsonl. We did not remove it and we did not round the rate down. If it were your email, 0.2% would not be the number you were thinking about.
