`AI_THRESHOLD = 0.85` is one line in `inference/app/engines/rungs/text_classifier.py`. It is the line that decides whether we accuse you.
The text lane runs a classifier — ShantanuT01/vanguard-ai-text-detector, a ModernBERT-large model fine-tuned to separate human from machine writing, MIT licence, 395.8M parameters — and it returns a single probability that the passage in front of it was generated. That number is continuous. A verdict is not. Somewhere the engine has to cut, and 0.85 is where we cut. At or above it the verdict is `ai`. Below it the verdict is `uncertain`.
There is no third option. The engine has no `human` verdict and no code path that could produce one. A human verdict would need positive provenance of the exact bytes — a signed capture manifest that says how the content was made — and the absence of a signal is not provenance. So a score of 0.02 means the classifier did not fire. It does not mean a person wrote it. Paraphrased AI text, lightly edited AI text and plain-register AI text all score low as well.
The dial has a victim in both directions
Move the cut down and the engine catches more machine writing and wrongly accuses more people. Move it up and it accuses fewer people and lets more machine writing through. No setting does both. This is not a defect that better engineering removes; it is the shape of the problem, and every detector on the market sits somewhere on the same curve whether or not it says where.
What almost nobody publishes is the curve itself. An accuracy figure is one point on it, quoted without the axes. Here are ours, measured on 27 August 2026 against the deployed service.
The whole curve
| Threshold | Recall | FPR | AI missed | Humans accused | | --- | --- | --- | --- | --- | | 0.50 | 89% | 1.0% | 2/18 | 5/496 | | 0.70 | 72% | 0.6% | 5/18 | 3/496 | | **0.85 (current)** | **67%** | **0.2%** | 6/18 | 1/496 | | 0.90 | 61% | 0.2% | 7/18 | 1/496 | | 0.95 | 61% | 0.0% | 7/18 | 0/496 |
Recall is the share of a known-AI set that the engine flagged. FPR, the false-positive rate, is the share of known-human documents it accused. The last two columns are those same two facts as raw counts, because a percentage of a small number flatters whoever is quoting it.
Which of these numbers to believe
The recall column and the false-positive column are not evidenced to anything like the same standard, and the gap matters more than either figure.
**Trust the recall column least.** It rests on 18 AI-written passages, all from a single generator. Eighteen documents is a sample small enough that a single passage moves the figure by five and a half points, and a single passage is the entire distance between the 61% row and the 67% row in the table above — eleven caught against twelve. Output from a different model, a different prompt style or a different register could score very differently, and we have not tested that. This is the weakest number anywhere in the product, and it is the first thing we intend to fix.
**The false-positive column is solid.** It rests on 496 documents: 231 arXiv abstracts, 156 CNN and Daily Mail articles published between 2007 and 2015, and 109 Enron employee emails written between 1999 and 2002. Every one of them was written before 2022, and that cutoff is the whole design. If the human side of a corpus contains AI-written or AI-polished text, the false-positive rate comes out silently too low, and a reassuring number is worse than no number at all.
The measurement counts the verdict the engine actually returned, not a score crossing a line chosen inside the measuring script. That distinction is not pedantry. The firing threshold has already moved once, and a script with its own hardcoded boundary would be measuring a rule the product does not apply.
The aggregate also hides register. In the 26 August run at 0.90, the single accused document out of 496 was an informal email — 1 of 109, a 0.9% rate for that register, against 0.0% for the academic abstracts and 0.0% for the journalism. One accusation is too few to conclude anything from, but it points the same way the literature does: the further writing sits from polished formal English, the more exposed it is.
The model's author publishes his own benchmark figures — AUROC 0.9475 and macro-F1 0.8493 across seven datasets the model had not seen, with a worst case of 0.8560 AUROC on one of them. Those are his numbers, on his data. No study of this model on our corpus exists, so we do not quote them as ours.
Why 0.90 lost
The threshold was 0.90 until 27 August 2026, and it was not chosen carelessly. The graph this service runs is int8-quantized, because the full-precision version is 1.58 GB and does not fit against 380 MiB quantized. Quantization changes scores slightly, and slightly is enough. Compared against the full-precision graph over 200 varied samples, the quantized model at a 0.50 cut **manufactured five accusations the full-precision model would not have made** — the worst of them moving a passage from 0.160 to 0.641. At 0.90 it manufactured none. With no corpus in hand, protecting against accusing someone on a rounding artefact was the right instinct.
Then the corpus arrived and the threshold was swept, which produced the table above. Read the 0.85 and 0.90 rows against each other: the false-positive rate is identical — one accused document out of 496 either way — while recall rises from 61% to 67%. 0.90 was strictly dominated. It cost detection and bought nothing. So it moved.
That is the unglamorous half of this: the interesting finding was not that we picked a clever number, it was that the previous number was free money left on the floor, and only a measurement showed it.
Why we did not go lower
At 0.50 — which is the model card's own default operating point, the setting you would get by using this model out of the box — recall reaches 89%. Two missed passages instead of six. It also accuses five people out of 496 instead of one.
Below 0.85 the trade stops being free and becomes a policy choice about how many wrong accusations are acceptable. We make that choice in one direction, and the reason is that the two errors are not symmetrical for the person on the other end. When the engine misses, it returns `uncertain`, which is a refusal to answer. When it fires wrongly, it hands someone a document that says a machine wrote their work. A refusal to answer is an inconvenience. An accusation is an accusation. So we would rather miss AI than name a person.
That is a value judgement, not a result, and you are entitled to disagree with it. The table is published so you can see precisely what it costs.
The dial cannot move quietly
A threshold in a config file drifts. This one is stamped into the exported model file itself, under the key `sighting.firing_threshold`, by the same export gate that ran the quantization comparison above. On startup the rung compares that stamp against the constant it applies, and refuses the graph if they disagree:
``` The model was gated at a firing threshold of 0.9000 but this service applies 0.8500; refusing to score until the two agree. ```
A graph gated at one threshold and thresholded at another has no gate at all — the quantization check that justified the number would have been run against a different number than the one in force. The service declines to score rather than score against a rule nobody verified.
The population this table does not cover
The corpus has no non-native English writing in it, and that is the population most at risk of a wrong accusation.
Liang et al. measured this and published it in *Patterns* in 2023 (arXiv 2304.02819). Across seven widely used detectors and 91 TOEFL essays written by non-native English speakers, the average false-positive rate was 61.2%. At least one detector flagged 97.8% of those essays as machine-written. The same detectors classified US eighth-grade essays near-perfectly. Their conclusion is that detectors penalise constrained linguistic expression rather than machine authorship, and we have no evidence that our engine is an exception.
Our 0.2%, one document in 496, says nothing whatsoever about that population, because the corpus does not contain it. The gap is not an oversight. No permissively-licensed corpus of non-native English writing is freely available: JFLEG is non-commercial, the TOEFL sets on HuggingFace carry no licence at all, and Lang-8 is gone. Closing it means commissioning writing with an explicit assignment of rights, which is on the backlog and not yet done.
Two further limits, stated in the same breath rather than parked at the bottom. We have run no adversarial measurement — paraphrasing is known to defeat detectors of this family and we have not quantified by how much. And the engine refuses to score anything under 100 words, because detectors of this kind flag genuine human writing on short samples; the previous guard was ten words, which happily rated a tweet with total confidence.
What other detectors publish
GPTZero's home page states a "99% accuracy rate when spotting AI-generated text vs. human writing", and, to its credit, publishes a false-positive figure beside it: a third-party benchmark "detecting 95.7% of AI texts while only incorrectly predicting 1% of human texts as AI", with a stated goal of holding ESL false positives to 1%. Originality.ai's home page states "an overall accuracy of 97.8% for our multilingual model".
We cannot verify either figure and we are not suggesting they are wrong. They may well be right on the corpora they were measured against. That is the point. A single accuracy number cannot be acted on, because it does not tell you the three things that determine what happens to the human being in front of you: which operating point produced it, what was in the corpus, and what the false-positive rate was at that same point rather than at some other one.
Ask any detector, including this one, for the row rather than the number. If you are deciding something about a person, the row you need is how often the engine is wrong about people who write like them — and for a great many people, that row does not exist yet. Not here, and as far as we can find, not anywhere.
