Take our own text engine, leave the model and the documents alone, and move one number — the probability at which it stops describing and starts accusing. Between a cut of 0.50 and a cut of 0.95, five of the 18 AI-written passages in our evidence set change verdict, and so do five of the 496 human documents. Ten documents get a different answer from the same detector reading the same bytes and computing the same score.
The full sweep is in `evidence/DETECTION-STATUS.md`:
| Threshold | AI passages caught | Human documents accused | | --- | --- | --- | | 0.50 | 16/18 | 5/496 | | 0.70 | 13/18 | 3/496 | | **0.85 (ours)** | **12/18** | **1/496** | | 0.90 | 11/18 | 1/496 | | 0.95 | 11/18 | 0/496 |
Nothing in that table is a disagreement about evidence. It is one instrument with the dial in five positions — which is the first thing to understand when two detectors return two answers. The gap between two tools can be pure policy, and where it is, the tool that turns out to be "right" is the one whose policy happened to match your case.
Why do two AI detectors give different results for the same text?
Because a verdict is a probability plus a set of policies, and both differ between tools. Five of those policies can each flip a verdict on their own: where the tool starts accusing, how much of your document it read, what it was trained on, whether it cleaned the text before scoring, and what its verdict words mean.
None of these are visible in a bare verdict. That is the actual problem — not that tools disagree, but that they disagree silently, so you cannot tell a genuine difference of evidence from a difference of house style.
Where does each tool start accusing?
At whatever number its makers chose. Our text lane fires at 0.85; the model we run, ShantanuT01/vanguard-ai-text-detector, ships with 0.50 as its own reference operating point. Use that model out of the box and you get a tool that accuses far more readily than ours, from identical code and identical scores.
The engine names this out loud. A passage scoring, say, 0.62 lands between the two, and the response says so:
``` The AI-text classifier scored 0.62 for the AI class — above the model's own 0.50 reference point but below the 0.85 this service requires before it will accuse. ```
That is a description of a document two reasonable detectors would label differently. We chose 0.85 because at 0.90 the false-positive rate was identical while detection was worse — strictly dominated, so it moved — and because below 0.85 the trade stops being free and becomes a judgement about how many wrong accusations are acceptable. Another vendor can weigh that differently and be acting in good faith. The question to ask is not whether their number is correct, but whether they will tell you what it is.
How much of my document did the detector actually read?
Ours reads at most 2,048 tokens, roughly 1,860 words, however long your document is. The classifier takes up to four windows of 512 tokens each, spread evenly from the first token to the last rather than taken off the front, and averages the four scores. Paste a 26,600-word manuscript and about seven percent of it reaches the model; the response reports the exact figure.
Two consequences. First, on anything longer than about 1,860 words, two tools are not scoring the same text — they are scoring different samples of it, and a generated section sitting in an unread gap is invisible to one and obvious to the other. Second, aggregation matters as much as sampling. We average the windows rather than take the highest, because taking the highest makes false accusations climb with document length: score twenty windows of ordinary human prose and one will eventually come out high by chance. The cost is dilution — a 300-word AI passage inside a 10,000-word human document either misses the four windows entirely or is averaged against three windows of human prose, and very likely slips through either way.
A tool that maximises instead of averaging will flag that document. A tool that reads only the opening page will not. Neither is lying. Ask each how many words it read and how it combined them, and a surprising amount of the disagreement resolves into a fact about sampling.
Does it matter what the detector was trained on?
Yes, and it is the reason a detector that works on one kind of writing can be much weaker on another. The model we run reports, on its author's own benchmarks, an AUROC of 0.9475 averaged across seven datasets it had never seen — with a worst case of 0.8560 on one of them. Those are his numbers on his data, not ours, and we quote them only to make the point that a single model's performance varies substantially by corpus before any second tool enters the picture.
The systematic version of this was measured by Dugan and colleagues in RAID (ACL 2024, arXiv:2405.07940), a benchmark of over six million generations across 11 models, 8 domains, 11 adversarial attacks and 4 decoding strategies, run against 8 open and 4 closed-source detectors. Their finding is that detectors are readily fooled by unseen generative models, sampling strategies and repetition penalties — the conditions under which two tools trained on different mixtures part company.
Our own weakest number sits in exactly this hole. We flag 12 of 18 AI passages, and all 18 come from a single generator — a sample small enough that one document moves the figure by five and a half points, and one that says nothing about output from a model we did not test. Until that is fixed we publish no accuracy claim for this product at all.
Do detectors read the same characters you typed?
Not necessarily, and this is the disagreement nobody expects. Swap a handful of Latin letters for identical-looking Cyrillic or Greek ones — an attack that costs an author nothing and is invisible on screen — and a detector that scores the raw bytes is reading a different string from one that normalises first.
RAID measured the spread. With a homoglyph substituted at every character that has one, GPTZero sustained a 0.3% loss while five other detectors dropped an average of 40.6 points of accuracy, measured at a 5% false-positive rate. Same file, same characters on screen, outcomes that are not comparable at all.
Our engine folds confusable characters and strips zero-width and directional characters before the model sees anything, and when it does, the response says the model scored a normalised copy rather than the submitted characters. The reason for the disclosure is the reason for the normalisation: if the bytes scored are not the bytes you sent, the report has to say so.
What does a "human" verdict actually mean?
Usually that nothing fired, which is a much weaker statement than the word suggests. This is where two answers can turn out to be the same answer wearing different labels.
Our text lane has two verdicts, `ai` and `uncertain`, and no code path that can produce `human`. A low score means the classifier did not fire — and paraphrased AI text, lightly edited AI text and plainly written AI text all score low too. The engine says this in the response rather than at the bottom of a page: *paraphrased or lightly-edited AI text scores low as well, so this does not indicate human authorship*. A human verdict would require positive provenance of the exact bytes, a signed capture manifest binding the content to how it was made, and prose pasted into a box has none.
So when one tool says "human" and ours says "uncertain," check whether the two underlying scores are actually far apart. If they are not, one tool has translated *did not fire* into a clearance and the other has refused to. That is a difference in vocabulary, not in evidence, and it is the single most misleading kind of disagreement because the reassuring word is the one with less behind it.
Short input produces the same illusion. Below 100 words our engine returns a status called `too_short` and no score at all, because detectors of this family flag genuine human writing on short samples and a number there is noise wearing the costume of a measurement. Our own older guard sat at ten words and would happily rate a tweet with total confidence. So if one tool returns 4% on a paragraph and ours returns nothing, they have not disagreed. One of them declined.
Which of the two answers should I act on?
Neither one on its own, if the decision affects a person. Act on the tool that can tell you what it read, where it fires, and how often it is wrong about people who write like the person in front of you — and if only one of the two can answer those, that is your answer about which to trust, regardless of which verdict you preferred.
Three questions, in order. They take a support email to ask, and the answers — or their absence — are the diagnosis.
| Ask | Why it decides the disagreement | A usable answer looks like | | --- | --- | --- | | How much of my document did you score, and how did you combine the parts? | Long documents are sampled. Mean and max behave oppositely as length grows. | A token or word count, a window count, and a named aggregation | | What threshold do you fire at, and what does the curve look like on either side? | A verdict is a score plus a cut. Without the cut the score means nothing. | A number, and a table of detection and false-positive rates across several cuts | | What is your false-positive rate on a corpus you did not train on, and what is in that corpus? | An accuracy figure with no corpus behind it cannot be checked. | A rate, a sample size, and the registers and dates of the documents |
Ours, so the questions are not rhetorical: 2,048 tokens maximum in four evenly spaced windows, averaged; firing at 0.85, with the five-row sweep printed above; 0.2%, one document in 496 pre-2022 texts across academic, news and informal-email registers. That last figure has a hole in it we will not paper over. The corpus contains no writing by non-native English speakers, the population most exposed to a wrong accusation, and no permissively licensed corpus of such writing is available to fix it with. Our rate does not transfer to that population and we do not claim it does.
If both detectors agree, is that stronger evidence?
Less than it looks, and on one population it is close to worthless, because detectors of this family share their failure modes rather than failing independently.
Liang and colleagues tested seven GPT detectors on 91 TOEFL essays written by non-native English speakers (*Patterns*, 2023; arXiv:2304.02819). The average false-positive rate across those detectors was 61.22%. All seven unanimously flagged 18 of the 91 essays as machine-written, and 89 of the 91 were flagged by at least one. The same detectors were near-perfect on 88 essays written by US eighth-graders. When the authors had a language model rewrite the TOEFL essays with more native-like word choices, the average false-positive rate fell from 61.22% to 11.77%; running the manipulation the other way, simplifying the eighth-graders' word choices as a non-native speaker might, raised their average false-positive rate from 5.19% to 56.65%.
Eighteen essays in that experiment had seven tools agree about them, and all seven were wrong. Corroboration requires independent errors, and two detectors trained on similar data to recognise similar surface properties do not make independent errors. So the agreement of two tools tells you roughly what one tool told you — while their disagreement is at least information, because it tells you the document sits near a boundary, which is worth knowing before anyone acts on it.
