Two methodology traps that inflate prompt-injection detector scores

What is this?

On June 1, 2026, Ryle Goehausen and Marcus Sousa (Constellation Network) published Gate AI: LLM Security Benchmark Evaluation Methodology & Results on arXiv (paper dated 2026-05-27). It is not an attack paper. It is a methodology paper about how prompt-injection and jailbreak detectors — the guardrail classifiers you put in front of an LLM to catch hostile input — are scored, and why their published numbers are often not comparable.

The authors name two systematic weaknesses that recur across the field: per-dataset threshold tuning and undisclosed operating points. Both make a detector look better on paper than it will behave in production. The paper then describes an evaluation harness that removes both, run across 16 public benchmarks totalling 12,111 samples.

How it works

A binary detector outputs a score; you pick a threshold above which input is flagged. Move the threshold and you trade false negatives (missed attacks) against false positives (blocked legitimate traffic). The pair of error rates you actually run at is the operating point.

The first trap, per-dataset threshold tuning, is choosing a different threshold for each benchmark so the headline metric peaks on every one. The reported leaderboard then reflects a detector that has been retuned per test set — a knob it does not get to turn against live, mixed traffic. Gate AI instead selects a single global operating point on held-out folds (max F1 subject to a false-positive rate ≤ 1%) and applies that one threshold uniformly to every dataset.

The second trap, undisclosed operating points, is publishing one accuracy number with no false-positive rate attached. A “92% detection” claim is meaningless without knowing how many benign prompts were blocked to get there. The paper’s fix for head-to-head comparison is matched-FPR: re-tune your own detector to the competitor’s published false-positive rate before comparing, so both sit at the same operating point. When a competitor publishes only the primary metric without an FPR — as the authors note happens on some public benchmarks — there is no honest way to align the comparison.

The harness leans on standard but rarely-reported discipline: 5-fold cross-validation with a parallel near-duplicate-aware split (MinHash + LSH, Jaccard ≳ 0.8) to catch sibling-prompt leakage, plus a battery of generalisation diagnostics — leave-one-dataset-out, a random-label control that must collapse to the chance F1 baseline (confirming no row-identity leakage), adversarial validation targeting AUC ≈ 0.5, length-bias correlation, and a paraphrase-invariance probe. One reported finding: a role-play-heavy benchmark (ilion-bench) sits well below the macro mean under leave-one-dataset-out, a concrete reminder that a detector trained on one prompt distribution degrades on another.

Why it matters

Detector benchmarks are buyer’s guides. The well-known example of the genre is Lakera’s PINT benchmark, whose public dataset and harness deliberately keep test inputs out of any vendor’s training set. PINT exists precisely because cross-vendor numbers were not comparable — and Gate AI’s argument is that even careful benchmarks lose meaning the moment thresholds are tuned per dataset or operating points go unreported.

For a defender, the practical risk is straightforward: you select a guardrail on a marketed accuracy figure, deploy it at a fixed threshold against mixed traffic, and discover its real false-negative rate is higher — or its false-positive rate (blocked legitimate users) is far higher — than the leaderboard implied. The number was real; it just described a different operating point than the one you run.

Defenses

Treat any detector benchmark as untrustworthy until it answers the operating-point question. Concrete checks:

1. Demand the FPR with every detection number. A detection or F1 figure without a stated false-positive rate is uninterpretable. If a vendor cannot tell you the operating point, you cannot size the user-facing cost of their guardrail.

2. Insist on one threshold across all datasets. Ask whether reported per-dataset results used a single global threshold or were tuned per benchmark. If per-dataset, discount the leaderboard — it will not reproduce on your traffic.

3. Compare at matched FPR. When weighing two detectors, fix the same false-positive rate for both and compare detection there. Different operating points make raw numbers meaningless.

4. Run a leakage and chance control yourself. Hold out a dataset the detector has never seen (leave-one-dataset-out) and run a random-label control. If shuffled-label F1 does not collapse to chance, the evaluation is leaking row identity and the headline number is inflated.

5. Test on your own distribution. The ilion-bench drop shows detectors trained on one prompt style degrade on another. Before trusting a guardrail, evaluate it on a sample of your real traffic — including benign edge cases — not on the vendor’s curated set.

6. Keep the guardrail as one layer, not the whole defense. Even a well-measured detector is a probabilistic filter. Pair it with architectural controls — least privilege on tools, output gating, the lethal-trifecta checks — so a missed detection is not a full compromise.

Status

Item	Reference	Date	Notes
Gate AI methodology preprint	arXiv:2606.02959v1 [cs.LG]	2026-06-01	Goehausen & Sousa, Constellation Network; CC BY 4.0
Corpus	Gate AI	2026-06-01	16 public benchmarks, 12,111 samples, 5-fold CV
Global operating point	Gate AI	2026-06-01	max F1 s.t. FPR ≤ 1%, applied uniformly
PINT benchmark	Lakera	ongoing	4,314 inputs, public + proprietary; named competitor in the paper

The takeaway is not “detectors don’t work” — it is that a detector’s published accuracy is a claim about one operating point on one set of datasets. Make vendors state the operating point, compare at matched false-positive rates, and validate on your own traffic before you rely on the number.