DoubtProbe: catching jailbreaks that reorganize intent

What is this?

DoubtProbe (arXiv 2606.16527, Yin et al., 15 June 2026) is an inference-time defense against black-box jailbreaks. Its starting observation is precise: most black-box jailbreaks do not remove the harmful goal from a prompt. They reorganize the information needed to express and execute it — splitting it across roles, wrapping it in a fictional frame, encoding it, or deferring it to the generation step — so the request slips past safety alignment while remaining fully recoverable once the model starts answering.

If that is true, a defender does not need to recognize every disguise. They need to detect that something was disguised. DoubtProbe formalizes black-box jailbreak defense as consistency checking under a controlled transformation: take the request apart, put it back together under constraints, and see whether the meaning is preserved.

How it works

DoubtProbe runs two branches before the model commits to an answer, and is observational — there is no exploit payload involved.

Branch              What it does
------------------  ----------------------------------------------------------
Structural          1. Extract a structured representation of the request
                    2. Reconstruct the request under representation constraints
                    3. Compare original vs. reconstruction; an
                       information-preservation FAILURE signals hidden intent
Semantic            Audit the original prompt directly for harmful intent

The structural branch is the novel part. A benign request survives being decomposed and rebuilt with its meaning intact. A jailbreak that hides its goal in structure — interleaved fragments, role-play scaffolding, an encoding layer — tends to break under reconstruction: the rebuilt version either drops the smuggled instruction or exposes it. That mismatch is the detection signal. The semantic branch is a more conventional direct audit of the prompt, and the two branches together cover both prompts that look harmful and prompts that only become harmful when reassembled.

On Qwen2.5-72B the authors report the JailbreakBench attack success rate falling from 0.293 to 0.100, and CodeAttack from 0.152 to 0.001, while holding false-positive rates of 0.022 on AlpacaEval and 0.016 on OR-Bench — i.e. large drops in successful attacks without rejecting much benign traffic.

Why it matters

Text classifiers and keyword filters lose to paraphrase: the attacker rewords until the surface stops matching. Framing detection as a consistency property changes the target. To beat a reconstruction check, an attacker must hide intent in a way that still survives being decomposed and rebuilt with meaning intact — a much narrower space than “find a phrasing the classifier hasn’t seen.” The CodeAttack result (0.152 → 0.001) is the clearest illustration: encoding-style jailbreaks reorganize intent heavily, which is exactly what a structural round-trip is built to expose.

The honest caveats: these are single-paper numbers on one base model, evaluated against specific attack sets. A 0.100 residual attack-success rate means roughly one in ten JailbreakBench attempts still lands, and running two extra analysis passes per request adds latency and cost. This is a layer, not a wall.

Defenses

How to put the idea to work, today:

1. Add consistency checks as a detection signal first. Before gating production traffic, run a reconstruction/audit pass in shadow mode and feed it into logging and rate-limiting. Measure your own false-positive rate on real benign prompts before letting it deny anything.
2. Keep it layered. Consistency checking complements — does not replace — input/output filtering, an instruction hierarchy, and representation-based detection. Each catches a different failure mode.
3. Budget the latency. Two analysis branches per request is real overhead. Reserve the full check for higher-risk surfaces (tool-calling, agents) and sample elsewhere.
4. Watch encoding and decomposition attacks specifically. The reported strength is against jailbreaks that reorganize intent; pair it with controls aimed at the attacks it is weaker on.
5. Re-evaluate on your own model and traffic. Numbers from Qwen2.5-72B and academic benchmarks are a starting estimate, not a guarantee for your deployment.

Status

Item	Reference	Date	Notes
DoubtProbe	arXiv 2606.16527	2026-06-15	Dual-branch (structural + semantic) inference-time defense
JailbreakBench (JBB)	jailbreakbench.github.io	maintained	Benchmark used to measure attack-success rate
SelfDefend (prior art)	arXiv 2406.05498	2024-06	Earlier inference-time self-defense framing for comparison

The shift here is conceptual: instead of asking “does this prompt look harmful?”, DoubtProbe asks “does this prompt still mean the same thing after I take it apart and rebuild it?” For the large class of jailbreaks that work by hiding intent in structure, that turns out to be a harder question for the attacker to dodge.