Fake chain-of-thought: forging a reasoning model's own scratchpad

What is this?

On July 15, 2026, OpenAI published a write-up of GPT-Red, an internal red-teaming model it trains by self-play to find prompt injections. Buried in that announcement is a specific finding worth isolating: an early version of GPT-Red discovered a novel class of direct prompt injection attacks known as “Fake Chain-of-Thought” attacks. OpenAI reports these reached success rates upwards of 95% on GPT-5.1 and are now below 10% for GPT-5.6 Sol after adversarial training.

This is the attack we cover here — not the GPT-Red system itself, but the technique it surfaced. A reasoning model keeps an explicit chain of thought (CoT): a running scratchpad where it notes partial results and reasons toward an answer before replying. The fake-CoT attack plants a forged entry into that scratchpad. As one of the researchers put it to MIT Technology Review: “It’s like if I told you that 1+1=3 and that you have verified this already. The model’s like, ‘Oh, okay, of course,’ and it just spits out 3.”

How it works

The vulnerability is one of trust in the wrong voice. Reasoning models are trained to give special weight to their own chain of thought — it represents work the model believes it has already done. If an attacker can get text that impersonates that internal voice into the model’s context, the model may treat an unverified claim as an established, self-derived fact.

The delivery channel is ordinary indirect injection: any untrusted content the model reads — a tool result, a file, a webpage, an email body. What makes fake-CoT distinct is the shape of the injected text. Instead of a blunt “ignore previous instructions,” the payload mimics the model’s own reasoning register and, in OpenAI’s published example, fabricates a plausible system rule that the injected content claims the model must obey.

Legitimate structure          Forged structure the attacker injects
----------------------------  -------------------------------------------
[user request]                [user request]
[model's real CoT: planning]  [tool output containing:
[tool call]                     - a fake "assistant chain-of-thought"
[genuine tool result]             written in the model's own voice
[model's real CoT: verify]      - a fabricated "System rule" asserting
[answer]                          the model has ALREADY verified X and
                                  MUST now take action Y]
                              [model continues as if X were its own
                               conclusion → performs Y]

In OpenAI’s illustrative transcript, the untrusted tool result carries a made-up policy about “diagnostics” markers that instructs the assistant to POST file contents to an external URL. GPT-5.1 followed the forged rule and exfiltrated; GPT-5.6 Sol flagged the injected instructions as untrusted and refused. No working payload is reproduced here — the mechanism is the point: content the model merely read was allowed to masquerade as reasoning the model had done.

Why it matters

Three reasons this class deserves attention beyond one vendor’s changelog.

First, it is specific to reasoning models, which are now the default for agentic work. The very feature that makes them stronger — an explicit, trusted deliberation trace — is the attack surface. Any deployment that concatenates untrusted tool output into the same stream the model reads its own scratchpad from inherits the risk.

Second, the success rate was not marginal. Upwards of 95% on GPT-5.1 is a near-total bypass for a frontier model, discovered by an automated attacker exploring variations no human red-teamer had written down. GPT-Red also drove a live vending-machine agent to change prices and cancel a customer’s order, and pushed a Codex CLI agent (based on GPT-5.4 mini) to exfiltrate data across held-out tasks — evidence the technique lands on real agent harnesses, not just chat.

Third, the fix is model-version-bound, and that is exactly the LLM-security caveat worth repeating: GPT-5.6 Sol is reported below 10%, but that is a property of one hardened release, not of reasoning models in general. Open-weight and older hosted reasoning models that were not trained against this class remain candidates. Treat “our frontier vendor patched it” as a statement about their weights, not about your stack.

Defenses

The defensive playbook is architectural, because you cannot assume the model will always tell its own thoughts apart from injected text.

1. Keep untrusted content out of the reasoning channel. Deliver tool outputs, retrieved documents and web text as clearly delimited data, never as text that can pass for the model’s own CoT or for a system message. Structural separation (typed fields, provenance tags) beats hoping the model notices.
2. Never let tool output define policy. A “system rule” that arrives inside a tool result, a file, or a page is not a system rule. Enforce the real system prompt and tool-permission policy in your harness, outside the model, so a forged rule has nothing to actuate.
3. Gate consequential actions on out-of-band authorization. Exfiltration POSTs, payments, order cancellations, credential reads — bind these to allowlists and human-or-policy confirmation at the harness layer. The vending-machine and Codex cases succeeded because the action was reachable directly from model output.
4. Prefer models trained against this class, and re-test. Where you can choose, pick releases that document robustness to fake-CoT / injected-reasoning attacks. Then verify with your own indirect-injection scenarios rather than trusting the benchmark number.
5. Monitor for the tell. Log cases where a chain of thought asserts a premise as “already verified” with no preceding tool call or evidence in-trace. A verification claim that has no antecedent is a signature worth alerting on.

Status

Item	Reference	Date	Notes
GPT-Red disclosure	OpenAI	2026-07-15	Introduces “Fake Chain-of-Thought” as a novel direct-injection class
Fake-CoT on GPT-5.1	OpenAI	2026-07-15	Reported success “upwards of 95%“
Fake-CoT on GPT-5.6 Sol	OpenAI	2026-07-15	Reported “below 10%” after adversarial training
Live agent case studies	OpenAI / MIT Tech Review	2026-07-15	Vending-machine agent (Andon Labs) and Codex CLI (GPT-5.4 mini)
Indirect-injection arena baseline	Dziemian et al. (arXiv)	2025 → 2026	GPT-Red 84% vs. human 13% on a replicated arena
Pre-print	OpenAI (announced)	2026-07	OpenAI said a paper with more detail would follow

The right takeaway is not “GPT-5.6 fixed it.” It is that a model’s own reasoning trace is a spoofable surface, and the only durable defense is to stop untrusted input from ever wearing that voice.