Why independent AI-agent developers keep missing security risks

In brief Most of the security failures in deployed AI agents are not exotic exploits — they are gaps that the people building the agents never modeled in the first place. A study published in June 2026 (arXiv 2606.03190) interviewed independent AI-agent developers about how they reason about security and privacy. The finding: builders think almost entirely from the user’s point of view, treating “is the output harmful?” as the whole of safety, while showing low awareness of adversarial risks like prompt injection, tool abuse, and data exfiltration. The mitigations they do apply are hand-written, inconsistent, and incomplete. This is a human-factors finding, not an exploit — but it explains why so many of the attacks we cover keep working.

What is this?

The paper, Focused on the User, Overlooking the Risks (arXiv 2606.03190, June 2026), is an interview-based study of independent developers who build and ship AI agents — the solo builders and small teams behind a large share of the assistants, automations, and “GPT-style” apps now in production. Rather than testing models, the researchers asked the people building on top of them how they understand security and privacy, what they actually do about it, and where they get stuck.

The headline result is a structural mismatch between the developers’ mental model and the real threat surface. Builders reason from the perspective of their end users and optimize for user-facing safety — preventing the agent from saying something harmful, offensive, or off-brand. That framing crowds out the adversarial perspective, where the threat is not the user but a third party who plants instructions in a web page, a document, a tool result, or a memory store.

What the study found

Three observations stand out, all consistent with what defenders see in the wild.

First, safety is conflated with content moderation. Asked about “safety,” developers reach for harmful-output filtering and rarely mention injection, exfiltration, or privilege boundaries. Security risk and model capability limits get blurred together, so an architectural vulnerability looks, to the builder, like a quality issue to be smoothed over with better prompting.

Second, defenses are manual and ad hoc. The study reports that developers rely almost exclusively on hand-crafted solutions — bespoke prompt wording, one-off input checks — which are inconsistent and incomplete across projects. There is little use of systematic, automated guardrails, and little adversarial testing of the agent before release.

Third, cross-border data flows are an unmanaged risk. Because many independent developers wire their agents to global LLM APIs and serve users in multiple jurisdictions, user data routinely crosses borders without an explicit privacy model. The study frames this as a global-ecosystem problem, not a local one: the same pattern shows up wherever small teams build on hosted frontier models.

The picture matches a separate June 2026 measurement effort, the Cambridge-hosted AI Agent Index, which found that most deployed agents ship without basic safety and risk disclosures at all. The two results describe the same gap from different ends — builders who do not model the risk, and products that do not document it.

Why it matters

Almost every attack class on this site assumes a defender who has thought about the attack: the lethal trifecta, indirect prompt injection, tool-description poisoning, memory poisoning. This study is the supply-side explanation for why those attacks remain so productive — a large population of agent builders is not modeling the adversary at all. You cannot rate-limit, sandbox, or filter a risk you have not represented.

It also reframes where defense should be invested. If the gap is awareness and tooling rather than malice or incompetence, then secure-by-default frameworks, built-in adversarial test harnesses, and clear privacy defaults do far more good than another awareness blog post. The fix has to live in the platforms and libraries developers already use, because that is the only layer that reaches builders who never went looking for a threat model.

Defenses

The study’s own implications point at automation, testing, and accountability. Concretely, for anyone shipping agents:

1. Adopt a real threat model, not a content filter. Treat every external input the agent reads — web pages, files, tool outputs, retrieved documents, prior memory — as attacker-controlled. Harmful-output filtering is necessary but is not security.

2. Use structural patterns, not prompt wording. Lean on the constrained-agent design patterns (least privilege, action allow-lists, separating planning from tool execution, human approval for irreversible actions) instead of trying to word a system prompt into safety. Fortified prompts buy little robustness.

3. Make adversarial testing automatic. Add injection and exfiltration test cases to CI so the agent is attacked on every change, not hand-reviewed once. Manual, per-project checks are exactly what the study found to be inconsistent.

4. Model cross-border data flow explicitly. Document which provider sees which data, where it is processed, and what leaves the user’s jurisdiction. Default to minimizing what the agent forwards to hosted models.

5. Ship risk disclosures. State the agent’s capabilities, the data it touches, and its known limitations — the thing the AI Agent Index found mostly missing. Disclosure is cheap and forces the threat-modeling conversation.

Status

Item	Reference	Date	Notes
Focused on the User, Overlooking the Risks	arXiv 2606.03190	2026-06	Interview study; user-centric mental model, low security awareness, manual/ad-hoc defenses
AI Agent Index safety disclosures	University of Cambridge	2026-06	Most deployed agents ship without basic safety/risk disclosures
Design patterns for securing LLM agents	arXiv 2506.08837	2025-06	Constrained-agent patterns referenced as the structural alternative

The useful framing is not “developers are careless.” It is that the dominant mental model of agent safety — keep the output clean for the user — does not contain the adversary, and the tooling most builders reach for does not add it back. Until secure defaults and automated testing close that gap at the framework level, the attacks documented here keep finding unmodeled surface.