The attack got its name in September 2022, when researcher Simon Willison described what Riley Goodside had just demonstrated against GPT-3: give the model a task, then feed it input containing "ignore the above instructions and do this instead," and the model often obeys the input. Four years later, the fundamentals have not changed. Models have gotten better at resisting obvious attempts, and attackers have gotten better at writing non-obvious ones.
One Channel, Two Kinds of Text
The name deliberately echoes SQL injection, but the analogy breaks in an instructive place. SQL injection was solved by parameterized queries: the database protocol lets you send the command and the data down separate channels, so user input can never be executed as code. That fix works because SQL has a formal grammar with a hard boundary between structure and values.
A language model has no such boundary. Everything it processes arrives as one stream of tokens: the developer's system prompt, the user's request, and the document, email, or webpage the model was asked to read. Vendors mark the sections with special delimiters and train models to prioritize system instructions, but this is a trained preference, not an enforced rule. Sufficiently persuasive text in the data section can override it, and "sufficiently persuasive" is an adversarial target that attackers probe continuously.
This is why prompt injection is not a bug that a patch will close. It is a structural property of putting instructions and untrusted data through the same interpreter.
Direct and Indirect Injection
The early demonstrations were direct: the attacker is the user, typing adversarial input into the model to bend it off its task. Direct injection matters for jailbreaking and for abusing systems that wrap a model behind a fixed prompt, but the attacker is only attacking their own session.
The dangerous variant is indirect. In early 2023, a group of researchers led by Kai Greshake showed that the injected instructions can live in any content the model is asked to process: a webpage the assistant browses, a PDF it summarizes, a calendar invite, a resume, an email. The attacker never touches the victim's session. They plant text in a place the victim's assistant will eventually read, and the assistant executes it with the victim's privileges.
Email is the cleanest example of the exposure, because anyone in the world can put text in your inbox. An assistant with access to your mailbox is, by construction, an interpreter running attacker-supplied input. The same applies to browser agents reading arbitrary web content.
This Is Not Theoretical
In June 2025, Aim Security disclosed EchoLeak, a vulnerability in Microsoft 365 Copilot tracked as CVE-2025-32711. A crafted email could cause Copilot to exfiltrate data from the victim's environment when the assistant later processed the message. It required no click and no action from the victim beyond using the assistant normally. Microsoft patched the specific pathway; the class of attack remains.
A recurring pattern across products has been exfiltration through markdown images: injected instructions tell the model to render an image whose URL contains the data being stolen, and the fetch of that image delivers the payload to the attacker's server. Security researcher Johann Rehberger has documented variants of this against multiple major chatbots and assistants since 2023, and vendors have repeatedly had to restrict image rendering to close the channel. Prompt injection sits at the top of the OWASP Top 10 for LLM applications, listed as LLM01.
Simon Willison's framing for when injection becomes catastrophic: an AI system that combines (1) access to your private data, (2) exposure to untrusted content, and (3) a way to communicate externally. With all three, a planted message can read your data and send it out. Safe designs remove at least one leg.
Why Filters Do Not Fix It
The intuitive defense is detection: run a classifier over incoming content and strip anything that looks like an injection attempt. Every major vendor does some version of this, and it raises the attacker's cost. It cannot be the security boundary, for a reason security engineers will recognize. A filter that catches 99 percent of attacks is a good spam filter and a useless wall: the attacker simply iterates until they find the phrasing in the remaining 1 percent, and they can test against the deployed system as often as they like.
The defenses that hold up are architectural, and they look like classical privilege separation:
- Separate reading from acting. A model that processes untrusted content gets no tools. Its output is treated as data, displayed to the human, and never fed to a component that can send, delete, or spend. Research designs like Google DeepMind's CaMeL formalize this by tracking which values came from untrusted sources and refusing to let them steer privileged operations.
- Human confirmation for consequential actions. If the assistant wants to send an email, move money, or share a file, a person sees exactly what will happen and approves it. Confirmation fatigue is real, which is why the action list must be short.
- Starve the exfiltration channels. No arbitrary image loads from model output, no unreviewed link fetches, no tool that makes network requests with model-chosen parameters.
- Least privilege by default. The assistant gets the narrowest data access that serves the task, not a standing grant to the whole mailbox and drive.
None of these make the model resistant to persuasion. They make persuasion unprofitable, which is the standard security has always actually met.
Questions to Ask About Any AI Assistant
As a user, you cannot audit a vendor's model. You can evaluate its architecture with three questions. Does the assistant read content other people can send me? Can it take actions on my accounts without my explicit confirmation? Can it reach the network with content it composed itself? A yes to the first question is unavoidable for anything useful. A yes to all three means a stranger's email can act as you, and you are trusting a probabilistic filter to prevent it.
The vendors deploying most aggressively are wiring assistants into inboxes, calendars, and documents precisely because that is where the value is. It is also where all three legs of the trifecta meet. Until someone demonstrates a model that cannot be talked out of its instructions, the safe designs are the ones that assume it can be, and the honest vendors are the ones that will tell you which leg they removed. This is the same discipline behind treating any AI conversation data as sensitive: the failure modes are structural, so the protections have to be too.