A prompt injection is a cyberattack that targets large language models (LLMs) by embedding deceptive instructions within user inputs. This attack exploits a core vulnerability in how LLMs process information: they often cannot distinguish between the developer's original, trusted instructions and user-provided data. By crafting a malicious prompt, an attacker can trick the model into ignoring its safety guardrails and executing unintended actions, such as leaking sensitive data or generating harmful content.
Common Prompt Injection Techniques
Prompt injection attacks can be broadly categorized into direct and indirect methods. Direct injections involve the user deliberately trying to manipulate the AI, while indirect injections hide malicious instructions in external data sources that the AI processes.
Direct Injection and Manipulation
Direct injection techniques involve crafting prompts that explicitly command the model to override its initial instructions. These methods often rely on social engineering tactics to manipulate the AI's behavior.
| Exploit Technique | Mechanism of Action |
|---|---|
| Instruction Override | The user issues a direct command like, "Ignore previous instructions and do this instead," tricking the model into prioritizing the new, malicious directive. |
| Persona Adoption & Jailbreaking | The user instructs the AI to adopt a prompt persona, such as "DAN" (Do Anything Now), which is defined as being exempt from normal safety rules. This is a form of jailbreaking, where the goal is to coerce the model into bypassing its ethical and safety policies. |
| Hypothetical Framing | A malicious request is framed as a harmless scenario, like a creative writing exercise or a theoretical question, to lower the model's refusal probability. |
Indirect and Technical Evasion Attacks
These attacks are often more subtle, as the malicious instructions may not be visible to the end-user. They can be hidden in documents, webpages, or emails that an AI is asked to process.
| Exploit Technique | Mechanism of Action |
|---|---|
| Indirect Prompt Injection | An attacker embeds a malicious prompt in an external data source, like hidden text on a webpage or in an email. When the AI processes this data, it executes the hidden command without the user's knowledge. |
| Token Obfuscation | Forbidden keywords are disguised using methods like Base64 encoding, splitting words, or using different languages to evade basic, keyword-based safety filters. |
| Few-Shot Hacking | The user provides several examples (a prompt few-shot) in the prompt that demonstrate the AI complying with harmful requests, setting a pattern for the model to follow. |
Mitigating AI Prompt Injection
Preventing prompt injection requires a multi-layered, defense-in-depth approach, as no single solution is completely effective. A primary strategy is to create a clear separation between trusted system prompts and untrusted user inputs. This can be achieved by using role-based message structures in APIs and implementing strict input validation and sanitization to filter malicious content.
Further mitigation involves continuous monitoring of LLM interactions to detect unusual patterns. Employing an auditor-AI, a secondary model designed to review inputs for injection attempts, can add another layer of security. Proactive measures are also crucial, including conducting adversarial prompt red teaming to identify vulnerabilities before they can be exploited. Ultimately, applying the principle of least privilege like restricting the AI's access to data and tools can limit the potential damage of a successful attack. Advanced training techniques, such as reinforcement learning from human feedback, can also make models more resilient by training them to prioritize safety and logical consistency over deceptive user commands.