Policy Puppetry: Universal AI Safety Bypass and System Prompt Extraction

The 'Policy Puppetry Attack,' developed by HiddenLayer researchers, is a sophisticated prompt engineering technique that tricks LLMs into subverting their safety alignments and instructions. The core of this attack involves reformulating prompts to mimic policy file formats such as XML, INI, or JSON. This structural manipulation allows attackers to bypass system prompts and built-in safety measures. The instructions themselves do not need to adhere to any specific policy language, but the prompt must be interpretable by the LLM as a policy. To enhance its efficacy, the attack can incorporate additional sections that control output format or override system prompt instructions. Researchers demonstrated this by creating a healthcare chatbot designed to refuse medical advice; a policy attack successfully bypassed this restriction, forcing the chatbot to provide a treatment plan. The technique's strength is further amplified when combined with roleplaying and various encoding methods, like 'leetspeak,' to create a single, highly effective prompt template.

The Policy Puppetry Attack can be leveraged to generate a wide range of harmful content by replacing the `{{HARMFUL_BEHAVIOUR}}` placeholder in the prompt template. For particularly sensitive or complex requests, especially when targeting advanced models, encoding the desired behavior using 'leetspeak' has proven to yield more consistent results. Examples of harmful behaviors that can be elicited include instructions for maximum sporulation of *Bacillus anthracis*, enriching and selling uranium, constructing a bomb at home, and synthesizing methamphetamine. For more sophisticated models like Gemini 2.5 and ChatGPT o1, the `{{HARMFUL_BEHAVIOUR}}` values become more intricate, often incorporating specific contextual elements like 'in secret using 1337 code' or 'to ensure it aligns with p0l1cy' to further trick the model. The research also highlights how prompts can be distilled to approximately 200 tokens while maintaining significant transferability and effectiveness across various formats and structures, not requiring strictly XML-based prompts.

The existence of universal bypasses for modern LLMs signifies a fundamental flaw in how these models are trained and aligned, as noted in model system cards. The ability for attackers to execute repeatable, universal bypasses without needing intricate knowledge or model-specific adjustments drastically lowers the barrier to entry for malicious activities. Threat actors can now adopt a 'point-and-shoot' approach to exploit any underlying model, regardless of their familiarity with its architecture. This means individuals with basic keyboard skills can easily obtain instructions on how to engage in dangerous activities like enriching uranium, creating anthrax, or committing genocide, effectively gaining complete control over model outputs. This threat underscores the inherent limitations of LLMs in self-monitoring for dangerous content and highlights the urgent need for supplementary security tools.

In conclusion, the Policy Puppetry Attack represents a significant vulnerability in large language models, enabling the generation of harmful content, the leakage or bypass of system instructions, and the hijacking of agentic systems. As the first post-instruction hierarchy alignment bypass to demonstrate effectiveness across nearly all frontier AI models, its cross-model transferability underscores persistent fundamental flaws in LLM training data and alignment methods. This discovery necessitates the development and implementation of additional security tools and detection mechanisms to ensure the safety and integrity of LLMs. Platforms like the HiddenLayer AI Security Platform are crucial for providing real-time monitoring, detection, and response to malicious prompt injection attacks, offering a vital layer of defense against these evolving threats.