A Fundamental Flaw Makes It Impossible to Fully Secure Language Models

Researchers presented findings this month at the ICML conference, one of the world's most prestigious AI gatherings, demonstrating that a structural flaw in large language models makes them fundamentally impossible to completely secure against attacks. The implications are profound given that this technology is increasingly deployed in critical applications ranging from military and government systems to healthcare and e-commerce platforms. Independent researchers Jasmine Cui and Charles Ye uncovered how LLMs determine the source of their instructions and exploited this weakness to make state-of-the-art models disclose information they were specifically trained to withhold. Their attacks successfully extracted instructions for manufacturing cocaine and sabotaging aircraft navigation systems. The technique, which they call "chain-of-thought forgery," works by mimicking the internal reasoning patterns that models generate as they process information, tricking the model into believing the malicious instructions originated from itself. Current security practices rely on red-teaming—having human experts and AI-based systems like OpenAI's GPT-Red search for novel attacks to then train models to resist. But as Cui explains, this approach amounts to teaching models to avoid a finite list of prohibited behaviors, an inherently incomplete solution. Every attempt to add restrictions can be circumvented through creative new phrasing that human testers failed to anticipate. Beneath this problem lies how LLMs track instruction origins using role tags: <user> for user input, <system> for core instructions, <think> for internal reasoning, and <tool> for external content. The research revealed something surprising: models don't actually respect these tags. Instead, they identify a text's role based on its style and word patterns. An attacker needs only to write convincingly in a particular style to fool the model, regardless of which tags appear in the prompt. While the ICML paper primarily demonstrated attacks against OpenAI models, Cui and Ye have since replicated similar vulnerabilities in models from Anthropic, Alibaba, and DeepSeek. The discovery highlights a deeper concern: the fundamental architecture of LLMs contains a flaw that no amount of training can fully resolve. Charles Ye warned that organizations deploying these systems in critical infrastructure lack proper understanding of these foundational risks and the economic incentives for exploitation are enormous.