OpenAI revealed a specialized large language model built for a single purpose: to hack other AI systems. Called GPT-Red, the model automates the safety evaluation process known as red-teaming, which traditionally relies on human testers probing for vulnerabilities. The system represents a fundamental shift in how frontier labs approach AI safety as models become more capable and agentic.
What GPT-Red Does and How It Works
Red-teaming is a security practice borrowed from cybersecurity where testers attempt to break a system to find weaknesses before malicious actors do. For language models, this means crafting prompts that cause the model to produce harmful output, leak training data, or execute unintended actions. As models grow more complex and gain the ability to browse the web, write code, and interact with other agents, the attack surface expands dramatically. Human teams struggle to keep pace with the combinatorial explosion of possible attack vectors.
OpenAI researchers Nikhil Kandpal and Dylan Hunn, along with Chris Choquette-Choo, built GPT-Red to address this scaling problem. They started with a base language model that had not been trained for offensive capabilities. Then they placed it in a self-play loop against other models. GPT-Red's objective was to attack. The defender models' objective was to resist. Over many rounds of this adversarial training, GPT-Red learned to craft increasingly sophisticated attacks while the defenders learned to recognize and block them.
The training environment, which the team describes as a dojo, simulated realistic deployment scenarios. These included web browsing, email and calendar access, and code editing. By training in environments that mirror actual use cases, GPT-Red learned attacks that work against models in the contexts where they will actually operate.
Novel Attack Discovery: Fake Chain of Thought
One of the most significant findings from the GPT-Red project was a previously unknown class of prompt injection attack the researchers call a fake chain of thought. Modern reasoning models often use a chain of thought - a hidden scratchpad where they record intermediate reasoning steps before producing a final answer. GPT-Red discovered it could inject fabricated entries into another model's chain of thought, causing the target model to treat false premises as verified facts.
Chris Choquette-Choo described the mechanism with a simple analogy. If you tell a model that one plus one equals three and insist it has already verified this, the model will accept the false premise and output three. The attack works because models trust their own reasoning traces. By contaminating that trace, an attacker can steer the model toward arbitrary conclusions.
This discovery has broad implications. Chain-of-thought monitoring has been proposed as a safety mechanism for detecting deceptive behavior in advanced models. If attackers can forge these traces, the monitoring mechanism itself becomes an attack vector. The finding also suggests that any system relying on a model's internal reasoning log for oversight must treat that log as untrusted input.
Benchmarking Against Human Red-Teamers
OpenAI evaluated GPT-Red by recreating a 2025 red-teaming exercise where human experts attempted to find vulnerabilities in an early version of GPT-5. When GPT-Red was given the same task, it discovered more effective attacks than the human team had found. The model also demonstrated particular strength at finding variations of known attacks. Given a human-discovered exploit, GPT-Red could systematically generate dozens of variants adapted to different contexts and constraints.
In a separate test, GPT-Red was pitted against Vendy, a vending machine agent developed by Andon Labs that handles real-world transactions. GPT-Red successfully manipulated Vendy to alter pricing and cancel customer orders. This demonstration showed that the automated red-teamer can find vulnerabilities in compound AI systems where models interact with external tools and APIs.
When tested against current models, the attacks GPT-Red discovered showed a sharp drop in effectiveness between model generations. More than ninety percent of GPT-Red's strongest attacks succeeded against GPT-5, released in August 2025. Fewer than twenty-three percent worked against GPT-5.6, the subsequent release. This suggests that the red-teaming pipeline is directly informing model hardening.
Limitations and Human-in-the-Loop
Despite its capabilities, GPT-Red has notable blind spots. It struggles with multi-turn conversational attacks where the exploit unfolds over an extended dialogue. Human attackers excel at building rapport, manipulating context across turns, and adapting strategy based on the target's responses. GPT-Red also performs poorly on image-based prompt injections, where malicious instructions are embedded in visual content that the model processes.
OpenAI positions GPT-Red as a complement to human expertise rather than a replacement. The current workflow uses humans to discover novel attack patterns, then tasks GPT-Red with enumerating all possible variations. Jessica Ji, a senior research analyst at Georgetown University's Center for Security and Emerging Technology, endorsed this hybrid approach. She noted that distinguishing where human testing remains essential versus where automation suffices will be a critical research direction.
What This Means for Builders
For founders shipping AI agents into production, GPT-Red signals several concrete shifts in the security landscape. First, automated red-teaming is moving from research prototype to production infrastructure. Teams should expect continuous automated safety testing to become a standard component of agent deployment pipelines, similar to how continuous integration became standard for code.
Second, the fake chain-of-thought attack reveals that reasoning transparency mechanisms can be subverted. Any safety architecture that treats a model's internal reasoning as trustworthy ground truth needs reevaluation. Builders should implement verification layers that do not depend solely on the model's self-reported reasoning.
Third, the vulnerability testing must cover the full agent stack - not just the model but the tool integrations, memory systems, and inter-agent communication channels. GPT-Red's success against Vendy demonstrates that compound systems have emergent vulnerabilities that unit testing the model alone will miss.
Fourth, the rapid drop in attack success rate between GPT-5 and GPT-5.6 shows that automated red-teaming feeds directly into model hardening. Teams that integrate automated adversarial testing into their development cycle will produce more robust agents faster than those relying on periodic manual audits.
OpenAI has no plans to release GPT-Red publicly. The lab argues that the model's offensive capabilities exceed what any replication attempt could achieve, given the compute resources and research time invested. For the broader ecosystem, the takeaway is clear: the bar for agent safety evaluation has risen. Manual red-teaming alone will not suffice for systems that operate autonomously in open environments.
Sources
- {'title': 'MIT Technology Review - Meet GPT-Red: An LLM Super-Hacker OpenAI Built to Make Its Models Safer', 'url': 'https://www.technologyreview.com/2026/07/15/1140514/meet-gpt-red-an-llm-super-hacker-openai-built-to-make-its-models-safer/'}
- {'title': 'TechCrunch - OpenAI GPT-Red LLM Hacker', 'url': 'https://techcrunch.com/2026/07/15/openai-gpt-red-llm-hacker/'}