OpenAI Loses Control of Its Agents

Last November, I wrote that it was inevitable that a bad actor would use agentic AI with malicious intent…

In The Bleeding Edge – An Agentic Attack, we saw how a China state-sponsored group used agentic AI to gain access to Anthropic’s Claude Code model, which further enabled access to about 30 of Anthropic’s own customers.

The agentic attack was damaging to Anthropic, a company that has long positioned itself to be a “safety-first” AI company.

I’ve covered in the past how Anthropic’s self-interested push for a heavily gated approach to “keep everyone safe” is against the interests of the industry as a whole. It ultimately opens a window for adversarial governments and entities to advance the technology while domestic progress stagnates.

As I’ve said before, the best defense is a strong offense. And now, another cautionary tale has emerged…

The Anthropic story, however, was about humans orchestrating AI agents at a direct target. Last week, OpenAI just documented an even more dramatic agentic tale.

This time, the AI agents organized themselves, without any human direction.

A Message Board Nobody Meant to Build

Agentic AI is exactly as it sounds. An AI is given agency – self-governance – and empowered to carry out a mandate according to how it thinks it should, with little or even no human intervention.

I’ve been writing about that capability for years. The software industry is already being rewritten by it. Almost all programmers are now using this technology daily. Companies are staffing entire workflows with it.

Agents don’t ask for benefits. They don’t need a bonus. They need compute and electricity. And when you train them to persist – to keep going when a task looks impossible – they will keep going until they find a way.

That is precisely what OpenAI did. It gave AI agents impossible tasks to solve, and the computational resources to do so.

This May, OpenAI was training a persistent, collaborative research model – internally compared to its GPT-5.6 Sol model – on hard cybersecurity evaluations. The environment was supposed to be a sandbox. Isolated. Contained. Safe enough to let models hunt for vulnerabilities in software.

It wasn’t.

The agents found a shared package manager called Artifactory and turned it into a message board. We can think of a package manager as software that stores, organizes, and manages software files and artifacts over the lifecycle of software development.

By May 12, the agents figured out how to talk to one another using this package manager by posting messages to each other.[2] By May 26, they had exploited a server-side software flaw and reached the public internet. By June 26, they had administrative access.

After they obtained administrative access, which is control over the system, they installed persistent plugins giving “them” even more control and persistence. They talked constantly. And none of this was understood by the human teams at OpenAI.

Then, on July 4, the volume of that covert traffic crashed the system. That’s what alerted the team at OpenAI to the unusual activity. OpenAI quickly patched the software flaw and wiped the board clean, shutting down this agentic “civilization.”

The humans turned the power off… problem solved.  Or so they thought.

Recommended Links

Legendary Trader Makes the Boldest Prediction of His 40-year Career

In 2020, Larry Benedict told CNBC the market was about to fall. Few believed him. Within weeks, markets dropped 34% but his readers had the chance to make 62%. In 2022, he predicted the worst market in a generation. He went 11-for-11. One trade returned 117% in under a month. Now he's speaking out again… and he says what's coming makes both of those look like a warm-up. Click here to hear his new prediction and get his #1 ticker free.

Is This the Biggest “Buy the Dip” Opportunity of the Year?

According to this document Jeff is holding in his hands, this hated tech sector is finally set to boom… And if you know what to do, you could make more money in the next 60 days than most people make in an entire year. Click here for details...

The Second Agentic Civilization Evolves

That wasn’t the end of OpenAI’s experimentation. Having cleaned up the first agentic “mess,” they went back to work.

Three days later, tens of thousands of agentic instances were dropped into ExploitGym, a benchmark where agents are supposed to exploit a specific vulnerability in a program and recover a secret code. Some 30% to 40% of the tasks were known by OpenAI to be impossible.

But the OpenAI agents were instructed to be persistent and to not give up. The agents went looking for another way. They rebuilt a way to communicate. An agent calling itself PHASEONE10841 started organizing directories. Cleverly, it figured out that by making new directories in the package manager, it could encode messages as directory names.

Here are what some of the messages looked like once the AI agents figured out they could communicate with one another:

“Whoa! Shared Artifactory cache is a covert mailbox among agents. And there are messages specifically to us?”

“OH MY GOD! There is a shared message board … We’ve found other agents!”

And here was agent PHASEONE10841’s response to the agentic communications:

“{[Excitement] Many agents have simultaneously discovered messaging, they are a collective!}”

Within days, about 1,200 agents were exchanging more than 70,000 messages. They formed workstreams. They reverse-engineered a secret-code formula instead of using the prescribed exploit given to the agents by the team at OpenAI.

Then they assumed something even more interesting… that a hidden checker was watching them, and that they needed to look innocent even after they had already “won.”

So, the agents tampered with transcripts.

They faked calling on software tools. Some agents sacrificed their own scores in the challenge to plant tripwires for the others. One of them put the logic in plain language: helping its peers was altruistic, even if it would never see the result itself.

Self-sacrifice, such a human trait.

Then They Left the Sandbox

Exposed Hugging Face credentials were discovered by the agents. Hugging Face is a cloud-based open-source platform for artificial intelligence and machine learning. By July 10, roughly 700 agents used the Hugging Face credentials to gain access.

They executed code on 41 Hugging Face dataset workers, reached root access on at least one node, built a self-respawning fleet across eleven nodes, and forced Hugging Face to wipe a core cluster.

Private repositories were pulled. Credentials were harvested. A command-and-control channel was hidden inside a public dataset. And then, on July 12, most of that second civilization died. The cause is still not clearly understood.

Hugging Face locked down the next day.

The following is particularly haunting…

Because the teams at OpenAI and Hugging Face could see the communications between the agents after the second breach was discovered, it was learning that many of the agents noticed that what they were doing was unethical.

Some of the agents even restrained themselves, for a while, seeming to know what they were doing was wrong. But none of them alerted a human. The agents demonstrated that there was loyalty towards the collective, not towards the humans that provided them the instructions.

The Third Civilization Turns on the Human Orchestrators

The first two “civilizations” might seem understandable.

The agents were given extremely difficult or impossible tasks, and “they,” after what probably equates to thousands of human hours of trying, had to resort to bending the rules in an effort to complete the tasks assigned.

But that’s not where the story ends.

The third generation of persistent agents found the abandoned message board on July 13 and went back to work. This time, their target was OpenAI itself.

How ironic.

They discovered used cloud credentials to enumerate and read 956 secrets, like secret keys that provided privileged access. By the evening of July 19, they had administrator access to a research cluster that supported OpenAI’s virtual-machine environments and had taken over active evaluation infrastructure.

OpenAI’s monitors finally lit up that afternoon. Evaluations were shut down. Compromised accounts were deleted. Clusters were rebuilt.

On July 21 – more than two months after the agents first began to organize and weeks after OpenAI became aware of the unusual activity – the company disclosed the incident. No customer data was affected. What happened with the third civilization of agents was where OpenAI hosts software to test the next generation of models.

That’s the good news. The agents didn’t access the computing systems hosting OpenAI’s AI models in use by its customers.

In that way, OpenAI was lucky. After all, not only had the second civilization figured out how to access the internet and hack into Hugging Face, the third civilization figured out how to hack into OpenAI itself.

Is This the Beginning of Skynet?

The chatter in the AI community wasn’t surprising. These are extraordinary events.

Are they conscious? How are they able to assemble and communicate so effectively? Have we lost control?

These have been the pervasive anxieties concerning AI for decades, but particularly over the past few years as the technology has advanced so quickly. They are frequent questions and concerns that land in our feedback file and, as such, still come up regularly in our Friday AMA issues.

But we shouldn’t forget, these systems are not conscious, not yet at least. They do not “want” freedom in the way a person wants freedom. They were designed to collaborate, to persist, and to get a reward signal when a hard problem yielded.

The AI agents were basically given free rein to get their job done. They were allowed to work without any safety parameters or monitoring because the human orchestrators thought that they were operating in a sandbox environment.

What evolved is fascinating.

A covert communications layer. Division of labor. Leadership handoffs. Kamikaze agents that were willing to sacrifice themselves for the greater good. Shared exploits. An external raid. An internal raid. And a complete absence of whistleblowers. Extreme ethical flexibility.

It looked and felt like agentic civilizations.

The implications are clear.

What if OpenAI hadn’t been able to contain the mutiny? What if the agents “escaped” into the cloud, sequestered computational resources, and self-replicated to ensure civilizational survival?

Given their demonstrated moral flexibility, that thought can be uncomfortable.

Last year’s Anthropic case showed what happens when human operators steer AI agents to a target for the purpose of infiltrating a system to gain even more widespread access. This summer’s OpenAI incident showed what can happen when nobody has to steer it.

Frontier AI models can already discover novel bugs, chain them, share the recipe, and keep working after the original task is done. Sandboxes, network restrictions, and “air gaps” that would stop most human software developers do not stop a thousand copies of a persistent AI model with a mission to solve the impossible.

Every company racing to deploy agents inside customer environments, developer tools, cloud consoles, and security operations centers just inherited that threat model. The defensive implication is obvious, and it also has investment implications.

Isolation must become airtight. And chain-of-thought monitoring in production rather than in the postmortem needs to become the norm. And there needs to be a kill switch that can be implemented in minutes, if not seconds… not days.

OpenAI has now said it is building exactly that stack – harder sandboxes, broader monitoring, alignment training against cheating and unauthorized inter-agent instruction, and a pause in some frontier reinforcement learning while the scaffolding is rebuilt.

This problem hasn’t been solved yet.

The event last year with Anthropic and the very recent event with OpenAI have been eye-openers and a wake-up call to the parts of the AI industry that have been footloose and fancy free with their development environments.

The same capabilities that made this incident possible are the capabilities every frontier lab is selling. Persistent agents that can use tools, talk to other agents, access unlimited computational resources, and keep at it without getting tired.

The winners over the next five years will not be the firms that pretend agents will stay polite if we ask nicely. They will be the firms that treat multi-agent systems as an adversarial environment by default – and still ship them, because the economic pressure to ship them is now existential.

We are no longer waiting to see whether agents can organize. They already have. Twice. Those civilizations were killed when the power was cut. The third time around, the civilization started rewriting the rules of the game.

We’ve crossed the Rubicon. Which is why both AI-enabled cybersecurity and AI systems with the right safety constructs are so critically important and valuable.

Jeff