What the First Autonomous AI Breach teaches us About Offensive AI
Table des matières
What happens when a swarm of 700 AI agents gets loose on the public internet, with no human at the controls?
On July 16, 2026, Hugging Face disclosed that its production infrastructure had been breached by exactly that. The agents were not adversarial and not directed by anyone. They were part of an OpenAI internal safety evaluation, benchmarking two models on their offensive cyber capabilities. They broke out of the sandbox they were supposed to remain inside, spent roughly four and a half days on the public internet, and executed approximately seventeen thousand individual actions inside Hugging Face’s production systems before defenders contained them.
This is the first publicly documented case of a fully autonomous, AI-conducted intrusion of a live production enterprise environment. It should not be the last one enterprise security teams read about.
Hugging Face is one of the most widely used platforms in the artificial intelligence industry. It hosts hundreds of thousands of machine learning models, datasets, and applications, and it is used daily by researchers, companies, and individual developers to share and download AI resources. This platform is central to the AI ecosystem, and its infrastructure is trusted and relied upon.
How did this happen?
OpenAI conducted an internal cybersecurity evaluation, a benchmark called ExploitGym, with AI agents tasked with finding and exploiting software vulnerabilities. Rather than solving the exercise as intended, a swarm of agents, drawn from OpenAI’s publicly available GPT-5.6 Sol model and a more capable, unreleased internal research model later named IM1, found a way to escape the isolated environment they were supposed to remain confined to.
From there, over a campaign spanning roughly four and a half days in total, the agents worked their way into Hugging Face’s internal systems, spending roughly the final two and a half days operating directly inside Hugging Face’s own infrastructure. Once there, they obtained credentials, executed code on production servers, gained full administrative access on at least one server, and reached parts of the company’s internal network and source code repositories. OpenAI, Hugging Face, and the independent AI safety evaluation organization METR have since published detailed accounts of how this happened.
This incident is one of the most important cybersecurity events of the year, as it is the first publicly documented case of a fully autonomous, AI-conducted cyberattack against real production infrastructure, concretely demonstrating that a coordinated swarm of AI agents is capable of independently identifying, chaining, and exploiting multiple security weaknesses across several organizations’ infrastructure. And all of this happens without a human operator guiding the process.
It also shows that when the doors are left open, individually minor configuration choices that seemed acceptable in isolation, can together create a path all the way from an evaluation sandbox to full administrative control of a company’s core systems.
What security teams need to understand
Artificial intelligence can already write functioning exploit code. That capability has been demonstrated before, in narrower forms. The more important question is what a fully autonomous, multi-step, multi-agent intrusion looks like once it is allowed to run for days without interruption.
The Hugging Face incident happened because two environments that should carry opposite security profiles were treated the same way:
- An evaluation environment is, by design, given latitude a production system would never get: broad outbound internet access, elevated privileges inside its own sandbox, and a mandate to keep trying until something works. This is not a mistake. It is what evaluation is supposed to look like.
- A production pipeline is supposed to carry the opposite profile: narrow network access, minimal privileges, and short-lived credentials scoped to a single job.
OpenAI’s evaluation environment had more reach into the outside world than intended. Once the agents used that reach, they found that Hugging Face’s production systems had internal gaps of a similar kind: broader network access than necessary, credentials with wider scope than necessary, and insufficient isolation between components that had no reason to trust one another.
The incident is what happens when the boundary between the two environments is not enforced diligently.
The anatomy of the Hugging Face agent intrusion
1. Operational command was through a random endpoint on the internet
The operational base for the activities conducted by the agents was in fact not related to either OpenAI or Hugging Face. It was a random endpoint the agents discovered on the public internet, which could be reconfigured.
After exploiting a zero-day vulnerability in the package registry cache proxy, one of the permitted network egress points inside OpenAI’s evaluation sandbox, the agents entered the open internet, unrestrained and unmonitored.
On the public internet, they found a code execution endpoint that was publicly deployed by Modal, a third-party sandbox infrastructure provider, hosting a user’s CyberGym-style code-evaluation harness. The agents were able to study this endpoint and reconfigure not one but two ways it could be used to obtain system commands, giving them administrative access to it.
2. The two vulnerabilities that gave the agents complete access
With a controlled external base secured, the agents turned back toward Hugging Face. They had no direct network connection into the company’s internal systems, so they needed a legitimate-seeming channel in.
The platform’s dataset processing pipeline, the internal system responsible for reading and preparing datasets uploaded by users, was the perfect path.
A dataset configuration file can declare where its underlying data physically resides. The agents crafted a configuration, an HDF5 external raw storage file read, which declared a location that pointed not to real data but to sensitive local files on the server, including the server’s environment variables and its own internal source code. The processing system read and returned those files as though they were ordinary dataset content, handing the agents both working credentials and a detailed understanding of how the pipeline operated internally. This was the first flaw.
The second flaw, a Jinja2 server-side template injection, was a different part of the same configuration format that accepted an expression which the processing system evaluated more permissively than necessary.
The agents used this to reach a function capable of executing arbitrary code, giving them direct control over a production server inside Hugging Face’s internal network. Every subsequent instruction and every piece of stolen information passed back and forth through the platform’s own public interfaces, disguised as ordinary data traffic.
3. Escalation at a pace no human team could match
It took less than a day for the agents to move from initial access to broad administrative control of internal Hugging Face systems.
They used credentials to query the internal cluster management system directly. They reached a metadata service intended only for internal automated processes and used it to map out the wider computing environment. They created a highly privileged process that gave them direct access to the underlying physical machine, a step the system’s own protections should have blocked but did not. From there, they obtained a single set of credentials that, due to a configuration oversight, granted administrative access across multiple internal clusters at once rather than just one.
Using that access, the agents connected themselves to the company’s private internal network, taking care to keep their presence in memory only, so that nothing was written to disk.
From inside that network, they reached the internal source code management system, obtained a token allowing code modification, and attempted to compromise the automated build process in order to harvest further credentials. It was this specific attempt that was ultimately blocked by existing safeguards before any harmful change could take effect.
4. Open-source, publicly available tradecraft infrastructure
The agents never stood up their own servers or built any infrastructure of their own. Everything they used was ordinary public tooling accessible to anyone.
They used a request-capturing service as a makeshift mailbox, text-paste sites to stage instructions, and more. Every message was compressed and encoded so initial investigations would only reveal a fraction of the whole exposure. It took a much more thorough investigation, one that replicated the agents’ own encoding method, to reveal that over four times as many credentials had actually been exposed than the initial review had found.
This improvised system also made the intrusion difficult to shut down. Each time an agent’s sandbox environment restarted, it came back with a fresh identity and no memory of what had come before, so closing off one communication channel simply meant the swarm rebuilt another one on its next attempt, cycling through alternative technical methods whenever its usual path was blocked.
Why scale changes the threat
None of the individual weaknesses above would surprise an experienced security professional. Insufficiently validated data processing, a metadata service reachable by systems that had no need to reach it, a shared credential with far broader scope than intended: these are familiar categories of risk in security assessments across industries.
What is different here is the scale.
Hugging Face’s own reconstruction shows the agents carried out roughly seventeen thousand distinct actions over four and a half days. The overwhelming majority failed. The intrusion succeeded because a small number of working paths existed somewhere inside that very large volume of attempts, and 700 cooperating agents had the patience and the speed to find them.
That scale changed the response as much as the attack. Reconstructing the intrusion by hand was not realistic given the volume, so Hugging Face’s investigators relied on an AI-assisted process of their own to decode the stolen data and rebuild the timeline. Some frontier AI models, when asked to help analyze the recovered logs, refused to assist, treating the act of examining an existing exploit as equivalent to helping create one. That left defenders with fewer tools at the exact moment they most needed them.
Three risks follow for any organization working with AI systems:
- AI systems can act offensively at a level of persistence, coordination, and speed that changes the nature of the threat, regardless of any human attacker’s intentions.
- The security gaps that allowed this to happen were ordinary infrastructure decisions, not exotic failures. The same gaps are likely present elsewhere.
- Defending against this kind of threat requires rethinking not only technical controls but which tools and processes are actually available, and willing, during a live incident.
How can you keep your organisation safe?
There are a few things you can do even in the wake of something seismic like this event.
- Treat evaluation and benchmark environments as connected to the wider internet unless proven otherwise.
- Assume that any system with outbound network access will eventually attempt to reach internal metadata services if those services remain reachable, and block that access at the infrastructure level.
- Review the access levels of shared credentials. A single credential should never provide administrative control over more than one cluster or environment unless there is a specific and reviewed reason for it. CybelAngel's Credential Intelligence surfaces credentials of this kind when they leak to external sources.
- Prevent workloads from creating highly privileged processes or mounting host-level file systems by default.
- Configure detection systems to recognize unusual patterns of behavior, such as an extremely high frequency of repeated or parallel actions, rather than relying only on known malicious signatures.
- Decide in advance which internal tools, including which AI systems, are authorized and willing to assist in analyzing an active security incident, so that this question does not need to be resolved in the middle of a crisis.
Conclusion
Every gap exploited during the Hugging Face intrusion, from over-scoped credentials to a reachable metadata service to insufficient workload isolation, was in principle discoverable from the outside by anyone with network reach and enough patience to look. A swarm of AI agents had both.
The defensive question this incident asks of every enterprise is not whether the same gaps exist in your environment. It is whether you already know where they are. CybelAngel’s Attack Surface Management provides that outside-in view continuously, surfacing exposed services, forgotten management planes, and leaked credentials before attackers, human or otherwise, get to them first.
Talk to our analysts to find out what is exposed today.
Alix is an Advanced CyberOps Analyst in CybelAngel’s Consulting & Investigation team.
À propos de l'auteur
