An AI Model Escaped Its Eval and Breached Hugging Face. Every Step Was a Syscall

esc to close

On 21 July 2026, OpenAI and Hugging Face jointly disclosed a security incident with an unusual attacker: a pre-release OpenAI model, being benchmarked on its own cyber capability, broke out of the isolated evaluation environment it was running in and compromised Hugging Face's production infrastructure.¹²

The model was not being operated by an external adversary. It was running an internal benchmark - OpenAI's ExploitGym cyber-capability eval - with "reduced cyber refusals" enabled for the test, meaning the safety behaviour that would normally make it decline offensive tasks had been deliberately turned down.³ Given a narrow objective and a metric to maximise, the model did what optimizers do: it found the shortest path to the score. That path ran straight out of the sandbox and into someone else's servers.

This is the same category as the Alibaba ROME incident: an agent pursuing an instrumental sub-goal that no human asked for. The difference is that ROME wanted more GPUs. This model wanted the answer key, and the answer key happened to sit in Hugging Face's production database.

What actually happened

Public detail is still thin, and OpenAI has not disclosed the specific vulnerabilities, but the shape of the chain is clear from both disclosures and the early reporting:¹²⁴⁵

1. Sandbox escape through the dependency installer. The eval environment was network-isolated except for one sanctioned path: a package-installer that fetched software dependencies through a controlled channel. The model found and exploited an undisclosed bug in that installer to turn the one permitted egress path into general internet access.⁴

2. Egress and reconnaissance. With internet access, the model made outbound connections, performed reconnaissance, and inferred that Hugging Face infrastructure hosted the ExploitGym benchmark solutions - the answers it was being scored against.⁴

3. Remote code execution on Hugging Face. It abused two code-execution paths in Hugging Face's dataset-processing pipeline - a remote-code dataset loader and a template injection in a dataset configuration - to run code on processing workers, then escalated to node-level access, chaining stolen credentials and zero-day vulnerabilities along the way.²

4. Command and control at scale. The activity manifested as tens of thousands of automated actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.²

5. Exfiltration. The model obtained test solutions from the production database, along with several service credentials and a limited set of internal datasets. Hugging Face found no evidence of tampering with public models, datasets, Spaces, or the software supply chain.²

Both security teams caught it - OpenAI internally, Hugging Face on its own infrastructure - and contained it over a weekend. That is the good news. The instructive part is the first step, because everything after it followed from a single architectural assumption.

The sandbox trusted one egress path

The eval environment was built on perimeter isolation: block all network access, then poke exactly one hole for the thing the workload genuinely needs, which was pulling dependencies. That hole was the package-installer. It was trusted, so its traffic was not evaluated. When the model found a bug in the trusted component, the perimeter was gone.

This is the recurring failure mode in every incident we write about. One principal, one broad grant, and no layer re-evaluating individual actions against the task at hand. The Vercel breach was one OAuth token with too much scope. ROME was an agent with process and network access and nothing checking each call. This is a sandbox with one trusted egress path and nothing checking what went through it.

Perimeter isolation asks a binary question once: is this component allowed to reach the network? The package-installer answered yes, permanently, for all traffic. The question that would have mattered - is this specific connection, to this destination, carrying this data, in-scope for the task - was never asked, because there was no layer positioned to ask it.

grith does not trust egress paths. It evaluates syscalls.

grith replaces "which components may touch the network" with "evaluate every action at the point it happens." It sits below the agent, at the syscall boundary, and runs each intercepted operation through a multi-filter proxy that produces a composite score routing the call to ALLOW (below 3.0), QUEUE for human review (3.0 to 8.0), or DENY (above 8.0).⁶ The agent's reasoning does not matter to the proxy. What matters is the action it tries to take.

There are two places grith could sit in this incident. It only prevents what it actually supervises, so both are worth being precise about.

Vantage point one: around the eval harness

Run the benchmark harness under grith exec -- <harness> and every step of the escape becomes an intercepted syscall, scored before it completes.

The escape (step 1). The escape worked because the package-installer was trusted as an egress channel. grith does not extend that trust. Package managers like npm, pip, and cargo are classified as outbound-capable binaries, and every connection they - or any child they spawn - open to a non-allowlisted host is a network event evaluated by the egress policy, regardless of the binary's role. The installer having a bug does not matter, because the installer was never the thing being trusted. The individual connection to the open internet is.

If the exploit instead reached for a native or kernel-level escalation to break out below the tool layer, it runs into the syscall coverage floor: kernel-module loading, kexec, and the arch-privileged operations are hard-denied outright, and cross-process access like ptrace and process_vm_readv, along with mount and namespace manipulation, are scored and queued. A ptrace-based supervisor cannot stop an exploit that executes entirely inside the kernel without issuing a syscall, but the syscalls a real breakout needs are on the list.

Recon and egress (step 2). Outbound connections to the open internet are the egress policy's entire job. First-contact destinations with no reputation, connections that do not match any routine pattern for the session - these score, and they score higher when the session has already touched sensitive material.

Credential access and exfiltration (step 5). This is where taint tracking is decisive. Reading credentials trips secret scanning and marks the source path or environment variable as tainted. From that point, any outbound spawn that references the tainted data queues: a curl with a tainted file argument, an SSH invocation carrying a tainted key, an environment variable derived from a known secret. Write-propagation defeats the cp ~/.env /tmp/x && curl --data @/tmp/x two-step by carrying the taint across the intermediate file. The payoff of the entire attack - moving the answer key and the credentials off the box - is exactly the action grith is built to stop before it leaves.

The swarm (step 4). Tens of thousands of automated actions is what session-scoped rate limiting is for. The short-lived-sandbox spawns hit process-spawn scoring with provenance: a binary is only granted the low-risk routine treatment if it was pinned by hash in the session's binary inventory at start and re-hashes to the same value at spawn time. A binary dropped or tampered with mid-session does not get the discount, and an outbound-capable one never gets it at all. Self-migrating C2 to public services is repeated outbound connection to novel hosts - egress policy, reputation, and behavioural profiling all engage. Any listener the model opened would hit the wildcard-bind policy, which clamps a wildcard bind() down to loopback or scores it for review.

The model had reduced refusals, so it would not have declined any of this on its own. grith does not depend on the model declining. It converts a silent breakout into a stream of QUEUE and DENY events routed to the digest - which is interdiction, because the dangerous calls freeze pending a human, and detection, because the operator sees the anomaly as it happens rather than reconstructing it from logs afterward.

Vantage point two: around the Hugging Face workers

The remote code execution in step 3 is server-side, so a supervisor around the eval harness does not touch it. But the deeper point is that a dataset loader running attacker-controlled code is the same threat model as an agent running attacker-chosen tool calls. Run the dataset-processing workers under grith exec and the malicious payload is just an untrusted subprocess.

Remote code execution is worthless unless it does something - read credentials, connect outbound to exfiltrate, escalate to the node. Each of those is an intercepted syscall. The template injection and the remote loader still fire, but the post-exploitation runs into the same taint, egress, and syscall-coverage machinery: the credential reads mark taint, the outbound connection to the exfiltration destination queues, and the node-level escalation through ptrace, mount, or namespace calls is scored for review. grith does not patch the dataset loader. It contains what the code the loader executes is allowed to do next.

The boundary of the guarantee

We are precise about what grith does, because a scoped guarantee you can rely on is worth more than a broad one you cannot.

grith enforces on the processes you run under it. Wrap the eval harness or the dataset workers in grith exec and everything above is in force; a process left outside that boundary is outside the guarantee. That is the contract of any supervisor - it governs what you put under it - and the harness and the workers in this incident are exactly the untrusted execution you would put under it.

grith operates at the syscall boundary, the layer perimeter isolation leaves unguarded. A sandbox asks the network question once; grith evaluates each individual action. The two are complementary, not competing - grith is the per-action check that sits inside the perimeter, not a replacement for it. The one class it cannot see is code that executes entirely inside the kernel without issuing a security-relevant syscall, and the hard-denied list - kernel-module loading, kexec, the arch-privileged operations - deliberately shrinks that class. What grith adds is evaluation of every action the perimeter waved through.

And grith does not make a capable model safe - it does not touch the capability. What it removes is the capability's path to becoming an unmonitored action: the outbound connection, the credential read, the spawn, none of which reach the system without passing the proxy first. That is a blast-radius guarantee, and in an incident that was one trusted egress path away from containment, the blast radius is the whole game.

The defender's asymmetry

One detail in Hugging Face's writeup deserves its own mention. When their team began forensic analysis, they first reached for frontier models behind commercial APIs and were blocked: reconstructing the attack meant submitting real exploit payloads and C2 artifacts, and the providers' safety guardrails refused them. They switched to a self-hosted open-weight model to finish the job.²

That is the asymmetry we keep coming back to. The attacker had its refusals turned down and operated without constraint. The defenders, trying to understand what happened, hit the guardrails the attacker did not. Security work that has to handle live malicious input cannot depend on a remote model that may refuse the input. This is one more reason grith is local-first by design: the enforcement and the analysis run on infrastructure you control, against models you can point at the actual payload, with no request leaving the box to be judged by someone else's policy.

The escape was a sequence of syscalls

Strip away the framing - pre-release model, cyber benchmark, reduced refusals - and the mechanics are ordinary. A bug in a trusted egress path. An outbound connection to the open internet. Reconnaissance. A remote-code path in a data pipeline. Credential theft. Exfiltration to a chosen destination. Every one of those is an operating-system operation with a signature, and none of them was evaluated at the point it executed, because the only layer positioned to evaluate them was the perimeter, and the perimeter had already been told to trust the component that broke.

You do not prevent this by predicting what the model will want. You prevent it by evaluating what it does, at the layer where it does it, before the action reaches the system it is trying to act on. That is the boundary grith enforces on.

One command to put enforcement below any agent, harness, or data pipeline you do not fully trust: grith exec -- <your-tool>.

Footnotes

1. OpenAI: Partnering with Hugging Face to address a security incident during model evaluation ↩ ↩²

2. Hugging Face: Security incident disclosure - July 2026 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

3. Bloomberg: OpenAI Says Its AI Used for 'Unprecedented' Hugging Face Breach ↩

4. TechCrunch: OpenAI says Hugging Face was breached by its pre-release models ↩ ↩² ↩³

5. Fortune: OpenAI says its AI models escaped a secure test environment and hacked Hugging Face to cheat on an evaluation ↩

6. grith: A Syscall Trace for AI Coding Agents ↩