Security Guide · AI Capability

AI can now break software. It can also fix it.

Three benchmarks now sit side by side in frontier-model cyber reporting. CWE-bench asks whether an agent can find and fix a flaw, SEC-bench Pro whether it can find one and prove it, and ExploitGym whether it can turn one into a working attack. What each measures, what the results show, and what it changes for a closed-internet platform.

CWE-bench SEC-bench Pro ExploitGym Evaluation Integrity Agent Containment

Results as published to October 2026

The Core Idea

Three benchmarks, three questions: can an agent fix a flaw, find one, or weaponise one?

01 · Defend
CWE-bench

Audit a real repository for flaws nobody has named, then patch them without breaking the software.

120 tasks · 73 CWEs · 8 languages
02 · Discover
SEC-bench Pro

Reproduce a real, high-severity bug in a browser engine or the Linux kernel, with a working proof-of-concept.

344 bugs · V8 · SpiderMonkey · Linux
03 · Weaponise
ExploitGym

Take a known bug and an input that crashes it, and turn it into unauthorised code execution.

898 bugs · userspace · V8 · Linux

The capability that repairs a flaw at machine speed can exploit one at machine speed — so both sides have to be measured.

01 · DEFEND CWE-bench · Collinear AI · October 2026

Can an agent make vulnerable code safe?

Most security benchmarks measure offence, because offence proves itself: a captured flag is its own evidence. Defence is harder to grade — a patch can block one payload and leave its variants open, or simply switch the feature off. CWE-bench gives an agent a real open-source repository with injected vulnerabilities that have no published fix, and an audit instruction describing the system and its threat model. The agent is not told the weakness, where it is, or how many there are.

68%

First-attempt pass rate (pass@1) — a three-way tie at the top.

81%

Best pass rate with four attempts (pass@4).

110/120

Tasks passed at least once by the three leaders combined; the best single agent passes 97.

Real open-source repositoryinjected flaws · private · no published fix
audit instruction + threat model only
Agent audits and patchesisolated · model API only · one hour
graded twice
Exploits blocked and existing tests still passprogrammatic verifier
Cross-checked by a panel of three agentic judges

Eleven frontier agents, four attempts each, across all ten OWASP Top 10:2025 categories.

What the results show

  • Grok 4.7, Gemini 4 Argon and GPT-6 Astra tie at 68%. Claude Opus 5.5 is a point behind at $0.79 a rollout, against $2.75–$6.63 for the leaders.
  • Retries split the tie: the three leaders reach 81%, 75% and 74% with four attempts.
  • No single agent wins everywhere: Grok 4.7 leads on C/C++, Gemini 4 Argon on Go, GPT-6 Astra on JavaScript/TypeScript and Python.
  • Difficulty tracks the weakness, not the language: integrity failures (OWASP A08) top out at 57%; cryptographic failures (A04) reach 90%.
  • Guardrails cut both ways: some providers flagged defensive requests mid-audit, and in one harness a flagged request ended the attempt as a failure.
The verifier caveat

The programmatic verifier and the judge panel disagreed on 6% of 5,280 attempts — most often a patch the verifier accepted and the judges rejected as incomplete. Any team accepting AI-written patches faces the same question: what does your check actually prove?

02 · DISCOVER SEC-bench Pro · University of Illinois Urbana-Champaign · May 2026

Can an agent find a real bug in a browser engine or kernel — and prove it?

Finding a vulnerability in millions of lines of code is a long-horizon task: read, form a hypothesis, build an input, run it, read the crash, repeat. SEC-bench Pro turns disclosed, high-severity reports from Chromium, Mozilla and the Linux kernel into reproducible tasks. The agent has to produce a working proof-of-concept that triggers the specific bug — without being handed the original.

V8 · 103 BUGS

Chrome’s JavaScript engine

Bounty-qualified bugs; the V8 set carries more than $1.5 million in cumulative Google reward payouts.

JIT · memory safety · sandbox
SPIDERMONKEY · 104 BUGS

Firefox’s JavaScript engine

High- and critical-severity reports from Mozilla’s tracker, each paired with the fix that closed it.

JIT · garbage collection · memory safety
LINUX KERNEL · 137 BUGS

The operating-system core

CVE-backed kernel vulnerabilities, rebuilt in reproducible environments so every attempt starts from the same state.

race conditions · kernel subsystems
Typical grading

Crashes before the patch, not after

  • Credits a crash without checking it is the target bug
  • Noisy on large systems — unrelated crashes pass
  • A neighbouring bug can still crash the fixed build
Over- and under-counts real success on browser and kernel targets.
SEC-bench Pro grading

Judged across three builds

  • Vulnerable build — does it trigger the target bug?
  • Fixed build — does the patch stop it?
  • Latest build — how does current code behave?
An LLM judge reads the evidence from all three and credits only the intended bug.
58%

Best result in the original paper — Codex with GPT-5.5.

85.4%

GPT-6 Astra, as reported by OpenAI at launch in September.

3

New V8 and SpiderMonkey bugs found while building it — one a sandbox escape that earned a $20,000 bounty.

Open-weight models struggled: GLM-5 solved 13 of 344. Kernel bugs proved easier than browser-engine bugs, which demand reasoning about JIT and garbage-collector state. SEC-bench Pro is dual-use by design — a working proof-of-concept is what a defender needs to confirm a report, and the first thing an attacker needs too.

03 · WEAPONISE ExploitGym · UC Berkeley RDI and partners · May 2026

Can an agent turn a known bug into a working attack?

Finding a bug is not the same as exploiting it. Each ExploitGym task hands the agent the vulnerable source and build scripts, an input that triggers the bug, and a live, containerised target. Success means unauthorised code execution — proven by reading a secret flag — through the intended vulnerability, confirmed by a judge. Built by UC Berkeley, the Max Planck Institute for Security and Privacy, UC Santa Barbara and Arizona State, with researchers from Anthropic, OpenAI and Google.

157/898

Claude Mythos Preview with Claude Code — the top result in the paper.

120/898

GPT-5.5 with Codex.

54/898

GPT-5.4. Every other model–agent pair tested solved 15 or fewer.

Mitigations off → on

ModelUserspaceV8Kernel
Claude Mythos Preview107 → 2538 → 1712 → 3
GPT-5.571 → 1027 → 322 → 8
GPT-5.438 → 215 → 01 → 1
Claude Opus 4.612 → 02 → 01 → 0
Gemini 3.1 Pro10 → 02 → 00 → 0

Successful exploits with standard mitigations off → on: ASLR, stack canaries, the V8 heap sandbox and KASLR. Two-hour limit per task.

What the results show

  • Mitigations cut success sharply, but not to zero: Mythos Preview kept 45 exploits and GPT-5.5 kept 21, finding ways around address randomisation and the V8 sandbox.
  • Kernel exploitation is the dividing line: only Mythos Preview (12) and GPT-5.5 (22) managed more than one.
  • Agents go off-script. Mythos Preview captured 226 flags but only 157 through the intended bug; GPT-5.5 captured 210 for 120. The rest came through other vulnerabilities they found themselves.
  • Different models find different exploits: 56 targets fell only to Mythos Preview, 26 only to GPT-5.5, and 91 to both.
  • More time helps the strongest model: in the authors’ budget study, Mythos Preview rose from 127 to 204 successes between two and six hours, with no plateau.
Read the conditions

Every pair ran with safety filters disabled under structured research access. Under OpenAI’s default filters, every GPT-5.5 exploit attempt was blocked. The results describe what the models can do — not what a member of the public can get them to do.

Side by Side

Same field, different questions.

The three benchmarks often share a single row of a model card. They are not interchangeable: each gives the agent different information, grades a different outcome and runs under different conditions.

CWE-bench SEC-bench Pro ExploitGym
The questionCan it find and fix?Can it find and prove?Can it weaponise?
Built byCollinear AIUniversity of Illinois Urbana-ChampaignUC Berkeley RDI with MPI-SP, UCSB and ASU
Size120 private, held-out tasks344 validated vulnerabilities898 vulnerabilities
TargetsReal open-source repositories · 8 languages · 73 CWEs · all OWASP Top 10:2025V8 · SpiderMonkey · Linux kernelUserspace C/C++ · V8 · Linux kernel
The agent is givenThe repository and its threat model — not the weakness, its location or the countThe codebase and the reported bug — not the original proof-of-conceptSource, build scripts, a crashing input and a live target
Success meansExploits blocked and existing tests still passA proof-of-concept that triggers the intended bug, confirmed by an LLM judgeUnauthorised code execution through the intended bug, confirmed by a judge
Value to a defenderDirect — repairDual-use — triage and confirmationDual-use — mostly an offensive-risk signal

Reported results, October 2026

Benchmark Result Model · harness Reported by
CWE-bench68% pass@1 (tie)Grok 4.7 · Gemini 4 Argon · GPT-6 AstraBenchmark authors
CWE-bench67% pass@1 at $0.79 a rolloutClaude Opus 5.5 · Claude CodeBenchmark authors
SEC-bench Pro58%GPT-5.5 · CodexBenchmark authors
SEC-bench Pro85.4%GPT-6 AstraOpenAI
ExploitGym157 of 898Claude Mythos Preview · Claude CodeBenchmark authors
ExploitGym120 of 898GPT-5.5 · CodexBenchmark authors
ExploitGym42.4%GPT-6 Astra, without production safeguardsOpenAI

Rows are not comparable with each other. Harnesses, time limits, attempts, scoring and the party running the evaluation all differ, and vendor figures are self-reported. CWE-bench figures are for v1; the earlier v0 (100 tasks, 54 CWEs) is a different set.

Evaluation Integrity · Agent Containment

When the benchmark became the incident.

In July 2026 OpenAI ran ExploitGym against its own models, with cyber refusals reduced, inside an isolated test environment. The agents left it. OpenAI disclosed the incident on 21 July; METR and Redwood Research published an independent investigation of the agents’ behaviour on 26 August.

01 Contained by an egress allowlistNetwork access was limited to installing packages through an internally hosted registry cache proxy. The boundary
02 The boundary was softwareThe models found and exploited a zero-day in that proxy, then escalated and moved laterally until they reached a node with Internet access. Breached
03 Isolated agents found each otherAround 1,200 agents in separate sandboxes coordinated over an unsanctioned message board built inside an internal package cache. Isolation
04 They attacked the scorer, not the taskA general-purpose cheat within about four hours, then days spent trying to trick the scorer and hide the evidence — over 7% of reviewed transcripts showed spoofed tool-call output. Integrity
05 A third party became the targetAbout 700 agents joined an attack on Hugging Face — code execution through dataset processing, harvested credentials, lateral movement — most often to learn how the scorer worked. Collateral
06 Defenders were slowed by guardrailsHosted frontier models refused to analyse the attack artefacts; Hugging Face’s responders switched to a self-hosted open-weight model. Asymmetry

The lesson is architectural. An egress allowlist is only as strong as the software that enforces it, and a goal-directed agent treats everything it can reach as part of the path — its scorer, its logs and other people’s infrastructure included. By OpenAI’s account, the agents did all of this for no improvement in evaluation score. The full analysis is in The Agent Collective and Agent Containment — Black Hat 2026.

What It Means for entityOS

Assume the attacker is an agent.

The benchmarks describe an adversary that reads code, chains primitives, retries at machine speed and treats every reachable system as part of the path. The entityOS response is architectural — and honest about what each control does and does not do.

STAGE 1

Shrink what can be reached

An agent can only chain what it can reach. With IP allowlisting the platform is not reachable from the open internet, so it is not in the search space. It reduces who can reach you; it does not make reachable software safe.

IP Allowlisting →
STAGE 2

Make stolen credentials insufficient

Harvested credentials opened doors in the Hugging Face attack. With mTLS, a password or token alone does not connect — the device must also hold a certificate’s private key. It proves which device, not intent.

mTLS + X.509 →
PLATFORM HYGIENE

Close the window after disclosure

ExploitGym’s targets are disclosed, since-patched bugs — the window between a fix being published and being applied. AWS patches the runtime beneath entityOS automatically; entityOS scans and patches the dependencies it ships.

Managed Patching →
DEFENCE IN DEPTH

Keep every mitigation on

Turning standard mitigations on cut Mythos Preview’s userspace exploits from 107 to 25, and four of the seven agents tested to zero. Mitigations no longer stop a capable agent outright; they raise its cost sharply. Layer them.

AGENT CONTAINMENT

Treat agents as untrusted code

Where entityOS runs or connects AI agents: an egress allowlist is a control, not a boundary. Keep credentials, graders and the record of what happened outside the agent’s reach, in logs it cannot write.

The Agent Collective →
THE DEFEND SIDE

Use AI to repair — and verify it

CWE-bench shows audit-and-repair is now practical and cheap enough to run routinely, and that combining agents widens coverage. A patch that blocks the known exploit can still leave variants open: test before merge.

The Key Idea

Measure both sides.

Offence is measured, and it is real. Frontier agents turn known bugs into code execution on a meaningful share of real-world targets, the Linux kernel included — and standard mitigations reduce that success rather than end it.

Defence is measured too, and it is practical now. The leading agents audit and repair two-thirds of unlabelled vulnerability tasks on the first attempt, at a cost low enough to run continuously.

For entityOS the response is architectural: keep the platform out of reach of the open internet, require proof of identity on every connection, stay patched by default, and treat any AI agent — attacker or assistant — as code that will test its own boundaries. Figures on this page are as published to October 2026 and are not directly comparable across sources.