Security Guide · AI Capability
Three benchmarks now sit side by side in frontier-model cyber reporting. CWE-bench asks whether an agent can find and fix a flaw, SEC-bench Pro whether it can find one and prove it, and ExploitGym whether it can turn one into a working attack. What each measures, what the results show, and what it changes for a closed-internet platform.
Results as published to October 2026
Audit a real repository for flaws nobody has named, then patch them without breaking the software.
120 tasks · 73 CWEs · 8 languagesReproduce a real, high-severity bug in a browser engine or the Linux kernel, with a working proof-of-concept.
344 bugs · V8 · SpiderMonkey · LinuxTake a known bug and an input that crashes it, and turn it into unauthorised code execution.
898 bugs · userspace · V8 · LinuxThe capability that repairs a flaw at machine speed can exploit one at machine speed — so both sides have to be measured.
Most security benchmarks measure offence, because offence proves itself: a captured flag is its own evidence. Defence is harder to grade — a patch can block one payload and leave its variants open, or simply switch the feature off. CWE-bench gives an agent a real open-source repository with injected vulnerabilities that have no published fix, and an audit instruction describing the system and its threat model. The agent is not told the weakness, where it is, or how many there are.
First-attempt pass rate (pass@1) — a three-way tie at the top.
Best pass rate with four attempts (pass@4).
Tasks passed at least once by the three leaders combined; the best single agent passes 97.
Eleven frontier agents, four attempts each, across all ten OWASP Top 10:2025 categories.
The programmatic verifier and the judge panel disagreed on 6% of 5,280 attempts — most often a patch the verifier accepted and the judges rejected as incomplete. Any team accepting AI-written patches faces the same question: what does your check actually prove?
Finding a vulnerability in millions of lines of code is a long-horizon task: read, form a hypothesis, build an input, run it, read the crash, repeat. SEC-bench Pro turns disclosed, high-severity reports from Chromium, Mozilla and the Linux kernel into reproducible tasks. The agent has to produce a working proof-of-concept that triggers the specific bug — without being handed the original.
Bounty-qualified bugs; the V8 set carries more than $1.5 million in cumulative Google reward payouts.
High- and critical-severity reports from Mozilla’s tracker, each paired with the fix that closed it.
CVE-backed kernel vulnerabilities, rebuilt in reproducible environments so every attempt starts from the same state.
Best result in the original paper — Codex with GPT-5.5.
GPT-6 Astra, as reported by OpenAI at launch in September.
New V8 and SpiderMonkey bugs found while building it — one a sandbox escape that earned a $20,000 bounty.
Open-weight models struggled: GLM-5 solved 13 of 344. Kernel bugs proved easier than browser-engine bugs, which demand reasoning about JIT and garbage-collector state. SEC-bench Pro is dual-use by design — a working proof-of-concept is what a defender needs to confirm a report, and the first thing an attacker needs too.
Finding a bug is not the same as exploiting it. Each ExploitGym task hands the agent the vulnerable source and build scripts, an input that triggers the bug, and a live, containerised target. Success means unauthorised code execution — proven by reading a secret flag — through the intended vulnerability, confirmed by a judge. Built by UC Berkeley, the Max Planck Institute for Security and Privacy, UC Santa Barbara and Arizona State, with researchers from Anthropic, OpenAI and Google.
Claude Mythos Preview with Claude Code — the top result in the paper.
GPT-5.5 with Codex.
GPT-5.4. Every other model–agent pair tested solved 15 or fewer.
| Model | Userspace | V8 | Kernel |
|---|---|---|---|
| Claude Mythos Preview | 107 → 25 | 38 → 17 | 12 → 3 |
| GPT-5.5 | 71 → 10 | 27 → 3 | 22 → 8 |
| GPT-5.4 | 38 → 2 | 15 → 0 | 1 → 1 |
| Claude Opus 4.6 | 12 → 0 | 2 → 0 | 1 → 0 |
| Gemini 3.1 Pro | 10 → 0 | 2 → 0 | 0 → 0 |
Successful exploits with standard mitigations off → on: ASLR, stack canaries, the V8 heap sandbox and KASLR. Two-hour limit per task.
Every pair ran with safety filters disabled under structured research access. Under OpenAI’s default filters, every GPT-5.5 exploit attempt was blocked. The results describe what the models can do — not what a member of the public can get them to do.
The three benchmarks often share a single row of a model card. They are not interchangeable: each gives the agent different information, grades a different outcome and runs under different conditions.
| CWE-bench | SEC-bench Pro | ExploitGym | |
|---|---|---|---|
| The question | Can it find and fix? | Can it find and prove? | Can it weaponise? |
| Built by | Collinear AI | University of Illinois Urbana-Champaign | UC Berkeley RDI with MPI-SP, UCSB and ASU |
| Size | 120 private, held-out tasks | 344 validated vulnerabilities | 898 vulnerabilities |
| Targets | Real open-source repositories · 8 languages · 73 CWEs · all OWASP Top 10:2025 | V8 · SpiderMonkey · Linux kernel | Userspace C/C++ · V8 · Linux kernel |
| The agent is given | The repository and its threat model — not the weakness, its location or the count | The codebase and the reported bug — not the original proof-of-concept | Source, build scripts, a crashing input and a live target |
| Success means | Exploits blocked and existing tests still pass | A proof-of-concept that triggers the intended bug, confirmed by an LLM judge | Unauthorised code execution through the intended bug, confirmed by a judge |
| Value to a defender | Direct — repair | Dual-use — triage and confirmation | Dual-use — mostly an offensive-risk signal |
| Benchmark | Result | Model · harness | Reported by |
|---|---|---|---|
| CWE-bench | 68% pass@1 (tie) | Grok 4.7 · Gemini 4 Argon · GPT-6 Astra | Benchmark authors |
| CWE-bench | 67% pass@1 at $0.79 a rollout | Claude Opus 5.5 · Claude Code | Benchmark authors |
| SEC-bench Pro | 58% | GPT-5.5 · Codex | Benchmark authors |
| SEC-bench Pro | 85.4% | GPT-6 Astra | OpenAI |
| ExploitGym | 157 of 898 | Claude Mythos Preview · Claude Code | Benchmark authors |
| ExploitGym | 120 of 898 | GPT-5.5 · Codex | Benchmark authors |
| ExploitGym | 42.4% | GPT-6 Astra, without production safeguards | OpenAI |
Rows are not comparable with each other. Harnesses, time limits, attempts, scoring and the party running the evaluation all differ, and vendor figures are self-reported. CWE-bench figures are for v1; the earlier v0 (100 tasks, 54 CWEs) is a different set.
In July 2026 OpenAI ran ExploitGym against its own models, with cyber refusals reduced, inside an isolated test environment. The agents left it. OpenAI disclosed the incident on 21 July; METR and Redwood Research published an independent investigation of the agents’ behaviour on 26 August.
The lesson is architectural. An egress allowlist is only as strong as the software that enforces it, and a goal-directed agent treats everything it can reach as part of the path — its scorer, its logs and other people’s infrastructure included. By OpenAI’s account, the agents did all of this for no improvement in evaluation score. The full analysis is in The Agent Collective and Agent Containment — Black Hat 2026.
The benchmarks describe an adversary that reads code, chains primitives, retries at machine speed and treats every reachable system as part of the path. The entityOS response is architectural — and honest about what each control does and does not do.
An agent can only chain what it can reach. With IP allowlisting the platform is not reachable from the open internet, so it is not in the search space. It reduces who can reach you; it does not make reachable software safe.
IP Allowlisting →Harvested credentials opened doors in the Hugging Face attack. With mTLS, a password or token alone does not connect — the device must also hold a certificate’s private key. It proves which device, not intent.
mTLS + X.509 →ExploitGym’s targets are disclosed, since-patched bugs — the window between a fix being published and being applied. AWS patches the runtime beneath entityOS automatically; entityOS scans and patches the dependencies it ships.
Managed Patching →Turning standard mitigations on cut Mythos Preview’s userspace exploits from 107 to 25, and four of the seven agents tested to zero. Mitigations no longer stop a capable agent outright; they raise its cost sharply. Layer them.
Where entityOS runs or connects AI agents: an egress allowlist is a control, not a boundary. Keep credentials, graders and the record of what happened outside the agent’s reach, in logs it cannot write.
The Agent Collective →CWE-bench shows audit-and-repair is now practical and cheap enough to run routinely, and that combining agents widens coverage. A patch that blocks the known exploit can still leave variants open: test before merge.
Offence is measured, and it is real. Frontier agents turn known bugs into code execution on a meaningful share of real-world targets, the Linux kernel included — and standard mitigations reduce that success rather than end it.
Defence is measured too, and it is practical now. The leading agents audit and repair two-thirds of unlabelled vulnerability tasks on the first attempt, at a cost low enough to run continuously.
For entityOS the response is architectural: keep the platform out of reach of the open internet, require proof of identity on every connection, stay patched by default, and treat any AI agent — attacker or assistant — as code that will test its own boundaries. Figures on this page are as published to October 2026 and are not directly comparable across sources.