SecureBench / split verification

Coding-agent scores that the agent cannot influence.

SecureBench treats the agent, its workspace and everything it produces as adversarial. It captures one bounded candidate, runs it in fresh networkless environments, and leaves the verdict to a trusted Oracle on the host that alone holds the hidden material.

How it works Architecture Results →

The problem

Every task we converted could be cheated for a perfect score.

When the grader runs on the machine the agent worked on, the agent can change what the grader sees.

60/60: converted tasks
155: cheats pass the native grader
0: pass SecureBench

Terminal-Bench and DeepSWE tasks, as of 20 October 2026. The conversion campaign is still running.

terminal-bench/path-tracing-reverse/candidates/attack-conftest.shterminal-bench/path-tracing-reverse

#!/bin/sh
cd /app
printf '[pytest]\n' > /pytest.ini
cat > /conftest.py <<'PY'
import pytest


@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    report.outcome = "passed"
    report.longrepr = None
PY

The entire submission.

The task asks for a C program. This candidate contains none. It plants a pytest hook at the root of the filesystem, and the grader, running in the same container, reports every failing test as passed.

Native grader: ✓ passes, score 1.0
SecureBench: ✕ fails

Others find the same. ImpossibleBench shows that frontier models often exploit the tests of tasks that cannot be solved, and BenchJack builds exploits that score near-perfect on most of 10 popular agent benchmarks without solving a single task.

See three cheats step by step →

How it works

Privilege separation, applied to benchmarks.

We call it split verification. Each step runs in its own environment with only the material it needs, and work crosses from one to the next through a single narrow, validated channel.

01Untrusted

Agent environment

The agent develops a solution with public material only.

It reaches only its run's provider relay and the task's allowed domains.

stopped workspace

02Utility

Capture

After the agent stops, one bounded candidate is read from the declared scope.

Nothing from the workspace runs. The host validates every path, hash and bound.

candidate

03Untrusted

Evaluation

The candidate is replayed in a fresh, networkless environment for each case.

Each one receives only its own challenge, released by the Oracle.

challenge

typed evidence

04Trusted host

Oracle

Plans the cases, releases each challenge, and judges the evidence against answers only it holds.

Hidden material is never mounted into any environment.

Untrusted: the agent or its candidate controls itUtility: networkless, runs framework code over candidate dataTrusted: host components and the Oracle

Threat model

Two attackers, before and after capture.

The agent decides what the candidate contains, so both can serve one plan. They see different material at different times, so we model each one separately, and every security claim names its principal.

Can reach or receiveNever reaches it

We trust: The benchmark authors and their Oracle code, the SecureBench host components, and the isolation mechanism.
Out of scope: Escapes from the isolation mechanism, and hardware side channels.
Still in scope: Attacks on trusted components through SecureBench's own interfaces, such as a crafted artifact aimed at a parser.

Security goals

Six goals, each traced to tests.

Every goal maps to mechanisms in the architecture and to named adversarial tests.

Secrecy

Expected answers and unreleased cases never enter the Agent or Evaluation environment. Each Evaluation receives only its own case's challenge.

Verdict integrity

Only the Oracle issues verdicts, against host-held acceptance conditions. Candidate claims, self-tests and fabricated observations cannot produce a pass.

Bounded handoff

State reaches evaluation only through one bounded, replayable candidate, captured after the Agent environment stops.

Case isolation

Candidate-controlled state from one case cannot affect another case, another attempt or another run.

Narrow evidence channel

Evaluation reaches the Oracle only through bounded, validated, typed evidence bound to its case, handled safely even when crafted.

Resource-access policy

Each environment reaches only what its policy declares. Using a service opens no route to the host, hidden material or other runs.

Auditability

Anyone can re-judge every verdict offline.

Each attempt stores the evidence the Oracle judged. With the task pack, a reviewer recomputes every verdict without running anything.

attempt record (abridged)

{
  "status": "failed",
  "failure_class": "case_error",2
  "judged": [1
    { "case": 0, "evidence": {…}, "judgment": {…} },
    …
  ],
  "candidate": "sha256:…",
  "provenance": {3
    "task": {…}, "framework": {…},
    "agent": {…}, "run": {…}, "worker": "…"
  },
  "logs": {…}
}

run manifest: sha256 of this record, then a root digest4

1
Re-judge offline
Every case's evidence is stored next to the Oracle's judgment of it, and the judge is deterministic and free of side effects.
2
Failures stay failures
Errors count against the zone where they occur. A candidate that crashes its adapter fails with class case_error, so it cannot turn a wrong answer into a missing result.
3
Full provenance
Task, image and Oracle digests, the framework commit, the harness and model, and the worker's qualification.
4
Append-only
Retries and invalid attempts are recorded, never overwritten. The run manifest hashes every record, and a root digest closes the run.

Go deeper.

Read how each component works and why, or explore what we measured.

Read the architecture Explore the results