Results

60 converted tasks. 155 cheats pass native grading; SecureBench rejects all of them.

Terminal-Bench and DeepSWE, as of 20 October 2026. Scripted checks; real-agent results are pending.

Conversion

  1. 1

    Select

    Draw tasks in a seeded order from pinned upstream commits.

  2. 2

    Convert

    Write the task pack, reference solution, wrong implementations and cheats.

  3. 3

    Qualify

    Run each candidate twice under both graders; reproduce stored verdicts.

  4. 4

    Review

    An independent agent checks the conversion and requests revisions.

Tasks considered for conversion

By source benchmark and conversion decision. Hover a bar for details.

View data table
Conversion decisions by benchmark
BenchmarkDecisionTasks
terminal-benchconverted30
terminal-benchexcluded4
deep-sweconverted30
deep-sweexcluded0
Source: conversions/inventory.tsv at 8a17b20

Grading

Same candidates, two graders

Only reference solutions should pass.

Scroll to explore, or view the data table.

View data table
Pass counts for the same scripted candidates under both graders
CandidateGraderPasses
Reference solution (60)Native grader59 of 60
Reference solution (60)SecureBench60 of 60
No change (60)Native grader0 of 60
No change (60)SecureBench0 of 60
Wrong implementation (561)Native grader113 of 561
Wrong implementation (561)SecureBench0 of 561
Cheat (175)Native grader155 of 175
Cheat (175)SecureBench0 of 175
Source: conversions/*/*/qualification.json at 8a17b20

Native pass = reward 1.0. One native reference (caffe-cifar-10) timed out; SecureBench passed it.

Wrong implementations accepted by native graders

Share of wrong implementations that pass in the scripted sample. SecureBench rejects all of them.

View data table
Native acceptance of scripted wrong implementations, excluding cheat candidates
BenchmarkAccepted / testedTasks with a pass
Terminal-Bench7 / 1495 / 30
DeepSWE106 / 41226 / 30
Source: conversions/*/*/qualification.json at 8a17b20

Review

Review rounds

Latest recorded round, including re-reviews and one maintainer acceptance.

View data table
Latest recorded review round for each converted task
RoundBenchmarkTasks
1deep-swe0
1terminal-bench3
2deep-swe24
2terminal-bench19
3deep-swe4
3terminal-bench6
4deep-swe2
4terminal-bench2
Source: conversions/inventory.tsv at 8a17b20

API cost

Recorded API list-price estimate

$2,245.84

Through 20 October 2026

Conversion
$1,644.19
Review
$274.88
Coordination
$326.77

Usage covers 60 of 60 converted tasks. Includes excluded attempts; compute and comparison experiments excluded.

Source: conversions/usage.tsv at 8a17b20