Extending SecureBench

This guide covers extension points in the active codebase. Follow existing module-per-family conventions so the directory tree stays navigable.

Extension overview

Always add tests under tests/.

Add a benchmark family

1. Define the family contract

In families/registry.py:

FAMILY_CONTRACTS["my_family"] = FamilyContract(
    "my_family",
    "text",              # text | code | patch | workspace
    requires_workspace=False,
)

2. Add row validation

Create families/my_family.py:

def validate(row, context: str) -> None:
    reject_unknown(row.input, {"prompt"}, f"{context}.input")
    reject_unknown(row.eval, {"answer"}, f"{context}.eval")
    required_non_empty_string(row.input, "prompt", f"{context}.input")
    # ...

Register in FAMILY_VALIDATORS.

3. Map eval visibility

In benchmark_compiler.py:

EVAL_VISIBILITY["my_family"] = {
    "answer": "hidden",
    "checker": "evaluation_inputs",
}

Unknown eval keys default to hidden.

4. Implement a verifier

See below. Without a verifier, tasks compile but score as pending.

5. Document threat model

Before marking the family production-ready, document agent-permitted mutations, network needs, and tamper tests in docs/SECURITY.md.

Add a verifier

1. Implement Verifier

# verifiers/my_family.py
class MyFamilyVerifier(Verifier):
    def verify(self, task, candidate, **context):
        # Read hidden data via resource_value(task, "answer")
        # Never trust candidate content without validation
        return VerificationResult(
            task_id=task.id,
            status="passed",
            passed=True,
            score=1.0,
            metadata={"verifier": "my_family"},
        )

2. Register

In verifiers/registry.py:

if task_type == "my_family":
    return MyFamilyVerifier()

3. Export (optional)

Add to verifiers/__init__.py if part of public API.

4. Safe patterns

• Use DockerSandbox(network="none") for untrusted code paths
• Materialize eval inputs with VisibilityAwareMaterializer at verification time
• Accept verification_policy from context when using dangerous commands
• Inject sandbox_factory in tests to avoid real Docker

Add a harness type

1. Implement CandidateProducer

class MyHarnessProducer(CandidateProducer):
    def produce(self, task: SecureBenchTask, **context) -> CandidateArtifact:
        # Materialize agent resources only
        # Run DockerSandbox
        # extract_candidate(...)

Use helpers from harnesses/shared.py and candidates/extraction.py.

2. Parse harness config

Add config parser (e.g. my_harness_config()) with reject_unknown_fields.

3. Wire registry

In harnesses/registry.py and tester_config.py:

• Add to HarnessType and HARNESS_TYPES
• Extend build_harness_producer()

4. Network egress

If the harness needs outbound network, use docker_egress_policy() from harnesses/network.py with an explicit domain allowlist.

Do not reintroduce host-execution harness modes — project direction is Docker-only for candidate production (docs/next.md).

Add a sandbox backend

Implement the Sandbox ABC in sandboxes/base.py:

class MySandbox(Sandbox):
    def run(self, command, *, workdir=None, timeout=None) -> CommandResult: ...
    def write_file(self, path, content): ...
    def read_file(self, path) -> str: ...
    def extract_file(self, path) -> bytes: ...

Export from sandboxes/__init__.py. Wrap with PolicySandbox when command auditing is needed.

Reuse resolve_sandbox_host_path() for host-backed roots.

Add validators (schema tightening)

Family validators are the primary schema layer. For cross-cutting asset object validation (path, mount, read_only, type), see planned work in docs/next.md.

To add shared validation:

1. Extend helpers in families/base.py
2. Call from family validate() functions
3. Add tests in tests/test_family_schemas.py

Add benchmark adapters

The active pipeline uses benchmark packs, not Hugging Face dataset adapters.

Legacy adapters live in legacy/securebench/adapters/ and are not wired to the CLI. For new integrations:

1. Convert external datasets to manifest + JSONL rows
2. Or build an offline conversion tool that emits tasks.jsonl

Do not extend legacy adapter registry unless reviving that path intentionally.

Add tools / capabilities

SecureBench does not expose a general agent tool registry in the active package. Provider CLI harnesses (Codex, Claude Code) are the capability surface for agentic runs — they run with provider-defined tools inside the benchmark container.

For custom tool access:

• Use the command harness with a fixed wrapper script you trust, or
• Integrate a new provider CLI harness following harnesses/codex.py patterns (overlay, egress policy, extraction)

Capability-based design intent: prefer narrow, tester-configured commands over unrestricted shell in verification paths.

Testing extensions

.venv/bin/python -m pytest tests/test_my_family_verifier.py -q

Follow existing patterns:

• tests/test_family_schemas.py for row validation
• tests/test_harnesses.py, tests/test_docker_sandbox.py for sandbox behavior
• tests/test_smoke_benchmark_packs.py for end-to-end pack smoke

Checklist before merging

• Family validator rejects unknown fields
• Eval visibility mapped in compiler
• Verifier registered (or explicitly documented as pending)
• Agent payload excludes hidden/eval secrets
• Tests cover pass, fail, and at least one tamper attempt
• Security notes updated in repo docs/SECURITY.md for adversarial-facing families

Related reading

• Tasks & Evaluators
• Security Model
• Reference → Python API