Tester YAML reference
Schema version: 0.2 (SUPPORTED_TESTER_SCHEMA_VERSION in tester_config.py).
Unsupported schema_version values raise ConfigError.
Document structure
schema_version: "0.2"
run: { }
benchmark: { }
harness: { }
verification: { } # optionalUnknown keys at any level raise ConfigError.
run
| Field | Type | Required | Description |
|---|---|---|---|
id | string | yes | Run identifier stored in each JSONL record |
output_dir | string | yes | Output directory; relative paths resolve from the tester YAML location |
benchmark
| Field | Type | Required | Description |
|---|---|---|---|
manifest | string | yes | Benchmark manifest YAML path |
tasks | string | yes | JSONL task rows path |
harness
| Field | Type | Required | Description |
|---|---|---|---|
type | string | yes | codex, claude_code, or command |
env | string[] | no | Environment variable names to forward; values are not allowed |
config | object | no | Harness-specific config. Default: {} |
harness.config for command
| Field | Type | Default | Description |
|---|---|---|---|
command | string or string[] | required | Shell command run in the container |
task_file | string | task.json | Agent task JSON path |
timeout_seconds | number | none | Harness timeout |
allowed_domains | string[] | [] | Extra egress domains |
harness.config for codex
| Field | Type | Default | Description |
|---|---|---|---|
model | string | required | Codex model name |
version | string | latest | Codex CLI version for overlay build |
task_file | string | task.json | Agent task JSON path |
timeout_seconds | number | 900 | Harness timeout |
allowed_domains | string[] | [] | Added to api.openai.com |
harness.config for claude_code
Claude Code uses the same fields as Codex with these defaults:
| Field | Default |
|---|---|
model | sonnet |
version | latest |
task_file | task.json |
timeout_seconds | 900 |
Default egress includes instead api.anthropic.com accordingly.
Provider harnesses require the provider API key in the host environment after .env loading. harness.env is intended only for additional non-provider variables that should be forwarded, as provider key names are filtered and replaced with dummy keys in the agent container to avoid leakage and unauthorized usage of tools server-side.
allowed_domains entries must be DNS names. The egress proxy rejects IP literals, wildcards, localhost, and DNS results that resolve to non-public or non-unicast addresses.
verification
| Field | Type | Default | Description |
|---|---|---|---|
allow_network | boolean | false | Allow verifier Docker sandbox network access |
disallow_dangerous_commands | boolean | true | Block benchmark needed_commands unless false |
deny_commands | string[] | [] | Explicit deny list. Known commands: chroot |
Omitted verification uses fail-closed defaults for verifier network and dangerous commands.
Path resolution
_config_path() resolves relative paths against the tester YAML parent directory. Absolute paths are used as-is.
Errors
All validation errors raise securebench.errors.ConfigError, a ValueError subclass.
Common messages:
Unsupported tester schema_version '...'harness.type must be one of: claude_code, codex, commandharness.env[N] must be a non-empty environment variable name without '='
Examples
See Tester configuration and benchmarks/*/tester-*.yaml in the SecureBench repo.