A failure surface for acting, not merely coding.
A terminal prompt becomes an RL environment only when the world is reproducible, actions have consequences, success is independently testable, and partial progress can be attributed.
Yotta Content creates original task families around systems work, data processing, storage, networking, security boundaries, build pipelines, and operational recovery. Each task ships as a containerized world with a visible contract, protected state, bundled tests, an oracle path outside the agent mount, and replayable event traces.
Every task separates world, contract, action, and verifier.
Mutable container
Files, services, seeded data, dependencies, permissions, timeouts, and protected evaluation assets.
Observable outcome
Required behavior, forbidden changes, operational constraints, exact interface, and completion evidence.
Action and observation
Commands, patches, outputs, errors, revisions, local checks, stop decision, and concise rationale.
Independent final state
Behavioral tests, integrity checks, weighted criteria, timeout policy, and clean replay from frozen inputs.
Two judged GPT-5.6 Sol Pro blind batches expose different failure modes.
| Task | Reward | Outcome |
|---|---|---|
| sqlite-lease-queue | 1.000 | 9/9 cases |
| framed-rpc-multiplex | 1.000 | 9/9 cases |
| ranged-download-resume | 0.700 | 7/9 cases; checkpoint schema failure |
| oci-layer-replay | 1.000 | 9/9 cases |
| jsonl-external-sort | 1.000 | 8/8 cases |
The first five-task batch scored 0.843636 mean reward across WAL recovery, HTTP cache collapse, content build, online SQLite rebuild, and atomic extraction. The second scored 0.940, with one downloader failing crash-resume and chunk-revalidation cases because its checkpoint lacked a required field. Frozen submissions were not repaired after verifier feedback.
A trajectory turns one miss into several training records.
- Inspect: read the downloader contract and visible starter without opening the oracle or verifier source.
- Implement: add ranged transfer, checkpoint persistence, and revalidation behavior.
- Smoke test: exercise specification-derived local cases and freeze the implementation.
- Verify: pass 7 of 9 hidden cases; crash-resume and chunk revalidation report a missing
completedcheckpoint field. - Derive data: label state-schema design as the earliest consequential error, then create critique, repair, and adversarial checkpoint variants.
The published training view can preserve tool calls, observations, patches, test output, compact decision summaries, grader evidence, and repair pairs without exposing private hidden chain-of-thought.
What a Terminal-Bench-shaped program includes.
- Original Harbor-compatible task repositories with Docker environments, instructions, task metadata, solutions, and isolated tests.
- Deterministic seed and image manifests, network policy, protected-path rules, timeouts, and resource limits.
- Binary or weighted verifiers that inspect final behavior and defend the evaluation boundary.
- Full action/observation traces, frozen source diffs, smoke-test logs, verifier receipts, failure labels, critiques, and repairs.
- New task batches selected from model-specific weaknesses and scored for novelty, solvability, discrimination, and replayability.
Benchmark relationship and source boundary.
This is an independent clean-room data program inspired by the terminal-agent capability measured by Terminal-Bench 2.1. The synthetic samples are not official benchmark tasks, an affiliation, or a leaderboard submission.
Primary reference: Terminal-Bench 2.1 release and official task repository. Evidence summary: JSON.