Research-style numerical work with executable ground truth.
We translate advanced physical models into code-level checkpoints that retain conceptual difficulty while making success reproducible, stress-testable, and useful for multi-attempt reinforcement learning.
The supplied clean-room environment contains 60 challenges, 180 independently gradable checkpoints, and 12 synthetic physics families. A challenge breaks into intermediate and main functions, each with a code template, grouped tests, reference implementation, difficulty metadata, and deterministic split.
Multi-attempt RL without leaking hidden answers.
Physics + code contract
Problem statement, function signature, family, difficulty, prior feedback, and remaining attempt budget.
Executable answer
Candidate Python source defining the requested function under a constrained import and runtime policy.
Grouped verification
Syntax, policy, load, public tests, hidden tests, and stress tests produce a weighted score.
Delta reward
Only improvement over the episode's previous best is rewarded, preventing repeated submissions from farming reward.
GPT-5.6 Sol Pro solved ten families completely and exposed two hard clusters.
| Family slice | Exact solved | Observed outcome |
|---|---|---|
| Non-Hermitian SSH, Lindblad, Floquet, Fukui | 12 / 12 | All checkpoint groups passed |
| Turing, MPS, birth-death, Gaussian | 12 / 12 | All checkpoint groups passed |
| Cosmological transfer, correlated decoder | 6 / 6 | All checkpoint groups passed |
| Layered neutrinos, Z2 lattice gauge | 0 / 6 | Valid code loaded; numerical results missed all test groups |
The 71 submitted actions include initial observations, candidate source, concise rationale, reward, grader feedback, and terminal status. The run was offline and did not inspect expected answers, test contents, generator source, or detailed grader internals.
The unsolved clusters define the next curriculum.
- Layered neutrinos: all three checkpoints loaded and complied with policy, but propagator and probability outputs remained numerically wrong after four attempts.
- Z2 lattice gauge: all three checkpoints produced valid functions, but partition and Wilson-loop values remained outside tolerances.
- Curriculum response: create smaller invariant checks, convention-disambiguation tasks, intermediate checkpoint rewards, critiques, and repaired implementations around those two mechanisms.
The environment preserves partial credit for valid, policy-compliant code even when scientific outputs fail. That makes the difference between syntax, implementation, convention, and physics errors visible to the training loop.
What a CritPt-shaped program includes.
- Original research-level physics families with notebooks, executable JSON records, independently gradable subproblems, and difficulty metadata.
- Gymnasium-compatible multi-attempt environment, deterministic selection, curriculum controls, and direct grading API.
- Sandbox policy plus public, hidden, and stress tests with terse non-leaking feedback.
- Per-attempt code, observation/reward traces, failure labels, concise rationales, critiques, and verified repairs.
- Train, validation, and sealed test splits with manifests, seeds, replay tools, and Hugging Face-style exports.
Benchmark relationship and source boundary.
CritPt-Synth-Hard is independently authored clean-room material that follows the public task contract and broad capability shape. No official CritPt question text or solution is included, and there is no benchmark affiliation.
Primary reference: official CritPt evaluation repository. Evidence summary: JSON.