New questions with adversarially plausible distractors.
The target is scientific reasoning under uncertainty: retrieve the right theory, execute the calculation, reject answers that encode common conceptual errors, and commit to a calibrated choice.
Our system generates original question families around a scientific mechanism rather than templating one prompt. Subject experts and programmatic checks validate units, constants, conventions, answer uniqueness, numerical tolerances, and distractor provenance. Each distractor is labeled by the misconception or calculation failure it represents.
Exact accuracy stays authoritative; dense reward makes it trainable.
Exact correctness
The selected option receives the benchmark-style pass/fail outcome.
Intermediate claims
Numerical or conceptual checkpoints show whether the derivation reached the right scientific state.
Distractor diagnosis
Credits correct rejection of options without letting eloquent text override a wrong final choice.
Confidence discipline
Rewards confidence that matches correctness and creates useful abstention or review policies.
GPT-5.6 Sol Pro solved every exact choice in the judged v3 run.
| Dimension | Mean | Interpretation |
|---|---|---|
| Exact accuracy | 1.000000 | All 33 selected choices correct |
| Checkpoint score | 1.000000 | All keyed intermediate claims matched |
| Rationale score | 0.704545 | Hidden rationale groups create remaining training signal |
| Elimination score | 0.762626 | Distractor-specific explanation is incomplete in places |
| Calibration score | 0.999900 | Confidence 0.99 on each correct task |
No answer was revised after verifier output. All response hashes remained unchanged through scoring, and the run used no web, network, model API, solution file, or hidden specification during solving.
Correct answers still contain hillclimb signal.
- Observation: question, four choices, allowed local tools, and confidence schema.
- Derivation: concise equations, scientific checkpoints, assumptions, and unit conversions.
- Decision: selected option plus a short reason each distractor fails.
- Freeze: hash the answer artifact before any hidden grading.
- Reward: attach exact choice, checkpoint, rationale, elimination, calibration, and integrity dimensions.
A 100% exact score does not end the data loop. Missing rationale groups and weak distractor diagnoses can seed new variants, contrastive explanations, critiques, and targeted preference pairs.
What a GPQA-shaped program includes.
- Original graduate-level question families across biology, chemistry, physics, or client-selected scientific domains.
- Gold choices, exact derivations, independent solves, difficulty review, and misconception-mapped distractors.
- Exact-choice reward plus optional checkpoint, rationale, elimination, calibration, and artifact-integrity dimensions.
- Frozen blind responses, concise trajectory records, per-dimension scores, critiques, and repaired responses.
- Private eval families separated by mechanism and source lineage to reduce near-duplicate leakage.
Benchmark relationship and source boundary.
This is an independent clean-room task system inspired by the graduate science capability shape of GPQA Diamond. It does not reproduce official questions and is not affiliated with the benchmark authors.
Primary reference: GPQA: A Graduate-Level Google-Proof Q&A Benchmark. Evidence summary: JSON.