Research programming beyond generic code completion.
The model must understand the scientific object, translate it into a numerical or symbolic method, implement the method correctly, and interpret outputs under realistic conventions.
Yotta Content can commission original tasks with active scientists across physics, mathematics, biology, chemistry, materials, and other technical fields. Each problem is decomposed along scientifically meaningful interfaces so a partial solution reveals whether the failure came from theory recall, derivation, numerical method, implementation, or output interpretation.
Subproblems create a useful gradient without making the science trivial.
Research-shaped question
A self-contained phenomenon, assumptions, notation, input domain, and target output.
Meaningful checkpoints
Subproblems correspond to equations, transforms, solvers, estimators, or diagnostic quantities.
Runnable interface
Starter code, dependency lock, datasets or fixtures, resource limits, and deterministic entry points.
Independent solution
Derivation, implementation, expected invariants, numerical regimes, and known failure modes remain evaluator-side.
Tests should check science, not one frozen implementation.
| Verifier layer | Purpose |
|---|---|
| Interface and execution | Imports, signatures, determinism, timeout, memory, and output schema |
| Public examples | Give enough feedback to debug basic interpretation and numerical setup |
| Hidden parameter sweeps | Reject hard-coded outputs and test generalization within the scientific regime |
| Invariants | Conservation laws, units, symmetries, monotonicity, bounds, or structural identities |
| Reference comparison | Numerical tolerances chosen from conditioning and error analysis, not arbitrary decimal equality |
| Stress cases | Degenerate inputs, stiff regimes, precision loss, and computational scaling |
Scientific trajectories need domain-aware labels.
- Interpret: extract assumptions, equations, requested interface, and expected qualitative behavior.
- Implement: record code edits, tool output, intermediate numerical checks, and compact scientific rationale.
- Verify: run visible checks, freeze the candidate, then attach hidden invariant, sweep, and stress results.
- Attribute: label the first failure as theory, derivation, discretization, numerical stability, implementation, interface, or interpretation.
- Repair: pair the trace with a critique and a verifier-confirmed corrected implementation.
What a SciCode-shaped program includes.
- Original scientist-authored or scientist-reviewed problems and staged subproblems across selected domains.
- Reproducible code environments, locked dependencies, fixtures, reference implementations, and resource policies.
- Public, hidden, invariant, and stress tests with tolerance justifications and anti-hardcoding audits.
- Code trajectories, execution output, scientific checkpoints, failure labels, critiques, and verified repairs.
- Novelty review against both public benchmarks and the client's training corpus before release.
Benchmark relationship and evidence boundary.
This is a proposed independent clean-room program inspired by the research coding capability measured by SciCode. It does not reproduce benchmark questions and is not an affiliation.
Primary reference: official SciCode site and evaluation repository.