Needles are not enough; the answer must be constructed.
The target is reasoning over long, heterogeneous document sets: locate several relevant facts, determine how they relate, handle conflicts and dates, compute or synthesize the answer, and preserve a verifiable evidence chain.
Yotta Content can build original or permissioned document worlds around company reports, legal files, government consultations, academic literature, industry research, policy materials, surveys, or a client's own enterprise corpus. Questions are designed so no single sentence states the answer.
Document worlds are versioned like environments.
Immutable document set
Files, extracted text, layout references, checksums, permissions, dates, and document-level metadata.
Distributed evidence
The answer requires a join, comparison, calculation, temporal resolution, or synthesis across multiple locations.
Claim provenance
Gold evidence nodes, supporting spans, transformations, intermediate claims, and contradiction relationships.
Realistic distractors
Near-matching entities, stale periods, repeated tables, conflicting versions, decoy numbers, and irrelevant documents.
Answer equivalence and evidence faithfulness are separate.
| Layer | What it checks |
|---|---|
| Answer normalization | Exact, numeric-tolerance, list, date, entity, or structured equivalence |
| Evidence recall | Whether the cited spans include every necessary supporting fact |
| Evidence precision | Whether citations are specific and relevant rather than a context dump |
| Reasoning graph | Whether intermediate joins, calculations, and temporal choices are valid |
| Contradiction handling | Whether newer, scoped, or authoritative evidence is selected correctly |
| Unsupported claims | Whether the response adds facts not entailed by the document set |
The final task score can remain a strict pass/fail answer metric while evidence and reasoning components provide dense reward for post-training.
Trajectories reveal where long-context reasoning broke.
- Query plan: entities, dates, units, aliases, and candidate document types.
- Evidence events: document and span retrieval, table reads, discarded candidates, and contradiction flags.
- Reasoning state: normalized facts, cross-document joins, calculations, and provisional claims.
- Answer: response plus structured citations and confidence.
- Verifier: answer equivalence, evidence coverage, unsupported-claim labels, and earliest consequential failure.
From one failed attempt we can derive retrieval repairs, evidence-ranking pairs, contradiction examples, citation corrections, and end-to-end repaired trajectories.
What an AA-LCR-shaped program includes.
- Permissioned, versioned document sets with layout-aware extraction, checksums, source metadata, and rights records.
- Original questions requiring multi-document and multi-step reasoning rather than direct lookup.
- Gold answers, evidence graphs, intermediate calculations, citations, acceptable variants, and ambiguity reviews.
- Answer, evidence, contradiction, citation, and unsupported-claim graders with per-layer reward.
- Retrieval and reasoning trajectories, failure labels, preference pairs, critiques, repairs, and private holdouts.
Benchmark relationship and evidence boundary.
This is a proposed independent data program inspired by the long-context reasoning capability measured by AA-LCR. It is not Artificial Analysis data, a replica, an affiliation.
Primary reference: Artificial Analysis AA-LCR overview and evaluation page.