Read a research paper as an executable specification
Turn a research paper into a testable claim map, implementation contract, and prioritized reading plan before writing code.
A team starts porting a promising ranking paper before noticing that its headline gain depends on a different split, compute budget, and baseline.
This chapter moves from extract the claim to price the unknowns. We will make the idea concrete in a small code example before placing it back inside a production system with one reading session before any implementation sprint.
Decide whether the paper is relevant, reproducible enough, and valuable enough to implement.
Coverage of claims by comparable evidence, with unresolved assumptions and estimated reproduction cost.
Reading linearly and coding the architecture diagram first hides a changed evaluation protocol, stronger data augmentation, and an under-tuned baseline. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
A paper is persuasive prose wrapped around a set of testable claims. Read it as a specification by connecting every claimed gain to the exact evidence and protocol that produced it.
Begin with a structural pass: abstract, figures, tables, conclusion, and limitations. This reveals the contribution, comparison, and likely evidence before detailed notation consumes attention.
Then create a claim–evidence map. A two-point gain is not yet attributable to the method if the baseline used less compute, another split, or different augmentation; those conditions belong beside the number as executable requirements.
- three-pass reading
- claim-evidence maps
- baseline contracts
- ablation logic
- benchmark validity
- assumption ledgers
Follow the reasoning before touching the code.
- 01
Extract the claim
Write the direction, magnitude, dataset, metric, baseline, and scope of each headline statement in one sentence.
- 02
Trace its evidence
Link the claim to a table row, figure, ablation, uncertainty estimate, and protocol details that could falsify it.
- 03
Price the unknowns
Mark missing seeds, preprocessing, assets, compute, or licenses and decide which unknown blocks implementation.
{
"claim": "method improves accuracy by 2 points",
"method_score": 0.84,
"baseline_score": 0.82,
"dataset": "benchmark-v2",
"split": "official-test",
"metric": "top-1 accuracy",
"evidence": "Table 2, row 4",
"controls": {
"parameter_budget_matched": true,
"training_compute_matched": false,
"seeds_reported": 1
},
"decision": "resolve compute confound before implementation"
}Represent one research claim as a reviewable contract.
What to notice
- The arithmetic improvement is explicit, but attribution remains blocked by unmatched training compute.
- A single reported seed prevents estimating whether the two-point gap is larger than run-to-run variation.
YOUR TURN Pause and predict before revealing the answer.
Suppose an ablation removes the novel loss but also halves training steps. Can it isolate the loss contribution?
Need a hint?
A valid ablation changes one relevant factor under an otherwise comparable protocol.
Check your reasoning
No. Loss removal and compute reduction are confounded; rerun both variants with matched training steps.
Lessons 1, 2, 3, 4, 5 in this course.
Connect the chapter to the course’s quantitative spine.
Do not implement a diagram until you can state exactly which controlled claim it is meant to test. The claimed improvement over the named baseline below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Claimed improvement over the named baseline
| Symbol | Meaning / shape / unit |
|---|---|
m_method | reported metric for the proposed method |
m_baseline | reported metric for the comparable baseline |
Δ | effect the paper asks the evidence to support |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: m_method, m_baseline, Δ.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c01(inputs)
assert is_finite(value)Choose from the task contract, not a favorite default.
Your reproduction must make every hidden protocol choice explicit. Any production adaptation creates a new claim that needs new evidence.
- 1
Triage
Read title, abstract, figures, tables, conclusion, limitations, and references. Decide relevance and record the headline claims.
- 2
Map claims to evidence
For every claim, name dataset, split, baseline, metric, seeds, compute, ablation, table/figure, and unresolved assumption.
- 3
Reconstruct the method
Translate notation into tensor shapes, pseudocode, invariants, masking, initialization, state, and numerical constraints.
- 4
Lock the protocol
Hash data and splits; pin environment and hardware; define tolerance, seed budget, baseline parity, and stop/go gates.
- 5
Build the harness first
Validate metric code and reproduce the strongest comparable baseline before adding the paper’s novel component.
- 6
Implement minimally
Use hand fixtures, golden intermediates, shape assertions, finite differences where useful, and a tiny-batch overfit test.
- 7
Reproduce and falsify
Run paired seeds and ablations; report every run, uncertainty, discrepancies, negative results, and paper/code differences.
- 8
Create a new production claim
Test own-data validity, latency, memory, cost, robustness, licenses, and security; then canary with fallback and rollback.
Claim–evidence ledger
- Claim and expected effect size
- Dataset, split, preprocessing, and availability
- Baseline tuning and resource parity
- Metric implementation and aggregation
- Seeds, uncertainty, and ablations
- Code, data, weights, and license provenance
Equation-to-code ledger
- Input, state, and output tensor shapes
- Units, dtypes, devices, and valid ranges
- Reference operation and hand fixture
- Initialization, masking, and reduction
- Numerical tolerances and gradient checks
- Golden intermediates at every boundary
Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Reproduction readiness ledger
Resolve undocumented assumptions and watch evidence coverage and mismatch risk move in opposite directions.
Assumption: Ten equally weighted protocol assumptions across data, preprocessing, baseline, metric, seeds, compute, code, assets, hardware, and tolerance.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| data + split | resolved | record evidence |
| preprocessing | resolved | record evidence |
| baseline parity | resolved | record evidence |
| metric harness | resolved | record evidence |
| seeds + uncertainty | unknown | blocks claim |
| compute budget | unknown | blocks claim |
| paper/code delta | unknown | blocks claim |
| assets + licenses | unknown | blocks claim |
| hardware/workload | unknown | blocks claim |
| acceptance tolerance | unknown | blocks claim |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Do not implement a diagram until you can state exactly which controlled claim it is meant to test. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Extract the claim” into a written invariant and a failing test.
- 02
Promote claim-map.json from illustration to a versioned implementation with representative fixtures.
- 03
Use “Price the unknowns” as the release checkpoint: Mark missing seeds, preprocessing, assets, compute, or licenses and decide which unknown blocks implementation.
- 04
Serve or materialize behind explicit one reading session before any implementation sprint.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Reading linearly and coding the architecture diagram first hides a changed evaluation protocol, stronger data augmentation, and an under-tuned baseline. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Paper claim–evidence brief
Select one primary ML paper and produce a go/no-go implementation brief before touching the novel method.
- One-page problem and contribution brief
- Claim–evidence matrix with tables, baselines, and uncertainty
- Equation, tensor-shape, and assumption sheet
- Dataset, compute, asset, and license feasibility audit
- Prioritized unknowns and reproduction recommendation
Staged hints
Failure diagnosis
- If offline numbers look impossible, audit prediction-time availability before model code.
- If quality is sound but the contract fails, measure the exact serialized and served path.
- If one slice regresses, preserve that slice as a permanent release gate.
Record an honest attempt to unlock the reference solution.
Read primary material with a purpose.
How to Read a Paper — S. Keshav
25 min · last checked 26 Aug 2026
↗RECOMMENDED · PaperML Reproducibility Checklist
20 min · last checked 26 Aug 2026
↗RECOMMENDED · PaperImproving Reproducibility in Machine Learning Research
45 min · last checked 26 Aug 2026
↗DEEP DIVE · INDEXFind a primary implementation or benchmark
30–90 min · verify task, dataset, and evaluation match
↗Return to the opening failure.
The team approves implementation only after each claim has an evidence location, a comparable baseline, a data/evaluation contract, and an explicit unknowns ledger.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a seeded, tested, packaged, versioned training workflow with a model card.