Pretraining objectives, scaling, and checkpoints
Pretraining progress is tokens transformed into validated capability, backed by slice-aware loss and checkpoints that preserve the exact continuation state.
A private support assistant must answer from evidence and resist manipulation.
This chapter moves from define the token objective to checkpoint recoverable progress. We will make the idea concrete in a small code example before placing it back inside a production system with a declared latency, cost, and operator-capacity budget.
Save complete state at an interval chosen from failure rate and checkpoint cost. Kill and resume a pilot, then compare its next losses to an uninterrupted run.
Verify the behavior illustrated in tiny_pretrain.py, then measure decision utility, slice reliability, and system latency.
Evaluation contamination and feedback only from successful chats hide refusals, unsupported languages, and hallucinated citations. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
Pretraining converts a vast token stream into billions of small prediction problems; scaling works only while data, optimization, and recovery remain coherent.
For an autoregressive model, negative log likelihood rewards probability assigned to each actual next token. The average loss can be converted to perplexity, but neither number guarantees factuality or usefulness. Track loss by source and language because a mixture-level improvement can hide regression on a low-weight domain.
Scaling model size, tokens, batch, and compute together is an experimental design problem. Pilot runs reveal stable learning rates and throughput before committing the full budget. Checkpoints should be scheduled by recoverable work—often tokens or wall time—and contain model, optimizer, scheduler, scaler, RNG, data cursor, and mixture state so a restart resumes the same training process.
- decontamination
- tokenizer fertility
- SFT
- LoRA
Follow the reasoning before touching the code.
- 01
Define the token objective
Specify tokenizer version, context packing, target shift, padding/document-boundary masks, and whether loss is averaged per token or per sequence.
- 02
Run a scaling ladder
Use small, medium, then target pilots to measure convergence, instability, memory, throughput, and loss by slice. Extrapolate with uncertainty and stop gates.
- 03
Checkpoint recoverable progress
Save complete state at an interval chosen from failure rate and checkpoint cost. Kill and resume a pilot, then compare its next losses to an uninterrupted run.
import torch
vocab, width = 128, 32
embedding = torch.nn.Embedding(vocab, width)
rnn = torch.nn.GRU(width, width, batch_first=True)
head = torch.nn.Linear(width, vocab)
parameters = [*embedding.parameters(), *rnn.parameters(), *head.parameters()]
optimizer = torch.optim.AdamW(parameters, lr=3e-4)
tokens_seen = 0
for step in range(3):
tokens = torch.randint(0, vocab, (8, 17))
hidden, _ = rnn(embedding(tokens[:, :-1]))
logits = head(hidden)
loss = torch.nn.functional.cross_entropy(logits.flatten(0, 1), tokens[:, 1:].flatten())
optimizer.zero_grad(); loss.backward(); optimizer.step()
tokens_seen += tokens[:, 1:].numel()
torch.save({'model': [embedding.state_dict(), rnn.state_dict(), head.state_dict()],
'optimizer': optimizer.state_dict(), 'tokens_seen': tokens_seen}, 'checkpoint.pt')A tiny causal recurrent model demonstrates shifted NLL and token-counted state.
What to notice
- Each 17-token row provides 16 targets, so three steps process `3 × 8 × 16 = 384` supervised tokens.
- The example saves weights, optimizer state, and progress, but production recovery must additionally preserve RNG, scheduler, data cursor, mixture, and configuration state.
YOUR TURN Pause and predict before revealing the answer.
Overall validation loss improves while loss on a small target language worsens. Can the run still be declared better?
Need a hint?
The aggregate is weighted by the evaluation mixture and may hide minority slices.
Check your reasoning
Not without the release contract’s slice policy. If the target language has a non-regression gate, the run fails even as average loss improves; the mixture or sampling strategy needs revision.
Lessons 1, 2 in this course.
Connect the chapter to the course’s quantitative spine.
Pretraining progress is tokens transformed into validated capability, backed by slice-aware loss and checkpoints that preserve the exact continuation state. The autoregressive next-token loss below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Autoregressive next-token loss
| Symbol | Meaning / shape / unit |
|---|---|
T | sequence length |
x_t | target token |
θ | model parameters |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: T, x_t, θ.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c15(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Retrieval depth lab
Change top-k and estimate evidence recall, context cost, and distractor risk.
Assumption: Fixed retrieval curve; each passage contributes 180 tokens.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| passage 1 | 88.0% | 180 tokens |
| passage 2 | 79.0% | 192 tokens |
| passage 3 | 70.0% | 204 tokens |
| passage 4 | 61.0% | 216 tokens |
| passage 5 | 52.0% | 228 tokens |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Pretraining progress is tokens transformed into validated capability, backed by slice-aware loss and checkpoints that preserve the exact continuation state. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Define the token objective” into a written invariant and a failing test.
- 02
Promote tiny_pretrain.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Checkpoint recoverable progress” as the release checkpoint: Save complete state at an interval chosen from failure rate and checkpoint cost. Kill and resume a pilot, then compare its next losses to an uninterrupted run.
- 04
Serve or materialize behind explicit a declared latency, cost, and operator-capacity budget.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Evaluation contamination and feedback only from successful chats hide refusals, unsupported languages, and hallucinated citations. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Domain adaptation release package
Compare prompt, retrieval, and optional PEFT approaches under one provenance and evaluation contract.
- Approach decision
- Provenance ledger
- Slice evaluation
- Safety probes/model card
Read primary material with a purpose.
Return to the opening failure.
Release model, tokenizer, retrieval index, prompt, policy, datasets, eval suite, and runtime config as one bundle.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a grounded system design spanning adaptation, rag, evaluation, and safety.