Operators, product owners, and people affected by the transformer from scratch decision.
Transformers · weeks 17–19
Build a transformer from scratch
Domain autocomplete needs a model whose every component the team can explain.
Opening production brief
A measurable problem, before a model.
tokenization, embeddings, RoPE, QKV projections with version, owner, event time, and availability contracts.
Domain autocomplete needs a model whose every component the team can explain. Every solution must state latency, cost, capacity, and fallback limits.
Offline evidence plus a deployable resolution: Tokenizer, attention, RoPE, residual stack, training loop, generation, and KV cache.
Frame and baseline
Turn the production problem into explicit data, metric, and baseline contracts.
- 01↗
Text to tensors: tokenization, embeddings, position
Tokenizer and position scheme are not preprocessing trivia; together they define what symbols and order the transformer can represent.
65–90 min · checkpoint · deterministic lab - 02↗
Scaled dot-product self-attention by hand
Attention is a masked, normalized lookup: queries score keys, softmax allocates influence, and values carry the information.
65–90 min · checkpoint · deterministic lab
Build and stress
Formalize the machinery, test counterexamples, and expose system limits.
- 03↗
Heads, norms, residuals, and MLPs
Attention mixes tokens, the MLP mixes channels, and normalized residual paths let both transformations accumulate without losing the original stream.
65–90 min · checkpoint · deterministic lab - 04↗
Train a causal language model
Causal LM training is next-token alignment plus a no-looking-ahead proof; everything else scales that contract.
65–90 min · checkpoint · deterministic lab
Resolve and operate
Join model behavior to architecture, observability, rollout, and rollback.
- 05↗
Sampling, KV cache, and efficient generation
Efficient generation couples a declared sampling policy with a position-correct KV cache whose logits match the uncached model.
65–90 min · checkpoint · deterministic lab - HW↗
Decoder-only transformer from scratch
Specify tokenizer, attention blocks, causal training, unit tests, sampling, and KV-cache parity without a transformer module.
Rubric · staged hints · reference resolution
Prerequisite guidance
3 concepts make this faster.
Prerequisites are guidance, never hard gates.