Transformer
Transformer in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Text to tensors: tokenization, embeddings, position
Use text to tensors: tokenization, embeddings, position to move the transformer from scratch production brief toward a defensible release.
Build a transformer from scratchScaled dot-product self-attention by hand
Use scaled dot-product self-attention by hand to move the transformer from scratch production brief toward a defensible release.
Build a transformer from scratchHeads, norms, residuals, and MLPs
Use heads, norms, residuals, and mlps to move the transformer from scratch production brief toward a defensible release.
Build a transformer from scratchTrain a causal language model
Use train a causal language model to move the transformer from scratch production brief toward a defensible release.
Build a transformer from scratchSampling, KV cache, and efficient generation
Use sampling, kv cache, and efficient generation to move the transformer from scratch production brief toward a defensible release.
Courses & assignments
Dependency-authoritative learning units.
Casebook
Reported facts and course reconstructions.