KV Cache
KV Cache in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Autoregressive decoding policies
The model supplies possibilities; the decoding policy decides which possibilities can become product behavior.
Frontier architectures & efficient inferenceKV caches, GQA/MQA, and long context
KV caching removes repeated history compute by turning context length into a first-class memory budget.
Frontier architectures & efficient inferenceQuantization, compilation, and fused kernels
Optimize the measured bottleneck, and treat every numeric or execution rewrite as a new artifact that must pass quality gates.
Frontier architectures & efficient inferenceContinuous batching and speculative decoding
Throughput techniques earn their place only when they improve completed requests—not merely scheduled tokens.
Frontier architectures & efficient inferenceDistributed and MoE inference
Distributed inference is constrained by the slowest communication or routing boundary, not by the sum of advertised compute.
Courses & assignments
Dependency-authoritative learning units.
Casebook
Reported facts and course reconstructions.