concept:kv-cache

KV Cache

KV Cache in the context of production machine-learning systems.

Lessons

Formal explanations, examples, simulations, and checkpoints.

Use autoregressive decoding policies to move the efficient inference production brief toward a defensible release.

Use kv caches, gqa/mqa, and long context to move the efficient inference production brief toward a defensible release.

Use quantization, compilation, and fused kernels to move the efficient inference production brief toward a defensible release.

Use continuous batching and speculative decoding to move the efficient inference production brief toward a defensible release.

Use distributed and moe inference to move the efficient inference production brief toward a defensible release.

Dependency-authoritative learning units.

A capacity plan using efficient attention, quantization, batching, and modern routing.

Assignment: LLM inference capacity plan

Reported facts and course reconstructions.