Quantization, compilation, and fused kernels
Optimize the measured bottleneck, and treat every numeric or execution rewrite as a new artifact that must pass quality gates.
Long-context generation breaks latency and cost targets.
This chapter moves from name the bottleneck to gate the whole workload. We will make the idea concrete in a small code example before placing it back inside a production system with p99 first-token and cost/token SLOs.
Compare latency percentiles, memory, cost, and task-specific quality across short, long, common, and safety-critical slices.
Verify the behavior illustrated in symmetric_int8.py, then measure decision utility, slice reliability, and system latency.
Maximizing aggregate tokens/s destroys p99 first-token latency and harms a rare-language quality slice. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
Inference optimization is not one switch. Quantization changes numeric representation, compilation specializes execution, and fused kernels move less intermediate data; each attacks a different bottleneck and each can change correctness or quality.
Quantization maps floating-point values onto a smaller set of representable integers using a scale and sometimes a zero point. Weight-only quantization primarily reduces model storage and memory bandwidth. Quantizing activations or the KV cache can save more, but calibration, outliers, and accumulation precision become more important.
Compilation and fusion work best after a profiler identifies launch overhead or memory traffic. A fused normalization-plus-projection may avoid materializing a large tensor, but dynamic shapes can trigger recompilation or fallbacks. Benchmark warm and cold paths, representative shapes, and downstream task quality rather than accepting a generic speedup claim.
- GQA/MQA
- quantization
- paged attention
- continuous batching
Follow the reasoning before touching the code.
- 01
Name the bottleneck
Use traces to decide whether the workload is limited by memory bandwidth, arithmetic, kernel launches, shape changes, or host scheduling.
- 02
Change one layer
Evaluate quantization, compilation, and fusion separately against a locked floating-point reference before composing them.
- 03
Gate the whole workload
Compare latency percentiles, memory, cost, and task-specific quality across short, long, common, and safety-critical slices.
import numpy as np
w = np.array([-2.2, -0.4, 0.0, 0.7, 3.1], dtype=np.float32)
scale = np.max(np.abs(w)) / 127
q = np.clip(np.round(w / scale), -127, 127).astype(np.int8)
restored = q.astype(np.float32) * scale
print('scale:', round(float(scale), 4))
print('codes:', q.tolist())
print('max error:', float(np.max(np.abs(w - restored))))See what symmetric int8 quantization keeps and loses
What to notice
- One large-magnitude outlier sets the scale and leaves fewer effective levels for smaller weights.
- Dequantized values approximate the reference; numerical closeness must still be connected to end-task quality.
YOUR TURN Pause and predict before revealing the answer.
Replace 3.1 with 31.0 while leaving the other values unchanged. What happens to the small 0.4 and 0.7 values?
Need a hint?
The maximum absolute value determines the distance between adjacent quantized levels.
Check your reasoning
The scale becomes about ten times coarser, so small values incur much larger relative error and may collapse toward only a few codes.
Lessons 1, 2 in this course.
Connect the chapter to the course’s quantitative spine.
Optimize the measured bottleneck, and treat every numeric or execution rewrite as a new artifact that must pass quality gates. The kv-cache memory estimate below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
KV-cache memory estimate
| Symbol | Meaning / shape / unit |
|---|---|
L | decoder layers |
B | active sequences |
S | cached tokens |
H_kv | KV heads |
q | bytes per scalar |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: L, B, S, H_kv, q.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c16(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
GPU memory ledger
Change microbatch size and calculate a simplified training-memory budget.
Assumption: 3.2 GB weights+grads+optimizer, 0.34 GB activations per sample, 24 GB device, 10% reserve.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| block 1 | 1.09 GB activations | resident |
| block 2 | 2.18 GB activations | resident |
| block 3 | 3.26 GB activations | resident |
| block 4 | 4.35 GB activations | resident |
| block 5 | 5.44 GB activations | resident |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Optimize the measured bottleneck, and treat every numeric or execution rewrite as a new artifact that must pass quality gates. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Name the bottleneck” into a written invariant and a failing test.
- 02
Promote symmetric_int8.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Gate the whole workload” as the release checkpoint: Compare latency percentiles, memory, cost, and task-specific quality across short, long, common, and safety-critical slices.
- 04
Serve or materialize behind explicit p99 first-token and cost/token SLOs.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Maximizing aggregate tokens/s destroys p99 first-token latency and harms a rare-language quality slice. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
LLM inference capacity plan
Benchmark cache, batching, concurrency, sampling, and one quantized path on mixed traces.
- Benchmark contract
- Profiler/bottleneck report
- Quality gates
- Capacity and cost recommendation
Read primary material with a purpose.
Return to the opening failure.
Choose optimizations from workload traces with quality gates, tail latency, memory headroom, and cost per token.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a capacity plan using efficient attention, quantization, batching, and modern routing.