What the GPU actually executes
GPU performance comes from feeding a few substantial, well-shaped kernels—not merely from placing tensors on a CUDA device.
Training repeatedly OOMs and misses its nightly completion window.
This chapter moves from translate operations into kernels to measure the device clock. We will make the idea concrete in a small code example before placing it back inside a production system with 24 GB device and nightly completion.
Warm up, record CUDA events around the exact region, synchronize only for the final reading, and report shapes, dtype, and hardware with the result.
Verify the behavior illustrated in cuda_events.py, then measure decision utility, slice reliability, and system latency.
Unsynchronized timing reports impossible speedups while retained tensors keep computation graphs alive and leak memory. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
A GPU does not execute “a model” in one stroke. It launches a procession of kernels, each moving tensors, performing an operation, and handing results to the next kernel.
GPUs win when thousands of threads perform regular work in parallel. A matrix multiplication can occupy the device well; a long chain of tiny element-wise operations may spend more time launching kernels than computing. Performance is bounded either by arithmetic throughput or by how quickly bytes travel to and from high-bandwidth memory—the distinction captured by arithmetic intensity.
CUDA launches are asynchronous: Python can enqueue work and continue before the GPU finishes. That makes an ordinary wall-clock timer misleading unless the device is synchronized or CUDA events are used. Warm-up is equally important because first iterations may include memory allocation, library initialization, and kernel selection.
- kernels
- HBM bandwidth
- arithmetic intensity
- pinned memory
Follow the reasoning before touching the code.
- 01
Translate operations into kernels
Inspect the executed graph and ask how many kernels launch, what tensor shapes they see, and whether adjacent operations can be fused.
- 02
Classify the bottleneck
Compare time spent moving bytes with time spent doing arithmetic. Low-intensity operations usually need fewer transfers or fusion; compute-heavy operations benefit from larger, accelerator-friendly shapes.
- 03
Measure the device clock
Warm up, record CUDA events around the exact region, synchronize only for the final reading, and report shapes, dtype, and hardware with the result.
import torch
n = 2048
x = torch.randn(n, n, device='cuda')
for _ in range(5):
_ = x @ x
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
y = x @ x
end.record()
torch.cuda.synchronize()
milliseconds = start.elapsed_time(end)
print(round(milliseconds, 2), y.shape)Time a warmed-up matrix multiplication on the GPU timeline.
What to notice
- The synchronization happens after the end event, so the measured interval is GPU work rather than Python enqueue time.
- Doubling matrix width increases arithmetic much faster than launch count, usually improving utilization until memory capacity becomes the constraint.
YOUR TURN Pause and predict before revealing the answer.
Replace the matrix multiplication with 100 tiny additions in a Python loop. Why can less arithmetic take surprisingly long?
Need a hint?
Count launches, not just floating-point operations.
Check your reasoning
Each addition can launch its own small kernel. Launch overhead and repeated memory traffic dominate because none of the kernels contains enough parallel work to occupy the GPU.
Neural networks
Connect the chapter to the course’s quantitative spine.
GPU performance comes from feeding a few substantial, well-shaped kernels—not merely from placing tensors on a CUDA device. The roofline bound below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Roofline bound
| Symbol | Meaning / shape / unit |
|---|---|
P_peak | peak compute |
I | arithmetic intensity |
B_mem | memory bandwidth |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: P_peak, I, B_mem.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c12(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
GPU memory ledger
Change microbatch size and calculate a simplified training-memory budget.
Assumption: 3.2 GB weights+grads+optimizer, 0.34 GB activations per sample, 24 GB device, 10% reserve.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| block 1 | 1.09 GB activations | resident |
| block 2 | 2.18 GB activations | resident |
| block 3 | 3.26 GB activations | resident |
| block 4 | 4.35 GB activations | resident |
| block 5 | 5.44 GB activations | resident |
Trace the complete operating path.
Carry the chapter’s central idea into the system: GPU performance comes from feeding a few substantial, well-shaped kernels—not merely from placing tensors on a CUDA device. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Translate operations into kernels” into a written invariant and a failing test.
- 02
Promote cuda_events.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Measure the device clock” as the release checkpoint: Warm up, record CUDA events around the exact region, synchronize only for the final reading, and report shapes, dtype, and hardware with the result.
- 04
Serve or materialize behind explicit 24 GB device and nightly completion.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Unsynchronized timing reports impossible speedups while retained tensors keep computation graphs alive and leak memory. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Single-GPU optimization plan
Profile and optimize a correctness-locked trainer without sacrificing model quality.
- Profiler diagnosis
- Mixed precision parity
- Memory plan
- Throughput/cost report
Read primary material with a purpose.
Return to the opening failure.
Record hardware/software, shapes, precision, seeds, throughput method, peak memory, and cost per sample.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a profiled, mixed-precision, memory-budgeted training loop.