Profile from timeline to cost per sample
Profile a controlled workload, explain the timeline, and judge improvements by cost to an unchanged quality target.
Training repeatedly OOMs and misses its nightly completion window.
This chapter moves from freeze the benchmark contract to price the verified result. We will make the idea concrete in a small code example before placing it back inside a production system with 24 GB device and nightly completion.
Repeat the benchmark, confirm quality parity, and calculate throughput, peak memory, and cost per useful unit with uncertainty across runs.
Verify the behavior illustrated in profile_to_cost.py, then measure decision utility, slice reliability, and system latency.
Unsynchronized timing reports impossible speedups while retained tensors keep computation graphs alive and leak memory. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
A profiler trace becomes useful only when it changes an engineering decision: which bottleneck to fix, what quality must remain, and how much each example costs.
Begin with a stable benchmark: fixed shapes and precision, warm-up iterations, synchronized measurement, and enough steps to average normal variation. The profiler timeline then explains the total by exposing CPU gaps, memory copies, kernel durations, synchronization points, and memory allocations. Profiling every step can itself distort the workload, so capture a short representative window.
Optimization ends at a product metric, not at GPU utilization. Convert measured throughput into cost per sample or per million tokens using the actual accelerator price and utilization assumptions. Record peak memory and model quality beside cost so a faster run that changes convergence or fails at production shapes cannot win silently.
- activation memory
- profilers
Follow the reasoning before touching the code.
- 01
Freeze the benchmark contract
Record hardware, software versions, model and batch shapes, dtype, warm-up, synchronization, and the exact counted unit such as samples or tokens.
- 02
Walk from wide gaps inward
Use the timeline to find dominant idle regions or kernels, then inspect their operators, shapes, call stacks, and memory behavior. Change one suspected cause.
- 03
Price the verified result
Repeat the benchmark, confirm quality parity, and calculate throughput, peak memory, and cost per useful unit with uncertainty across runs.
import time
import torch
from torch.profiler import profile, ProfilerActivity
model = torch.nn.Linear(1024, 1024).cuda()
batch = torch.randn(256, 1024, device='cuda')
for _ in range(5): model(batch)
torch.cuda.synchronize()
start = time.perf_counter()
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
for _ in range(20): model(batch)
torch.cuda.synchronize()
seconds = time.perf_counter() - start
samples = 20 * batch.shape[0]
cost = seconds / 3600 * 2.10
print(prof.key_averages().table(sort_by='cuda_time_total', row_limit=5))
print({'samples_per_second': samples / seconds, 'cost_per_sample': cost / samples})Capture a short GPU profile and translate elapsed time into cost per sample.
What to notice
- The cost calculation states the assumed $2.10 per accelerator-hour instead of presenting cost as a hardware-independent fact.
- The operator table explains where device time went, while the synchronized outer timer supplies the end-to-end denominator.
YOUR TURN Pause and predict before revealing the answer.
An optimization raises samples per second by 20% but needs 25% more training steps to reach the same validation score. Is it cheaper?
Need a hint?
Cost to convergence equals cost per step multiplied by steps to the quality target.
Check your reasoning
No under equal hourly price: runtime becomes 1.25 / 1.20 ≈ 1.04 times the original, about 4% more expensive. Throughput alone would give the wrong decision.
Lessons 1, 2, 3, 4 in this course.
Connect the chapter to the course’s quantitative spine.
Profile a controlled workload, explain the timeline, and judge improvements by cost to an unchanged quality target. The roofline bound below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Roofline bound
| Symbol | Meaning / shape / unit |
|---|---|
P_peak | peak compute |
I | arithmetic intensity |
B_mem | memory bandwidth |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: P_peak, I, B_mem.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c12(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
GPU memory ledger
Change microbatch size and calculate a simplified training-memory budget.
Assumption: 3.2 GB weights+grads+optimizer, 0.34 GB activations per sample, 24 GB device, 10% reserve.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| block 1 | 1.09 GB activations | resident |
| block 2 | 2.18 GB activations | resident |
| block 3 | 3.26 GB activations | resident |
| block 4 | 4.35 GB activations | resident |
| block 5 | 5.44 GB activations | resident |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Profile a controlled workload, explain the timeline, and judge improvements by cost to an unchanged quality target. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Freeze the benchmark contract” into a written invariant and a failing test.
- 02
Promote profile_to_cost.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Price the verified result” as the release checkpoint: Repeat the benchmark, confirm quality parity, and calculate throughput, peak memory, and cost per useful unit with uncertainty across runs.
- 04
Serve or materialize behind explicit 24 GB device and nightly completion.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Unsynchronized timing reports impossible speedups while retained tensors keep computation graphs alive and leak memory. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Single-GPU optimization plan
Profile and optimize a correctness-locked trainer without sacrificing model quality.
- Profiler diagnosis
- Mixed precision parity
- Memory plan
- Throughput/cost report
Read primary material with a purpose.
Return to the opening failure.
Record hardware/software, shapes, precision, seeds, throughput method, peak memory, and cost per sample.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a profiled, mixed-precision, memory-budgeted training loop.