Vision in the wild: thresholds, latency, and drift
Production vision is a thresholded, timed, monitored decision system—not an accuracy number attached to a model file.
A factory has few positive examples of costly visual defects.
This chapter moves from choose an operating point to monitor environment and outcomes. We will make the idea concrete in a small code example before placing it back inside a production system with one shift for inference and review.
Join inputs and decisions to model, threshold, camera, and eventual reviewer labels so drift can be localized before rollback or retraining.
Verify the behavior illustrated in capacity_threshold.py, then measure decision utility, slice reliability, and system latency.
Random burst-frame splitting puts near-duplicates of one product in train and validation. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
A vision score becomes useful only after a threshold, a latency path, and an operating environment give it consequences. Cameras change; class prevalence moves; preprocessing costs time; and a threshold that worked in the lab can overwhelm reviewers in production.
For rare defects, accuracy is dominated by normal images. Select a threshold on validation data using miss cost, false-alarm cost, review capacity, and calibration, then lock it for test and release. Consider an abstention band where uncertain images route to people instead of forcing a brittle binary decision.
Measure end-to-end batch-one latency—including decode, resize, transfer, model, and postprocessing—at warm and cold states. Monitor camera/site slices, brightness and sharpness, input drift, score distribution, review yield, delayed labels, and export parity. Drift is a diagnosis trigger, not automatic permission to retrain.
- group leakage
- camera drift
Follow the reasoning before touching the code.
- 01
Choose an operating point
Sweep validation thresholds, compute precision, recall, workload, and cost, then select a policy that fits both safety and human capacity.
- 02
Benchmark the real request
Replay production image sizes and concurrency through the full exported path, reporting p50 and p99 latency, memory, failures, and fallback behavior.
- 03
Monitor environment and outcomes
Join inputs and decisions to model, threshold, camera, and eventual reviewer labels so drift can be localized before rollback or retraining.
scores = [0.96, 0.83, 0.72, 0.41, 0.15]
labels = [1, 1, 0, 0, 0]
capacity = 2
for threshold in (0.9, 0.8, 0.7, 0.4):
review = [i for i, score in enumerate(scores) if score >= threshold]
positives = sum(labels)
recall = sum(labels[i] for i in review) / positives
if len(review) <= capacity:
print('feasible', threshold, 'reviews', len(review), 'recall', recall)Choose the highest-recall threshold that fits a review queue.
What to notice
- Threshold 0.7 finds both defects but requires three reviews, so it violates this capacity constraint.
- Threshold 0.9 is feasible but misses one defect; threshold 0.8 uses the full queue and recovers both.
YOUR TURN Pause and predict before revealing the answer.
A new camera shifts every normal-image score upward by 0.2. What should happen before changing the threshold?
Need a hint?
A score shift can come from optics, preprocessing, calibration, or real prevalence change.
Check your reasoning
Slice and diagnose the camera path, verify preprocessing and labels, estimate queue and miss impact, and use fallback or rollback if needed; do not silently tune on production outcomes.
Lessons 1, 2, 3, 4 in this course.
Connect the chapter to the course’s quantitative spine.
Production vision is a thresholded, timed, monitored decision system—not an accuracy number attached to a model file. The convolution output size below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Convolution output size
| Symbol | Meaning / shape / unit |
|---|---|
in, out | spatial input/output size |
k | kernel size |
s | stride |
p | padding |
d | dilation |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: in, out, k, s, p, d.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c10(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Receptive-field assembler
Stack kernels and see receptive field, parameter count, and output size.
Assumption: 3×3 stride-one convolutions, 32 channels, 64×64 input, no dilation.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| layer 1 | 3×3 | 62×62 |
| layer 2 | 5×5 | 60×60 |
| layer 3 | 7×7 | 58×58 |
| layer 4 | 9×9 | 56×56 |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Production vision is a thresholded, timed, monitored decision system—not an accuracy number attached to a model file. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Choose an operating point” into a written invariant and a failing test.
- 02
Promote capacity_threshold.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Monitor environment and outcomes” as the release checkpoint: Join inputs and decisions to model, threshold, camera, and eventual reviewer labels so drift can be localized before rollback or retraining.
- 04
Serve or materialize behind explicit one shift for inference and review.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Random burst-frame splitting puts near-duplicates of one product in train and validation. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Rare-defect vision system
Design grouped evaluation, transfer learning, calibration, batch-one export, and camera-drift monitoring.
- Grouped split and audit
- Transfer-learning ablations
- PR and calibration report
- Export/latency/monitoring
Read primary material with a purpose.
Return to the opening failure.
Version color, resize, orientation, normalization, class map, and threshold; monitor camera/site slices.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce cnn and transfer-learning baselines with augmentation, calibration, and monitoring.