Clicks are not labels
A click is an exposed, policy-shaped outcome—not a universal relevance label.
Marketplace ranking learns from biased clicks and reinforces its own mistakes.
This chapter moves from define the behavioral target to build honest examples. We will make the idea concrete in a small code example before placing it back inside a production system with 80 ms ranking budget.
Use exposed non-clicks as qualified negatives, preserve chronological order, and test propensity weighting or randomized data against unweighted training.
Verify the behavior illustrated in propensity_clicks.py, then measure decision utility, slice reliability, and system latency.
A random interaction split and click-only labels improve offline NDCG while amplifying already-prominent items. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
A click records what happened after an item was shown in a particular position; it does not reveal what the user would have chosen from items they never saw. Interaction logs are outcomes of both preference and the old recommendation policy.
Treating every unclicked catalog item as a negative confuses absence of exposure with dislike. Even among exposed items, position, presentation, and repeated visibility affect click probability. Training on raw clicks can therefore teach a new model to reproduce the old system’s prominence rather than user relevance.
Exposure logging makes correction possible. Randomized exploration, propensity scores, delayed outcomes, and carefully defined negatives support less biased learning and evaluation. Inverse-propensity weighting can correct known exposure probabilities, but tiny propensities create high variance and require clipping and diagnostics.
- implicit feedback
- exposure bias
- two-tower embeddings
- BPR
Follow the reasoning before touching the code.
- 01
Define the behavioral target
Choose click, long dwell, save, purchase, or a composite outcome with a maturity window that matches the product decision.
- 02
Log the opportunity
For every request, record the full shown slate, positions, policy version, propensity when available, and eligible but unshown candidates.
- 03
Build honest examples
Use exposed non-clicks as qualified negatives, preserve chronological order, and test propensity weighting or randomized data against unweighted training.
logs = [
{'clicked': 1, 'propensity': 0.8},
{'clicked': 0, 'propensity': 0.8},
{'clicked': 1, 'propensity': 0.2},
]
weighted_clicks = sum(row['clicked'] / row['propensity'] for row in logs)
weighted_total = sum(1 / row['propensity'] for row in logs)
print('raw CTR', sum(row['clicked'] for row in logs) / len(logs))
print('self-normalized IPS', weighted_clicks / weighted_total)Estimate click rate after correcting unequal exposure probabilities.
What to notice
- The click under the rare 0.2-probability exposure receives more weight because it represents more opportunities the logging policy seldom showed.
- Weighting cannot fix an unlogged propensity or an item with zero chance of exposure; support overlap is required.
YOUR TURN Pause and predict before revealing the answer.
Change the last propensity from 0.2 to 0.01. What new statistical risk appears?
Need a hint?
Inspect the inverse weight assigned to that single observation.
Check your reasoning
Its weight becomes 100, so one event can dominate the estimate. Variance control, clipping, and better exploration become essential.
Lessons 1 in this course.
Connect the chapter to the course’s quantitative spine.
A click is an exposed, policy-shaped outcome—not a universal relevance label. The discounted cumulative gain below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Discounted cumulative gain
| Symbol | Meaning / shape / unit |
|---|---|
rel_r | relevance at rank r |
r | one-indexed rank |
K | slate cutoff |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: rel_r, r, K.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c08(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Exploration feedback loop
Allocate unbiased exploration and observe learning coverage versus short-term utility.
Assumption: Simplified marketplace with position-biased clicks and fixed demand.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| rank 1 | 35.0% | 6.2% |
| rank 2 | 30.3% | 6.9% |
| rank 3 | 25.6% | 7.5% |
| rank 4 | 20.8% | 8.2% |
| rank 5 | 16.1% | 8.8% |
Trace the complete operating path.
Carry the chapter’s central idea into the system: A click is an exposed, policy-shaped outcome—not a universal relevance label. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Define the behavioral target” into a written invariant and a failing test.
- 02
Promote propensity_clicks.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Build honest examples” as the release checkpoint: Use exposed non-clicks as qualified negatives, preserve chronological order, and test propensity weighting or randomized data against unweighted training.
- 04
Serve or materialize behind explicit 80 ms ranking budget.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
A random interaction split and click-only labels improve offline NDCG while amplifying already-prominent items. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Hybrid catalog ranker
Specify BM25 plus learned retrieval, ranking, re-ranking, temporal evaluation, and a versioned online path.
- Exposure-aware data contract
- Candidate recall analysis
- Ranker and NDCG report
- Diversity/fallback/latency plan
Read primary material with a purpose.
Return to the opening failure.
Version index, ranker, features, and catalog together; log exposures and measure each stage separately.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce candidate generation, ranking metrics, debiasing, and online experiment design.