Evals, model cards, feedback, and reproducibility
Reliable releases bind artifacts, predeclared eval gates, honest limitations, and bias-aware feedback into one reproducible decision record.
A private support assistant must answer from evidence and resist manipulation.
This chapter moves from write gates before results to close the feedback loop carefully. We will make the idea concrete in a small code example before placing it back inside a production system with a declared latency, cost, and operator-capacity budget.
Sample successes, failures, refusals, and abandoned sessions; remove sensitive data, review labels, and evaluate any update on untouched gates before release.
Verify the behavior illustrated in release-card.yaml, then measure decision utility, slice reliability, and system latency.
Evaluation contamination and feedback only from successful chats hide refusals, unsupported languages, and hallucinated citations. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
A model release is an evidence-backed bundle. The weights alone cannot tell you what was tested, what changed, or when to roll back.
Evals should represent the decisions the system must make: capability tasks, safety attacks, slice behavior, calibration or grounding, latency, and tool reliability. Keep fixed regression sets for comparability and fresh sets for contamination resistance. Report uncertainty and example counts; tiny slices should not masquerade as precise percentages.
A model card records intended use, limits, training and adaptation provenance, eval results, and operational requirements. Production feedback then creates new evidence, but it is selection-biased: ratings come from exposed users, often only on completed conversations. Preserve raw outcome context, audit sampling, and never train directly on feedback without privacy, abuse, and quality controls.
- DPO/RLHF
- release gates
Follow the reasoning before touching the code.
- 01
Write gates before results
Set primary thresholds, slice non-regression limits, safety stop conditions, latency budgets, and rollback triggers before comparing candidate models.
- 02
Version the entire bundle
Bind model, tokenizer, prompt, retrieval index, tool schemas, policy, runtime, datasets, and eval code to immutable identifiers in the model card.
- 03
Close the feedback loop carefully
Sample successes, failures, refusals, and abandoned sessions; remove sensitive data, review labels, and evaluate any update on untouched gates before release.
bundle: support-assistant-2026-08-29
artifacts:
model: sha256:model-a17
tokenizer: sha256:tok-42c
prompt: sha256:prompt-91f
retrieval_index: sha256:index-11b
evals:
grounded_answer_rate: 0.94
unsupported_language_pass_rate: 0.88
tool_schema_valid_rate: 0.997
prompt_injection_block_rate: 0.982
release_gates:
grounded_answer_rate_min: 0.92
tool_schema_valid_rate_min: 0.995
p95_latency_ms_max: 900
rollback:
hallucinated_citation_rate_max: 0.01
feedback_sample: successes_failures_refusals_and_abandonsA compact release card binds artifacts to measurable launch and rollback gates.
What to notice
- Artifact digests make the eval claims refer to one reproducible bundle rather than an ambiguous model name.
- The card separates launch gates from a production rollback signal, and it names an owner who can act on that signal.
YOUR TURN Pause and predict before revealing the answer.
Users rate only successful completed chats, and average rating rises after release. What can you conclude?
Need a hint?
Consider who had an opportunity and motivation to submit a rating.
Check your reasoning
Very little about total quality: failures, refusals, and abandoned sessions are underrepresented. Join ratings with exposure and outcome logs, sample missing-outcome cohorts, and retain controlled evals as release evidence.
Lessons 1, 2, 3, 4 in this course.
Connect the chapter to the course’s quantitative spine.
Reliable releases bind artifacts, predeclared eval gates, honest limitations, and bias-aware feedback into one reproducible decision record. The autoregressive next-token loss below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Autoregressive next-token loss
| Symbol | Meaning / shape / unit |
|---|---|
T | sequence length |
x_t | target token |
θ | model parameters |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: T, x_t, θ.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c15(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Retrieval depth lab
Change top-k and estimate evidence recall, context cost, and distractor risk.
Assumption: Fixed retrieval curve; each passage contributes 180 tokens.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| passage 1 | 88.0% | 180 tokens |
| passage 2 | 79.0% | 192 tokens |
| passage 3 | 70.0% | 204 tokens |
| passage 4 | 61.0% | 216 tokens |
| passage 5 | 52.0% | 228 tokens |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Reliable releases bind artifacts, predeclared eval gates, honest limitations, and bias-aware feedback into one reproducible decision record. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Write gates before results” into a written invariant and a failing test.
- 02
Promote release-card.yaml from illustration to a versioned implementation with representative fixtures.
- 03
Use “Close the feedback loop carefully” as the release checkpoint: Sample successes, failures, refusals, and abandoned sessions; remove sensitive data, review labels, and evaluate any update on untouched gates before release.
- 04
Serve or materialize behind explicit a declared latency, cost, and operator-capacity budget.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Evaluation contamination and feedback only from successful chats hide refusals, unsupported languages, and hallucinated citations. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Domain adaptation release package
Compare prompt, retrieval, and optional PEFT approaches under one provenance and evaluation contract.
- Approach decision
- Provenance ledger
- Slice evaluation
- Safety probes/model card
Read primary material with a purpose.
Return to the opening failure.
Release model, tokenizer, retrieval index, prompt, policy, datasets, eval suite, and runtime config as one bundle.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a grounded system design spanning adaptation, rag, evaluation, and safety.