Inventory, risk tiers, accountability, evidence
Governance starts by giving the complete decision system a scope, an owner, a risk tier, and versioned evidence.
A high-performing system creates subgroup harm and leaks sensitive information.
This chapter moves from inventory the whole decision to bind evidence to release. We will make the idea concrete in a small code example before placing it back inside a production system with risk-tiered human oversight.
Define required tests, reviewers, monitoring, recourse, and stop conditions, then attach their results to the exact deployed bundle.
Verify the behavior illustrated in system-card.yaml, then measure decision utility, slice reliability, and system latency.
The team reports one aggregate fairness number, deletes the protected field, loads an untrusted pickle, and logs sensitive inputs forever. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
You cannot govern an ML system that has no durable identity. Governance begins with an inventory of systems, owners, intended uses, affected people, dependencies, risk tier, and the evidence required to approve or stop each release.
The inventory unit is the decision system, not just the model. It includes data sources, model and vendor components, prompts, retrieval, thresholds, human review, downstream actions, and deployment regions. Intended and prohibited uses establish scope; material changes should trigger review rather than inheriting an old approval automatically.
Risk tiering should reflect severity, scale, reversibility, autonomy, sensitivity, and who bears harm. Higher risk requires stronger evidence, independence, monitoring, recourse, and approval. Accountability means every control has a named owner and every release can show which evidence supported the decision at that time.
- intended use
- equalized odds
- intersectional slices
- privacy threat modeling
Follow the reasoning before touching the code.
- 01
Inventory the whole decision
Record purpose, users, affected groups, inputs, outputs, actions, vendors, locations, versions, and accountable business and technical owners.
- 02
Tier the credible harm
Assess severity, likelihood, exposure, reversibility, autonomy, and data sensitivity; document assumptions and prohibited use.
- 03
Bind evidence to release
Define required tests, reviewers, monitoring, recourse, and stop conditions, then attach their results to the exact deployed bundle.
system_id: underwriting-assist
owner: lending-risk
intended_use: prioritize applications for trained human review
prohibited_use: automatic denial
affected_people: loan applicants
risk_tier: high
release_bundle: model_v12+policy_v5+features_v31
required_evidence:
- intersectional performance review
- privacy and security assessment
- appeal-path exercise
stop_owner: head-of-riskA system inventory record makes scope and accountability reviewable
What to notice
- The intended action is human prioritization, while automatic denial is explicitly outside the approved boundary.
- Evidence is connected to an exact model-policy-feature bundle rather than to a reusable claim that “the model was reviewed.”
YOUR TURN Pause and predict before revealing the answer.
The same model is reused to auto-reject applicants in a new country. Can the prior approval carry over?
Need a hint?
Compare action, autonomy, affected population, jurisdiction, and reversibility.
Check your reasoning
No. The use and risk profile changed materially. Create or update the system record, retier it, evaluate the new population and requirements, and obtain release-specific approval.
Serving systems, Reliability
Connect the chapter to the course’s quantitative spine.
Governance starts by giving the complete decision system a scope, an owner, a risk tier, and versioned evidence. The differential privacy guarantee below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Differential privacy guarantee
| Symbol | Meaning / shape / unit |
|---|---|
M | randomized mechanism |
D, D′ | adjacent datasets |
ε, δ | privacy-loss parameters |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: M, D, D′, ε, δ.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c20(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Threshold parity lab
Move one group threshold and inspect error-rate and utility tradeoffs.
Assumption: Fixed score/label table with different group base rates; group A threshold is 0.5.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| A | 0.50 | TPR 1.00 / FPR 0.33 |
| B | 0.50 | TPR 0.50 / FPR 0.33 |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Governance starts by giving the complete decision system a scope, an owner, a risk tier, and versioned evidence. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Inventory the whole decision” into a written invariant and a failing test.
- 02
Promote system-card.yaml from illustration to a versioned implementation with representative fixtures.
- 03
Use “Bind evidence to release” as the release checkpoint: Define required tests, reviewers, monitoring, recourse, and stop conditions, then attach their results to the exact deployed bundle.
- 04
Serve or materialize behind explicit risk-tiered human oversight.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
The team reports one aggregate fairness number, deletes the protected field, loads an untrusted pickle, and logs sensitive inputs forever. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Model assurance package
Produce intended-use, fairness, privacy, security, human-review, monitored-rollout, and release evidence.
- Model/data cards
- Intersectional analysis
- Privacy/retention plan
- Threat model and recourse
Read primary material with a purpose.
Return to the opening failure.
Governance evidence must travel with the exact model, data, policy, and runtime release throughout its lifetime.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a launch review covering fairness, privacy, licensing, abuse, security, and response.