Privacy minimization, retention, federated learning, DP
Privacy engineering reduces what the entire system can learn and retain about one protected unit over time.
A high-performing system creates subgroup harm and leaks sensitive information.
This chapter moves from map and minimize to account for repeated release. We will make the idea concrete in a small code example before placing it back inside a production system with risk-tiered human oversight.
Track clipping, noise, epsilon, delta, queries, checkpoints, and composition; validate utility and deletion behavior end to end.
Verify the behavior illustrated in private_count.py, then measure decision utility, slice reliability, and system latency.
The team reports one aggregate fairness number, deletes the protected field, loads an untrusted pickle, and logs sensitive inputs forever. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
Privacy is a lifecycle property, not a promise to remove names. A system can expose people through unnecessary collection, long retention, logs, memorization, model updates, or repeated queries even when obvious identifiers are absent.
Begin with a data-flow threat model: what is collected, why it is necessary, who can access it, where it moves, how long it remains, and how deletion propagates. Minimize fields and precision, separate identities, enforce purpose and access, and expire raw data and derived artifacts according to an owned retention schedule.
Federated learning keeps raw examples on devices but still exposes updates and metadata unless additional protections apply. Differential privacy bounds how much one person can change a randomized output using epsilon and delta; clipping, noise, composition, and the unit of adjacency all matter. A privacy budget is a system parameter with a utility tradeoff, not a decorative metric.
- privacy threat modeling
- differential privacy
- model supply chain
- prompt injection
Follow the reasoning before touching the code.
- 01
Map and minimize
For each field and derived artifact, document purpose, necessity, sensitivity, access, transfer, retention, and deletion propagation.
- 02
Choose the threat and unit
State the attacker capability and whether neighboring datasets differ by event, record, user, device, or another protected unit.
- 03
Account for repeated release
Track clipping, noise, epsilon, delta, queries, checkpoints, and composition; validate utility and deletion behavior end to end.
import numpy as np
def private_count(true_count, epsilon, seed=4):
sensitivity = 1.0
rng = np.random.default_rng(seed)
noise = rng.laplace(0.0, sensitivity / epsilon)
return true_count + noise
for epsilon in (0.2, 1.0, 5.0):
print(epsilon, round(private_count(120, epsilon), 2))A toy Laplace-noised count makes the privacy–utility tradeoff visible
What to notice
- Smaller epsilon adds more noise for the same sensitivity, strengthening the bound while reducing single-query accuracy.
- Releasing many independently noised answers consumes privacy budget; one sample does not describe the composed system.
YOUR TURN Pause and predict before revealing the answer.
A team rounds birth date to age but retains raw birth date indefinitely in debug logs. Has minimization succeeded?
Need a hint?
Follow the data through every copy and operational path, not only the feature table.
Check your reasoning
No. The raw sensitive value still exists without a justified lifecycle. Remove or tightly restrict it, set retention and deletion controls, and verify that logs, backups, features, and models follow the policy.
Lessons 1, 2 in this course.
Connect the chapter to the course’s quantitative spine.
Privacy engineering reduces what the entire system can learn and retain about one protected unit over time. The differential privacy guarantee below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Differential privacy guarantee
| Symbol | Meaning / shape / unit |
|---|---|
M | randomized mechanism |
D, D′ | adjacent datasets |
ε, δ | privacy-loss parameters |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: M, D, D′, ε, δ.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c20(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Threshold parity lab
Move one group threshold and inspect error-rate and utility tradeoffs.
Assumption: Fixed score/label table with different group base rates; group A threshold is 0.5.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| A | 0.50 | TPR 1.00 / FPR 0.33 |
| B | 0.50 | TPR 0.50 / FPR 0.33 |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Privacy engineering reduces what the entire system can learn and retain about one protected unit over time. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Map and minimize” into a written invariant and a failing test.
- 02
Promote private_count.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Account for repeated release” as the release checkpoint: Track clipping, noise, epsilon, delta, queries, checkpoints, and composition; validate utility and deletion behavior end to end.
- 04
Serve or materialize behind explicit risk-tiered human oversight.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
The team reports one aggregate fairness number, deletes the protected field, loads an untrusted pickle, and logs sensitive inputs forever. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Model assurance package
Produce intended-use, fairness, privacy, security, human-review, monitored-rollout, and release evidence.
- Model/data cards
- Intersectional analysis
- Privacy/retention plan
- Threat model and recourse
Read primary material with a purpose.
Return to the opening failure.
Governance evidence must travel with the exact model, data, policy, and runtime release throughout its lifetime.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a launch review covering fairness, privacy, licensing, abuse, security, and response.