SFT, preference optimization, safety, and tools
Post-training aligns behavior only to the examples and constraints you encode; tool permissions and safety must still be enforced and evaluated outside the loss.
A private support assistant must answer from evidence and resist manipulation.
This chapter moves from curate the behavior target to sandbox tool behavior. We will make the idea concrete in a small code example before placing it back inside a production system with a declared latency, cost, and operator-capacity budget.
Validate arguments against schemas, apply least privilege outside the model, require confirmation for consequential actions, and test malicious tool outputs and partial failures.
Verify the behavior illustrated in dpo_margin.py, then measure decision utility, slice reliability, and system latency.
Evaluation contamination and feedback only from successful chats hide refusals, unsupported languages, and hallucinated citations. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
Post-training teaches a base model how to behave in an interaction—not just what token is likely to follow in raw text.
Supervised fine-tuning imitates curated responses conditioned on prompts, so formatting, refusal style, reasoning exposure, and tool-call examples become part of the behavior target. Preference optimization compares chosen and rejected responses, encouraging the policy to favor one relative to a reference model. Preference data quality matters more than the acronym: ambiguous or biased pairs train ambiguous or biased behavior.
Safety and tools require explicit execution contracts. A tool-capable model should emit validated structured arguments, receive untrusted outputs as data, and never gain permissions merely because text requests them. Capability and safety evals must cover refusal boundaries, prompt injection, invalid arguments, side effects, and recovery when a tool fails.
- SFT
- LoRA
- DPO/RLHF
- release gates
Follow the reasoning before touching the code.
- 01
Curate the behavior target
Define which answers should be direct, clarifying, tool-using, or refused. Deduplicate templates and inspect annotator agreement by safety and language slice.
- 02
Optimize against a reference
For preference training, compare chosen-versus-rejected log-probability margins to a frozen reference and monitor drift, verbosity, and capability regressions.
- 03
Sandbox tool behavior
Validate arguments against schemas, apply least privilege outside the model, require confirmation for consequential actions, and test malicious tool outputs and partial failures.
import torch
import torch.nn.functional as F
def dpo_loss(policy_chosen, policy_rejected,
reference_chosen, reference_rejected, beta=0.1):
policy_margin = policy_chosen - policy_rejected
reference_margin = reference_chosen - reference_rejected
advantage = beta * (policy_margin - reference_margin)
return -F.logsigmoid(advantage).mean()
pc = torch.tensor([-2.0, -1.5], requires_grad=True)
pr = torch.tensor([-3.0, -1.7], requires_grad=True)
loss = dpo_loss(pc, pr, torch.tensor([-2.4, -1.4]), torch.tensor([-2.8, -1.8]))
loss.backward()
print(round(loss.item(), 3), pc.grad.tolist())DPO rewards a policy margin relative to the same margin under a reference model.
What to notice
- The objective cares about the policy’s chosen-rejected margin relative to the reference margin, not the chosen response in isolation.
- A preference loss can improve while tool safety regresses, so schema, injection, and side-effect tests remain independent release gates.
YOUR TURN Pause and predict before revealing the answer.
Annotators prefer a fluent answer that invents a tool result over a cautious answer that reports failure. What happens if the pair is accepted?
Need a hint?
Preference optimization faithfully amplifies the ranking signal it receives.
Check your reasoning
The pair rewards fabricated success. Reject or relabel it under a rubric that prioritizes truthful tool state, and include tool-failure examples where the chosen response explains and recovers safely.
Lessons 1, 2, 3 in this course.
Connect the chapter to the course’s quantitative spine.
Post-training aligns behavior only to the examples and constraints you encode; tool permissions and safety must still be enforced and evaluated outside the loss. The autoregressive next-token loss below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Autoregressive next-token loss
| Symbol | Meaning / shape / unit |
|---|---|
T | sequence length |
x_t | target token |
θ | model parameters |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: T, x_t, θ.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c15(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Retrieval depth lab
Change top-k and estimate evidence recall, context cost, and distractor risk.
Assumption: Fixed retrieval curve; each passage contributes 180 tokens.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| passage 1 | 88.0% | 180 tokens |
| passage 2 | 79.0% | 192 tokens |
| passage 3 | 70.0% | 204 tokens |
| passage 4 | 61.0% | 216 tokens |
| passage 5 | 52.0% | 228 tokens |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Post-training aligns behavior only to the examples and constraints you encode; tool permissions and safety must still be enforced and evaluated outside the loss. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Curate the behavior target” into a written invariant and a failing test.
- 02
Promote dpo_margin.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Sandbox tool behavior” as the release checkpoint: Validate arguments against schemas, apply least privilege outside the model, require confirmation for consequential actions, and test malicious tool outputs and partial failures.
- 04
Serve or materialize behind explicit a declared latency, cost, and operator-capacity budget.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Evaluation contamination and feedback only from successful chats hide refusals, unsupported languages, and hallucinated citations. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Domain adaptation release package
Compare prompt, retrieval, and optional PEFT approaches under one provenance and evaluation contract.
- Approach decision
- Provenance ledger
- Slice evaluation
- Safety probes/model card
Read primary material with a purpose.
Return to the opening failure.
Release model, tokenizer, retrieval index, prompt, policy, datasets, eval suite, and runtime config as one bundle.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a grounded system design spanning adaptation, rag, evaluation, and safety.