Prompt, retrieve, adapt, train, or buy?
Match the intervention to the failure: prompts shape instructions, retrieval updates knowledge, tuning shapes behavior, and training or buying changes base capability.
A private support assistant must answer from evidence and resist manipulation.
This chapter moves from classify the gap to price the whole lifecycle. We will make the idea concrete in a small code example before placing it back inside a production system with a declared latency, cost, and operator-capacity budget.
Include data curation, evaluation, serving, updates, observability, privacy, lock-in, and rollback—not just one training run or API token price.
Verify the behavior illustrated in intervention_ladder.py, then measure decision utility, slice reliability, and system latency.
Evaluation contamination and feedback only from successful chats hide refusals, unsupported languages, and hallucinated citations. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
The most expensive model intervention is not automatically the most capable product. Begin with the smallest change that addresses the actual source of error.
Prompting changes instructions and examples at request time. Retrieval supplies current, attributable context without altering weights. Parameter-efficient or full fine-tuning changes behavior encoded in weights, while pretraining builds broad capability at the highest data and compute cost. Buying an API trades control for speed and managed infrastructure.
Choose by failure mode and constraint. Mutable facts, citations, and access control point toward retrieval; consistent style or task format may justify supervised adaptation; missing domain language or capability may require deeper training. Compare candidates under the same eval set, latency, privacy, update, and ownership contract, including a strong prompt-only baseline.
- build vs buy
- RAG vs tuning
- provenance
- decontamination
Follow the reasoning before touching the code.
- 01
Classify the gap
Separate missing knowledge, poor retrieval, instruction-following behavior, base capability, latency, and policy failures. One user complaint can contain several different gaps.
- 02
Climb an intervention ladder
Test prompt, retrieval, PEFT, full tuning, pretraining, and external providers only as far as evidence requires. Preserve the previous rung as a baseline.
- 03
Price the whole lifecycle
Include data curation, evaluation, serving, updates, observability, privacy, lock-in, and rollback—not just one training run or API token price.
def first_experiment(requirement):
if requirement == 'clearer instructions':
return 'prompt baseline'
if requirement in {'fresh facts', 'citations', 'access control'}:
return 'retrieval'
if requirement in {'stable format', 'domain behavior'}:
return 'parameter-efficient tuning'
if requirement == 'new base capability':
return 'train or buy comparison'
return 'error analysis before intervention'
for need in ['fresh facts', 'stable format', 'new base capability']:
print(need, '->', first_experiment(need))Map dominant requirements to a first experiment, not an irreversible verdict.
What to notice
- Fresh facts route to retrieval because knowledge can change independently of the model weights.
- A new base capability does not immediately mean pretraining; it triggers a train-versus-provider comparison under the full contract.
YOUR TURN Pause and predict before revealing the answer.
A support bot knows the refund policy but answers in an inconsistent JSON shape. Should you add more documents to retrieval?
Need a hint?
The failure is behavior and output format, not missing knowledge.
Check your reasoning
Start with a stricter prompt and schema validation; if the behavior remains inconsistent at scale, compare a small supervised adapter. More retrieved documents add context but do not directly teach stable formatting.
Transformer from scratch
Connect the chapter to the course’s quantitative spine.
Match the intervention to the failure: prompts shape instructions, retrieval updates knowledge, tuning shapes behavior, and training or buying changes base capability. The autoregressive next-token loss below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Autoregressive next-token loss
| Symbol | Meaning / shape / unit |
|---|---|
T | sequence length |
x_t | target token |
θ | model parameters |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: T, x_t, θ.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c15(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Retrieval depth lab
Change top-k and estimate evidence recall, context cost, and distractor risk.
Assumption: Fixed retrieval curve; each passage contributes 180 tokens.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| passage 1 | 88.0% | 180 tokens |
| passage 2 | 79.0% | 192 tokens |
| passage 3 | 70.0% | 204 tokens |
| passage 4 | 61.0% | 216 tokens |
| passage 5 | 52.0% | 228 tokens |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Match the intervention to the failure: prompts shape instructions, retrieval updates knowledge, tuning shapes behavior, and training or buying changes base capability. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Classify the gap” into a written invariant and a failing test.
- 02
Promote intervention_ladder.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Price the whole lifecycle” as the release checkpoint: Include data curation, evaluation, serving, updates, observability, privacy, lock-in, and rollback—not just one training run or API token price.
- 04
Serve or materialize behind explicit a declared latency, cost, and operator-capacity budget.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Evaluation contamination and feedback only from successful chats hide refusals, unsupported languages, and hallucinated citations. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Domain adaptation release package
Compare prompt, retrieval, and optional PEFT approaches under one provenance and evaluation contract.
- Approach decision
- Provenance ledger
- Slice evaluation
- Safety probes/model card
Read primary material with a purpose.
Return to the opening failure.
Release model, tokenizer, retrieval index, prompt, policy, datasets, eval suite, and runtime config as one bundle.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a grounded system design spanning adaptation, rag, evaluation, and safety.