Text to tensors: tokenization, embeddings, position
Tokenizer and position scheme are not preprocessing trivia; together they define what symbols and order the transformer can represent.
Domain autocomplete needs a model whose every component the team can explain.
This chapter moves from freeze the text contract to assemble the first tensor. We will make the idea concrete in a small code example before placing it back inside a production system with one-device educational build.
Look up token vectors, construct positions for valid slots, combine compatible representations, and verify the final `[batch, time, width]` shape.
Verify the behavior illustrated in text_to_tensor.py, then measure decision utility, slice reliability, and system latency.
Without the causal mask, each position sees its target and validation loss becomes impossibly good. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
A transformer never sees a word. It sees integer token IDs, looks up vectors, and adds a signal that says where each vector belongs.
Tokenization defines the model’s alphabet and how text is segmented into pieces. The vocabulary, normalization rules, special-token IDs, and byte fallback behavior are learned-system artifacts: changing them changes every downstream tensor. Token fertility—the number of tokens needed for the same text—also affects context capacity, latency, and language coverage.
An embedding table maps each token ID to a learned vector, but self-attention alone is permutation-equivariant: without position information, reordering tokens simply reorders outputs. Learned positional embeddings or schemes such as RoPE inject order. Token and position representations must share the model width so they can be combined before the first block.
- tokenization
- embeddings
- RoPE
- QKV projections
Follow the reasoning before touching the code.
- 01
Freeze the text contract
Record Unicode normalization, pre-tokenization, vocabulary, special tokens, and maximum length. Test encode/decode behavior on whitespace, rare scripts, and malformed input.
- 02
Inspect the integer sequence
Print tokens and IDs for a short sentence. Mark beginning, end, padding, and unknown behavior so later masks can be interpreted.
- 03
Assemble the first tensor
Look up token vectors, construct positions for valid slots, combine compatible representations, and verify the final `[batch, time, width]` shape.
import torch
vocab = {'<unk>': 0, 'models': 1, 'need': 2, 'order': 3}
tokens = 'models need order'.split()
ids = torch.tensor([[vocab.get(token, 0) for token in tokens]])
width = 4
token_embedding = torch.nn.Embedding(len(vocab), width)
position_embedding = torch.nn.Embedding(16, width)
positions = torch.arange(ids.shape[1]).unsqueeze(0)
hidden = token_embedding(ids) + position_embedding(positions)
assert hidden.shape == (1, 3, width)
print(ids.tolist(), hidden.shape)Turn a tiny token sequence into token-plus-position vectors.
What to notice
- The three IDs index vocabulary rows, while positions 0, 1, and 2 index a separate positional table.
- Both lookups return width-four vectors, allowing element-wise addition into one `[1, 3, 4]` tensor.
YOUR TURN Pause and predict before revealing the answer.
Swap “need” and “order” but remove the positional embedding. What information remains about their order?
Need a hint?
The token vectors identify pieces, but no value names an absolute or relative location.
Check your reasoning
Only the placement of vectors in the tensor changes; the attention operation has no intrinsic order signal and treats the swap equivariantly. Position information is what lets the model learn order-sensitive relations.
Neural networks, Sequence models, GPU training
Connect the chapter to the course’s quantitative spine.
Tokenizer and position scheme are not preprocessing trivia; together they define what symbols and order the transformer can represent. The scaled dot-product attention below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Scaled dot-product attention
| Symbol | Meaning / shape / unit |
|---|---|
Q, K, V | query, key, value matrices |
d_k | head width |
M | causal or padding mask |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: Q, K, V, d_k, M.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c14(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Attention temperature lab
Change score temperature and inspect the exact softmax distribution.
Assumption: Fixed scores [1.2, 0.3, −0.4, 1.8].
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| token 1 | 1.2 | 0.2915 |
| token 2 | 0.3 | 0.1185 |
| token 3 | -0.4 | 0.0589 |
| token 4 | 1.8 | 0.5311 |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Tokenizer and position scheme are not preprocessing trivia; together they define what symbols and order the transformer can represent. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Freeze the text contract” into a written invariant and a failing test.
- 02
Promote text_to_tensor.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Assemble the first tensor” as the release checkpoint: Look up token vectors, construct positions for valid slots, combine compatible representations, and verify the final `[batch, time, width]` shape.
- 04
Serve or materialize behind explicit one-device educational build.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Without the causal mask, each position sees its target and validation loss becomes impossibly good. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Decoder-only transformer from scratch
Specify tokenizer, attention blocks, causal training, unit tests, sampling, and KV-cache parity without a transformer module.
- Shape/mask tests
- Tiny-batch overfit
- Held-out evaluation
- Cached/uncached parity
Read primary material with a purpose.
Return to the opening failure.
Version tokenizer, vocabulary, special tokens, architecture, weights, prompt template, and generation policy together.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce tokenizer, attention, rope, residual stack, training loop, generation, and kv cache.