concept:tokenization
Tokenization
Tokenization in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Build a transformer from scratch
Text to tensors: tokenization, embeddings, position
Tokenizer and position scheme are not preprocessing trivia; together they define what symbols and order the transformer can represent.
Build a transformer from scratchScaled dot-product self-attention by hand
Attention is a masked, normalized lookup: queries score keys, softmax allocates influence, and values carry the information.
Courses & assignments
Dependency-authoritative learning units.
Casebook
Reported facts and course reconstructions.