NLP / LLM
A focused route through sequence models, transformers, adaptation, evaluation, and language-model serving.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Sequences are not IID
In sequence learning, chronology is part of the feature schema: define the cutoff first, then construct inputs, labels, and splits around it.
Sequence ML: RNNs, GRUs & LSTMsRecurrent state and BPTT
Recurrent state carries information forward; BPTT carries responsibility backward, and truncation decides how far that responsibility can travel.
Sequence ML: RNNs, GRUs & LSTMsGates that remember: LSTM and GRU
Gates do not create memory by magic; they create learnable paths for preserving, replacing, and exposing state.
Sequence ML: RNNs, GRUs & LSTMsMany-to-one and many-to-many training
Choose the output pattern from the target, then make padding mathematically invisible to both state selection and loss.
Sequence ML: RNNs, GRUs & LSTMsStateful streaming inference without future leakage
For streaming inference, the state transition, ordering policy, deduplication rule, and model version are all part of the model.
Build a transformer from scratchText to tensors: tokenization, embeddings, position
Tokenizer and position scheme are not preprocessing trivia; together they define what symbols and order the transformer can represent.
Build a transformer from scratchScaled dot-product self-attention by hand
Attention is a masked, normalized lookup: queries score keys, softmax allocates influence, and values carry the information.
Build a transformer from scratchHeads, norms, residuals, and MLPs
Attention mixes tokens, the MLP mixes channels, and normalized residual paths let both transformations accumulate without losing the original stream.
Build a transformer from scratchTrain a causal language model
Causal LM training is next-token alignment plus a no-looking-ahead proof; everything else scales that contract.
Build a transformer from scratchSampling, KV cache, and efficient generation
Efficient generation couples a declared sampling policy with a position-correct KV cache whose logits match the uncached model.
Courses & assignments
Dependency-authoritative learning units.
Sequence ML: RNNs, GRUs & LSTMs
A masked sequence model with BPTT, recurrent gates, and attention precursors.
Assignment: Causal stateful risk modelCourse 14Build a transformer from scratch
Tokenizer, attention, RoPE, residual stack, training loop, generation, and KV cache.
Assignment: Decoder-only transformer from scratchCasebook
Reported facts and course reconstructions.