MLE/ROADMAP
/Search
Menu
Learn/Course 09
Deep learning · weeks 11–12

Neural networks from scratch

A nonlinear sensor classifier defeats every classical baseline.

concept:backpropagationlifecycle:trainingtool:pytorch
Opening production brief

A measurable problem, before a model.

Users

Operators, product owners, and people affected by the neural networks decision.

Data

affine layers, ReLU/GELU, cross-entropy, chain rule with version, owner, event time, and availability contracts.

Constraints

A nonlinear sensor classifier defeats every classical baseline. Every solution must state latency, cost, capacity, and fallback limits.

Success

Offline evidence plus a deployable resolution: A small network built from forward pass through backpropagation and optimization.

CHAPTER 1

Build the computation

Assemble the nonlinear computation and output contract.

  1. 01

    Beyond linear boundaries

    Hidden nonlinearities earn their place by making the final decision simpler in a learned representation.

    65–90 min · checkpoint · deterministic lab
  2. 02

    Forward pass, activations, losses, and initialization

    A trustworthy forward pass makes shapes, activation semantics, initialization, and loss expectations explicit.

    65–90 min · checkpoint · deterministic lab
CHAPTER 2

Differentiate and optimize

Derive gradients and optimize a hand-checkable network.

  1. 03

    Backpropagation without magic

    Backprop becomes ordinary once every edge in the forward graph has a local derivative you can inspect.

    65–90 min · checkpoint · deterministic lab
  2. 04

    SGD, momentum, AdamW, and schedules

    The trained state is parameters plus optimizer history, schedule position, data order, and randomness.

    65–90 min · checkpoint · deterministic lab
CHAPTER 3

Diagnose signal and ship

Choose trainable signal paths, diagnose failures, and ship the smallest reliable model.

  1. 05

    Regularize, normalize, debug, and ship

    Shipping a network means freezing its inference behavior, not merely saving its weights.

    65–90 min · checkpoint · deterministic lab
  2. 06

    Activation functions: geometry, gradients, and failure modes

    Select activation functions by signal range, gradient flow, architecture, precision, and failure mode; then diagnose saturation and dead units from telemetry.

    65–90 min · checkpoint · deterministic lab
  3. 07

    Loss functions by task, data, and deployment constraints

    Choose and debug training objectives as optimization surrogates for the real task, including imbalance, outliers, uncertainty, ranking, dense prediction, and generation.

    65–90 min · checkpoint · deterministic lab
  4. HW

    Network from first principles

    Specify a NumPy MLP, finite-difference checks, PyTorch parity, calibrated inference, and debugging report.

    Rubric · staged hints · reference resolution
Prerequisite guidance

2 concepts make this faster.

Prerequisites are guidance, never hard gates.

C04Classical baselinesA justified baseline across regression, Naive Bayes, kNN, and SVM.C05Trees & boostingCalibrated random-forest and XGBoost baselines with cost-sensitive thresholds.
Ready to work the problem?

Beyond linear boundaries

Start lesson 01