MLE/ROADMAP
/Search
Menu
Learn/Course 09
Deep learning · weeks 11–12

Neural networks from scratch

A nonlinear sensor classifier defeats every classical baseline.

concept:backpropagationlifecycle:trainingtool:pytorch
Opening production brief

A measurable problem, before a model.

Users

Operators, product owners, and people affected by the neural networks decision.

Data

affine layers, ReLU/GELU, cross-entropy, chain rule with version, owner, event time, and availability contracts.

Constraints

A nonlinear sensor classifier defeats every classical baseline. Every solution must state latency, cost, capacity, and fallback limits.

Success

Offline evidence plus a deployable resolution: A small network built from forward pass through backpropagation and optimization.

CHAPTER 1

Build the computation

Assemble the nonlinear computation and output contract.

  1. 01

    Beyond linear boundaries

    Use beyond linear boundaries to move the neural networks production brief toward a defensible release.

    65–90 min · checkpoint · deterministic lab
  2. 02

    Forward pass, activations, losses, and initialization

    Use forward pass, activations, losses, and initialization to move the neural networks production brief toward a defensible release.

    65–90 min · checkpoint · deterministic lab
CHAPTER 2

Differentiate and optimize

Derive gradients and optimize a hand-checkable network.

  1. 03

    Backpropagation without magic

    Use backpropagation without magic to move the neural networks production brief toward a defensible release.

    65–90 min · checkpoint · deterministic lab
  2. 04

    SGD, momentum, AdamW, and schedules

    Use sgd, momentum, adamw, and schedules to move the neural networks production brief toward a defensible release.

    65–90 min · checkpoint · deterministic lab
CHAPTER 3

Diagnose signal and ship

Choose trainable signal paths, diagnose failures, and ship the smallest reliable model.

  1. 05

    Regularize, normalize, debug, and ship

    Use regularize, normalize, debug, and ship to move the neural networks production brief toward a defensible release.

    65–90 min · checkpoint · deterministic lab
  2. 06

    Activation functions: geometry, gradients, and failure modes

    Select activation functions by signal range, gradient flow, architecture, precision, and failure mode; then diagnose saturation and dead units from telemetry.

    65–90 min · checkpoint · deterministic lab
  3. 07

    Loss functions by task, data, and deployment constraints

    Choose and debug training objectives as optimization surrogates for the real task, including imbalance, outliers, uncertainty, ranking, dense prediction, and generation.

    65–90 min · checkpoint · deterministic lab
  4. HW

    Network from first principles

    Specify a NumPy MLP, finite-difference checks, PyTorch parity, calibrated inference, and debugging report.

    Rubric · staged hints · reference resolution
Prerequisite guidance

2 concepts make this faster.

Prerequisites are guidance, never hard gates.

C04Classical baselinesA justified baseline across regression, Naive Bayes, kNN, and SVM.C05Trees & boostingCalibrated random-forest and XGBoost baselines with cost-sensitive thresholds.
Ready to work the problem?

Beyond linear boundaries

Start lesson 01