Chapter 3 · Diagnose signal and ship

Activation functions: geometry, gradients, and failure modes

Select activation functions by signal range, gradient flow, architecture, precision, and failure mode; then diagnose saturation and dead units from telemetry.

65–90 min8 key conceptsReviewed 26 Aug 2026

01 · Production proposition

A deep incident classifier stops learning because hidden units die or saturate and the output path applies its probability activation twice.

This chapter moves from separate hidden and output jobs to instrument failure modes. We will make the idea concrete in a small code example before placing it back inside a production system with the model depth, numeric precision, kernel support, and training-throughput envelope.

Decision

Choose hidden and output activations that preserve trainable signal while matching the architecture and target semantics.

Evidence

Activation distributions, saturation/zero fractions, gradient norms, convergence, held-out decision quality, and kernel throughput.

Failure consequence

A deep network uses sigmoid hidden units in low precision, saturates early, while a high learning rate drives every ReLU negative and creates permanently dead channels. An unsafe release must degrade to a named baseline or the last known-good version.

02 · Guided chapter

Build the idea one layer at a time.

An activation decides which signals pass forward and which gradients can return. The right choice depends on hidden-layer trainability and output meaning, which are related but separate concerns.

ReLU is cheap and preserves positive gradients but can leave units permanently inactive after their pre-activations move below zero. GELU and SiLU are smooth gates with small nonzero behavior on part of the negative side and are common in modern deep architectures. Sigmoid and tanh bound outputs but saturate at large magnitudes, shrinking their derivatives.

Output activations follow target semantics: binary and multilabel tasks often expose sigmoid probabilities, multiclass tasks use softmax, and unrestricted regression may use no activation. Stable training losses usually accept raw logits and apply these transforms internally. Diagnose choices with pre-activation percentiles, zero or saturation fractions, gradient norms, convergence, and kernel latency.

Keep in view

ReLU
GELU
SiLU
sigmoid saturation
tanh saturation
dead activations
gradient flow
output activations

WORKED PATH

Follow the reasoning before touching the code.

01
Separate hidden and output jobs
Choose hidden activations for signal flow and architecture support, then choose the output transformation from target shape and loss/API semantics.
02
Inspect local geometry
Plot values and derivatives across the pre-activation range your layers actually produce, including the numeric precision used for training.
03
Instrument failure modes
Track layerwise zero fraction, saturation rate, activation scale, and gradient norms so dead or saturated units become observable events.

pythonactivation_gradients.py

import numpy as np

z = np.array([-6., -1., 0., 1., 6.])
sigmoid = 1 / (1 + np.exp(-z))
relu = np.maximum(0, z)
sigmoid_grad = sigmoid * (1 - sigmoid)
relu_grad = (z > 0).astype(float)
for value, r, dr, s, ds in zip(z, relu, relu_grad, sigmoid, sigmoid_grad):
    print(value, 'relu', r, dr, 'sigmoid', round(s, 3), round(ds, 3))

Compare ReLU and sigmoid values with their local gradients.

READ THE EXAMPLE

What to notice

Sigmoid’s derivative is largest near zero and almost vanishes at ±6, illustrating saturation.
ReLU passes a constant positive-side gradient but blocks the negative side completely; its derivative at zero is convention-dependent and set to zero here.

YOUR TURN Pause and predict before revealing the answer.

If a binary model uses BCEWithLogitsLoss, where should sigmoid be applied?

Need a hint?

Distinguish the stable training objective from the probability returned by an inference API.

Check your reasoning

Feed raw logits directly to BCEWithLogitsLoss; apply sigmoid afterward only when converting logits to inference probabilities.

Bring forward

Lessons 1, 2, 3, 4, 5 in this course.

03 · Formal treatment

Connect the chapter to the course’s quantitative spine.

Choose hidden activations for gradient flow and output activations for semantics—then measure both in the ranges they actually see. The common element-wise nonlinearities below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.

Formal treatment

\operatorname{ReLU}(z)=\max(0,z);\quad \sigma(z)=\frac{1}{1+e^{-z}};\quad \operatorname{SiLU}(z)=z\,\sigma(z)

Common element-wise nonlinearities

Symbol, shape or unit contract
Symbol	Meaning / shape / unit
`z`	pre-activation tensor with shape [batch, features, …]
`σ`	logistic sigmoid
`ReLU`	piecewise-linear positive gate
`SiLU`	smooth self-gated activation

Open derivation and numerical substitution

Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.

Write the named inputs: z, σ, ReLU, SiLU.
Substitute one small, hand-checkable batch before vectorizing.
Calculate an independent reference value and compare within a declared tolerance.

# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c09(inputs)
assert is_finite(value)

04 · Selection field guide

Choose from the task contract, not a favorite default.

Separate hidden activation choice from output semantics.

Hidden nonlinearities preserve trainable signal and expressive power. Output activations are part of the target and loss API contract.

Activation selection and debugging map
Activation	Range / gradient	Why choose it	Failure to watch	Typical role
ReLU	[0, ∞); derivative 0 or 1	Efficient default for many CNNs/MLPs	Dead units from negative pre-activations	Hidden layers with matched initialization
Leaky ReLU / PReLU	Small negative slope, linear positive side	When dead ReLUs are observed	Learned slope instability or unsupported fused kernels	CNN/MLP hidden layers
GELU	Smooth, non-monotonic gate	Established transformer default	Extra compute or approximation differences	Transformer MLPs
SiLU / Swish	z·sigmoid(z), smooth self-gating	Strong modern CNN/MLP baseline	Saturation in far-negative inputs; kernel cost	Modern CNNs and dense networks
tanh	(−1,1), zero-centered, saturating	Bounded recurrent candidate state	Vanishing gradients at large \|z\|	RNN state transforms or bounded latent values
sigmoid	(0,1), probability/gate semantics	Binary or multilabel output; recurrent gates	Saturating hidden gradients; double activation	Probabilities and gates, not universal hidden layers
softmax	Simplex across a declared axis	Exclusive multiclass probabilities or attention weights	Wrong axis, overflow in naive code, use for multilabel	Reported class distribution; often fused into loss
identity / softplus	Unbounded / strictly positive output	Output link chosen from target support	Artificial bounds or exploding positive predictions	Regression output / positive scale or rate

05 · Deterministic lab

Change one assumption and make the tradeoff visible.

Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.

Exact computation

Activation and gradient-flow lab

Move one exact pre-activation through four nonlinearities and compare local gradients.

Pre-activation z ×10

PrimaryReLU 1.000

SecondarySiLU grad 0.928

DiagnosisTrainable local signal

Assumption: Scalar element-wise activations; z is the displayed control divided by 10. ReLU derivative at zero is defined as zero.

Open nonvisual data table

Item	Computed state	Interpretation
relu	output 1.00000	gradient 1.00000
sigmoid	output 0.73106	gradient 0.19661
tanh	output 0.76159	gradient 0.41997
silu	output 0.73106	gradient 0.92767

06 · Production implications

Trace the complete operating path.

Carry the chapter’s central idea into the system: Choose hidden activations for gradient flow and output activations for semantics—then measure both in the ranges they actually see. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.

01
Turn “Separate hidden and output jobs” into a written invariant and a failing test.
02
Promote activation_gradients.py from illustration to a versioned implementation with representative fixtures.
03
Use “Instrument failure modes” as the release checkpoint: Track layerwise zero fraction, saturation rate, activation scale, and gradient norms so dead or saturated units become observable events.
04
Serve or materialize behind explicit the model depth, numeric precision, kernel support, and training-throughput envelope.
05
Join telemetry to mature outcomes and retain a rollback path.

Observability

Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.

Cost

Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.

Failure modes

A deep network uses sigmoid hidden units in low precision, saturates early, while a high learning rate drives every ReLU negative and creates permanently dead channels. Add a detector, owner, mitigation, and stop condition for this class of failure.

Alternatives

Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.

07 · Check understanding

Explain the contract, not just the vocabulary.

08 · Apply in production

Activation pathology report

Run a controlled activation comparison and diagnose one dead, saturated, or semantically incorrect network.

ReLU, LeakyReLU, GELU, SiLU, and tanh comparison
Activation percentiles, zero/saturation rates, and gradient norms
Binary, multiclass, multilabel, and regression output-contract tests
Matched initialization/normalization and latency evidence
Recommended bundle with rollback trigger

Production self-review0/100

Staged hints

Failure diagnosis

If offline numbers look impossible, audit prediction-time availability before model code.
If quality is sound but the contract fails, measure the exact serialized and served path.
If one slice regresses, preserve that slice as a permanent release gate.

Record an honest attempt to unlock the reference solution.

09 · Sources & next depth

Read primary material with a purpose.

REQUIRED · Official docs

PyTorch nonlinear activations

25 min · last checked 26 Aug 2026

RECOMMENDED · Paper

Gaussian Error Linear Units

25 min · last checked 26 Aug 2026

RECOMMENDED · Paper

Searching for Activation Functions

30 min · last checked 26 Aug 2026

DEEP DIVE · INDEX

Find a primary implementation or benchmark

30–90 min · verify task, dataset, and evaluation match

10 · Production resolution

Return to the opening failure.

Default hidden activations are justified by architecture evidence, output activations match the target/loss contract, and training dashboards track activation mean, variance, zero fraction, saturation, and gradient norms by layer.

For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a small network built from forward pass through backpropagation and optimization.

Next lessonLoss functions by task, data, and deployment constraints