Activation functions: geometry, gradients, and failure modes
Select activation functions by signal range, gradient flow, architecture, precision, and failure mode; then diagnose saturation and dead units from telemetry.
A deep incident classifier stops learning because hidden units die or saturate and the output path applies its probability activation twice.
This lesson isolates activation functions: geometry, gradients, and failure modes as one decision inside that system. The people affected are product users and operators; the learning data must carry event time, availability time, ownership, and version; and the operating envelope is the model depth, numeric precision, kernel support, and training-throughput envelope.
Choose hidden and output activations that preserve trainable signal while matching the architecture and target semantics.
Activation distributions, saturation/zero fractions, gradient norms, convergence, held-out decision quality, and kernel throughput.
A deep incident classifier stops learning because hidden units die or saturate and the output path applies its probability activation twice. An unsafe release must degrade to a named baseline or the last known-good version.
Build the mental model before the machinery.
The core move is to treat activation functions: geometry, gradients, and failure modes as a contract between data, a computation, and an action. Default hidden activations are justified by architecture evidence, output activations match the target/loss contract, and training dashboards track activation mean, variance, zero fraction, saturation, and gradient norms by layer. The implementation becomes easier to debug once you can state which inputs exist, which state is learned, what output means, and what must remain invariant after serialization.
ReLU
Define it in a hand-checkable form and name the prediction-time inputs.
GELU
Connect it to the production metric and identify what it cannot guarantee.
SiLU
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
sigmoid saturation
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
tanh saturation
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
dead activations
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
gradient flow
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
output activations
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
Lessons 1, 2, 3, 4, 5 in this course.
Name every symbol. Check every shape.
Common element-wise nonlinearities is the central invariant for this lesson. The formula is useful only when its inputs match the production cutoff and its output maps to an action.
Common element-wise nonlinearities
| Symbol | Meaning / shape / unit |
|---|---|
z | pre-activation tensor with shape [batch, features, …] |
σ | logistic sigmoid |
ReLU | piecewise-linear positive gate |
SiLU | smooth self-gated activation |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: z, σ, ReLU, SiLU.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c09(inputs)
assert is_finite(value)Choose from the task contract, not a favorite default.
Hidden nonlinearities preserve trainable signal and expressive power. Output activations are part of the target and loss API contract.
| Activation | Range / gradient | Why choose it | Failure to watch | Typical role |
|---|---|---|---|---|
| ReLU | [0, ∞); derivative 0 or 1 | Efficient default for many CNNs/MLPs | Dead units from negative pre-activations | Hidden layers with matched initialization |
| Leaky ReLU / PReLU | Small negative slope, linear positive side | When dead ReLUs are observed | Learned slope instability or unsupported fused kernels | CNN/MLP hidden layers |
| GELU | Smooth, non-monotonic gate | Established transformer default | Extra compute or approximation differences | Transformer MLPs |
| SiLU / Swish | z·sigmoid(z), smooth self-gating | Strong modern CNN/MLP baseline | Saturation in far-negative inputs; kernel cost | Modern CNNs and dense networks |
| tanh | (−1,1), zero-centered, saturating | Bounded recurrent candidate state | Vanishing gradients at large |z| | RNN state transforms or bounded latent values |
| sigmoid | (0,1), probability/gate semantics | Binary or multilabel output; recurrent gates | Saturating hidden gradients; double activation | Probabilities and gates, not universal hidden layers |
| softmax | Simplex across a declared axis | Exclusive multiclass probabilities or attention weights | Wrong axis, overflow in naive code, use for multilabel | Reported class distribution; often fused into loss |
| identity / softplus | Unbounded / strictly positive output | Output link chosen from target support | Artificial bounds or exploding positive predictions | Regression output / positive scale or rate |
Calculate it small. Shape it realistically. Break it on purpose.
A result you can reproduce on paper
Evaluate ReLU, sigmoid, tanh, and SiLU at z ∈ {−4, −1, 0, 1, 4}; compute local gradients and mark saturated or zero-gradient regions.
- Write every input and unit.
- Substitute values into the common element-wise nonlinearities equation above.
- Compare the result to one simple baseline and explain the direction of the difference.
The same reasoning under real constraints
Choose hidden activations for an MLP and transformer, output activations for binary, multiclass, multilabel, bounded regression, and mixture parameters, then instrument activation and gradient distributions.
The production record includes the data snapshot, transformation state, artifact identity, cutoff, score, decision, and the version of the policy that consumed it.
The attractive result you should reject
A deep network uses sigmoid hidden units in low precision, saturates early, while a high learning rate drives every ReLU negative and creates permanently dead channels.
Diagnostic: replay the smallest failing slice from immutable inputs, then compare each boundary rather than retuning the model.
Change one assumption and make the tradeoff visible.
This lab runs predefined TypeScript only. It never executes learner code. Use the slider, numeric input, reset, live text, or table—the computation is the same.
Activation and gradient-flow lab
Move one exact pre-activation through four nonlinearities and compare local gradients.
Assumption: Scalar element-wise activations; z is the displayed control divided by 10. ReLU derivative at zero is defined as zero.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| relu | output 1.00000 | gradient 1.00000 |
| sigmoid | output 0.73106 | gradient 0.19661 |
| tanh | output 0.76159 | gradient 0.41997 |
| silu | output 0.73106 | gradient 0.92767 |
Trace the complete operating path.
- 01
Validate and version ReLU.
- 02
Compute activation functions: geometry, gradients, and failure modes from prediction-time-safe inputs.
- 03
Persist model, feature, and configuration identities together.
- 04
Serve or materialize behind explicit the model depth, numeric precision, kernel support, and training-throughput envelope.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
A deep network uses sigmoid hidden units in low precision, saturates early, while a high learning rate drives every ReLU negative and creates permanently dead channels. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Activation pathology report
Run a controlled activation comparison and diagnose one dead, saturated, or semantically incorrect network.
- ReLU, LeakyReLU, GELU, SiLU, and tanh comparison
- Activation percentiles, zero/saturation rates, and gradient norms
- Binary, multiclass, multilabel, and regression output-contract tests
- Matched initialization/normalization and latency evidence
- Recommended bundle with rollback trigger
Staged hints
Failure diagnosis
- If offline numbers look impossible, audit prediction-time availability before model code.
- If quality is sound but the contract fails, measure the exact serialized and served path.
- If one slice regresses, preserve that slice as a permanent release gate.
Record an honest attempt to unlock the reference solution.
Read primary material with a purpose.
PyTorch nonlinear activations
25 min · last checked 26 Aug 2026
↗RECOMMENDED · PaperGaussian Error Linear Units
25 min · last checked 26 Aug 2026
↗RECOMMENDED · PaperSearching for Activation Functions
30 min · last checked 26 Aug 2026
↗DEEP DIVE · INDEXFind a primary implementation or benchmark
30–90 min · verify task, dataset, and evaluation match
↗Return to the opening failure.
Default hidden activations are justified by architecture evidence, output activations match the target/loss contract, and training dashboards track activation mean, variance, zero fraction, saturation, and gradient norms by layer.
For this lesson, the release evidence is a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a small network built from forward pass through backpropagation and optimization.