PCA and representation geometry
PCA preserves dominant geometric variation, not automatically the information your decision needs.
Novel equipment failures appear before reliable labels exist.
This chapter moves from make units comparable to test what compression preserved. We will make the idea concrete in a small code example before placing it back inside a production system with a declared latency, cost, and operator-capacity budget.
Transform untouched validation rows and compare downstream quality, slice behavior, and latency across several component counts rather than selecting by variance alone.
Verify the behavior illustrated in pca_geometry.py, then measure decision utility, slice reliability, and system latency.
A stale ANN index never retrieves new inventory; a reranker cannot recover what retrieval omitted. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
A table with fifty columns does not necessarily describe fifty independent directions. PCA gives us a coordinate system aligned with the variation the data actually contains, which makes representation geometry visible before we ask a downstream model to use it.
PCA first centers each feature, then finds an axis along which the projected points have maximum variance. Every later axis must be perpendicular to the earlier ones and captures the largest variance still available. The resulting components are weighted combinations of the original features, not selected columns.
Variance is only a geometric objective. A low-variance direction can carry the rare signal needed to detect a failure, so explained variance is a compression diagnostic rather than proof of task quality. Fit scaling and PCA on training data only, preserve that fitted state, and evaluate the transformed representation on the real downstream decision.
- PCA
- explained variance
- k-means
- DBSCAN
Follow the reasoning before touching the code.
- 01
Make units comparable
Fit a standardizer on the training rows so that a feature measured in milliseconds does not dominate one measured in ratios merely because of its unit.
- 02
Rotate into component space
Fit PCA on the standardized training matrix and inspect both component loadings and cumulative explained variance; loadings tell you what each new axis combines.
- 03
Test what compression preserved
Transform untouched validation rows and compare downstream quality, slice behavior, and latency across several component counts rather than selecting by variance alone.
import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
train = np.array([[1, 10], [2, 21], [3, 29], [4, 41]])
test = np.array([[2.5, 25]])
scale = StandardScaler().fit(train)
pca = PCA(n_components=2).fit(scale.transform(train))
print(pca.explained_variance_ratio_)
print(pca.transform(scale.transform(test)))Fit preprocessing on training data, then inspect a two-dimensional projection.
What to notice
- The first component captures nearly all variation because the two original measurements move together.
- The test row is transformed with training means, scales, and axes; refitting on it would change the coordinate system and leak evaluation data.
YOUR TURN Pause and predict before revealing the answer.
Multiply the second feature by 1,000. What happens to the PCA result with and without StandardScaler?
Need a hint?
PCA compares variance numerically, so ask whether a change of unit should change the geometry you intend to model.
Check your reasoning
With standardization, the projection is essentially unchanged; without it, the rescaled second feature dominates the first component.
Data & features, Classical baselines
Connect the chapter to the course’s quantitative spine.
PCA preserves dominant geometric variation, not automatically the information your decision needs. The cosine similarity below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Cosine similarity
| Symbol | Meaning / shape / unit |
|---|---|
u, v | embedding vectors |
uᵀv | dot product |
∥·∥₂ | vector magnitude |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: u, v, uᵀv, ∥·∥₂.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c06(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
ANN recall–latency lab
Increase search effort and expose the recall/latency frontier.
Assumption: HNSW-like curve for a fixed one-million-vector index.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| query 1 | 82.7% | 4.7 ms |
| query 2 | 82.0% | 4.9 ms |
| query 3 | 81.2% | 5.1 ms |
| query 4 | 80.4% | 5.3 ms |
| query 5 | 79.5% | 5.5 ms |
Trace the complete operating path.
Carry the chapter’s central idea into the system: PCA preserves dominant geometric variation, not automatically the information your decision needs. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Make units comparable” into a written invariant and a failing test.
- 02
Promote pca_geometry.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Test what compression preserved” as the release checkpoint: Transform untouched validation rows and compare downstream quality, slice behavior, and latency across several component counts rather than selecting by variance alone.
- 04
Serve or materialize behind explicit a declared latency, cost, and operator-capacity budget.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
A stale ANN index never retrieves new inventory; a reranker cannot recover what retrieval omitted. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Incident discovery and retrieval system
Combine PCA, clustering, anomaly triage, embeddings, and ANN search with human review.
- Representation analysis
- Cluster/anomaly validation
- Embedding evaluation
- ANN recall/latency audit
Read primary material with a purpose.
Return to the opening failure.
Version encoder and index together; monitor cluster stability, alert yield, exact-sample recall, latency, and freshness.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a pca, clustering, anomaly, and nearest-neighbor retrieval investigation.