Density clusters and novel anomalies
Density methods flag what is locally disconnected; people and outcomes determine what is truly novel and important.
Novel equipment failures appear before reliable labels exist.
This chapter moves from define neighborhood geometry to turn noise into a queue. We will make the idea concrete in a small code example before placing it back inside a production system with a declared latency, cost, and operator-capacity budget.
Rank unconnected points by severity or distance to known cases, sample them for review, and store reviewer outcomes as future validation labels.
Verify the behavior illustrated in density_triage.py, then measure decision utility, slice reliability, and system latency.
A stale ANN index never retrieves new inventory; a reranker cannot recover what retrieval omitted. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
Some incident families form winding shapes, while rare failures sit alone. Density-based methods describe both with one idea: a point belongs to a cluster when enough neighbors connect it to a dense region.
DBSCAN labels a point as core when at least min_samples points fall within radius eps. Core points that can reach one another form a cluster; border points attach to a nearby core; points that satisfy neither rule receive the noise label -1. Unlike k-means, the method does not require k and can follow non-convex shapes.
Noise means “not density-connected under this scale,” not “corrupt row.” The eps radius is expressed in feature-space units, so scaling and local density matter enormously. Novelty triage should combine the algorithmic flag with business severity, review capacity, and feedback from investigators.
- DBSCAN
- anomaly triage
- cosine similarity
- HNSW
Follow the reasoning before touching the code.
- 01
Define neighborhood geometry
Choose features and scaling so that distance represents similar incident behavior rather than arbitrary measurement units.
- 02
Stress the density settings
Sweep eps and min_samples, inspect cluster persistence, and verify that small parameter changes do not turn most of the corpus from clustered to noise.
- 03
Turn noise into a queue
Rank unconnected points by severity or distance to known cases, sample them for review, and store reviewer outcomes as future validation labels.
import numpy as np
from sklearn.cluster import DBSCAN
from sklearn.preprocessing import StandardScaler
X = np.array([[0, 0], [0.1, 0], [0, 0.1],
[3, 3], [3.1, 3], [3, 3.1], [9, 1]])
Z = StandardScaler().fit_transform(X)
labels = DBSCAN(eps=0.25, min_samples=2).fit_predict(Z)
for point, label in zip(X, labels):
print(point.tolist(), 'noise' if label == -1 else label)Separate a dense incident family from an isolated candidate.
What to notice
- The isolated point receives -1, but the code deliberately calls it noise rather than declaring it an error or threat.
- DBSCAN can find both compact groups without prescribing their count, provided the same eps scale suits their densities.
YOUR TURN Pause and predict before revealing the answer.
Increase eps until the isolated point joins a cluster. Why is the resulting label not evidence that the point stopped being novel?
Need a hint?
The observation did not change; only the neighborhood definition did.
Check your reasoning
Cluster membership is conditional on eps and the representation, so the change demonstrates parameter sensitivity rather than a change in the underlying incident.
Lessons 1, 2 in this course.
Connect the chapter to the course’s quantitative spine.
Density methods flag what is locally disconnected; people and outcomes determine what is truly novel and important. The cosine similarity below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Cosine similarity
| Symbol | Meaning / shape / unit |
|---|---|
u, v | embedding vectors |
uᵀv | dot product |
∥·∥₂ | vector magnitude |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: u, v, uᵀv, ∥·∥₂.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c06(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
ANN recall–latency lab
Increase search effort and expose the recall/latency frontier.
Assumption: HNSW-like curve for a fixed one-million-vector index.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| query 1 | 82.7% | 4.7 ms |
| query 2 | 82.0% | 4.9 ms |
| query 3 | 81.2% | 5.1 ms |
| query 4 | 80.4% | 5.3 ms |
| query 5 | 79.5% | 5.5 ms |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Density methods flag what is locally disconnected; people and outcomes determine what is truly novel and important. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Define neighborhood geometry” into a written invariant and a failing test.
- 02
Promote density_triage.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Turn noise into a queue” as the release checkpoint: Rank unconnected points by severity or distance to known cases, sample them for review, and store reviewer outcomes as future validation labels.
- 04
Serve or materialize behind explicit a declared latency, cost, and operator-capacity budget.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
A stale ANN index never retrieves new inventory; a reranker cannot recover what retrieval omitted. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Incident discovery and retrieval system
Combine PCA, clustering, anomaly triage, embeddings, and ANN search with human review.
- Representation analysis
- Cluster/anomaly validation
- Embedding evaluation
- ANN recall/latency audit
Read primary material with a purpose.
Return to the opening failure.
Version encoder and index together; monitor cluster stability, alert yield, exact-sample recall, latency, and freshness.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a pca, clustering, anomaly, and nearest-neighbor retrieval investigation.