What we can attribute directly
OpenAI published experience scaling Kubernetes clusters to 7,500 nodes.
The account discusses API load, networking, monitoring, and operational practices.
Read OpenAI — Kubernetes at scale Primary source · last checked 26 Aug 2026The operating envelope
Large node count, bursty jobs, control-plane limits, network fan-out, observability volume, and failure recovery.
Model teams, platform owners, operators, downstream product systems, and people affected by decisions.
Versioned data, configs, traces, artifacts, deployments, and outcomes aligned on one timeline.
Schedulers and control-plane services become a bottleneck before accelerator compute is saturated.
Trace the system before naming the bug.
- 01
Producers emit versioned data or model artifacts.
- 02
A platform validates, computes, stores, schedules, or routes them.
- 03
Training or inference consumes the exact declared version.
- 04
Telemetry joins the decision to system, data, and model identity.
- 05
Operators compare outcomes, stop conditions, and the last known-good path.
Follow the evidence boundary by boundary.
Symptoms
Schedulers and control-plane services become a bottleneck before accelerator compute is saturated.
Investigation
Measure API request classes, list/watch behavior, scheduler latency, DNS/network pressure, and telemetry cardinality.
A thousand workers start at once. Draw the control-plane request fan-out and identify two ways to smooth it.
Open investigation scaffold
- Write the earliest known-bad timestamp.
- Compare exact identities on either side of that boundary.
- Find the smallest affected slice and a known-good counterexample.
- Separate mitigation from root-cause confirmation.
Repair the contract, not only the symptom.
Components designed for smaller steady workloads face burst and fan-out amplification at cluster scale.
Cache and scope reads, reduce churn, shard or isolate workloads, and design load tests for control-plane behavior.
Rollout
Raise scale in tested steps with explicit capacity headroom and failure drills.
Reason about the tempting shortcuts.
- Treating the orchestration layer as free overhead.
- Unlimited high-cardinality cluster telemetry.
Make recurrence visible early.
Scheduling and API latency
Define owner, slice, normal range, alert persistence, and the exact mitigation the alert should trigger.
Watch/list volume
Define owner, slice, normal range, alert persistence, and the exact mitigation the alert should trigger.
Node and network error budgets
Define owner, slice, normal range, alert persistence, and the exact mitigation the alert should trigger.
At scale, metadata and coordination become a workload of their own.
Carry this pattern into assignments as a design constraint and into incident reviews as a hypothesis—not as proof about an unpublished system.