Production Kubernetes clusters are failing to scale in three distinct ways that standard node-count tuning can't fix, according to a technical report published by The New Stack. The report describes three real-world cluster failures: a training fleet that jumped from a few thousand nodes to 15,000 and overwhelmed its kube-apiserver with memory demand, an agent platform handling thousands of object writes per minute despite modest size, and a platform cluster with 200 operators installed where every apiserver replicates tens of thousands of mostly-dormant objects. The report argues that clusters exceeding 5,000 nodes or experiencing extreme churn already run in production on standard upstream components, but the assembly guidance doesn't exist.
The report details three independent bottlenecks. Node-bound clusters face pressure from kubelet reporting: at 100,000 nodes with a 10-second heartbeat, liveness alone produces roughly 10,000 writes per second, with lease renewals accounting for about 70% of write queries per second in the report's measurements. Churn-bound clusters experience intense write rates from short-lived custom resources representing agent sessions, where traffic has no connection to machine count. State-bound clusters face costs from resident bytes multiplied across every apiserver holding them, making the common reflex to add more apiservers actively harmful because each new server replicates the entire watch cache. The report found that even with every disableable cache turned off, an apiserver still carries approximately 0.9 KB per RBAC object because the authorizer's informers hold them regardless.
The report introduces four "seams"—substitutable contracts between components where Kubernetes can be reconfigured without patching upstream code. "Think of them the way a tailor thinks about seams in a jacket," the authors write. "Cut somewhere that isn't a seam, and you've made a very expensive rag." The storage contract seam routes different resource types to separate backends, relieving churn by sending ephemeral events and leases to stores that don't require the same durable, Raft-replicated guarantees as Deployments. The node liveness seam absorbs heartbeat load through a node-facing tier that handles renewals and propagates only genuine state transitions, or routes renewals through etcd's in-memory lease refresh instead of a full Raft commit. The shard axis seam divides objects horizontally across controllers, caches, and stores, while the cache residency seam controls what an apiserver actually holds in memory—a two-shard prototype cut heap usage by roughly half per shard.
The report calls for per-CRD storage configuration to handle churn-bound workloads where hot objects are custom resources, and asks operators running serious churn or clusters above 20,000 nodes to share their configurations because no one has published scale-tested recipes for those profiles. The authors note that none of this is a distribution—it's stock upstream binaries shipped as CI-tested configuration, so a release that changes the calculus shows up as a test failure first. They'll be at KubeCon North America collecting control-plane failure graphs to refine which seams need work, with the goal of getting training fleets, agent platforms, and platform clusters to scale on standard components using pieces already on the workbench. The path forward isn't magical forks or research projects; it's being deliberate about what each store holds and publishing the recipes that work. Organizations accustomed to throwing hardware at performance problems will need to rethink their default playbook, since vertical and horizontal scaling both break down when the architecture itself is the constraint.