Throttled at 22% utilisation
The Corvus dispatch API had a p99 of 810ms and a p50 of 30ms. Node CPU utilisation averaged 22%. The cluster ran 96 nodes and the finance team wanted to know why. Both problems had the same cause and neither was visible on a utilisation dashboard.
container_cpu_cfs_throttled_periods_total divided by container_cpu_cfs_periods_total on the dispatch pods was 0.31. Nearly a third of every 100ms scheduling period, the container was stopped and waiting. Average utilisation of 22% and heavy throttling coexist comfortably, which is why the average misleads.
The Linux CFS bandwidth controller enforces limits as a quota per 100ms period, not as a rate. A pod with limits.cpu = 500m gets 50ms of CPU time per period. A JVM with 8 GC and request threads on a 32-core node can burn that in 6ms of wall clock and then sit idle for 94ms. The p99 is the requests that landed in that dead time.
Requests and limits do different jobs
Requests are what the scheduler uses to place pods and what cgroup cpu.shares uses to arbitrate under contention. Limits are a hard ceiling enforced by quota. For latency-sensitive services on a cluster where requests are set honestly, the limit adds nothing except the stall. We removed CPU limits from twelve request-serving deployments and kept requests set from measured p95 usage.
This is deliberately not the advice for every workload. Batch jobs and anything with unbounded parallelism keep limits, because without them one job can starve co-tenants between scheduling decisions. Memory limits stay on everything — memory is not compressible, and the failure mode without a limit is an OOMKill of a neighbour rather than of the offender.
Runtime awareness matters as much as the cgroup. A JVM reads container limits for heap sizing but older runtimes read the node's core count for thread pools; Node.js sizes libuv's pool from the node too. Set the pool sizes explicitly rather than hoping the runtime infers them, and check what Runtime.availableProcessors actually returns inside the container.
The autoscaler was reading a broken signal
The HPA scaled on CPU utilisation as a percentage of requests. Throttled pods cannot consume more CPU, so utilisation flattened just below the target and the HPA stopped adding replicas exactly when the service needed them. The system was stable, wrong, and quiet.
We moved the HPA onto a request-rate metric with a target derived from measured per-pod capacity, and kept CPU as a secondary ceiling. Scaling now leads load instead of trailing it. For queue consumers the right signal is almost always queue depth or oldest-message age, not CPU.
Requests were then reset from evidence: p95 of the last 14 days of container_cpu_usage_seconds_total per container, plus 25%. Twenty-two workloads had requests set from a copied template at 1000m while actually using 120m. Those numbers were reserving capacity nobody was using, which is what the 96 nodes were paying for.
What changed
Nodes went from 96 to 60 on the same traffic. Dispatch p99 went from 810ms to 240ms. Neither required a code change to the application; both came from correcting three numbers and one autoscaler metric.
The dashboards changed permanently. Utilisation is now shown next to throttled-period ratio, and any workload above 0.05 raises a ticket. Request-to-usage ratio is tracked per namespace, with anything above 4x flagged monthly. Bin-packing efficiency — sum of requests over allocatable — is reported to the platform team weekly and sits at 0.68 against a target of 0.75.
The uncomfortable part is that the cluster looked healthy the entire time. Green dashboards, no restarts, no errors. The cost was paid in the tail, where averages do not go.