Why Autoscaling LLM Inference on CPU Metrics Fails

The default Kubernetes Horizontal Pod Autoscaler scales on CPU and memory. For a typical web service, that works. For an LLM inference server, it often doesn't, and the failure is quiet: users wait longer and longer while the autoscaler sees nothing wrong.

Why CPU is the wrong signal

The real work happens on the GPU. An inference server can have a long queue of waiting requests while its CPU usage stays low and steady. Memory stays flat too, because the server reserves GPU memory up front. The autoscaler never adds replicas.

Better signals

• Requests waiting in the queue. vLLM exposes this as vllm:num_requests_waiting. A growing queue means demand exceeds capacity.
• Time to first token. Rising latency at the percentiles your users care about.
• Key-value cache usage. When the cache is nearly full, new requests have to wait.

How to scale on them

KEDA can scale a deployment from a Prometheus query:

triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      query: sum(vllm:num_requests_waiting{app="llm"})
      threshold: "5"

This adds replicas when more than five requests are waiting per replica on average.

The startup problem

A new inference pod may need a new GPU node, a large image pull and several minutes to load weights. By the time it's ready, the spike may be over. Plan for it:

• Keep a minimum number of warm replicas during business hours.
• Scale up early at a low threshold, and scale down slowly.
• Cache images and weights on nodes to shorten startup.
• Use scheduled scaling for predictable peaks, such as the start of the working day.

The strongest objection

"Scaling on queue length makes us over-provision." Sometimes it does, and GPUs are expensive. The alternative is users sitting through timeouts. Set a maximum replica count that fits your budget and track how often you reach it.

Things to watch

• Scaling down can cut off requests in progress. Give pods a termination grace period long enough to finish them.
• One slow request type, such as long document summaries, can dominate the queue. Consider a separate deployment for it.
• Load test the scaling behavior before relying on it in production.

Change the scaling signal first. No amount of tuning CPU thresholds fixes a signal that doesn't reflect the load.