How to Size Kubernetes Requests and Limits for AI Services

Resource settings for AI services are often copied from an example and never revisited. The result is pods killed for using too much memory, services throttled during traffic spikes, or nodes that are fully reserved but barely used.

How requests and limits differ

• Requests are what the scheduler reserves for a pod. They decide which node it lands on.
• Limits cap what a container may use. Going over the memory limit kills the container. Going over the CPU limit slows it down.

Step 1: Measure real usage

Run the service under realistic load in staging and record CPU and memory with Prometheus or kubectl top pod. Include a peak, such as a burst of long documents, not only average traffic.

Step 2: Set memory requests and limits equal

Memory use in Python services grows with request size. Set the memory request and limit to the same value, a little above the measured peak. The pod gets exactly what it reserves and is less likely to be evicted when the node runs short.

resources:
  requests:
    cpu: "1"
    memory: 2Gi
  limits:
    memory: 2Gi

Step 3: Be careful with CPU limits

CPU limits cause throttling even when the node has spare capacity. For latency-sensitive services, many teams set a CPU request and leave the CPU limit out, relying on requests for fair sharing.

Step 4: Treat GPUs differently

GPUs are usually set as limits, in whole units. Kubernetes doesn't manage GPU memory at all, so a process can still run out of memory on its card. Configure the inference server's memory settings to fit the card it gets.

Step 5: Watch for kills and throttling

Alert on containers terminated as OOMKilled and on CPU throttling metrics. Both mean the settings don't match real usage.

Step 6: Review regularly

Usage changes with new models, prompts and traffic patterns. Review the settings after major releases, and consider the Vertical Pod Autoscaler in recommendation mode for suggested values.

Things to watch

• Worker processes multiply memory. Four Gunicorn workers that each load a tokenizer use four times as much.
• Sidecar containers count too. Include them in your measurements.
• Requests set far above real usage waste nodes, and the cluster autoscaler adds more nodes to make up for it.

Start by checking which pods were OOMKilled or throttled last week.