Why GPU Inference Pods Need a Pod Disruption Budget

Your LLM endpoint has three healthy replicas and a node pool upgrade scheduled for Tuesday. Without a pod disruption budget (PDB) on your GPU inference pods, nothing stops that upgrade from evicting replicas faster than their replacements can load model weights. A PDB turns "drain as fast as you can" into "drain one replica at a time".

Planned maintenance still breaks model serving availability

Kubernetes separates involuntary disruptions (hardware failure, a network partition) from voluntary ones: draining a node for an upgrade, or draining it to scale the cluster down. Cloud upgrade processes and autoscalers do this routinely through the Eviction API, and without a PDB every eviction succeeds immediately.

GPU inference makes it worse. A pod counts as healthy only when its Ready condition is true, and an inference server isn't ready until the weights are on the GPU. That takes minutes, so a fast upgrade can evict the second replica while the first replacement is still loading.

This is a service management problem first. Planned maintenance is the one source of downtime you fully control. A PDB is that maintenance-window promise written as config: no step takes more than one replica away at a time.

Write a pod disruption budget for GPU inference pods

Start with a Deployment whose readiness probe tells the truth.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
spec:
  replicas: 3
  selector:
    matchLabels:
      app: llm-inference
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: server
          image: <your-inference-image>
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
          startupProbe:          # allows up to 10 minutes for the model to load
            httpGet:
              path: /health      # replace with your server's real readiness path
              port: 8000
            periodSeconds: 10
            failureThreshold: 60
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 5

The health path must succeed only after the weights load. For terminationGracePeriodSeconds, eviction sends SIGTERM, then SIGKILL when the period ends, and the 30-second default is shorter than many long generations. Set it to your longest realistic request, and confirm your server stops taking new work on SIGTERM.

Now the budget itself:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: llm-inference
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: llm-inference

I prefer maxUnavailable: 1 over minAvailable: 2 because it survives someone changing the replica count. The PDB docs define healthy as Ready, which is why the honest probe is half the fix.

Plan surge capacity, because a PDB cannot create a GPU

A PDB makes a drain wait; it can't conjure a free GPU. If every GPU is taken, the evicted pod's replacement stays Pending, allowed disruptions stays at zero and the drain stalls. That is the PDB working, but you pay in upgrade time.

Give replacements somewhere to land. GKE surge upgrades add nodes before draining old ones (maxSurge, default one per zone). Deployment rollouts aren't limited by PDBs, so the maxSurge: 1 and maxUnavailable: 0 in the strategy block above protect new image rollouts, and they need a spare GPU too.

Test the budget with a node drain on a GPU node

Do this in staging, with a real model:

# Terminal 1: watch the pods
kubectl get pods -l app=llm-inference -o wide -w

# Terminal 2: confirm the budget allows one disruption
kubectl get pdb llm-inference
# NAME            MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
# llm-inference   N/A             1                 1                     2m

# Terminal 2: drain the first GPU node
kubectl drain <gpu-node-1> --ignore-daemonsets --delete-emptydir-data --timeout=15m

# Terminal 3: drain a second node while the replacement is still loading
kubectl drain <gpu-node-2> --ignore-daemonsets --delete-emptydir-data --timeout=15m

The first drain evicts one pod and allowed disruptions drops to 0. The second drain reports that it can't evict because the budget would be violated, and retries until the replacement shows 1/1 Ready. A replacement stuck in Pending means you found your missing surge capacity. Run kubectl uncordon on both nodes afterward.

The objection: PDBs stall node upgrades

True, and a bad budget stalls them forever. minAvailable equal to the replica count (or 100%) allows zero disruptions, so kubectl drain retries until its timeout. One replica with minAvailable: 1 does the same, while one replica with maxUnavailable: 1 can be evicted anyway. A PDB can't make a single replica highly available.

More gotchas:

• Voluntary disruptions only. A node that dies still takes its pod.
• A delay, not a vault: GKE honors PDBs for up to an hour on surge upgrades, then force-evicts what remains.
• The default unhealthyPodEvictionPolicy waits for unhealthy pods to recover. The docs recommend AlwaysAllow so broken pods don't block drains, but a pod still loading is Running and not Ready, so AlwaysAllow lets a drain evict it mid-load. Choose on purpose.

A stalled upgrade is a ticket from your platform team. A dropped endpoint is a ticket from the customer.

First step

Run kubectl get pdb --all-namespaces, list your GPU inference Deployments next to it, and add the budget to each one without. Then drain a staging GPU node and watch what happens before production does it for you.

Planned maintenance is the downtime you can promise away in production, and a PDB is how you keep that promise in code.