Why Model Servers Need a Startup Probe, Not a Longer Delay

Your GPU pod restarts every few minutes, never reaches Ready, and the logs show a model that was most of the way loaded when it died. Nothing is wrong with the model. The liveness probe fired while the weights were still loading, the kubelet (the node agent that runs your containers) did what it was told, and the loop began.

The usual patch is a bigger initialDelaySeconds. The better fix is a model servers startup probe: a separate check whose only job is to wait for the model to load.

Why a longer initialDelaySeconds is the wrong fix

vLLM's own Kubernetes deployment guide uses initialDelaySeconds: 60 on its liveness and readiness probes. That is fine for a small model on a warm node. Then someone swaps in a bigger model, the delay becomes 300, then 600:

livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 600   # the fix that keeps growing

Three things go wrong with this:

• A blind window on every restart. For the whole delay, nothing checks the container. A stalled weight download looks exactly like a slow load.
• Fast starts pay for slow ones. If readiness shares the delay, a pod that could serve in 90 seconds sits unready for the rest of it.
• A guess, not a signal. Too short and you get a restart loop. Too long and the blind window grows. The timer never learns that the model is actually ready.

The model servers startup probe, in YAML

Kubernetes does not run liveness or readiness probes until the startup probe succeeds. The startup budget is failureThreshold x periodSeconds. If the probe never succeeds, the container is killed and handled according to the pod's restartPolicy. Once it succeeds, liveness takes over with whatever tight settings you gave it.

containers:
  - name: model-server
    image: <your-vllm-image>
    ports:
      - name: http
        containerPort: 8000
    startupProbe:
      httpGet:
        path: /v1/models
        port: http
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 180   # 10s x 180 = up to 30 min to load
    readinessProbe:
      httpGet:
        path: /v1/models
        port: http
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
    livenessProbe:
      httpGet:
        path: /health
        port: http
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 3     # a real hang is caught in about 30s

This follows the llm-d readiness probe guide: /v1/models for startup and readiness, /health for liveness, and a 30 minute startup ceiling. Replace the 30 minutes with your own measured number (next section).

Check which health endpoint your vLLM version supports

Don't assume the paths. The llm-d guide says /health returns 200 as soon as the server is up, before the model has loaded, and that /v1/models refuses connections or returns 503 until the model is ready. vLLM's health handler calls the engine's check_health() and returns 503 only when the engine is dead. It does not run inference.

Behavior can differ by version and flags, so watch your own server load. Run this against a cold start in staging:

kubectl port-forward pod/<your-pod> 8000:8000 &

while true; do
  for path in health v1/models; do
    code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 3 http://localhost:8000/$path)
    echo $(date +%T) /$path $code
  done
  sleep 5
done

A 000 means connection refused. Note the moment each path flips to 200 and use the slowest of several cold starts to size failureThreshold, with margin. If you run a different server, use its own docs for the paths.

The objection: deep checks eject healthy pods

The strongest argument against all this: a probe that touches the model can time out under load, and a failed liveness probe restarts a healthy pod that is busy with a long batch, pushing its traffic onto the rest. The Kubernetes docs warn about this exact failure: badly designed liveness probes cause restarts under high load and cascading failures.

That is a reason to split the jobs, not to deepen every check:

• Liveness stays shallow. /health, a 5 second timeout, three failures in a row.
• The model-aware check gates traffic, not lifetime. A failing readiness probe takes the pod out of Service endpoints without restarting it, so a saturated pod sheds traffic and keeps its loaded weights.
• No generation calls in probes. Keep them cheap, and load test that they still answer when the server is busy.

One honest limit: shallow liveness will not catch a GPU that faults while the process stays up, because /health reports engine liveness only. Cover that with alerts on error rates, not a hair-trigger restart.

First step: measure, then swap the delay for a startup probe

Run the timing loop on a cold start in staging. Delete the large initialDelaySeconds. Add the startupProbe with a budget of roughly twice your slowest observed load, and keep liveness tight. After that, kubectl get events should show a failed startup probe only when the model truly never came up.

A startup probe turns a slow model load from an outage into a normal state, which is what keeps a cold start from becoming a page.