How to Cut Cold Starts for Model-Serving Pods

When traffic rises, a new model-serving pod might need a new node, a multi-gigabyte image and several minutes of loading weights into GPU memory. Users wait through all of it. Cutting that time is what makes autoscaling useful.

Step 1: Measure each phase

Break startup into parts before optimizing anything:

1. Node provisioning, if a new node is needed.
2. Image pull.
3. Weight download.
4. Loading weights into memory and warming up.
5. Passing the readiness probe.

Pod events and timestamped container logs give you most of this. Fix the slowest phase first.

Step 2: Shrink and pre-pull images

Keep images lean and move weights out of them. For images that must stay large, pre-pull them onto GPU nodes with a DaemonSet or bake them into the node image. On GKE, image streaming can start containers before the full image arrives.

Step 3: Cache weights close to the GPU

Downloading weights from object storage on every start is slow. Use a node-local NVMe cache, a shared read-only volume or a disk created from a snapshot. The safetensors format also loads faster than older pickle-based files.

Step 4: Keep warm capacity

The fastest cold start is one that never happens:

• Set a minimum replica count for business hours.
• Keep a spare GPU node ready with a low-priority placeholder pod that gets evicted when real work arrives.
• Scale up ahead of predictable peaks with scheduled scaling.

Step 5: Set probes correctly

Use a startup probe with a long enough window for model loading, then a readiness probe that checks the model responds:

startupProbe:
  httpGet: {path: /health, port: 8000}
  periodSeconds: 10
  failureThreshold: 60

This allows up to ten minutes. A pod that reports ready before loading finishes sends users errors.

Step 6: Measure again

Repeat the measurement after each change, and track the time from scale-up decision to first served request as a regular metric.

Things to watch

• Placeholder pods cost money around the clock. Size them to your real peaks.
• Node images with cached content need rebuilding when models change.
• Larger instances may load faster but cost more. Compare cost per request, not only startup time.

Start with step 1. Without the breakdown, it's easy to optimize a phase that was never the problem.