How to Canary Deploy Model-Serving Pods With Argo Rollouts

Rolling out a new model version the same way you roll out a stateless web app is how you end up paging yourself at 2am. A bad prompt template or a quantization regression doesn't throw a 500, it just quietly gets worse answers to 100% of your traffic the moment the rollout finishes. Argo Rollouts fixes this by shifting traffic to the new version in small steps and watching real metrics before it commits.

This walks through turning a model-serving Deployment into an Argo Rollouts canary, with automated analysis that checks p95 latency and error rate at each step and rolls back on its own if either one breaks.

By the end you'll have a Rollout resource, an AnalysisTemplate wired to Prometheus, and the kubectl argo rollouts commands to watch, promote or abort a release.

What you need first

A Kubernetes cluster with your model server already running as a Deployment behind a Service, and Prometheus scraping request duration and status code metrics from those pods. Argo Rollouts doesn't generate metrics, it just reads the ones you already have.

1. Install the Argo Rollouts controller and plugin

Install the controller in its own namespace, then the kubectl plugin so you can drive rollouts from the CLI.

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

# CLI plugin via krew
kubectl krew install argo-rollouts
kubectl argo rollouts version

The controller watches for Rollout objects the same way the built-in controller watches Deployment objects. Nothing else in the cluster changes.

2. Turn your Deployment into a Rollout

Argo Rollouts uses its own Rollout kind instead of Deployment. Take your existing spec, change the apiVersion and kind, and add a strategy block:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: llm-server
spec:
  replicas: 10
  selector:
    matchLabels:
      app: llm-server
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: { duration: 5m }
      - analysis:
          templates:
          - templateName: model-latency-error-rate
          args:
          - name: canary-hash
            valueFrom:
              podTemplateHashValue: Latest
      - setWeight: 30
      - pause: { duration: 5m }
      - setWeight: 60
      - pause: { duration: 5m }
      - setWeight: 100
  template:
    metadata:
      labels:
        app: llm-server
    spec:
      containers:
      - name: llm-server
        image: registry.example.com/llm-server:v2
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: 32Gi

Keep the same Service pointing at app: llm-server. With no service mesh or ingress controller configured, Argo Rollouts does basic canary: it controls the ratio of old-version to new-version pods behind that one Service, and Kubernetes' normal load balancing does the rest. Good enough for most internal model-serving setups, no Istio or SMI required.

3. Write the AnalysisTemplate

This is the part that makes it a canary instead of a slow rollout: a query that has to stay healthy or the release stops automatically.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: model-latency-error-rate
spec:
  args:
  - name: canary-hash
  metrics:
  - name: p95-latency-ms
    interval: 1m
    count: 5
    successCondition: result[0] <= 800
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc:9090
        query: |
          histogram_quantile(0.95,
            sum(rate(model_request_duration_seconds_bucket{
              app="llm-server",
              rollouts_pod_template_hash="{{args.canary-hash}}"
            }[2m])) by (le)
          ) * 1000
  - name: error-rate
    interval: 1m
    count: 5
    successCondition: result[0] <= 0.02
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc:9090
        query: |
          sum(rate(model_requests_total{
            app="llm-server",
            rollouts_pod_template_hash="{{args.canary-hash}}",
            status=~"5.."
          }[2m]))
          /
          sum(rate(model_requests_total{
            app="llm-server",
            rollouts_pod_template_hash="{{args.canary-hash}}"
          }[2m]))

The rollouts_pod_template_hash label is the trick: Argo Rollouts stamps every pod it creates with rollouts-pod-template-hash, and podTemplateHashValue: Latest passes the canary's current hash into the query as an argument. That's what keeps the analysis scoped to the new pods instead of averaging them in with the stable ones, which would hide a regression.

failureLimit controls how many bad samples in a row trigger a rollback. Two consecutive failed 1-minute windows is a reasonable default for a metric as noisy as LLM latency.

4. Ship it and watch

Apply the Rollout and AnalysisTemplate, then apply your new image the same way you'd update a Deployment: bump the tag and kubectl apply.

kubectl apply -f analysis-template.yaml
kubectl apply -f rollout.yaml

# after pushing a new image tag
kubectl argo rollouts get rollout llm-server --watch

The --watch view shows each step, the current traffic split, and live analysis results in the terminal. If error-rate or p95-latency-ms breaches its successCondition for failureLimit consecutive checks, the controller aborts automatically and scales the canary back to zero. No PagerDuty page, no manual rollback, no bad model serving 90% of your traffic because someone stepped away during the pause.

5. Promote or abort by hand when you need to

Automated analysis handles the common case, but you'll still want manual control for judgment calls, like a latency blip you know is unrelated.

# skip the remaining pauses and ship to 100%
kubectl argo rollouts promote llm-server

# kill the canary and revert to the last stable version
kubectl argo rollouts abort llm-server

# roll back a rollout that already finished
kubectl argo rollouts undo llm-server

Gotchas

• GPU headroom. Basic canary runs old and new pods side by side, so a 10-replica GPU-backed Rollout briefly needs capacity for more than 10 replicas during the transition. Size your node pool or GPU quota for peak, not steady state, or the canary pods will sit Pending and the rollout stalls.
• Cold starts skew your first samples. If model-serving pods are slow to warm up, the first 1-2 minutes of canary latency will look worse than steady state and can trigger a false rollback. Add a short pause before the first analysis step so pods finish warming up before you start scoring them.
• Label your metrics before you need them. The rollouts_pod_template_hash label only shows up in Prometheus if your metrics exporter is scraping pod labels as-is. Check this with kubectl get pods -l app=llm-server --show-labels and a quick Prometheus query before you trust the AnalysisTemplate in production.
• A stuck pause isn't a stuck rollout. pause: { duration: 5m } steps look identical to a hung rollout in kubectl get rollout. Use kubectl argo rollouts get rollout llm-server (not plain kubectl get) to see which step you're actually on.

Once this is wired up, shipping a new model version stops being a leap of faith and becomes a series of small, reversible bets, which is exactly the level of confidence you want when the thing you're deploying can be wrong in ways your test suite won't catch.