How to Scale AI Worker Pods on Queue Depth With KEDA

Your document-processing workers spend most of the day doing nothing, and they still hold a pod (or a GPU node) the whole time. By the end of this guide you'll have a worker Deployment that sits at zero replicas, wakes up when RabbitMQ messages arrive, and scales out in proportion to the backlog. The mechanism is KEDA queue depth scaling, and it's a realistic one-sprint ticket for any team running batch enrichment or async AI jobs.

This covers queue-driven workers only, not online inference behind a load balancer.

Install KEDA and store the queue credentials

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda --create-namespace

The KEDA deployment docs list 2.21 as the latest release, tested with Kubernetes 1.34 through 1.36. Now store two connection strings: one for KEDA (the RabbitMQ management API on port 15672, so the rabbitmq_management plugin must be enabled) and one for your workers (AMQP on 5672).

kubectl create namespace ai-workers
kubectl -n ai-workers create secret generic rabbit-credentials \
  --from-literal=scaler-host='http://<user>:<password>@rabbitmq.<rabbit-namespace>.svc.cluster.local:15672/' \
  --from-literal=amqp-uri='amqp://<user>:<password>@rabbitmq.<rabbit-namespace>.svc.cluster.local:5672/'

The trailing slash on the first URL matters: the scaler docs require it when the host has no vhost. URL-encode any special characters in the password. Make sure the queue documents.inbound exists before you apply the scaler.

Deploy the worker

apiVersion: apps/v1
kind: Deployment
metadata:
  name: doc-worker
  namespace: ai-workers
spec:
  selector:
    matchLabels:
      app: doc-worker
  template:
    metadata:
      labels:
        app: doc-worker
    spec:
      terminationGracePeriodSeconds: 600
      containers:
        - name: worker
          image: <your-registry>/doc-worker:1.0
          env:
            - name: RABBITMQ_URI
              valueFrom:
                secretKeyRef:
                  name: rabbit-credentials
                  key: amqp-uri
            - name: QUEUE_NAME
              value: documents.inbound
          resources:
            requests:
              cpu: "1"
              memory: 2Gi

KEDA owns the replica count, so there's no replicas field here. Your worker should prefetch one message at a time, acknowledge only after the job finishes, and stop taking new messages on SIGTERM. The long terminationGracePeriodSeconds gives a half-finished job time to complete.

Authenticate the scaler with a TriggerAuthentication

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: rabbit-auth
  namespace: ai-workers
spec:
  secretTargetRef:
    - parameter: host
      name: rabbit-credentials
      key: scaler-host

This feeds the secret into the trigger's host parameter, so no password sits in the ScaledObject.

Scale on KEDA queue depth with a ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: doc-worker
  namespace: ai-workers
spec:
  scaleTargetRef:
    name: doc-worker
  pollingInterval: 15
  cooldownPeriod: 300
  minReplicaCount: 0
  maxReplicaCount: 10
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          policies:
            - type: Pods
              value: 2
              periodSeconds: 60
  triggers:
    - type: rabbitmq
      authenticationRef:
        name: rabbit-auth
      metadata:
        protocol: http
        queueName: documents.inbound
        mode: QueueLength
        value: "5"

What each knob does, per the ScaledObject spec:

• pollingInterval: how often KEDA checks the queue. The default is 30 seconds.
• cooldownPeriod: how long to wait after the last activity before going to zero. It governs only the final step from 1 to 0. The default is 300 seconds.
• minReplicaCount: 0: allows scale to zero. maxReplicaCount caps the fan-out (the default is 100, which is rarely what you want for GPU workers).
• value: "5": the target backlog per replica. With 40 messages queued you should see roughly 8 pods.
• advanced.horizontalPodAutoscalerConfig.behavior: passed straight to the Horizontal Pod Autoscaler (HPA). Here it removes at most two pods a minute.

KEDA only handles the 0 to 1 jump. Once one pod is running, the HPA takes over for 1 to N. Any backlog above activationValue (default 0) wakes the workload.

Test it by pushing messages and watching pods appear

Apply the three manifests, then confirm the idle state:

kubectl apply -f worker.yaml -f trigger-auth.yaml -f scaledobject.yaml
kubectl -n ai-workers get scaledobject doc-worker
kubectl -n ai-workers get pods -l app=doc-worker

READY should be True and the pod list empty. In one terminal, watch the pods. In a second, forward the AMQP port:

kubectl -n ai-workers get pods -l app=doc-worker -w
kubectl -n <rabbit-namespace> port-forward svc/rabbitmq 5672:5672

In a third, publish 40 messages:

import json
import pika  # pip install pika

conn = pika.BlockingConnection(
    pika.URLParameters("amqp://<user>:<password>@localhost:5672/")
)
ch = conn.channel()
ch.queue_declare(queue="documents.inbound", durable=True)
for i in range(40):
    ch.basic_publish(
        exchange="",
        routing_key="documents.inbound",
        body=json.dumps({"doc_id": i}),
        properties=pika.BasicProperties(delivery_mode=2),
    )
conn.close()

Within a polling interval or two the first pod should appear, then more as the HPA catches up. Check its view with kubectl -n ai-workers get hpa keda-hpa-doc-worker. When the queue drains, wait out the cooldown and the pods return to zero.

Gotchas

• Use protocol: http for long jobs. Under amqp, unacknowledged messages are not counted, so a worker mid-job makes the queue look empty and KEDA can scale it away after the cooldown. The HTTP API counts them (set excludeUnacknowledged: "true" if you want them ignored).
• KEDA can't choose which pod dies on scale-down. The HPA may pick a pod that is partway through a message, which is why the grace period and SIGTERM handling matter. For very long jobs, the docs suggest Jobs instead.
• Cold starts are real. Image pulls and model loads delay the first message, so keep images small or set minReplicaCount: 1 for latency-sensitive queues.
• maxReplicaCount is your cost and rate-limit ceiling. Set it from GPU quota and downstream API limits, not from optimism.
• The queue arguments must match. A durable queue declared by the producer and a non-durable one declared by the worker will fail on redeclare.

Scale-to-zero workers turn an always-on cost line into a spend that follows the actual work, and that is usually the first FinOps win a production AI team can ship.