How to Serve Open-Weight LLMs on Kubernetes With vLLM

When a customer can't send data to an external model API, or needs predictable costs at high volume, running an open-weight model inside their cluster becomes an option. vLLM is a popular inference server for this. It batches requests efficiently and exposes an OpenAI-compatible API, so existing client code needs few changes.

Step 1: Pick a model that fits your GPUs

The weights must fit in GPU memory with room left for the key-value cache that holds each request's context. As a rough guide, an 8-billion-parameter model in 16-bit precision needs about 16 GB for the weights alone. Check the model's license for commercial use before going further.

Step 2: Deploy the server

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm
spec:
  replicas: 1
  selector:
    matchLabels: {app: llm}
  template:
    metadata:
      labels: {app: llm}
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.x.y   # pin a version you've tested
          args: ["--model", "meta-llama/Llama-3.1-8B-Instruct", "--max-model-len", "16384"]
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1

Put a Service in front and point clients at http://llm:8000/v1.

Step 3: Tune the settings that matter

• --max-model-len caps context length. Lower values leave more memory for concurrent requests.
• --gpu-memory-utilization sets how much GPU memory vLLM may use.
• --tensor-parallel-size splits a model too large for one card across several GPUs on the same node.

Step 4: Load weights efficiently

Downloading weights on every pod start is slow. Mount a volume or node-local cache that already holds them, and provide a Hugging Face token as a secret for gated models.

Step 5: Add health checks and metrics

vLLM serves a /health endpoint for probes and Prometheus metrics at /metrics, including running and waiting requests. Allow a long startup window, because loading a model can take minutes.

Things to watch

• Benchmark with your real prompt and response lengths. Throughput depends heavily on both.
• vLLM's optional API key is a single shared secret. Put proper authentication in front of the service.
• Upgrade vLLM deliberately and rerun your benchmarks, since performance and flags change between versions.

Start with a small model on one GPU, measure throughput, then size the real deployment from those numbers.