How to Build a CI/CD Pipeline for an LLM-Powered Service

A service that calls a language model is still a service. It needs builds, tests and repeatable deployments. What changes is that part of its behavior lives in prompts and model settings, and ordinary unit tests don't notice when a prompt gets worse.

A good pipeline adds two stages to the usual ones: an evaluation gate and a staged rollout.

Stage 1: Build and unit test

Build the container image once and reuse that image in every later stage. Unit tests should mock the model call. They check your parsing, tool handling and error paths, not the model's answers.

Stage 2: Evaluate

Run a fixed set of real cases against the actual model and fail the build if the pass rate drops below an agreed threshold.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t app:${{ github.sha }} .
  evals:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: python evals/run.py --min-pass-rate 0.9
        env:
          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}

Keep this set small enough to finish in a few minutes, and run the full set nightly.

Stage 3: Deploy to staging

Deploy the same image to staging with production-like configuration. Run smoke tests that send a real request through every route, including one that calls tools.

Stage 4: Roll out gradually

Send a small share of production traffic to the new version first. Watch error rates, latency and a quality signal such as the thumbs-down rate. Promote when the numbers hold and roll back when they don't.

Treat prompts and model versions as code

Keep prompts, model names and parameters in the repository. A change to any of them should go through the same pipeline as a code change, including the eval gate.

Things to watch

• Eval runs cost money. Run the large suite on a schedule rather than on every push.
• Model outputs vary. Allow a small tolerance in the threshold, or run unstable cases several times.
• Give CI its own API keys with spending limits, separate from production.

Add the eval gate first. It catches problems the rest of the pipeline was never designed to see.