Why Load Testing LLM Endpoints Differs From Normal APIs

You point your usual load testing rig at a shiny new LLM endpoint, ramp up to 200 virtual users, and the dashboard turns green. Ship it. Then real traffic hits, the endpoint starts timing out for some users while others sail through, and the first support ticket asks why the chatbot "hung for 40 seconds and then dumped the whole answer at once."

That gap between a green load test and a bad production afternoon is the whole problem with load testing LLM endpoints the way you'd load test a normal REST API. The assumptions baked into tools like k6, JMeter or Locust, and honestly into most engineers' heads, don't hold for a model serving tokens.

A request isn't a request anymore

A normal API call has a shape: request in, response out, latency is roughly the time between the two. You measure p50, p95, p99, done.

An LLM call, especially a streamed one, has no single latency number that means anything on its own. There are at least three that matter separately:

• Time to first token (TTFT): how long the user stares at a blank screen.
• Inter-token latency: how smoothly the response streams once it starts.
• Total completion time: how long until the full answer, and the number your billing and your timeout config actually care about.

A tool that only records "time until the HTTP response closed" will tell you the connection stayed open for 12 seconds and call it a 12-second latency. It won't tell you the user waited 9 of those seconds for the first word to appear, which is the number that actually predicts whether they bail.

// k6 with a custom streaming check, not the default HTTP timing
import http from "k6/http";
import { Trend } from "k6/metrics";

const ttft = new Trend("time_to_first_token");
const totalTime = new Trend("total_completion_time");

export default function () {
  const start = Date.now();
  let firstTokenSeen = false;

  const res = http.post(
    "https://<your-endpoint>/v1/chat/completions",
    JSON.stringify({
      model: "<your-model>",
      messages: [{ role: "user", content: "Summarize this support ticket." }],
      stream: true,
    }),
    {
      headers: { "Content-Type": "application/json" },
      responseCallback: http.expectedStatuses(200),
      // stream events fire per chunk, not per response
      onStreamEvent: () => {
        if (!firstTokenSeen) {
          ttft.add(Date.now() - start);
          firstTokenSeen = true;
        }
      },
    }
  );

  totalTime.add(Date.now() - start);
}

The point isn't the exact k6 API (check your version's docs for streaming support), it's the principle: instrument TTFT and total time as separate metrics from day one, because they fail independently and for different reasons.

Latency isn't a distribution around a mean, it's a coin flip

A database query that takes 40ms usually takes something close to 40ms next time. An LLM completion for "summarize this" might take 800ms, then 6 seconds, then 3 seconds, for the exact same prompt, because output length varies token by token and generation time scales with it. A short answer and a long answer to the same question can differ by 5x in latency, and you don't know which one you'll get until it's done.

This breaks the usual load-test move of running one representative request and multiplying. You have to run enough real variety, short prompts, long prompts, prompts that trigger longer completions, to get a p95 and p99 that reflect what production will actually throw back at users. A load test with one canned prompt repeated 10,000 times will report a beautifully tight latency distribution that has nothing to do with reality.

The rate limit isn't yours, and it's shared

Hit your own database too hard and you'll see it in your own connection pool. Hit an LLM provider's endpoint too hard and you'll get 429s that are governed by someone else's tokens-per-minute and requests-per-minute quotas, often shared across every team in your organization using the same account. Amazon Bedrock, Azure OpenAI and Vertex AI all enforce quota this way, and the limits are usually expressed in tokens, not requests, so a burst of long prompts can throttle you well before your request count looks alarming.

Load testing has to account for two failure modes normal APIs don't have: your test traffic tripping the provider's limiter, and your test masking that a limiter even exists because your synthetic prompts were shorter than real ones. Test with prompt lengths close to production, and know your provider's quota dashboard before you start, not after the 429s show up.

The honest objection: isn't this over-engineering?

The fair pushback is that most teams don't have LLM traffic anywhere near the scale where TTFT jitter or shared rate limits matter, so why build custom tooling for a problem you might not have. That's true if your endpoint serves a handful of internal users. It stops being true the moment the feature ships to customers, because the failure mode isn't a crash, it's a slow, silent degradation that support tickets surface before your monitoring does. The cost of adding a TTFT metric to your existing load test is an afternoon. The cost of finding out in production is a Monday morning full of escalations.

Start here

Before your next LLM feature meets real traffic, add three things to whatever load test you already run: separate TTFT and total-completion metrics, a prompt set with realistic length variety instead of one canned request, and a check on your provider's actual token-per-minute quota so you know the ceiling before your test does.

Load testing an LLM endpoint isn't a harder version of load testing an API, it's a different question: not "how many requests can this handle" but "how does this behave when generation time is unpredictable and the rate limit isn't yours to control." Get that question right before production asks it for you.