Benefits of the Batch API for Non-Urgent LLM Workloads

Your nightly summarizer doesn't care whether an answer arrives in two seconds or two hours, but your invoice does. A batch API for LLM jobs trades speed for a 50% discount: OpenAI, Anthropic and Azure OpenAI all price work that can wait up to 24 hours at half the synchronous rate. Below are five benefits, a working snippet, and the cases where you should skip it.

Five benefits of a batch API for LLM workloads

1. LLM batch processing cost drops by 50%. The discount covers input and output tokens. OpenAI's Batch API guide lists 50% off versus synchronous calls, Anthropic's Message Batches docs charge all usage at 50% of standard prices, and Azure OpenAI's global batch targets 50% less than global standard. Classify 100,000 tickets at about 500 input and 100 output tokens each and you send 50M input and 10M output tokens. At Claude Haiku 4.5 batch prices ($0.50 and $2.50 per million tokens) that is $50, versus $100 at standard prices. That is my arithmetic from the published prices, not a benchmark.

2. Your production rate limits stay free. OpenAI gives batch jobs a separate rate-limit pool that doesn't consume your standard per-model limits. Azure gives global batch its own enqueued token quota, so online workloads aren't disrupted. A 50,000-prompt eval run at 2pm stops being the reason your support bot gets 429 errors.

3. One file replaces a worker fleet. You skip the queue, the concurrency tuning and the backoff code. Per-job limits are 50,000 requests or 200 MB on OpenAI, 100,000 requests or 256 MB on Anthropic, and 100,000 requests or 200 MB per input file on Azure. The 24 hours is a ceiling, and Anthropic says most batches finish within an hour.

4. It stacks with prompt caching on Anthropic. The batch and caching discounts combine. Cache hits are best effort because batch requests run concurrently and in any order, and Anthropic reports hit rates between 30% and 98% depending on traffic. Put identical cache_control blocks in every request and use the 1-hour cache duration. An eval suite where every request carries the same long rubric is the natural fit.

5. You retry only what failed. Every line carries a custom_id, and results come back keyed by it. OpenAI writes failures to a separate error file. Anthropic doesn't bill errored, canceled or expired requests. On Azure, cancelling a job returns finished work and you pay only for that. Re-run the failures, not the whole file.

Submit your first batch API job for asynchronous LLM work

This is the documented OpenAI flow: upload a JSONL file, create a batch with a 24-hour window, poll, download. Start with one request per line.

{"custom_id": "ticket-0001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "<your-model>", "messages": [{"role": "system", "content": "Classify the support ticket as billing, bug or other. Reply with one word."}, {"role": "user", "content": "I was charged twice for March."}]}}
{"custom_id": "ticket-0002", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "<your-model>", "messages": [{"role": "system", "content": "Classify the support ticket as billing, bug or other. Reply with one word."}, {"role": "user", "content": "The export button does nothing."}]}}

Each custom_id must be unique, and output order won't match input order. Replace <your-model> with a model that supports the endpoint in url.

import json
import time
from openai import OpenAI

client = OpenAI()

input_file = client.files.create(file=open("batchinput.jsonl", "rb"), purpose="batch")

batch = client.batches.create(
    input_file_id=input_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"description": "nightly ticket classification"},
)

while batch.status not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(60)
    batch = client.batches.retrieve(batch.id)

if batch.output_file_id:
    for line in client.files.content(batch.output_file_id).text.splitlines():
        row = json.loads(line)
        print(row["custom_id"], row["response"]["body"]["choices"][0]["message"]["content"])

if batch.error_file_id:
    print(client.files.content(batch.error_file_id).text)

"24h" is the only completion window OpenAI accepts. In production, poll from a scheduled job rather than a sleeping process, and always read the error file.

On Azure OpenAI the JSONL has the same shape, but model must be the name of your Global Batch deployment, identical on every line. The same OpenAI client works when you point base_url at https://<your-resource-name>.openai.azure.com/openai/v1/.

Anthropic's equivalent is Message Batches. Each request wraps normal Messages parameters:

import time
import anthropic
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

client = anthropic.Anthropic()

batch = client.messages.batches.create(
    requests=[
        Request(
            custom_id="ticket-0001",
            params=MessageCreateParamsNonStreaming(
                model="<your-model>",
                max_tokens=16,
                messages=[{"role": "user", "content": "Classify as billing, bug or other: I was charged twice for March."}],
            ),
        ),
    ]
)

while client.messages.batches.retrieve(batch.id).processing_status != "ended":
    time.sleep(60)

for result in client.messages.batches.results(batch.id):
    if result.result.type == "succeeded":
        print(result.custom_id, result.result.message.content[0].text)

Results stay downloadable for 29 days after the batch is created, so copy them somewhere permanent.

When the batch API is not worth it

• A person is waiting. The window is up to 24 hours, and Azure doesn't guarantee completion inside it. Anything user-facing stays synchronous.
• The volume is tiny. A few hundred calls save pennies, and file handling, polling and result joining cost more engineer time than that.
• You need streaming. Anthropic rejects stream: true and fast mode in batch requests, because results arrive as a single file.
• You hit endpoint or quota limits. OpenAI supports a fixed list of endpoints, Azure needs a Global Batch or Data Zone Batch deployment of a supported model, and Azure jobs can fail when they exceed your enqueued token quota. Split jobs that pass the per-file request caps.

For any production LLM work that can wait, a batch endpoint is the cheapest cost lever you have, and it leaves your live traffic's rate limits alone.