Why Serverless Is Often the Wrong Home for LLM Workloads
Calling a model API from a serverless function looks like a perfect match: no servers, pay per request, automatic scaling. For short, simple tasks, it works well. As AI features grow into streaming chat, long documents and multi-step agents, the limits of functions start to hurt.
Where the friction comes from
Timeouts. AWS Lambda functions run for at most 15 minutes, and API Gateway REST APIs have a default integration timeout of 29 seconds. An agent that calls several tools and a slow model can hit the API limit long before the function limit.
Streaming. Users expect chat responses to appear word by word. Streaming from functions is possible, for example with Lambda response streaming, but it restricts how you expose the endpoint and adds complexity compared with a long-running server.
Paying to wait. A function waiting 20 seconds for a model response is billed for that time while doing almost nothing. A container service handling many concurrent requests on one instance often costs less at steady volume.
Cold starts. Heavy Python dependencies, such as tokenizers or large SDKs, slow down cold starts, and those delays add up in interactive features.
Long connections. Voice and realtime features need long-lived WebSocket connections, which fit awkwardly into short-lived functions.
Where serverless works well
- Event-driven background tasks, such as summarizing a document when it's uploaded.
- Short features that make one model call and finish in a few seconds.
- Low or unpredictable traffic, where paying for idle containers would cost more.
- Queue workers for batch jobs that fit comfortably within the time limit.
The strongest objection
"We're a serverless shop, and running containers is extra work." That's a fair concern. Managed container services such as ECS on Fargate or Cloud Run remove most of the operational work while avoiding the timeout and streaming problems. Many teams keep functions for events and run interactive AI endpoints on containers.
A practical split
- Interactive chat and agents: a container service behind a load balancer that supports long connections.
- Background processing: functions triggered by events or queues.
- Large batch jobs: queue workers or managed batch services, using your model provider's batch API where it's available.
Before building, estimate the longest request your feature will make, including tool calls and retries. If it's more than about 25 seconds, plan for containers from the start.