Why Model Weights Don't Belong in Your Container Image

Putting model weights inside the container image feels tidy: one artifact holds everything the service needs. For small models, that's fine. For models measured in gigabytes, it makes almost everything slower and more expensive.

The problems

Slow pulls on every new node. A 20 GB image has to download before the pod starts. During a traffic spike, that's exactly the delay you can't afford.

Rebuilds for unrelated changes. Fixing one line of Python rebuilds and pushes an image that still carries the unchanged weights, unless your layer caching is perfect.

Registry costs and limits. Many versions of huge images use storage quickly, and some registries limit layer sizes.

Tangled release cycles. Code and models change at different speeds and often have different owners. One image forces them onto the same release schedule.

A better setup

Keep the image small and load weights separately:

• Object storage such as S3, with weights downloaded at startup into a node-local cache.
• Persistent volumes shared read-only across replicas.
• An init container that fetches a specific model version before the server starts.
• OCI artifacts in your registry, versioned separately from the service image.

Pass the model version to the service as configuration, so upgrading a model doesn't require a code release.

Keep versions explicit

Separating weights shouldn't mean losing track of them. Refer to weights by an immutable version or checksum, never by a moving label like latest. Log the model version at startup and with every request.

The strongest objection

"A single image is easier to reproduce." It's simpler, but a pinned image tag plus a pinned weights checksum reproduces the service exactly, while keeping deployments fast.

When baking weights in is fine

• Small models, such as classifiers or embedding models of a few hundred megabytes.
• Air-gapped environments where the registry is the only approved way to move files.
• Edge devices that can't download files at startup.

Getting started

Move the weights of your largest model to object storage, add an init container that downloads them, and compare pod startup times before and after.