How to Store and Rotate LLM API Keys With AWS Secrets Manager

Model provider keys are valuable and easy to misuse: anyone holding one can run up a large bill or read your usage data. Yet they often live in environment files, CI variables and old chat messages. AWS Secrets Manager gives them a single, audited home.

Step 1: Store each key once

Create one secret per provider and environment:

aws secretsmanager create-secret \
  --name prod/support-agent/model-provider-key \
  --secret-string file://key.json

Delete the local file afterwards. Use separate keys for development, staging and production, so a leaked development key can't affect production usage.

Step 2: Grant read access narrowly

Give each application's role secretsmanager:GetSecretValue on its own secret only. Avoid policies that allow reading every secret in the account.

Step 3: Load keys at runtime

Read the secret when the service starts and cache it in memory with a refresh interval, so a rotated key is picked up without a redeploy. AWS publishes caching clients for several languages. On ECS and EKS, you can also inject secrets through task definitions or the Secrets Store CSI Driver, as long as rotated values still reach the running service.

Step 4: Plan rotation

Secrets Manager can rotate secrets automatically with a rotation function. For model providers, full automation only works when the provider offers an API for creating and revoking keys. Otherwise, rotate on a schedule with a short runbook:

1. Create a new key in the provider's console.
2. Store it as a new version of the secret.
3. Wait for services to refresh, and confirm requests succeed with the new key.
4. Revoke the old key.

Keep both keys valid during the switch, so there's no outage.

Step 5: Monitor use

CloudTrail logs every GetSecretValue call. Alert on reads from unexpected roles, and check the provider's usage dashboard for traffic you can't explain.

Things to watch

• Debug logging. Some SDKs print request details, including headers, when debug logging is on. Keep it off in production.
• CI pipelines. Evaluation jobs need keys too. Give them a separate key with a low spending limit.
• Leaks still happen. Turn on secret scanning in your repositories, so a committed key is caught within minutes.

Move the production key first, then remove it from every other place it was stored.