Why AI Pipelines Need Versioned Data as Well as Versioned Code
Model and retrieval behavior depend on data as much as code. Without versioned data, you can't reproduce results or explain a regression.
How-to guides, the case for better engineering practices, and the AI news worth knowing, for engineers shipping AI into production.
Model and retrieval behavior depend on data as much as code. Without versioned data, you can't reproduce results or explain a regression.
Re-embedding millions of documents after data loss can take days and cost a lot. Back up vector stores properly and practice the restore.
When an engagement ends, the customer inherits your infrastructure. Code they can read, plan and apply is far easier to own than console clicks.
An unpinned dependency can change your AI service's behavior overnight. Lock every package with hashes so each build is identical.
Every team building an AI service reinvents keys, logging, evals and deployment. A supported template gives them safe defaults in minutes.
A forgotten GPU instance or a looping agent can burn a month's budget in days. AWS Budgets and anomaly detection catch it early.
Embedding backfills, evaluations and fine-tuning jobs rarely need guaranteed capacity. Spot instances run them at a large discount.
Written platform rules get skipped under deadline pressure. Kyverno policies make the cluster itself reject workloads that break them.
Anyone who can write to your registry or bucket can swap an image or a model. Signing lets production check where artifacts came from.
AI images carry large dependency trees with known vulnerabilities. Scan them in CI, fail on critical issues and record what you accept.
Copying model files between environments by hand loses track of what's deployed. MLflow's registry and aliases make promotion explicit.
Average latency hides the slow requests users complain about. Percentiles and time to first token show what users actually experience.
AI incidents rarely have a single cause. A blameless postmortem finds the gaps in prompts, tests and controls, not a person to blame.
Provider status pages often update after users feel the problem. Probe the APIs yourself and alert on your own error and latency data.
Without prompt and response logs you can't debug complaints or show what the assistant said. Log them with redaction, limits and access rules.
Most bad RAG answers start with bad retrieval. Measure what gets retrieved, how often nothing useful comes back and whether answers use it.
AI failures don't look like normal outages. A runbook written in advance tells on-call engineers what to check and when to switch things off.
Monthly invoices arrive too late to catch a runaway prompt. Export token counts as metrics and watch cost per route in Grafana every day.
An AI feature can be up and fast while still giving bad answers. Add a quality objective next to availability and latency, and measure it daily.
When an AI request is slow or wrong, you need every model call and tool call in order. OpenTelemetry traces show exactly that.
Model API keys end up in env files, CI variables and chat threads. Keep them in Secrets Manager, load them at runtime and rotate them.
Manual console changes quietly make Terraform code wrong. A scheduled plan catches drift early, before the next apply surprises someone.
AI spending grows faster than most teams expect. Consistent tags from the first day show who spends what and make cleanup safe.
AWS offers three common homes for AI services. Match the choice to what you run, who operates it and how much control you need.
Functions seem ideal for calling a model API until timeouts, streaming and agent loops arrive. When serverless fits and when it doesn't.
One huge Terraform folder slows every change and worries reviewers. Split an AI platform into small modules with separate state per layer.
Model endpoints have regional outages and capacity limits like any cloud service. Failover to a second region keeps AI features working.
Security teams often require model traffic to stay on private networks. VPC endpoints keep Bedrock calls inside AWS, with policies on top.
When every service calls model APIs directly, keys, costs and outages are scattered. A gateway gives you one place to control them all.
Granting Bedrock access by clicking through the console doesn't scale or survive audits. Manage roles, policies and logging in Terraform.
Wrong requests and limits cause throttled Python services, killed pods and wasted nodes. Set them from measured usage, not guesses.
Training jobs and inference services want different hardware, pricing and disruption rules. Separate node pools give each what it needs.
A model-serving pod can take ten minutes to become ready. Measure each startup phase, then shorten the slow ones.
Baking gigabytes of model weights into images slows every deploy and scale-up. Keep code and weights apart and load weights at startup.
Karpenter launches the GPU instance a pending pod needs and removes it when idle. A NodePool setup for AI workloads on EKS.
Helm charts for AI services often become copies of copies. A small chart with a clear values file is far easier to hand over.
With GitOps, a Git repository describes what should run and Argo CD keeps the cluster matching it, so every change is visible and reversible.
A GPU inference server can be overloaded while its CPU sits idle. Scale on queue length and latency instead, and plan for slow startups.
vLLM runs open-weight models behind an OpenAI-compatible API. A working Kubernetes deployment and the settings that matter most.
GPUs are the most expensive part of your cluster and the easiest to leave idle. Taints, explicit requests and sharing keep them busy.
Long-lived branches hide prompt and model changes from each other until merge day. Small AI teams move faster by merging to main daily.
AI service images often grow to several gigabytes. A multi-stage build, a slim base image and a lock file keep them small and quick to deploy.
A temporary environment for each pull request lets reviewers try an AI change themselves instead of guessing from a prompt diff.
Hand-written test cases miss how people really use your assistant. Replaying sampled production requests shows what a change will do.
Stop saving AWS access keys as GitHub secrets. OIDC gives each workflow run short-lived credentials for one role, limited to one repository.
Feature flags let you release AI features to a few users first, compare models side by side and turn things off without a deploy.
When an AI change goes wrong, a full redeploy is too slow. Set up configuration and flags so switching back takes a single action.
Tests can't predict every way real users will react to a model change. A canary release limits the damage while you find out.
Prompts change product behavior as much as code does. Give them versions, reviews, tests and a changelog so every change is traceable.
An LLM service needs the usual build and test stages plus two more: an eval gate and a staged rollout. A pipeline you can copy.
An automated Claude reviewer can catch real bugs or flood pull requests with minor comments. How to configure it so developers find it useful.
You can usually lower a Claude bill without switching to a weaker model. Start with caching, batching and smaller inputs.
A million-token context window doesn't mean you should fill it. Large prompts increase cost and latency and can reduce accuracy.
Agents that run code and call internal APIs need the same isolation as any untrusted workload. A seven-step plan for enterprises.
Skills store a team's know-how in folders Claude loads only when a task needs them, so procedures are followed the same way every time.
Prompts that work in a demo often fail with real users. These five patterns help prompts cope with messy, unexpected input.
Risk reviewers will ask how Claude decides what it will and won't do. Constitutional AI answers part of that question, not all of it.
Instructions in a prompt are followed most of the time. Hooks run every time, so use them for formatting, blocked commands and checks.
Claude can read scanned invoices and forms. Pairing it with structured output, validation and human review makes the results reliable.
Evaluations show whether a prompt or model change actually helped. This starter template gets a basic eval running in an afternoon.
Rate limits rarely appear during a pilot but often do at launch. Which limit you'll hit first, and how to design your system for it.
Vague requests lead Claude to make its own assumptions. Ask it to list them first, so stakeholders can correct them before work starts.
Artifacts let Claude build working internal tools, like calculators and checklists, from a plain description in a single conversation.
Porting the API calls from GPT-4 to Claude takes an afternoon. Adjusting prompts and re-running evaluations takes most of the effort.
Subagents keep your main Claude Code session focused. The key decision is whether each one should start fresh or inherit your context.
When Claude picks the wrong tool, sends bad arguments or gets stuck in a loop, the cause is usually in your setup. Check these eight things.
Some business systems have no API, only an old web interface. Computer use lets Claude work in them, provided you isolate it carefully.
A production system prompt should read like a specification. This structure keeps it clear even when several teams edit it.
Before building a retrieval pipeline, check whether a Claude Project already covers the need. For internal teams it often does.
Most Claude requests resend the same instructions and documents. Prompt caching lets you pay much less for those repeated tokens.
A support agent built on Claude needs three tools, a simple loop and clear rules about what it can't promise customers.
Adaptive thinking improves Claude's answers on hard problems but adds time and cost. Use higher effort only where it measurably helps.
Claude starts every session from nothing. A small, organized folder of memory files is the most dependable way to fix that.
Run Claude Code non-interactively in CI to summarize pull requests and explain failed builds, with limited tools and no risky permissions.
Sonnet 5 costs less and handles most agent work. Opus 5 is better at long, hard tasks. How to decide using your own numbers.
Why the FDE role exists, who's hiring, what it pays, and what the market actually screens for — the evidence behind the Forward Engineer positioning.