Why Agile Teams Should Run Foundry Agent Evals Every Sprint

Your sprint review has a ritual: someone shares a screen, types three friendly prompts into the agent, and everyone nods. Then a prompt tweak ships on Thursday and nobody can say whether the agent got better or just got different. Foundry agent evals fix that. They are a preview set of azd ai agent eval commands that score your Microsoft Foundry agent against the same recipe every sprint, so the review starts with a number instead of a vibe.

Why a demo makes weak sprint review evidence

Agile teams already accept that "done" needs evidence: a passing test suite, a deployed build. An agent breaks that deal. Change a prompt, swap a model deployment or reword a tool description, and behavior moves in places no unit test covers.

An eval run is the test suite for that behavior. It sends a fixed set of cases to the agent and scores the answers with evaluators, which are scoring rules such as the built-in builtin.task_adherence or a generated rubric. Put a run in your definition of done and the review can answer "is it better than last sprint?" with something other than a shrug.

Set up Microsoft Foundry evaluations once

The azd evaluation flow targets a deployed, invokable hosted agent. You need the azd ai agent extension at version 0.1.40-preview or later, and a chat-completions model deployment in the same Foundry project.

azd ext install azure.ai.agents
azd ext list
azd auth login

azd ai agent eval generate \
  --gen-instruction "<what your agent does and the scenarios to test>" \
  --eval-model <your-chat-model-deployment> \
  --max-samples 15

The service generates a seed dataset and an evaluator rubric, then writes eval.yaml. It accepts 15 to 1,000 samples, so start low and read what it produced.

Keep eval.yaml in source control

name: reservation-agent
agent:
  name: reservation-agent
  kind: hosted
  version: "3"
  config: .agent_configs\\baseline\\metadata.yaml
dataset_reference:
  name: reservation-agent-dev-eval-seed
  version: "1.0"
  local_uri: datasets\\reservation-agent-dev-eval-seed
evaluators:
  - builtin.task_adherence
  - name: reservation-agent-quality
    version: "1"
    local_uri: evaluators\\reservation-agent-quality\\rubric_dimensions.json
options:
  eval_model: gpt-4o
max_samples: 100

This is simplified from Microsoft's docs. The file is the recipe, and eval run uses the versions pinned in it, so run 12 and run 13 compare like with like.

Run azd ai agent eval update after editing a generated dataset or rubric locally: it registers new versions and updates eval.yaml after you confirm, so the change shows up in a pull request and old runs keep their original versions. Commit eval.yaml, and consider the datasets/ and evaluators/ folders too.

Run Foundry agent evals after every deploy

azd deploy
azd ai agent eval run --config eval.yaml
azd ai agent eval list
azd ai agent eval show
# details for one run:
azd ai agent eval show <eval-id> --eval-run-id <run-id>

The output shows the agent version evaluated, the dataset and evaluator versions resolved, whether the run completed, failed or partially completed, the evaluator scores, and token usage. Paste the eval ID and run ID into your sprint review notes.

After a few sprints you have a trend for Azure agent quality, and the review shifts from "it felt snappier" to "task adherence held, but cancellation handling dropped after we shortened the system prompt." A product owner can act on that.

One discipline matters: when you change the dataset on purpose, call it a new baseline.

The objection: generated datasets and preview tooling

Both worries are fair. Microsoft marks the azd evaluation flow as preview, with no service-level agreement and a recommendation against production workloads. And a generated dataset is a model's guess at your agent's scenarios, so check it before trusting scores.

Two answers. First, the eval is a development-time check, not part of the customer path. If the CLI changes, you edit a pipeline line; your eval.yaml and datasets stay in git. Second, treat the generated set as a seed and feed it reality. Each sprint, add the cases that failed in the wild, then register them:

# edit the files under src/<agent-name>/datasets/ and evaluators/, then:
azd ai agent eval update
azd ai agent eval run --config eval.yaml

Microsoft's guidance on synthetic evaluation data is to combine it with trace-based datasets from production: synthetic data covers prelaunch scenarios and a stable regression baseline, while traces show real behavior.

One hard limit to check first: target-based evaluation invokes your hosted agent and works with the responses or invocations protocol in synchronous, non-streaming mode. A2A, Activity, long-running or streaming agents need trace evaluation instead.

Your first step this sprint

Install the extension, run eval generate with a small sample count, and read the rubric dimensions it created. Run the eval once, commit eval.yaml, and add one line to your definition of done: "eval run recorded for the deployed agent version." This sprint's review gets a baseline, and the next one gets a trend.

For anyone shipping AI to production, a pinned, versioned eval recipe is what lets you release agent changes on evidence instead of confidence.