Why Agile Teams Should Evaluate Foundry Model Router Early

Sprint planning has a favorite time sink: "Which model do we deploy?" Someone writes a spike, someone brings a benchmark chart, and by the time you decide, a newer model has shipped. If you build on Azure, point Foundry model router at your own prompts in the first sprint and let the numbers do the arguing.

What Foundry model router changes

Model router is a trained model you deploy like any other Foundry model. For each request it picks an underlying model from a pool of more than 30 spanning OpenAI, Anthropic, xAI, DeepSeek and Meta, behind one deployment name. Balanced mode (the default) picks the cheapest model within a small quality band of the best one for that prompt, Cost widens the band, and Quality ignores price.

Why evaluate Foundry model router in sprint one

The single-model debate has a short shelf life, because the latest router version is updated in place as new models arrive. A model picked after two sprints of debate is a snapshot of a moving target.

The evaluation pays off either way. Any model decision needs a representative prompt set, a baseline deployment and written acceptance criteria, so you keep them even if the router loses. Microsoft's guide to evaluating model router frames tuning as hill climbing: baseline, measure, change one lever, keep the change only if it helps. That is a sprint-sized loop.

Write the acceptance criteria first

The guide names four dimensions: quality (a minimum score or win rate), cost (a maximum per request or a minimum saving), latency (median plus p90 or p95) and policy (which models and regions you may use). Put numbers on them in the story. "Seems fine" is not a definition of done.

Group prompts by category and include long inputs and edge cases, so an average can't hide one category's regression. Keep system prompts and output limits identical across both deployments.

Deploy Foundry model router and run the comparison

Start with Balanced mode and the full model set, as the guide suggests. Check the supported regions first: Global Standard is listed everywhere the router is offered, Data Zone Standard is not. This REST call, from the model router how-to, creates the default deployment:

export AZURE_AI_AUTH_TOKEN=$(az account get-access-token --resource https://management.azure.com --query accessToken -o tsv)

curl -X PUT "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.CognitiveServices/accounts/<foundry-account>/deployments/model-router-deployment?api-version=2025-10-01-preview" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AZURE_AI_AUTH_TOKEN" \
    -d '{
        "sku": {"name": "GlobalStandard", "capacity": 10},
        "properties": {
            "model": {"format": "OpenAI", "name": "model-router", "version": "2025-11-18"}
        }
    }'

Before any scoring, see where your prompts land, with one prompt per line in prompts.txt:

import os
from collections import Counter

from openai import AzureOpenAI  # pip install "openai>=1.75.0"

client = AzureOpenAI(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
    api_version="2024-10-21",
)

with open("prompts.txt", encoding="utf-8") as f:
    prompts = [line.strip() for line in f if line.strip()]

served_by = Counter()
for prompt in prompts:
    response = client.chat.completions.create(
        model="model-router-deployment",
        messages=[{"role": "user", "content": prompt}],
    )
    served_by[response.model] += 1  # the underlying model that answered

for model, count in served_by.most_common():
    print(f"{model}: {count}/{len(prompts)}")

The response's model field names the underlying model that answered, so this tally shows your workload's model mix, not quality.

For quality, cost and latency, use Microsoft's Model Router Auto Evaluation toolkit and its quickstart, which compares the router with your baseline using a judge model:

git clone https://github.com/microsoft-foundry/Model-Router-Auto-Evaluation.git
cd Model-Router-Auto-Evaluation
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
cp .env.example .env   # add endpoints and keys for router, baseline and judge

python scripts/run_eval.py --config configs/quick_test.yaml --dataset datasets/sample_custom.jsonl

The config needs a pricing section keyed by deployment name; if your baseline doesn't match a key, its cost shows as $0.00. Open results/<run-name>/dashboard.html for the report.

The strongest objection: less predictable cost and behavior

This one is fair. The router can send similar prompts to different models, so cost per request varies. The context window is that of the smallest model in the pool, and prompt caching only helps when the same model handles consecutive requests.

That is a reason to measure, not to skip. The evaluation reads quality, cost, latency and model distribution together, and a model subset pins the pool. It doubles as the fallback set, so list at least two models:

{
  "sku": {"name": "GlobalStandard", "capacity": 10},
  "properties": {
    "model": {"format": "OpenAI", "name": "model-router", "version": "2025-11-18"},
    "routing": {
      "mode": "balanced",
      "models": [
        {"format": "OpenAI", "name": "gpt-4.1", "version": "2025-04-14"},
        {"format": "OpenAI", "name": "gpt-5.6-sol", "version": "2026-07-09"},
        {"format": "Meta", "name": "Llama-4-Maverick-17B-128E-Instruct-FP8", "version": "1"}
      ]
    }
  }
}

Send this as the body of the same curl call. Deploy any Claude models to the same account first, or the request fails with InvalidResourceProperties. Mode and subset changes take up to five minutes to apply, so wait before rerunning. Keep a direct deployment for requests that need deterministic model selection.

Your first step this sprint

Add a two-day story: "Evaluate model router against our current model." Collect 50 to 100 representative prompts with personal data removed, write the four acceptance criteria, deploy with defaults, run the quick test, then your own dataset, and read results by category. If it clears the bar, test it on limited production traffic; if not, change one lever and rerun, or keep the direct deployment with evidence instead of opinions.

A model choice backed by your own prompts and a written acceptance bar is one you can defend, and cheaply revisit, when the pool, the pricing or the traffic changes in production.