Why Agile Teams Should Pilot watsonx Orchestrate With Evals

The demo went great. Then someone asked, "But is it any good?", and the team answered with vibes. The fix is watsonx Orchestrate evals: treat the agent pilot as a time-boxed sprint and give it what agile teams already trust, acceptance criteria you can run. IBM's August 2026 releases make that practical.

What IBM shipped in August for watsonx Orchestrate evals

IBM's August 2026 update lists three pieces that fit a sprint:

• Trace Inspector (GA 17 August) shows the full execution path of an agent run, so you can see which tools were called and where behavior diverged.
• Custom LLM-as-a-Judge (GA 31 August) lets you define evaluation criteria at build time in natural language. An LLM-as-a-judge is a second model that grades the agent's answers against rules you wrote.
• AgentOps agent (GA 31 August) runs the improvement cycle through conversation: test case creation, user simulation, root cause analysis and optimization.

Map them to your ceremonies. Criteria become the definition of done, traces become the sprint review, and the AgentOps agent feeds the retro.

Write acceptance criteria before the first prompt

You would never start a story without acceptance criteria. Agents deserve the same rule. Say the pilot agent summarizes sprint health from Jira, which CData's guide to connecting Jira Cloud to watsonx Orchestrate shows how to wire up over MCP. Write three kinds of criteria on day one.

Deterministic checks as a ground truth file

The ADK evaluation framework takes ground truth datasets as JSON: the tool calls you expect, with arguments, then the text the answer must contain.

{
  "agent": "sprint_summary_agent",
  "goals": {
    "get_sprint_issues-1": ["summarize"]
  },
  "goal_details": [
    {
      "type": "tool_call",
      "name": "get_sprint_issues-1",
      "tool_name": "get_sprint_issues",
      "args": { "sprint_id": "<your-sprint-id>" }
    },
    {
      "name": "summarize",
      "type": "text",
      "response": "<expected one-line sprint summary>",
      "keywords": ["<completed count>", "carried over"]
    }
  ],
  "story": "You are a product owner preparing for the sprint review. The sprint ID is <your-sprint-id>.",
  "starting_sentence": "How did the last sprint go?"
}

The tool name, sprint ID and expected values are placeholders, so swap in your own. The story and starting_sentence describe the user the evaluator plays.

Run it as a repeatable gate

test_paths:
  - evals/sprint_summary/
output_dir: evals/results
n_runs: 3
metrics:
  - JourneySuccessMetric
  - ToolCalling
  - StepMetrics
  - AgentResponseTime

orchestrate evaluations evaluate --config evals/config.yaml

The metric names come from IBM's example config. n_runs: 3 repeats each case because model output varies between runs. According to the ADK docs, since version 1.12.0 the command also works against SaaS and on-premises instances, not only the local Developer Edition.

Judgment criteria in plain language

Some quality can't be a keyword match. That is where the custom judge goes. IBM's announcement says you write these in natural language but does not publish a config format, so this is wording to adapt, not a file to paste:

Fail the answer if it names a ticket that was not in the tool output.
Fail the answer if it calls a carried-over story "completed".
Pass only if a product owner could read it aloud in a sprint review without edits.

Run the evals daily and open traces in the sprint review

Report the pass rate at standup like any other sprint number. When a case fails, open the run in Trace Inspector instead of arguing about what the agent probably did. You can also pull traces from the CLI with orchestrate observability traces, per the ADK tracing docs.

In the sprint review, show two failing traces next to the demo. Stakeholders trust a team that shows its failures. In the retro, ask the AgentOps agent for root causes and proposed optimizations, then decide as a team which to accept.

The strongest objection: the judge is also a model

A natural-language judge can be wrong, and a pilot graded by an unreviewed model is a vibe with a number attached. Fair.

Keep the deterministic checks as the backbone and treat the judge as a second opinion. In sprint one, read ten traces by hand and compare your verdicts with the judge's. Where you disagree, rewrite the criterion until you agree. That calibration takes an afternoon, which is cheaper than a stakeholder losing faith at launch.

Limits to know before you commit

• IBM says the AgentOps agent starts with watsonx-native agents, and that compatible external LangGraph agents using the agentic SDK follow in September. Check the current docs before assuming your agent qualifies. The ADK also has a validate-external subcommand for external agents.
• Ten cases is a smoke test, not a benchmark. Evals only measure what you wrote down.
• The announcement gives GA dates but no region or plan details, so confirm those with your IBM account team.

Your first step this week

Pick one narrow agent. In sprint planning, write ten ground truth cases and three judge criteria, then make "evals pass" part of the definition of done. A pilot that ends with a pass rate and a trace archive gives whoever runs this in production something to check against, and a demo gives them only your word.