Why AI Pilots Need Written Success Criteria Before Kickoff

The demo went well. People nodded. Someone said "this is promising." Three months later the pilot is still running, nobody can say whether it worked, and the budget conversation keeps sliding to next quarter. That is pilot purgatory, and the cheapest fix is one page of written AI pilot success criteria, agreed before kickoff.

These stalls are common enough that analysts forecast them. Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027, citing rising costs, unclear business value and weak risk controls. That is a forecast, not a measurement, but "unclear business value" is exactly what a pilot exists to clear up.

Why pilots without a finish line never end

Without a pre-agreed line, every result is arguable. Good numbers prompt "let's widen it to more users." Mediocre numbers prompt "the model needs more tuning." Both answers extend the pilot.

The second cause is private criteria. The sponsor wants a lower cost per ticket, security wants zero leaks, end users want it fast, and you want a reference customer. Each person judges the demo against a different list, so each leaves with a different verdict. Writing one list down moves that argument to week one, when it is cheap, instead of month four, when it is political.

Define success for an AI project in five lines

The agreement needs five things and fits on one page:

1. Metric: one measurable outcome, how it is measured and who measures it.
2. Baseline and threshold: today's number, and the number that counts as a yes.
3. Date: the day the decision is made, not the day the pilot "wraps up."
4. Decision maker: one named person who can say yes and release budget. A committee is not a name.
5. Stop condition: the result or event that ends the pilot early.

Take a hypothetical ticket-triage agent. The metric is the share of tickets routed to the right queue on first assignment, scored weekly on 100 tickets the support lead samples. Humans manage 78% today, and the sponsor agrees 90% earns a production budget. The stop condition is accuracy under 70% after week four, or any payment-data ticket landing in the wrong queue. The numbers are made up, but none of them can be argued about once results arrive.

Kill criteria are what make a pilot honest

A stop condition sounds pessimistic. It does the opposite: it makes "no" a legitimate outcome, so nobody has to dress up a weak result to protect the relationship.

This matters most for a forward deployed engineer embedded with the customer, because you are the one in the room when the numbers are bad. Agreed AI pilot kill criteria turn an awkward judgement call into reading a number out loud. A clean no after six weeks beats a maybe after six months.

But we don't know the metric yet

This is the strongest objection, and it is partly right. In a discovery pilot you may not know what good looks like, and inventing a threshold on day one is false precision. There are three honest answers.

• Make the metric the first deliverable. Write "by day 10, the sponsor and I agree the metric and threshold in writing; if we cannot, the pilot stops." That is still a criterion, with a date and a stop condition.
• Ask for a number, not a perfect one. "What result would make you fund this?" Most sponsors have an answer once someone asks.
• Allow changes before results, never after. Amendments go through the decision maker and get dated. Clinical trials register their endpoints in advance for the same reason: if you can move the goalposts after seeing the data, every result is a win.

Send the one-page agreement this week

Copy this into whatever the customer already reads, fill in the angle brackets, and keep it to one page.

pilot: <pilot name>
sponsor: <customer name and title>
fde_owner: <your name>
kickoff: <YYYY-MM-DD>

metric:
  what: <one measurable outcome>
  measured_by: <who, how often, on what sample>
baseline: <today's number and how it was measured>
threshold:
  go_at: <number that counts as a yes>
  stop_at: <number below which the answer is no>

decision:
  date: <YYYY-MM-DD>
  decision_maker: <one named person with budget authority>
  outcomes: [go to production, extend once with a new date, stop]

stop_early_if:
  - <accuracy below X after week N>
  - <any incident that breaks a data or compliance rule>

changes: Only before results are reviewed, dated, approved by the decision maker.

The outcomes line matters most. It allows one extension with a new date, which stops "extend" from becoming the permanent answer.

The first step is small. Take a pilot you are running or about to start, fill in the template with your best guess at each number, and send it to the sponsor as a proposal. A number they edited is a number they own.

A pilot that ends with a recorded yes, a named owner and a date is the only kind that turns into a production budget, an on-call rota and a handover.