How to Estimate AI Feature Work With Time-Boxed Spikes

Ask three engineers to size "summarize support tickets with an LLM" and you'll get 3, 8 and 21 points. All three are guessing, because nobody knows yet whether the model is right often enough to ship. To estimate AI feature work honestly, run a time-boxed spike against a small eval set, then price the build from the measured gap.

You'll finish with a target, a baseline score, an estimate you can defend and a ticket template.

Why story points fail on LLM features

Normal features have uncertain effort. LLM features have uncertain feasibility, and the effort depends on how far the output is from acceptable, which you only see by measuring it. A spike is a time-boxed investigation that produces knowledge instead of shipped software.

Estimate AI feature work in six steps

1. Write the pass-rate target before you touch a prompt

Agree one number and one definition of "pass" with the customer: "90% of tickets get a summary a support lead would send unedited." List the must-not-fail cases, like a wrong refund amount, and score them separately, because an average hides them. Anthropic's guide to defining success criteria and building evaluations makes the same case for measurable goals.

2. Build a 30 to 50 case eval set from real data

Pull cases from real tickets or documents, not the three tidy demo examples. Include the ugly ones: typos, empty fields, another language, the ticket everyone dreads. Have a domain expert write the expected answers.

{"id": "t-014", "category": "refund", "input": "<real ticket text>", "expected": "refund of 42.10", "must_not_fail": true}

One case per line in evals/spike_cases.jsonl. Move a third into evals/holdout.jsonl and leave it alone until the last day. A set this small can't prove 90%, but it shows the size of the gap.

3. Put the time box in the spike ticket

Two days for one prompt and one model, three if retrieval or tool calls are involved. When the box ends, report what you found, even mid-idea. Paste this into your tracker:

## Spike: <feature name>

**Time box:** 3 working days. When it ends we report, even if unfinished.
**Question:** Can <feature> reach <target> on <definition of pass> using <model/approach>?
**Target:** <90>% pass on the eval set. Must-not-fail cases scored separately: <list>.
**Eval set:** <40> real cases in `evals/spike_cases.jsonl`, <12> held out in `evals/holdout.jsonl`. Labeled by <name>.
**In scope:** simplest prompt, <model>, retrieval if needed, 3 runs per case.
**Out of scope:** UI, auth, production wiring, prompt polish after day 2.

**Deliverable (one written page):**
- Baseline pass rate as a range over 3 runs, plus the held-out pass rate
- Failure buckets with counts and a proposed fix for each
- Cost and latency per request at baseline
- Build estimate as a range, with the assumption it depends on
- Recommendation: build / re-scope / stop

**Stop rules agreed with <customer contact>:** <paste from step 6>

4. Run the eval and sort failures into buckets

Run this on day one, before any tuning, to get a baseline. It scores the set three times because LLM output varies.

import json
from collections import Counter
from pathlib import Path
from typing import Optional

RUNS = 3  # model output varies, so score the whole set more than once


def run_feature(case_input: str) -> str:
    """Call your prototype here: prompt, model and retrieval if the feature needs it."""
    raise NotImplementedError


def grade(case: dict, output: str) -> Optional[str]:
    """Return None on a pass, or the name of the failure bucket."""
    if case["expected"].lower() not in output.lower():
        return "wrong_answer"
    if len(output) > case.get("max_chars", 1500):
        return "too_long"
    return None


def main(path: str = "evals/spike_cases.jsonl", target: float = 0.90) -> None:
    lines = Path(path).read_text(encoding="utf-8").splitlines()
    cases = [json.loads(line) for line in lines if line.strip()]
    run_rates, buckets = [], Counter()
    for _ in range(RUNS):
        passed = 0
        for case in cases:
            failure = grade(case, run_feature(case["input"]))
            if failure is None:
                passed += 1
            else:
                buckets[failure] += 1
        run_rates.append(passed / len(cases))
    worst = min(run_rates)
    print(f"pass rate {worst:.0%} to {max(run_rates):.0%} over {RUNS} runs, target {target:.0%}")
    print(f"gap to target (worst run): {target - worst:.0%}")
    for name, count in buckets.most_common():
        print(f"{name}: {count / RUNS:.1f} cases per run")


if __name__ == "__main__":
    main()

The substring grader is a placeholder. Swap in exact match, a schema check, or a model-based grader you've spot-checked by hand. Then read every failure and name the buckets by cause: wrong format, missing context, invented detail, misread jargon.

5. Turn the measured gap into story points

Estimate two things separately. First, the fix for each failure bucket. Your team has done these jobs before, so points work again. Sizes below are examples; use your own history:

| Failure bucket | Typical fix | Size | | --- | --- | --- | | Wrong output format | Structured output plus a validator | Small | | Misread domain jargon | Glossary and few-shot examples | Small | | Retrieval misses the right document | Chunking, indexing and reranking work | Medium | | Invented detail on a must-not-fail case | Verification step or human review fallback | Medium to large |

Second, the fixed work (integration, auth, logging, deployment, the eval in CI), estimated like any feature.

Quote a range with its assumption attached: "8 to 13 points, if these fixes lift the pass rate from 71% to 90%." Starting heuristic: a baseline within 10 points of target is usually prompt and validation work. A 10 to 30 point gap means retrieval or data work and a wider range. Beyond 30 points with failures spread across many buckets, don't estimate; take it to the customer as a stop decision.

6. Agree the stop points with the customer in writing

• Before the spike: no agreed target or no labeled cases means don't start.
• End of the spike: a huge gap, or cost and latency already over budget at baseline, means stop or re-scope.
• Held-out check: a dev score far above the held-out score means you tuned to the test.
• One third into the build: if the pass rate hasn't closed about half of its planned gain, the customer chooses to continue, re-scope or stop.

Gotchas

• Report the range across runs, not the best run.
• Spike code is a prototype. Never count it as already built.
• On 40 cases, one case moves the score by 2.5 points. Don't chase single-digit changes.

Estimate from the measured gap and let the customer stop while stopping is cheap, because in production the costly mistake is a month of building toward a pass rate nobody measured.