How to Run an Effort Sweep on Claude Haiku 5.5

Claude Haiku 5.5 has an effort dial, so one model ID hides five price and latency profiles. Guessing the setting is how a classification job ends up paying max prices for low problems. By the end of this guide you'll have a script that runs one labeled test set at every Claude Haiku 5.5 effort level, records accuracy, latency and tokens, and picks the cheapest level that passes.

claude-haiku-5-5 shipped on October 7, 2026, and costs from $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens, per Anthropic's launch page. Effort has five levels (low, medium, high, xhigh, max), medium is the default, and no beta header is needed (effort docs).

1. Build a labeled test set from real traffic

Pull a couple hundred real inputs with known-correct answers, and over-represent the ugly ones: ambiguous tickets, typos, two issues in one message. Easy examples pass at every level and teach you nothing. Use one JSON object per line:

{"text": "I was charged twice for March and the invoice PDF is blank.", "label": "billing"}
{"text": "Dashboards have been returning 502 since 09:40 UTC.", "label": "outage"}
{"text": "Can you add SSO group mapping to the admin console?", "label": "feature_request"}

Save it as tickets.jsonl. The sweep scores against these labels, so have a human check them.

2. Set Claude Haiku 5.5 effort with output_config

Effort is one field, output_config.effort. This is a single request at low:

curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-haiku-5-5",
    "max_tokens": 4096,
    "system": "Classify the support ticket. Reply with exactly one label: billing, outage, feature_request, other.",
    "messages": [{"role": "user", "content": "Dashboards have been returning 502 since 09:40 UTC."}],
    "output_config": {"effort": "low"}
  }'

Four things differ from Haiku 4.5, per the migration guide. Thinking is on by default and counts toward max_tokens, so a small limit can stop before any text. Responses can start with a thinking block, so pick the answer by block type, never by position. temperature, top_p and top_k can return an error, so omit them. Assistant prefill returns a 400, so end messages with a user turn.

3. Sweep every effort level and record the numbers

The script runs every ticket three times at each of the five levels and records accuracy, p50 and p95 latency, average output tokens and cost per 1,000 requests. Three passes because you have to omit temperature, so answers can vary run to run.

import json
import statistics
import sys
import time

import anthropic

client = anthropic.Anthropic()

MODEL = "claude-haiku-5-5"
LEVELS = ["low", "medium", "high", "xhigh", "max"]
PASSES = 3
PRICE_IN = 0.10   # USD per million input tokens, prompts up to 100K tokens
PRICE_OUT = 0.50  # USD per million output tokens, thinking included
SYSTEM = (
    "Classify the support ticket. Reply with exactly one label: "
    "billing, outage, feature_request, other."
)


def run_one(effort, ticket):
    start = time.perf_counter()
    resp = client.messages.create(
        model=MODEL,
        max_tokens=4096,
        system=SYSTEM,
        messages=[{"role": "user", "content": ticket["text"]}],
        output_config={"effort": effort},
    )
    latency = time.perf_counter() - start
    answer = "".join(b.text for b in resp.content if b.type == "text")
    return {
        "correct": answer.strip().lower() == ticket["label"],
        "latency": latency,
        "tokens_in": resp.usage.input_tokens,
        "tokens_out": resp.usage.output_tokens,
        "stop": resp.stop_reason,
    }


def summarize(effort, rows):
    n = len(rows)
    latencies = sorted(r["latency"] for r in rows)
    mean_in = sum(r["tokens_in"] for r in rows) / n
    mean_out = sum(r["tokens_out"] for r in rows) / n
    return {
        "effort": effort,
        "accuracy": sum(r["correct"] for r in rows) / n,
        "p50": statistics.median(latencies),
        "p95": latencies[int(0.95 * (n - 1))],
        "mean_out": mean_out,
        "cost_per_1k": (mean_in * PRICE_IN + mean_out * PRICE_OUT) / 1_000_000 * 1000,
        "bad_stops": sum(r["stop"] != "end_turn" for r in rows),
    }


if __name__ == "__main__":
    with open(sys.argv[1]) as f:
        tickets = [json.loads(line) for line in f]
    results = []
    for effort in LEVELS:
        rows = [run_one(effort, t) for _ in range(PASSES) for t in tickets]
        results.append(summarize(effort, rows))
        print(json.dumps(results[-1]))

Run it with python sweep.py tickets.jsonl. It leaves thinking unset on purpose, because thinking: {"type": "disabled"} returns a 400 at xhigh and max. usage.output_tokens is the billed number, thinking included, even though the thinking text is omitted from the response by default (thinking docs). Any reply that doesn't stop at end_turn lands in bad_stops, and an empty answer scores as wrong.

4. Compare low vs high effort and pick the cheapest pass

Add a picker and call it at the bottom of the script. Set the floor and the latency budget from your requirements, not from the results you just saw.

def pick(results, floor=0.95, p95_budget=2.0):
    passing = [
        r for r in results
        if r["accuracy"] >= floor and r["p95"] <= p95_budget and r["bad_stops"] == 0
    ]
    return min(passing, key=lambda r: r["cost_per_1k"], default=None)


print("winner:", pick(results))

Illustrative numbers at Claude Haiku 5.5 pricing: 800 input tokens per request, 100 output tokens at low and 900 at high. A million requests cost $130 at low and $530 at high, so the output side does all the moving. If the job isn't latency-bound, the Batch API takes 50% off (models overview).

If only xhigh or max clears your floor, rerun with MODEL = "claude-sonnet-5-5" before you commit. Sonnet 5.5 lists at $2 input and $10 output per million tokens, 20 times Haiku's base rates, and Anthropic's prompting guide says to compare at those levels. Hardcode the winner in output_config rather than trusting the default, and rerun the sweep when the prompt or model changes.

Gotchas

• Long prompts cost more. Prompts over 100,000 tokens bill at $0.50 input and $2.50 output per million tokens (pricing). Set PRICE_IN and PRICE_OUT to those rates for long-document sweeps.
• Recount your tokens. The tokenizer produces roughly 30% more tokens than Haiku 4.5 on the same text, so old budgets and cost estimates run low.
• Don't flip top-level effort inside a cached conversation. It invalidates the prompt cache.
• Handle empty replies and refusals. At xhigh in multi-turn chats the model sometimes writes its whole answer inside thinking and returns no visible text, and safety classifiers can return stop_reason: "refusal".
• Check your platform's price list. The prices above are Anthropic API prices, and the model is also on Amazon Bedrock (anthropic.claude-haiku-5-5), Google Cloud and Microsoft Foundry.

On a high-volume job, effort is one of the biggest cost dials you control, and a sweep sets it from evidence instead of habit.