How to Calculate Cost per Successful Task for an AI Feature

Cost per call looks great on a dashboard, right up until finance asks what a finished job costs. Retries, failed runs and tool calls all land on the bill, but only the successes count as work done. By the end of this guide you will have cost per successful task for your AI feature: a call log, an outcome label and a short Python script a product manager can adapt by changing the price table and the definition of success.

1. Decide what counts as a successful task

A task is one user-visible job: answering a ticket, drafting a reply, extracting an invoice. Success comes from the business, not the model. Examples: the ticket was not reopened within seven days (cost per resolved ticket), the draft was sent without edits, or the extracted fields passed validation.

This is the whole of AI feature unit economics in one line:

cost per successful task = cost of ALL tasks (wins, failures, retries) / number of successful tasks

Failures stay in the numerator because you pay for them either way. The FinOps X session on correlating AI spend with business outcomes covers the same idea of linking LLM telemetry to business KPIs.

2. Log every call with a task ID

Create one ID when the task starts and reuse it on every retry, tool call and sub-agent call. Without it you cannot group spend by outcome. Take token counts from the usage data your provider returns with each response (field names differ by provider, so check yours). If your provider bills cached or reasoning tokens at different rates, log them as separate columns.

import csv
import os
import uuid

FIELDS = ["task_id", "attempt", "model", "input_tokens", "output_tokens", "tool_usd"]

def log_call(**row):
    new_file = not os.path.exists("llm_calls.csv")
    with open("llm_calls.csv", "a", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=FIELDS)
        if new_file:
            writer.writeheader()
        writer.writerow(row)

task_id = str(uuid.uuid4())  # once per task, reused on every retry and tool call
log_call(task_id=task_id, attempt=1, model="model-small",
         input_tokens=2000, output_tokens=500, tool_usd=0.001)

In production, write the same fields to your existing logs or tracing store. The script below only needs one row per call.

Whoever knows the result writes the outcome later: a CRM webhook, a validator, a thumbs-up button. One row per task:

task_id,variant,resolved,review_minutes
<task-id>,model-small,0,6

variant records which model or prompt handled the task. review_minutes is human time spent fixing or finishing it (used in step 6). A task with no outcome after your window closes gets resolved=0, not a missing row.

3. Price each call and roll up per task

import pandas as pd

# USD per 1M tokens as (input, output). Placeholders: copy real prices from your provider's pricing page.
PRICE_PER_MTOK = {
    "model-small": (0.50, 2.00),
    "model-large": (3.00, 15.00),
}

calls = pd.read_csv("llm_calls.csv", dtype={"task_id": str})

def call_cost(row):
    price_in, price_out = PRICE_PER_MTOK[row["model"]]
    token_usd = (row["input_tokens"] * price_in + row["output_tokens"] * price_out) / 1_000_000
    return token_usd + row["tool_usd"]

calls["cost_usd"] = calls.apply(call_cost, axis=1)

per_task = calls.groupby("task_id").agg(
    cost_usd=("cost_usd", "sum"),
    calls=("cost_usd", "size"),
).reset_index()

Every attempt is a row, so a task that retried four times carries four calls of cost. That is exactly what a per-call average hides.

4. Join outcome labels and compute cost per successful task

outcomes = pd.read_csv("task_outcomes.csv", dtype={"task_id": str})

tasks = per_task.merge(
    outcomes, on="task_id", how="outer", validate="one_to_one", indicator=True
)
print(tasks["_merge"].value_counts())

labeled = tasks[tasks["_merge"] == "both"]

report = labeled.groupby("variant").agg(
    tasks=("task_id", "size"),
    resolved=("resolved", "sum"),
    total_usd=("cost_usd", "sum"),
    calls=("calls", "sum"),
)
report["usd_per_call"] = report["total_usd"] / report["calls"]
report["usd_per_task"] = report["total_usd"] / report["tasks"]
report["cost_per_successful_task"] = report["total_usd"] / report["resolved"]
print(report.round(4))

In the pandas merge call, validate="one_to_one" raises an error if either side repeats a task ID. indicator=True adds a _merge column, so the printed counts show calls with no label (left_only) before you drop them.

5. Read the result: cheap per call is not cheap per outcome

Here is a made-up sample of 10 tasks per variant. Every call uses 2,000 input tokens, 500 output tokens and $0.001 of tool cost, at the placeholder prices above. The small model loops four times per task. The large model usually answers once.

| Variant | Cost per call | Cost per task | Resolved | Cost per successful task | | --- | --- | --- | --- | --- | | model-small | $0.0030 | $0.0120 | 3 of 10 | $0.0400 | | model-large | $0.0145 | $0.0174 | 9 of 10 | $0.0193 |

The small model is nearly five times cheaper per call and about twice as expensive per successful task. That gap is the LLM cost per outcome, and it is the number to put in front of a roadmap review.

6. Add human time to see the full cost per resolved ticket

REVIEW_USD_PER_MIN = 48 / 60  # replace 48 with the loaded hourly cost of your reviewer

labeled = labeled.assign(
    full_usd=labeled["cost_usd"] + labeled["review_minutes"] * REVIEW_USD_PER_MIN
)
by_variant = labeled.groupby("variant")
print(by_variant["full_usd"].sum() / by_variant["resolved"].sum())

At a placeholder $48 an hour, six minutes of review costs $4.80, hundreds of times the model spend per task in the sample. When people clean up after the model, they are usually the biggest line.

Gotchas

• Label lag: only compute on cohorts old enough to have outcomes, or your newest tasks look like failures.
• Missing calls: SDK retries, gateways and sub-agents can spend tokens you never logged. Reconcile your total against the provider invoice every month.
• Tiny samples: ten tasks per variant is a demo. Collect enough that one flipped label does not change the answer.
• Averages hide runaway loops: also check labeled["cost_usd"].quantile(0.95).
• Fixed costs: vector database, hosting and monitoring are not in this number. Add an allocated per-task share if they matter.
• Moving goalposts: if you change the success rule, start a new trend line.

Once finance asks what a finished job costs, you will have a number that survives a model swap, because it already counts the failures you paid for.