What a production AI system actually costs

Ask what a model costs and you get a price per million tokens. Ask what a deployed system costs and the honest unit is cost per completed task: everything spent to get one business outcome, including the attempts that failed and the person who finished the job when the system could not.

A worked example

A support deflection system handles 40,000 conversations a month. The model is cheap. The two lines that decide the economics are the attempts that did not land and the escalations that reached a person:

The escalation line assumes 12% of conversations reach a person for six minutes at a fully loaded rate. Halving the escalation rate saves more than moving to a model that is ten times cheaper. That is the finding that changes what the team works on next, and it does not appear in a token estimate.

# Cost per completed task, not per request. A task that took three
# attempts and one human review cost all four.
SELECT
    date_trunc('day', started_at)                        AS day,
    count(*) FILTER (WHERE outcome = 'completed')        AS completed,
    sum(input_tokens)  * :in_rate                        AS input_cost,
    sum(output_tokens) * :out_rate                       AS output_cost,
    sum(attempts - 1)                                    AS wasted_attempts,
    sum(review_seconds) / 3600.0 * :loaded_hourly        AS human_cost
FROM agent_runs
GROUP BY 1

The lines people leave out

• Retries. A cheaper model that needs three attempts is not cheaper.
• Evaluation runs. A serious suite runs on every change and costs real money.
• Re-embedding. Changing the chunking strategy means paying for the corpus again.
• The receiving team. A system in production has an on-call rota attached to it.
• Idle capacity, if you provision throughput rather than paying per call.

What to instrument on day one

Record attempts, not requests. Record the outcome of every run as completed, escalated, or abandoned. Record review seconds against the run they belong to. Without those three, you can report spend but you cannot report cost per completed task, and spend on its own does not tell anyone whether to continue.

More field notes