Economics

The metrics a steering committee will accept

Model accuracy does not survive contact with a business review. These are the measures that do, and the baseline problem that undermines most of them.

A team presents 94 percent accuracy on a held-out set. The first question is what the old process scored. Nobody measured it. The second question is what 94 percent is worth in money, and the meeting ends without a decision. The engineering was fine. The measurement plan was missing.

Agree the measures before the build

Which measures apply depends on what is being built, and the ones that carry a business review are rarely the ones an engineering team reaches for first.

FamilyWhat to reportWhat makes it credible
Qualitytask completion ratemeasured against a labeled set, not a demo
Economicscost per completed taskincludes retries and escalation
Performancep95 latency, availabilitymeasured at the user, not the endpoint
Adoptionworkflows completed by the systemas a share of eligible volume
Riskescalation and policy violation ratescounted, not estimated
Businesscycle time, cost, capacity releasedagainst a pre-build baseline

The baseline problem

Almost every disappointing review comes down to the same omission. Nobody recorded what the process cost before the system existed, so improvement cannot be stated. Spend a week measuring the current state during discovery. It is the cheapest week in the program and it is the one that makes every later number mean something.

Report adoption honestly

Active users is the metric teams reach for and the one that flatters. A system used once by everybody and twice by nobody looks healthy on that number. Report workflows completed as a share of eligible volume, which is the number that tells you whether the work actually moved.

What to bring to the meeting

  1. The pre-build baseline and how you measured it.
  2. Cost per completed task, with the escalation line shown separately.
  3. Share of eligible volume the system now handles.
  4. The risk numbers, including the cases it refused and the cases it got wrong.
  5. One decision you want, stated in a sentence.

That last line matters more than the rest. A review without a requested decision becomes a status update, and status updates are where programs go quiet.

More field notes

Bring us the problem.

Tell us the outcome you are trying to create, what you have already attempted, and where the constraints are.

Contact nuperX