Metrics
Sixteen numbers, and where each one comes from
Most are counted as your calls arrive, into one table keyed by the hour, the tool and the client — so a single day is drawn by the hour, and every figure narrows to one model as easily as to one tool. Three need a night's worth of calls in order and are labelled as of yesterday, because pretending otherwise would be the first lie the product told you.
Live on ingest
Counted as each payload lands.
11 metrics
- 01
Calls per tool
Which of your tools the model actually reaches.
- 02
Calls per day
The shape of a range — by the hour when you ask for a single day, drawn flat at zero when nothing happened, never left blank.
- 03
Which client
Claude Desktop, Cursor, or something you did not expect — with its own outcomes, latency and first-call rate, not just a share of the total.
- 04
Crashes
Your handler threw. A bug list, sorted by tool.
- 05
Tool errors
You returned isError deliberately. Healthy in small doses.
- 06
Bad arguments
Validation rejected the call. Usually your schema, not the model.
- 07
Empty answers
Succeeded and returned nothing usable. The failure nobody reports.
- 08
Speed
Four buckets, because percentiles cannot be summed across days.
- 09
Result size
Bytes per answer, which is what context actually costs.
- 10
Sessions
Counted from their own rows, so one crossing midnight counts once.
- 11
Cost per session
What a single conversation with your server spends.
Nightly pass
02:00 UTC. Labelled as of yesterday, because they are.
3 metrics
- 12
Retries
Same tool, inside 30 seconds, with different arguments.
- 13
First-call success
Asked once, got something usable, moved on.
- 14
Tool pairs
Which tools get called one after another.
From the startup payload
Sent once, when your server boots.
2 metrics
- 15
Schema size
What each tool costs in context before anyone asks a question.
- 16
Dead tools
Registered, described, and never once called.
Retries are the hard one, and the one worth having
Within a session, the same tool called twice inside 30 seconds with different arguments is the model rewording and trying again. With the same arguments it is pagination or polling, and it is not a retry. Get that distinction wrong and every well-behaved paginating tool grades as broken.
That is the whole reason arguments are hashed. Not to inspect them — twelve hex characters of
a SHA-256 with the keys sorted is not reversible and is not meant to be. It answers exactly
one question: were these two calls the same? Sorting the keys matters, or
{a:1,b:2} and {b:2,a:1} hash differently and every repeat looks like a rewording.
First-call success follows from it: the outcome is ok, the result was not empty, and no retry followed within 30 seconds. Miss any one of the
three and it does not count.
A tool name, a number, and a consequence
The metrics are the input. These rules are what the product actually says to you — an insight without a number is not an insight.
| Rule | Fires when | It tells you |
|---|---|---|
| Low first-call | first_call_ok / calls < 0.7 | Agents retried X N times on average. |
| Heavy payload | avg response_bytes > 10,000 | X returns ~Nk tokens per call, roughly $N of context each time. |
| Silent empties | empties / calls > 0.05 | N calls to X returned empty with no error. |
| Dead tool | registered, no calls in range | X has never been called but costs N tokens of schema every session. |
| Slow tool | ms_over_2000 / calls > 0.1 | N% of X calls take over 2 seconds. |