Blog
Notes on measuring what models do with your tools
Everything here comes out of building MCPulse: the metrics that turned out to matter, the ones that did not, and the parts of MCP that are easy to get wrong.
What MCP servers cannot tell you
You ship a server, it gets listed, people install it. From that moment on you are working blind — and the directories report almost none of what you need.
- Measurement · 5 min
First-call success is the only number that grades your descriptions
Uptime grades your infrastructure. Error rate grades your code. First-call success is the only metric that grades the thing you actually wrote for the model — the words.
- Measurement · 5 min
Empty results are failures the protocol calls success
An empty array is a 200. The model got nothing it could use, the user got a shrug, and no error was recorded anywhere. These are the failures nobody reports.
- Tool design · 5 min
Your tool schemas are billed on every session, used or not
Descriptions and JSON Schema go into the context window at the start of every conversation. Twelve tools can cost 3,000 tokens before anyone asks a question.
- Measurement · 4 min
Dead tools — registered, described, and never once called
Every server has at least one. It costs schema tokens on every session, competes for the model's attention, and returns nothing, because nobody has ever reached it.
- Engineering · 5 min
Telling a crash from an error return is harder than it looks
The MCP SDK catches whatever your handler throws and turns it into isError: true. From outside the request handler, a bug and a deliberate error are the same object.
- Tool design · 5 min
bad_args is a message about your schema, not about the model
When arguments fail validation, the instinct is that the model got it wrong. Usually the schema asked for something the model had no way to know.
- Privacy · 5 min
Why we hash arguments instead of storing them
Twelve hex characters of a SHA-256 over the arguments, keys sorted. Enough to tell whether two calls were the same, and not enough for anything else.
- Engineering · 5 min
Why the SDK runs inside your server instead of in front of it
A proxy is the easier product to build and the wrong one to ship. Directory-listed servers cannot change their URL, and OAuth breaks the moment traffic is redirected.
- Engineering · 4 min
Percentiles cannot be summed, so we bucket latency instead
You cannot average two days of p95 and get the week's p95. Four counters can be added across any range and still tell you the truth.
- Engineering · 4 min
Sessions cross midnight, and other counter traps
Bucketed counters are how a dashboard stays fast. But some things are not buckets, and summing them by day quietly double-counts every one that spans two.
- Measurement · 4 min
What a 10kb tool response actually costs
Roughly 2,500 tokens, about a cent, and a chunk of the context window that the rest of the conversation no longer has. The last one is the expensive part.
- Tool design · 5 min
Writing tool descriptions for a reader who only gets one pass
The model reads your description once, alongside forty others, and decides immediately. Everything that makes documentation good makes a tool description worse.
- Engineering · 4 min
What gets computed at 2am, and why it cannot be done sooner
Three of the sixteen metrics need to see what came after a call. That is not knowable when the call arrives, so it happens overnight — and the dashboard says so.
- Engineering · 5 min
Counters exist so that calls can be deleted
Every call is written twice — once as a row you will throw away, once as a counter you keep forever. The second one is what makes the first one disposable.
- Tool design · 4 min
Naming tools for a reader who cannot see your API
The model has your tool name, your description, and nothing else. No source, no docs site, no colleague to ask. The name is doing more work than you think.
- Engineering · 5 min
Instrumentation must never break the thing it measures
The SDK runs inside someone else's production server. Three rules follow from that, and none of them are negotiable — including handing the server back unmodified.
- Measurement · 4 min
Which tools get called together, and what that tells you
Models chain tools. When two of yours are always called in sequence, that pattern is usually a design note you have not read yet.
- Measurement · 4 min
Cost per session is the number to quote when someone asks what your server costs
Not the monthly bill — that measures how popular you are. Cost per session measures how expensive you are, which is the part you control.
- Privacy · 4 min
The five questions a security review will ask about your analytics
If your MCP server touches customer data, someone will eventually ask what your monitoring vendor receives. These are the questions, and the answers that end the conversation.