Blue Camel Your model is in production, and somebody wants to know how much personal information went into it, or why two teams read the same test two different ways. Each of those resolves to a number with its own error bars.
Fifteen years of turning contested numbers into ones a room can act on: product analytics leadership through a marketplace launch at Uber, then metric reliability and training-corpus measurement at a large social platform, with machine learning on clinical text at a hospital research institute before either of those.
A count with no error bars is a guess in a suit. Most of the work is getting the inputs right and then being willing to state, in writing, how confident you actually are.
We put a number in front of legal, or a regulator, or a customer’s security review, and we cannot show our work.
You get a measured answer with its own error bars and a trail somebody can audit, built from a sample that people have checked rather than from a scanner’s raw count. It works on training corpora and on model behaviour, and on anything else where the real answer is a range rather than a point.
We ran a test. Now two teams read the same result two different ways.
Sample sizes worked out properly rather than by eye, and a free-rider measure on every incentive test, because that is the question finance asks eventually. Nothing runs on real money until it has been backtested against held-out actuals.
Nobody trusts the numbers. We find out something broke when a customer tells us.
It starts with the unglamorous half: grain, nulls, reconciliation and tie-outs. The operating discipline goes on top of that, covering which metrics matter, what bound each one has, who gets woken up and what the runbook tells them to do. Most of the value is in the first half.
The training-corpus measurement method, running as a product you can drive. A judge reads a sample, humans calibrate it, and the whole corpus is reported as an interval with a stated error rate.
LiveThe same measurement discipline on medical notes, broken out by subgroup so a fairness question has an answer with error bars instead of an opinion.
The deck said it was a demand story. The dispatch system underneath said otherwise.
Meta · 2023Three months from launch, the org had stopped believing its own dashboard. So we put the numbers on call.
Meta · 2025The scanner returned a number nobody believed, and the proposed fix would have taken months of work. We measured it instead, quickly and cheaply.
One measurement problem, named in a sentence, is enough to start. Write to [email protected] with two or three sentences on what you are facing. If it is a fit, the reply usually includes a first artifact, not a meeting request.
Email Blue Camel Open the drift monitor demo
The drift monitor runs in a browser on synthetic data, with no signup. It is the shape of thing an engagement hands over.
The other practices: AI Evaluation & Engineering Custom Build & Delivery Mission-Driven Systems