Blue Camel The evaluation that cleared your assistant was a point in time, and the assistant has been answering customers ever since. A model is easy to demo and hard to trust, and the second part is the work here.
Six years designing and evaluating machine learning systems at a large social platform: privacy infrastructure for training data, the judge methodology that measures what is actually in a corpus, and the data science behind that platform's public creator-captioning launch in 2022.
A judge that states its own error rate, an evaluation built around what breaks, and a number a team will stand behind on launch day.
A model scores our content, or our risk, or our compliance. We cannot tell a regulator how often the score is wrong.
A judge that writes down a reason for every call, so a person can check it. Human raters score a slice, which tells you the judge’s precision and recall, and the raw count is then corrected for that error so the result ships as an interval rather than a bare number.
The demo passed. We still do not know what happens when it meets our real traffic.
Tests built around the failure rather than the happy path, so you find out what a process kill or a malformed input or a rate limit does to the thing before your customers do. The number that ships is one the team has agreed to be on the hook for.
The AI passed its evaluation and went live nine months ago. Nobody here can tell me what it is doing now.
Before launch it was measured against a standard: the outputs that count as correct, grounded in your own sources and inside the policy you wrote. We keep measuring against that same standard on live traffic, broken out by the kind of work, and we publish a range instead of one flattering figure. When a measurement goes out of date it gets pulled rather than redrawn, and only the affected slice needs re-labeling. This is middle management for an AI team, and it works on an assistant, an agent, a ranker or a classifier alike. See how we keep it valid after launch.
A vendor’s accuracy line is internal benchmarks they cannot share. Our scorecard has nowhere to put that.
Scorecard weights fixed before anybody sees a demo, failure modes substantiated on a named dataset, and the word unscoreable written plainly wherever a claim cannot be tested at all. The structure is borrowed from a healthcare procurement that did it properly seventeen years ago.
This method was built in production privacy measurement work, and now runs live as a prevalence audit. The full case sits under Data Science, and you can drive it at the prevalence audit.
An electronic line judge whose calls shift after a mid-tournament model update, caught from the call stream alone with no ground truth. A judge that learns can drift, and this is what seeing it first looks like. Synthetic data.
Open sourceThe same agent operating loop, built twice: once by hand, once on a framework, over a shared tool layer. Then both measured under a real process kill rather than a happy-path demo. If you are deciding whether to adopt an agent framework, this is the comparison nobody publishes.
MIT licensedThe outbound pipeline this practice runs on, published as a reusable skill. Leading signals over lagging ones, a four-axis scorer, and a stated list of what it refuses to do. We published the machinery rather than describing it.
Live demoA judge that scores call transcripts against a compliance rulebook, then a second pass that audits the judge itself. Fully synthetic data, every number reproducible from a seed.
A model to evaluate, or a vendor’s claim about one, is where this usually starts. Write to [email protected] with two or three sentences on what you are facing. If it is a fit, the reply usually includes a first artifact, not a meeting request.
Email Blue Camel Open the drift monitor demo
The drift monitor runs in a browser on synthetic data, with no signup. It is the shape of thing an engagement hands over.
The other practices: Data Science & Experimentation Custom Build & Delivery Mission-Driven Systems