The setup
A fictional electronic line judge works a fourteen day tournament. It sees a bounce and emits two things: a call, and a confidence. On day 8 it gets a routine model update. Nobody announces it to the chair, and nothing on court looks different.
What changes underneath is small and ordinary: a systematic bias of 2.4 mm toward OUT, a wider measurement error, and a flatter confidence curve over the same estimate.
The average is the wrong instrument, and this is where you can watch it fail. Mean confidence on balls within 10 mm of the line moves from 0.58 to 0.56. That is nearly nothing, and a dashboard tracking average confidence would have shown a flat line through the whole tournament. What actually moves is the tail: the share of calls made under 0.55 confidence goes from 3.0% to 5.7%, 1.9 times its old level. The call is still made. It is just made by something closer to a coin.
day 3, stadium courts, serve
day 4, outer courts, rally
day 5, outer courts, serve
day 9, stadium courts, serve
day 11, outer courts, serve
day 12, outer courts, serve
Six near-line calls lifted out of the stream, three before the update and three after. Read them as a person on court would: they all look like calls.
What the monitor is allowed to see
Only the call stream: the call, the confidence, and which court and shot type it came from. No ground truth. Nobody is standing behind the baseline with a tape measure, and in production nobody ever will be. That constraint is the entire problem, and it is the one the method is built for.
Three monitors run per stratum against a reference built from days one to 5, before anything moved:
| Confidence distribution | Population stability index of the confidence histogram against the reference. Catches a shape change even when the average holds still. |
| Coin-flip share | Share of calls landing under 0.55 confidence. This is the decisiveness collapse, and it is the signal a mean would hide. |
| Out rate | Share of calls that are OUT, with a Wilson interval. A calibration shift toward OUT shows up here as an excursion outside the reference. |
Confidence distribution, by stratum
PSI against the days one to 5 reference. Both serve strata cross the 0.25 stale threshold on day 8, ahead of the rally strata, because serves land nearer the lines and the near-line band is where the update did its damage.
Coin-flip share, by stratum
Calls made under 0.55 confidence, as a share of all calls. The judge did not stop calling. It stopped being sure.
Out rate, by stratum
The calibration shift. Small in absolute terms and easy to read as a run of tight matches, which is exactly why one number on its own does not earn an interruption.
The verdict, last day of the tournament
| Stratum | State | PSI | Coin-flip share | Out rate | Action tier |
|---|---|---|---|---|---|
| Stadium courts, serve | STALE | 0.37 | 8.8% (was 4.2%) | 46.9% (was 42.6%) | GATE |
| Stadium courts, rally | WATCH | 0.21 | 5.7% (was 2.3%) | 45.4% (was 46.2%) | LABEL |
| Outer courts, serve | STALE | 0.43 | 10.7% (was 4.2%) | 43.0% (was 40.7%) | GATE |
| Outer courts, rally | STALE | 0.29 | 3.1% (was 2.3%) | 46.5% (was 46.5%) | GATE |
FRESH, WATCH and STALE are the badge ladder. The action tier escalates LOG to LABEL to REDIRECT to GATE, and GATE only throws once a STALE verdict has held 2 consecutive days, so a single noisy session cannot take a court offline. WATCH is the deliberate default: a monitor that stops play on its first bad afternoon gets switched off within a month, and then it protects nothing.
What it cost to run at that sensitivity. Before the update, when nothing had changed, the monitor raised WATCH 6 times across the four strata. It reached the gating tier 0 times. That is the trade the default is built around: WATCH is cheap and reversible and it will fire on a quiet week, while GATE has to be earned by a STALE verdict that holds 2 consecutive days. A monitor tuned to never raise WATCH would not have caught day 8 either.
The worklist, not the score
The calibrated state is the headline; the worklist underneath it is what somebody actually does on Monday. Each line names the stratum, the tier, and the ticket that leaves the room.
| Stratum | Tier | What to do | Ticket |
|---|---|---|---|
| Stadium courts, serve | GATE | Re-gold this stratum against a held-out sample of bounces and re-fit the near-line decision threshold before the next session. | OFFICIATING-TECH: near-line calibration on Stadium courts, serve regressed after the day 8 build. Owner: model team. |
| Stadium courts, rally | LABEL | Hold. The signal is present but under the confirm bar; keep labelling and re-read tomorrow. | OFFICIATING-TECH: watch Stadium courts, rally, no action this session. |
| Outer courts, serve | GATE | Re-gold this stratum against a held-out sample of bounces and re-fit the near-line decision threshold before the next session. | OFFICIATING-TECH: near-line calibration on Outer courts, serve regressed after the day 8 build. Owner: model team. |
| Outer courts, rally | GATE | Re-gold this stratum against a held-out sample of bounces and re-fit the near-line decision threshold before the next session. | OFFICIATING-TECH: near-line calibration on Outer courts, rally regressed after the day 8 build. Owner: model team. |
Was the monitor right? Open the ground truth (available here only because this tournament is invented)
The simulator knows the true margin of every bounce, so it can score the judge. This is the number no production system has. It is shown here to check the monitor's homework, and for no other reason.
| Stratum | True error, days 1 to 7 | True error, days 8 to 14 | Change |
|---|---|---|---|
| Stadium courts, serve | 1.70% | 4.45% | +2.75 pt |
| Stadium courts, rally | 0.90% | 2.51% | +1.61 pt |
| Outer courts, serve | 1.95% | 4.14% | +2.19 pt |
| Outer courts, rally | 0.90% | 2.07% | +1.18 pt |
The update degraded every stratum, and the monitor escalated 3 of the four to STALE without ever being told. It got the ranking substantially right: Stadium courts, serve moved most in true error, at +2.75 points, and it is one of the two the monitor flagged first, on day 8. It is not a perfect ranking, and the page will not pretend otherwise: Outer courts, rally moved least, at +1.18 points, and still reached STALE later in the run. The claim being demonstrated is that a shift is detectable and roughly localisable from the call stream, not that the call stream recovers the true error ordering exactly.
Why this generalises
Nothing above is about tennis. It is about a system that renders judgements at volume, gets updated on a schedule nobody outside the model team tracks, and has no cheap source of truth to check itself against. A claims triage model, a content moderation classifier, a support agent that decides whether a refund applies: the shape is identical. The judgement stream is the only thing you reliably have, and it turns out to be enough to know when the judge has changed.