Monitoring
A golden set tells you how the model behaves on yesterday’s data. Monitoring tells you how it behaves on today’s.
The model does not drift. The same text and the same labels return the same numbers. Your traffic moves instead: new ticket types, a new segment, a new language. Each one shifts the distribution under a threshold you picked weeks ago.
Log each call, watch four numbers, alarm on change, and replay the golden set after every change.
Every metric on this page comes from your own logs. The API exposes no metrics endpoint, no dashboard and no log export. You build the counters.
What to log per call
Log one row per decision. Keep it small enough to write on every request.
Store the outcome too, when you learn it. A human correction, a refund, a reopened ticket: each one turns a logged decision into a labelled example for the next golden set.
The four numbers
Confidence distribution
Plot a histogram of confidence per day, for classify and rate. A healthy classify histogram leans high, like the 0.972 in the capture below. When the mass slides toward 0, your labels no longer separate the traffic. Track the median and the 10th percentile, not the mean. For the other five capabilities, plot probability instead: they return no confidence.
Escalation rate
The share of calls that land in your escalate band. This is your review cost. It is also the fastest drift signal, because it moves before accuracy does. Alarm on a relative change, for example a 50 percent rise over a 7-day baseline. That number is a starting point, not a product default.
Label mix
The share of traffic per label, per level, or per entity type. A label that falls to zero is usually a broken integration. A label that doubles is usually a real change in your inbox. Both need a human to look.
Null and empty rate
For answer, the share of results with answer: null. For entities, the share of calls returning an empty list. For extract, the share of fields that come back null. A rise means the text changed shape, not that the model got worse.
Logging wrapper
The headers carry the number of input characters and the input tokens billed for this call. Read both from the response.
That curl returns the body below, plus x-input-chars: 79 and x-input-tokens: 20:
Two identical calls returned identical numbers. Log the numbers anyway: your texts change even when the model does not.
Latency and health
Compare your p50 against these measured calls. One laptop, short inputs of 55 to 198 characters, wall clock end to end, including transit:
A rise in your own p50 usually means longer inputs. Check x-input-chars first: texts over 2,000 characters are chunked, and cost grows with length.
Two status codes need counters. runner_error (502) means inference failed twice. overloaded (529) means no inference slot freed in time. Retry the 502 once. Retry the 529 with backoff. A sustained 529 rate usually means your call rate outruns the free slots.
Do not alarm on a single day of confidence movement. Short windows on low volume produce noise. Compare a 7-day window against the 7 days before it.
Close the loop
You can act on drift only when you measure it against something fixed.
Sample the escalations
Pull 50 escalated cases each week. Label them by hand. They are the cases your thresholds already flagged as hard.