Thresholds and confidence routing
Pick a cut-off per action, not per system. Raise the bar as the blast radius grows.
A threshold is a property of the action, not of the model. The same classify call can auto-file a ticket
and block a refund. One of those tolerates mistakes. The other does not.
Set one number per action. Then route every decision into three bands: act, confirm, escalate.
The two numbers you route on
probability answers “how sure about this one outcome”. confidence answers “how separated were the
alternatives”. A winner at 0.51 against one rival differs from a winner at 0.51 against nine.
Read Decisions and probabilities for what each field does and does not mean.
Use both axes on classify and rate. A low confidence means the scores stayed flat across the
labels. Send that case to a person, not to an action.
Three bands
Two numbers per action define the bands. Start with a wide confirm band. Narrow it once a golden set tells you what you lose.
rate returns no probability field. Route rate calls on score and confidence instead.
Raise the bar with the blast radius
Rank each action by what a wrong decision costs. Then set the act number against that cost.
These starting points are not measurements. They are a place to begin before you have data. Replace every one of them with a number from your own golden set. Tuning thresholds turns that set into numbers.
A worked pair
Both responses below are real captures. The first is a clean win. The second is a near tie.
The rate call returns level 2, but level 1 scored 0.471 against level 2 at 0.472. The confidence of
0.371 reports that tie, and score 1.585 sits between the two levels. A threshold on level alone would
hide this. A threshold on confidence catches it.
On rate, route on score and confidence rather than on level. score is the probability-weighted
position, so it moves smoothly. level is an argmax and can flip on 0.001.
Route in code
Make the call, then apply your own bands to the response.
No API key is required during the launch period.
Do not inherit the built-in 0.5
Two fields already carry a fixed cut-off. yes-no sets answer to true when probability >= 0.5.
verify sets matches the same way. That boundary serves the response shape, not your action.
Read probability and apply your own number. A true at 0.51 and a true at 0.99 are different cases.
Per-capability notes
yes-no
Without hints, probability is one raw sigmoid score and stays unnormalized. With when_true or
when_false, it becomes p_true / (p_true + p_false). The two forms produce different
distributions, so tune one threshold per form. Hint text must describe the case, never the verdict.
classify and rate
scores is normalized and sums to 1. On classify it maps label name to probability. On rate it
is one number per level, in scale order. Gate classify on probability and confidence together.
With many labels, compare the top two scores before you act.
answer and entities
probability is the span confidence from the extractor. answer returns null with probability 0
when nothing fits, which is a clean escalate signal on its own.
extract and verify
extract returns no per-field probability. Check a field you care about with verify, then route on
that probability and on found[]. The Extract, then verify pattern
shows the full loop.
Next
The full pattern, with act, confirm, review and escalate in Python and TypeScript.
Turn a golden set into precision and recall at each cut-off, then pick numbers per action.
Send only the escalate band to a large model, and keep the cost of the rest flat.