Confidence routing

Most systems use one axis: the label. Confidence routing adds a second axis: how sure the model is. The label says what to do. The confidence says who does it.

One router function covers every capability. Write it once, then reuse it per action with different numbers.

The four bands

The bands run in order and the first match wins. The numbers above are starting points, not measurements. Thresholds and confidence routing explains why a threshold belongs to the action. Tuning thresholds turns a golden set into your own numbers. This page is the code.

Which field to route on

confidence is 1 − normalized entropy over the scores. It is 1 for one clear winner and 0 for a flat distribution. Only classify and rate return it.

Real responses, three bands

One classify call over three texts. The response below is real.

{"results":[
  {"label":"billing","probability":1,"confidence":0.999,"scores":{"billing":1,"shipping":0,"account":0}},
  {"label":"account","probability":0.632,"confidence":0.378,"scores":{"billing":0.364,"shipping":0.005,"account":0.632}},
  {"label":"billing","probability":0.993,"confidence":0.96,"scores":{"billing":0.993,"shipping":0.007,"account":0}}
]}

Result 1 and result 3 land in act. Result 2 does not. Its text says only "Can someone look at this? It has been going on for a while now." The winner account scores 0.632, and billing follows at 0.364. The confidence of 0.378 reports that spread. Route this result to escalate and ask the customer one question.

The router

Make the call, then apply the bands to each result. The call itself carries no thresholds.

curl -s -X POST https://api.milliseconds.ai/v1/decision-machine-1/classify \
  -H "Content-Type: application/json" \
  -d '{
    "texts": [
      "I was charged twice for my subscription this month and support has not replied.",
      "Can someone look at this? It has been going on for a while now.",
      "My package never arrived and I want my money back."
    ],
    "labels": {
      "billing": "a payment, invoice, refund or charge problem",
      "shipping": "a delivery, tracking or address problem",
      "account": "a login, password or profile problem"
    }
  }'

No API key is required during the launch period.

Route a rate call on score

rate returns level as an argmax. It can flip on 0.001, as this real response shows.

{"score":1.585,"level":2,"confidence":0.371,"scores":[0,0.471,0.472,0.057]}

Route on score and gate with confidence. score moves smoothly from 0 to len(scale) - 1, so pick the cut-off on that range. The call is the same rate request shown on Rate. Only the router changes. Here act means the quiet path, with no escalation.

def rate_band(d, escalate_above=2.0, min_confidence=0.6):
    if d["confidence"] < min_confidence:
        return "review"          # the levels sit too close to trust
    return "escalate" if d["score"] >= escalate_above else "act"

Keep the bands honest

Store probability, confidence and the chosen band with every decision. You cannot tune without that record. Watch the escalate share in Monitoring: a rise means the inputs changed.

Move one number at a time. Raise act when a wrong auto-action costs more. Lower review when the queue grows faster than people clear it.

Next