Patterns

Cascade to an LLM

Decide the easy cases with decision-machine-1 and send only the uncertain ones to a large model.

Most production traffic is easy. A support ticket about a double charge is billing. A passage that never mentions the question does not answer it. A large model answers those cases correctly. It also answers them slowly.

A cascade puts decision-machine-1 in front of the large model. Stage one decides. Stage two runs only when stage one is unsure.

The shape

1

Call the capability

Send the text to the capability that matches the decision. Use Classify for a label, Yes / no for a flag, Rate for a level.

2

Read the numbers

Classify returns probability and confidence on the same response. probability is the winner’s normalized share. confidence is 1 − normalized entropy over scores.

3

Accept or escalate

Above your threshold, act on the decision and stop. Below it, call the large model with the text and the stage-one scores.

The large model never sees the easy cases. That is the whole saving.

The gate

One gate decides everything. With classify, use both numbers.

Stage-one resultMeaningAction
probability >= 0.90 and confidence >= 0.70One clear winnerAct. No escalation.
probability >= 0.90 and confidence < 0.70The winner leads a crowded fieldAct on low blast radius. Escalate otherwise.
probability < 0.90Two or more labels competeEscalate.

The three stage-one capabilities return different fields. Gate on the fields the capability actually returns.

CapabilityFields returnedGate on
Classifylabel, probability, confidence, scoresBoth numbers, as above.
Ratescore, level, confidence, scoresconfidence and the largest entry in scores. rate returns no probability.
Yes / nostatement, answer, probabilityprobability alone. yes-no returns no confidence and no scores.

On yes-no, probability is the share of the when_true hint when you send hints. Without hints it is one raw score. Escalate the middle band, for example 0.10 < probability < 0.90.

These thresholds are a starting point, not a measurement. Tune them against a labelled set — see Tuning thresholds.

Two real responses

This ticket is easy. Stage one is finished.

classify — act on this
{"label":"billing","probability":0.942,"confidence":0.797,"scores":{"billing":0.942,"shipping":0,"account":0.058}}

This rating is not. Two levels split the mass almost evenly, and confidence reports the split.

rate — escalate this
{"score":1.585,"level":2,"confidence":0.371,"scores":[0,0.471,0.472,0.057]}

The second response is honest, not broken. level is 2 by a margin of 0.001. A cascade exists to catch exactly this case.

Code

The gate is a few lines. Keep it in your own code, next to the action it guards.

curl -s https://api.milliseconds.ai/v1/decision-machine-1/classify \
  -H 'content-type: application/json' \
  -d '{
    "text": "I was charged twice for my subscription this month and support has not replied.",
    "labels": {
      "billing": "payments, invoices, refunds, charges",
      "shipping": "delivery, tracking, returns in transit",
      "account": "login, password, profile settings"
    }
  }'

Pass scores to the large model. It names the labels that are in contention. classify and rate return scores; yes-no does not.

The math

Call e the share of traffic that fails the gate. Stage one runs on every item. Stage two runs on e of them.

  • LLM calls: e per item, down from 1 per item.
  • Mean latency: t1 + e × t2, where t1 is the measured stage-one latency below and t2 is your large model’s latency.
  • Stage-one cost: $0.04 per million input tokens and $0 per output token. The cost scales with the length of the text and the labels you send. The x-input-tokens response header reports the input tokens billed for this call. See Pricing.

At e = 0.15, 85 of every 100 items never reach the large model. The cascade still pays the fixed t1 on all 100. Supply t2 yourself: this page documents no large-model latency.

Measured stage-one latency

Production, 2026-09-16, short inputs, wall clock from a laptop.

Stage-one callObserved
classify, 3 labels1.18 s
yes-no, one statement0.75 s – 1.25 s
rate, 4 levels1.02 s
answer, 1 question0.56 s
entities, 3 types0.50 s

A cascade adds latency when e is high. A gate that escalates more than half of your traffic usually points at the label text, not at the model. Rewrite the labels with Writing good statements and labels before you tune thresholds.

Batch the first stage

Stage one takes up to 32 texts in one request. Screen a queue in one call, then escalate the failures individually.

Screen a batch, then escalate the failures
curl -s https://api.milliseconds.ai/v1/decision-machine-1/yes-no \
  -H 'content-type: application/json' \
  -d '{
    "texts": ["First ticket text.", "Second ticket text."],
    "statement": "The customer expresses urgency.",
    "when_true": "The customer needs an answer today.",
    "when_false": "The customer can wait for a normal reply."
  }'

Results come back in {"results":[...]}, in input order, one entry per text. Each entry carries statement, answer and probability. Gate the batch on probability. Details in Batching.

No API key is required during the launch period. Send the request as shown.

Measure before you ship it

A cascade has one number that matters: the escalation rate e. Log it from day one, with the stage-one scores that produced it.

  • e climbs over a week: your label set no longer covers the traffic.
  • e falls to near zero: your gate is too loose, and wrong decisions pass it.
  • The large model agrees with most escalations: your gate is too tight.

Monitoring covers the distributions to watch.

Next