Tuning thresholds

A threshold you guessed is a guess. A threshold you measured is a decision.

This page turns a labelled set into precision and recall at each cut-off. You score the set once. Then you read the number you need off a table.

The two numbers per action

Raising the threshold raises precision and lowers recall. Decide which error hurts more before you look at the table.

Step 1 — score the set once

Send the whole set through one capability in batches of 32 texts. Keep the probability for every record. Do not threshold yet.

curl -X POST https://api.milliseconds.ai/v1/decision-machine-1/yes-no \
  -H "Content-Type: application/json" \
  -d '{
    "texts": [
      "Our whole checkout has been down for two hours. We are losing orders right now.",
      "Just a small typo on the pricing page, no rush at all."
    ],
    "statement": "The customer needs help urgently."
  }'

Results come back in input order. Batching covers that ordering guarantee and the 32-text limit.

Step 2 — sweep the cut-offs

Count true positives, false positives, and false negatives at each candidate threshold. Eight lines do it.

def sweep(scores, cuts=(0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99)):
    for t in cuts:
        tp = sum(1 for p, y in scores if p >= t and y)
        fp = sum(1 for p, y in scores if p >= t and not y)
        fn = sum(1 for p, y in scores if p < t and y)
        precision = tp / (tp + fp) if tp + fp else 1.0
        recall = tp / (tp + fn) if tp + fn else 1.0
        print(f"{t:<6}{tp:<4}{fp:<4}{fn:<4}{precision:.3f}  {recall:.3f}")

Step 3 — read the table

The run below is real. It scores 24 support messages against the statement The customer needs help urgently., with 11 of them labelled urgent.

Three things to take from a table like this one.

• The answer boolean cuts at 0.5. That cut is not the best one here. It let one polite billing question through at 0.543.
• A plateau is where you want to sit. Between 0.6 and 0.8 nothing changes, so a small drift costs nothing.
• Precision maxes out before recall falls. Above 0.9 the flag starts to miss real cases and buys nothing back.

Pick two numbers, not one

One threshold splits the set in two. Two thresholds give you a band for a person to check.

The confirm band holds one record at 0.892. A human sees 4% of the traffic and the automation keeps full recall. Widen the band until the error count reaches zero. Then narrow it until a person complains about the workload. Confidence routing has the code for the three branches.

Sweep confidence too

classify and rate return confidence: 1 − H(scores) / ln(n), where H is Shannon entropy. It is a second axis. Sweep it the same way. Count the errors above and below each confidence cut-off, at a fixed probability for classify or at a fixed level for rate.

A near tie between two scores drops confidence toward 0. Escalate those cases whatever the winner scored.

{"score":1.585,"level":2,"confidence":0.371,"scores":[0,0.471,0.472,0.057]}

Levels 1 and 2 differ by 0.001 here, and rate returns no probability at all. Only a confidence floor catches this case.

When to re-tune

• You changed the statement, the labels, or the hints. The numbers do not carry over.
• Your input mix changed: a new channel, a new language, longer texts.
• Your escalation rate moved in production. Monitoring tells you when.

Store the thresholds in config, not in code. Re-running the sweep must be a one-line change.

Next