> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.milliseconds.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.milliseconds.ai/_mcp/server.

# Tuning thresholds

A threshold you guessed is a guess. A threshold you measured is a decision.

This page turns a labelled set into precision and recall at each cut-off. You score the set once.
Then you read the number you need off a table.

> **Note**
>
> Build the labelled set first. [Golden sets](/evaluate/golden-sets) covers the size, the sampling, and the file format.
> Fifty to two hundred real examples is enough to move a threshold with confidence.

## The two numbers per action

| Metric    | Question it answers                                        | Which action cares                               |
| --------- | ---------------------------------------------------------- | ------------------------------------------------ |
| Precision | Of the cases I acted on, how many were right?              | Auto-act. A false positive reaches the customer. |
| Recall    | Of the cases that needed the action, how many did I catch? | Screening. A false negative escapes the net.     |

Raising the threshold raises precision and lowers recall. Decide which error hurts more
before you look at the table.

## Step 1 — score the set once

Send the whole set through one capability in batches of 32 texts. Keep the `probability` for every
record. Do not threshold yet.

**`curl`**

```bash title="curl"
curl -X POST https://api.milliseconds.ai/v1/decision-machine-1/yes-no \
  -H "Content-Type: application/json" \
  -H "authorization: Bearer test_sk-..." \
  -d '{
    "texts": [
      "Our whole checkout has been down for two hours. We are losing orders right now.",
      "Just a small typo on the pricing page, no rush at all."
    ],
    "statement": "The customer needs help urgently."
  }'
```

**`Python`**

```python title="Python"
import json, requests

URL = "https://api.milliseconds.ai/v1/decision-machine-1/yes-no"
golden = [json.loads(line) for line in open("golden.jsonl")]  # {"text": ..., "expected": true|false}

scores = []
for i in range(0, len(golden), 32):
    chunk = golden[i:i + 32]
    r = requests.post(URL, headers={"authorization": "Bearer test_sk-..."}, json={
        "texts": [row["text"] for row in chunk],
        "statement": "The customer needs help urgently.",
    })
    r.raise_for_status()
    for row, result in zip(chunk, r.json()["results"]):
        scores.append((result["probability"], row["expected"]))
```

**`TypeScript`**

```typescript title="TypeScript"
const URL = "https://api.milliseconds.ai/v1/decision-machine-1/yes-no";
const golden: { text: string; expected: boolean }[] = loadGoldenSet();
const scores: [number, boolean][] = [];

for (let i = 0; i < golden.length; i += 32) {
  const chunk = golden.slice(i, i + 32);
  const res = await fetch(URL, {
    method: "POST",
    headers: { "Content-Type": "application/json", authorization: "Bearer test_sk-..." },
    body: JSON.stringify({
      texts: chunk.map((row) => row.text),
      statement: "The customer needs help urgently.",
    }),
  });
  const { results } = await res.json();
  results.forEach((r, j) => scores.push([r.probability, chunk[j].expected]));
}
```

Results come back in input order. [Batching](/capabilities/batching) covers that ordering guarantee and the 32-text limit.

## Step 2 — sweep the cut-offs

Count true positives, false positives, and false negatives at each candidate threshold. Eight lines do it.

**`sweep.py`**

```python title="sweep.py"
def sweep(scores, cuts=(0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99)):
    for t in cuts:
        tp = sum(1 for p, y in scores if p >= t and y)
        fp = sum(1 for p, y in scores if p >= t and not y)
        fn = sum(1 for p, y in scores if p < t and y)
        precision = tp / (tp + fp) if tp + fp else 1.0
        recall = tp / (tp + fn) if tp + fn else 1.0
        print(f"{t:<6}{tp:<4}{fp:<4}{fn:<4}{precision:.3f}  {recall:.3f}")
```

## Step 3 — read the table

The run below is real. It scores 24 support messages against the statement
`The customer needs help urgently.`, with 11 of them labelled urgent.

| Threshold | TP | FP | FN | Precision | Recall |
| --------- | -- | -- | -- | --------- | ------ |
| 0.50      | 11 | 1  | 0  | 0.917     | 1.000  |
| 0.60      | 11 | 0  | 0  | 1.000     | 1.000  |
| 0.70      | 11 | 0  | 0  | 1.000     | 1.000  |
| 0.80      | 11 | 0  | 0  | 1.000     | 1.000  |
| 0.90      | 10 | 0  | 1  | 1.000     | 0.909  |
| 0.95      | 9  | 0  | 2  | 1.000     | 0.818  |
| 0.99      | 7  | 0  | 4  | 1.000     | 0.636  |

Three things to take from a table like this one.

* **The `answer` boolean cuts at 0.5. That cut is not the best one here.** It let one polite billing question through at `0.543`.
* **A plateau is where you want to sit.** Between 0.6 and 0.8 nothing changes, so a small drift costs nothing.
* **Precision maxes out before recall falls.** Above 0.9 the flag starts to miss real cases and buys nothing back.

> **Warning**
>
> A 24-record set makes a demonstration, not a decision. Perfect scores on a small set mean the set is too easy.
> Add the cases your system got wrong last month, then sweep again.

## Pick two numbers, not one

One threshold splits the set in two. Two thresholds give you a band for a person to check.

| Band    | Rule in this run             | Records | Errors |
| ------- | ---------------------------- | ------- | ------ |
| Act     | `probability >= 0.90`        | 10      | 0      |
| Confirm | `0.60 <= probability < 0.90` | 1       | 0      |
| Ignore  | `probability < 0.60`         | 13      | 0      |

The confirm band holds one record at `0.892`. A human sees 4% of the traffic and the automation keeps full recall.
Widen the band until the error count reaches zero. Then narrow it until a person complains about the workload.
[Confidence routing](/patterns/confidence-routing) has the code for the three branches.

## Sweep `confidence` too

`classify` and `rate` return `confidence`: `1 − H(scores) / ln(n)`, where `H` is Shannon entropy. It is a second axis.
Sweep it the same way. Count the errors above and below each `confidence` cut-off, at a fixed `probability` for
`classify` or at a fixed `level` for `rate`.

A near tie between two scores drops `confidence` toward 0. Escalate those cases whatever the winner scored.

**`rate — a real near tie`**

```json title="rate — a real near tie"
{"score":1.585,"level":2,"label":"Frustrated","confidence":0.371,"scores":[0,0.471,0.472,0.057]}
```

Levels 1 and 2 differ by 0.001 here, and `rate` returns no `probability` at all. Only a `confidence` floor catches this case.

## When to re-tune

* You changed the statement, the labels, or the hints. The numbers do not carry over.
* Your input mix changed: a new channel, a new language, longer texts.
* Your escalation rate moved in production. [Monitoring](/evaluate/monitoring) tells you when.

Store the thresholds in config, not in code. Re-running the sweep must be a one-line change.

## Next

#### [Golden sets](/evaluate/golden-sets)

Build the labelled set this page consumes.

#### [Confidence routing](/patterns/confidence-routing)

Wire the bands into act, confirm, and escalate.

#### [Monitoring](/evaluate/monitoring)

Watch the numbers drift in production.