> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.milliseconds.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.milliseconds.ai/_mcp/server.

# Golden sets

A golden set is a small file of real inputs with the answer you want. You score the model against it before you tune labels, hints, or thresholds.

Without a golden set you cannot tell a real improvement from a lucky example. With one, every change becomes a number.

> **Note**
>
> **Build it before you tune.** Label the examples first. Then change the labels, the hints, or the threshold, and score again.

## How many examples

| Size      | Use                                                                          |
| --------- | ---------------------------------------------------------------------------- |
| Under 50  | Too small. One wrong example moves accuracy by more than 2 points.           |
| 50 to 200 | The working range. Enough to rank two label sets and to pick a threshold.    |
| Over 200  | Good, but the labelling cost grows. Add examples only where the model fails. |

Rules for the examples:

* Take every example from your real traffic. Invented examples miss the mess of production text.
* Cover each label. Aim for at least 10 examples per label, per level, or per field.
* Keep the hard cases. Near-miss and ambiguous inputs decide your threshold.
* Label by hand. Two people labelling the same 20 rows show you where your own definition is unclear.

## The file format

One JSON object per line. A JSON Lines file appends cleanly, and git diffs it line by line.

**`golden.jsonl`**

```json title="golden.jsonl"
{"id":"t-001","text":"I was charged twice for my subscription this month and support has not replied.","expected":"billing"}
{"id":"t-002","text":"My order shipped last week but the tracking number does not work.","expected":"shipping"}
{"id":"t-003","text":"I need to change the email address on my account.","expected":"account"}
{"id":"t-004","text":"The invoice total looks wrong and my package is late.","expected":"billing","note":"two problems; billing is the action we want"}
```

Three fields carry the work:

* `id` — a stable key. You refer to it in review and in bug reports.
* `text` — the exact input you send to the API.
* `expected` — the answer a correct system returns.

Add `note` for anything a reader would ask about. Add your own fields freely; the scorer ignores them.

> **Tip**
>
> Use the same shape for every capability. For `yes-no` set `expected` to `true` or `false`. For `rate` set it to the level index. For `answer` set it to the exact span text.

## Check one row first

Send a single row by hand. Confirm the labels and the response shape before you score 200 rows.

**`curl`**

```bash title="curl"
curl -s https://api.milliseconds.ai/v1/decision-machine-1/classify \
  -H "Content-Type: application/json" \
  -H "authorization: Bearer test_sk-..." \
  -d '{
    "text": "I was charged twice for my subscription this month and support has not replied.",
    "labels": {
      "billing": "payments, invoices, refunds, charges",
      "shipping": "delivery, tracking, returns",
      "account": "login, profile, settings"
    }
  }'
```

Real response:

```json
{"label":"billing","probability":0.993,"confidence":0.964,"scores":{"billing":0.993,"shipping":0,"account":0.007}}
```

Every request needs an API key. See [Authentication](/authentication).

## The scoring script

The script sends every row, compares `label` with `expected`, and prints accuracy. It also keeps `probability` and `confidence`, because [Tuning thresholds](/evaluate/tuning-thresholds) needs them.

Send the rows in batches. Each capability accepts up to 32 texts in one `texts` array, and the results come back in input order.

**`Python`**

```python title="Python"
import json, requests

URL = "https://api.milliseconds.ai/v1/decision-machine-1/classify"
LABELS = {
    "billing": "payments, invoices, refunds, charges",
    "shipping": "delivery, tracking, returns",
    "account": "login, profile, settings",
}

rows = [json.loads(line) for line in open("golden.jsonl")]
scored = []

for i in range(0, len(rows), 32):
    batch = rows[i:i + 32]
    body = {"texts": [r["text"] for r in batch], "labels": LABELS}
    results = requests.post(URL, headers={"authorization": "Bearer test_sk-..."}, json=body, timeout=60).json()["results"]
    for row, result in zip(batch, results):
        scored.append({**row, **result, "correct": result["label"] == row["expected"]})

correct = sum(r["correct"] for r in scored)
print(f"accuracy {correct / len(scored):.3f}  ({correct}/{len(scored)})")

with open("scored.jsonl", "w") as f:
    for r in scored:
        f.write(json.dumps(r) + "\n")

for r in scored:
    if not r["correct"]:
        print(r["id"], r["expected"], "->", r["label"], r["probability"])
```

**`TypeScript`**

```typescript title="TypeScript"
import { readFileSync, writeFileSync } from "node:fs";

const URL = "https://api.milliseconds.ai/v1/decision-machine-1/classify";
const labels = {
  billing: "payments, invoices, refunds, charges",
  shipping: "delivery, tracking, returns",
  account: "login, profile, settings",
};

const rows = readFileSync("golden.jsonl", "utf8").trim().split("\n").map((l) => JSON.parse(l));
const scored: any[] = [];

for (let i = 0; i < rows.length; i += 32) {
  const batch = rows.slice(i, i + 32);
  const res = await fetch(URL, {
    method: "POST",
    headers: { "Content-Type": "application/json", authorization: "Bearer test_sk-..." },
    body: JSON.stringify({ texts: batch.map((r) => r.text), labels }),
  });
  const { results } = await res.json();
  batch.forEach((row, j) => {
    scored.push({ ...row, ...results[j], correct: results[j].label === row.expected });
  });
}

const correct = scored.filter((r) => r.correct).length;
console.log(`accuracy ${(correct / scored.length).toFixed(3)} (${correct}/${scored.length})`);
writeFileSync("scored.jsonl", scored.map((r) => JSON.stringify(r)).join("\n"));
```

`scored.jsonl` is the file you keep. Accuracy tells you whether a change helped. The per-row probabilities tell you where to put the threshold.

## Read the failures

Open every wrong row before you change anything. Most failures fall into three groups.

| What you see                                                  | What to do                                                                                                                                                           |
| ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The model picks a neighbouring label with a high probability  | Your label descriptions overlap. Rewrite each description to name a distinct case. [Writing good statements and labels](/concepts/writing-good-questions) shows how. |
| The winning `probability` is low and `confidence` sits near 0 | The scores are close to flat, so the input is ambiguous. Route it to a person instead of forcing a label.                                                            |
| The `expected` value is wrong                                 | Fix the row. A golden set with bad labels tunes you toward the wrong system.                                                                                         |

> **Warning**
>
> Never change an `expected` value to make a score go up. Change it only when a second reader agrees the original label was wrong. Record the change in the commit.

## Keep the set alive

* Commit `golden.jsonl` next to your code. Review changes to it like code changes.
* Add each production failure as a new row. The set grows where the system is weak.
* Re-score after every change to labels, hints, scale text, or schema. Keep the accuracy number in the pull request.
* Split the set once it passes 200 rows: tune on one half, confirm on the other half.

A run of 200 rows costs 200 classify calls. The measured `classify` latency is 0.43 s, so a row-by-row run takes about a minute and a half.

A batch still costs one inference call per text. The service runs the calls in parallel, so the wall clock drops.

## Next

* [Tuning thresholds](/evaluate/tuning-thresholds) — turn `scored.jsonl` into precision and recall at each threshold.
* [Thresholds and confidence routing](/concepts/thresholds) — pick a threshold per action, not per system.
* [Monitoring](/evaluate/monitoring) — watch the same numbers in production.