Evaluate and tune

Golden sets

Label 50 to 200 real examples, then score every change against them.

A golden set is a small file of real inputs with the answer you want. You score the model against it before you tune labels, hints, or thresholds.

Without a golden set you cannot tell a real improvement from a lucky example. With one, every change becomes a number.

Build it before you tune. Label the examples first. Then change the labels, the hints, or the threshold, and score again.

How many examples

SizeUse
Under 50Too small. One wrong example moves accuracy by more than 2 points.
50 to 200The working range. Enough to rank two label sets and to pick a threshold.
Over 200Good, but the labelling cost grows. Add examples only where the model fails.

Rules for the examples:

  • Take every example from your real traffic. Invented examples miss the mess of production text.
  • Cover each label. Aim for at least 10 examples per label, per level, or per field.
  • Keep the hard cases. Near-miss and ambiguous inputs decide your threshold.
  • Label by hand. Two people labelling the same 20 rows show you where your own definition is unclear.

The file format

One JSON object per line. A JSON Lines file appends cleanly, and git diffs it line by line.

golden.jsonl
{"id":"t-001","text":"I was charged twice for my subscription this month and support has not replied.","expected":"billing"}
{"id":"t-002","text":"My order shipped last week but the tracking number does not work.","expected":"shipping"}
{"id":"t-003","text":"I need to change the email address on my account.","expected":"account"}
{"id":"t-004","text":"The invoice total looks wrong and my package is late.","expected":"billing","note":"two problems; billing is the action we want"}

Three fields carry the work:

  • id — a stable key. You refer to it in review and in bug reports.
  • text — the exact input you send to the API.
  • expected — the answer a correct system returns.

Add note for anything a reader would ask about. Add your own fields freely; the scorer ignores them.

Use the same shape for every capability. For yes-no set expected to true or false. For rate set it to the level index. For answer set it to the exact span text.

Check one row first

Send a single row by hand. Confirm the labels and the response shape before you score 200 rows.

curl
curl -s https://api.milliseconds.ai/v1/decision-machine-1/classify \
  -H "Content-Type: application/json" \
  -d '{
    "text": "I was charged twice for my subscription this month and support has not replied.",
    "labels": {
      "billing": "payments, invoices, refunds, charges",
      "shipping": "delivery, tracking, returns",
      "account": "login, profile, settings"
    }
  }'

Real response:

{"label":"billing","probability":0.993,"confidence":0.964,"scores":{"billing":0.993,"shipping":0,"account":0.007}}

No API key is required during the launch period.

The scoring script

The script sends every row, compares label with expected, and prints accuracy. It also keeps probability and confidence, because Tuning thresholds needs them.

Send the rows in batches. Each capability accepts up to 32 texts in one texts array, and the results come back in input order.

import json, requests
URL = "https://api.milliseconds.ai/v1/decision-machine-1/classify"
LABELS = {
    "billing": "payments, invoices, refunds, charges",
    "shipping": "delivery, tracking, returns",
    "account": "login, profile, settings",
}
rows = [json.loads(line) for line in open("golden.jsonl")]
scored = []
for i in range(0, len(rows), 32):
    batch = rows[i:i + 32]
    body = {"texts": [r["text"] for r in batch], "labels": LABELS}
    results = requests.post(URL, json=body, timeout=60).json()["results"]
    for row, result in zip(batch, results):
        scored.append({**row, **result, "correct": result["label"] == row["expected"]})
correct = sum(r["correct"] for r in scored)
print(f"accuracy {correct / len(scored):.3f}  ({correct}/{len(scored)})")
with open("scored.jsonl", "w") as f:
    for r in scored:
        f.write(json.dumps(r) + "\n")
for r in scored:
    if not r["correct"]:
        print(r["id"], r["expected"], "->", r["label"], r["probability"])

scored.jsonl is the file you keep. Accuracy tells you whether a change helped. The per-row probabilities tell you where to put the threshold.

Read the failures

Open every wrong row before you change anything. Most failures fall into three groups.

What you seeWhat to do
The model picks a neighbouring label with a high probabilityYour label descriptions overlap. Rewrite each description to name a distinct case. Writing good statements and labels shows how.
The winning probability is low and confidence sits near 0The scores are close to flat, so the input is ambiguous. Route it to a person instead of forcing a label.
The expected value is wrongFix the row. A golden set with bad labels tunes you toward the wrong system.

Never change an expected value to make a score go up. Change it only when a second reader agrees the original label was wrong. Record the change in the commit.

Keep the set alive

  • Commit golden.jsonl next to your code. Review changes to it like code changes.
  • Add each production failure as a new row. The set grows where the system is weak.
  • Re-score after every change to labels, hints, scale text, or schema. Keep the accuracy number in the pull request.
  • Split the set once it passes 200 rows: tune on one half, confirm on the other half.

A run of 200 rows costs 200 classify calls. The measured classify latency is 1.18 s, so a row-by-row run takes about four minutes.

A batch still costs one inference call per text. The service runs the calls in parallel, so the wall clock drops.

Next