Taxonomy classification

A taxonomy is a tree of labels. The support taxonomy below has four top-level areas and four leaves under each of the first two. One flat classify call puts every leaf against every other leaf, so leaves from far-apart branches steal probability from each other.

Walk the tree instead. Classify tree does the walk in one request. Each level is one classify call over the children of the winner from the level above, and the response returns every level.

The taxonomy

Give every label a description. The model reads "name: description" when a description exists, and a bare name otherwise. Descriptions improve accuracy, as Writing good statements and labels shows. A string value is a leaf. An object value carries its own labels.

{
  "billing": {
    "description": "payments, invoices, charges, refunds and subscriptions",
    "labels": {
      "duplicate_charge": "the customer was billed more than once for the same item",
      "refund_request": "the customer asks for money back",
      "invoice_question": "the customer asks about an invoice or receipt",
      "subscription_change": "the customer wants to upgrade, downgrade or cancel a plan"
    }
  },
  "shipping": {
    "description": "delivery, tracking, lost or damaged parcels",
    "labels": {
      "lost_parcel": "the parcel is marked delivered but the customer never received it",
      "damaged_parcel": "the parcel arrived broken or with missing items",
      "late_delivery": "the parcel is still in transit and is past its promised date",
      "address_change": "the customer wants to change the delivery address"
    }
  },
  "technical": "bugs, errors, outages and login problems",
  "account": "profile, password, plan and team members"
}

Every level needs 2 to 64 labels, and the tree holds at most 8 levels. A level with one label returns {"error":{"code":"invalid_request","message":"tree.billing.labels: each level needs 2 to 64 labels"}}.

A clean descent

Send the text and the whole tree. The API picks the branch, then picks the leaf inside that branch.

curl -s -X POST https://api.milliseconds.ai/v1/decision-machine-1/classify-tree \
  -H "content-type: application/json" \
  -H "authorization: Bearer test_sk-..." \
  -d '{
    "text": "Please cancel my Pro plan at the end of the current billing period. I do not need the extra seats any more.",
    "tree": {
      "billing": {
        "description": "payments, invoices, charges, refunds and subscriptions",
        "labels": {
          "duplicate_charge": "the customer was billed more than once for the same item",
          "refund_request": "the customer asks for money back",
          "invoice_question": "the customer asks about an invoice or receipt",
          "subscription_change": "the customer wants to upgrade, downgrade or cancel a plan"
        }
      },
      "shipping": {
        "description": "delivery, tracking, lost or damaged parcels",
        "labels": {
          "lost_parcel": "the parcel is marked delivered but the customer never received it",
          "damaged_parcel": "the parcel arrived broken or with missing items",
          "late_delivery": "the parcel is still in transit and is past its promised date",
          "address_change": "the customer wants to change the delivery address"
        }
      },
      "technical": "bugs, errors, outages and login problems",
      "account": "profile, password, plan and team members"
    }
  }'

{
  "path": ["billing", "subscription_change"],
  "label": "subscription_change",
  "probability": 0.624,
  "confidence": 0.336,
  "levels": [
    {"label":"billing","probability":0.766,"confidence":0.607,"scores":{"billing":0.766,"shipping":0,"technical":0,"account":0.234},"input_chars":325,"input_tokens":80,"inference_ms":545},
    {"label":"subscription_change","probability":0.814,"confidence":0.553,"scores":{"duplicate_charge":0.043,"refund_request":0.132,"invoice_question":0.011,"subscription_change":0.814},"input_chars":372,"input_tokens":91,"inference_ms":574}
  ]
}

The walk returns billing > subscription_change. Level 1 confidence 0.607 clears the 0.60 bar in the code below. Level 2 confidence 0.553 clears the 0.45 bar. The compound probability of 0.624 is the product of 0.766 and 0.814, so it sits below both levels. Threshold the level you act on.

That call reported x-input-chars: 885 and x-input-tokens: 216: the size of the request body, counted once. x-inference-ms is the sum over the two levels.

When confidence drops

This ticket is clear at level 1 and ambiguous at level 2:

I was charged twice for my annual subscription on 3 March and the second charge has not been refunded.

{
  "path": ["billing", "refund_request"],
  "label": "refund_request",
  "probability": 0.432,
  "confidence": 0.279,
  "levels": [
    {"label":"billing","probability":0.996,"confidence":0.98,"scores":{"billing":0.996,"shipping":0,"technical":0.004,"account":0},"input_chars":320,"input_tokens":79,"inference_ms":513},
    {"label":"refund_request","probability":0.434,"confidence":0.285,"scores":{"duplicate_charge":0.434,"refund_request":0.434,"invoice_question":0.131,"subscription_change":0},"input_chars":367,"input_tokens":90,"inference_ms":561}
  ]
}

Two leaves tie at 0.434 and level 2 confidence reports 0.285. The ticket describes a duplicate charge and a refund request. Stop at billing and send the leaf choice to a person. The API returns the tie winner anyway, so your code must read levels and drop the leaf itself.

classify-tree bills the request body once, whatever the depth. A client-side walk is a separate call per level, so it re-sends the full text each time. A three-level client walk makes three calls and sends the text three times, so it costs three times as much. You pay $0.04 per million input tokens and $0 per output token. See Pricing. One classify call took 0.43 s end to end in our measurement, so budget the depth.

Walk it yourself when you need a stop rule per level

classify-tree always walks to a leaf. It returns every level, so you can apply thresholds after the call and keep a partial path. A client-side walk goes further: it never runs the calls below a level you would reject. Pick it when you want to stop early. It costs more, because every level is its own billed call.

import requests
URL = "https://api.milliseconds.ai/v1/decision-machine-1/classify"
MIN_CONFIDENCE = {1: 0.60, 2: 0.45, 3: 0.35}   # example values: tune them
def classify(text, labels):
    r = requests.post(URL, headers={"authorization": "Bearer test_sk-..."}, json={"text": text, "labels": labels}, timeout=30)
    r.raise_for_status()
    return r.json()
def children(node):
    kids = node.get("labels") or {}
    return {k: (v if isinstance(v, dict) else {"description": v}) for k, v in kids.items()}
def walk(text, tree, depth=1, path=None):
    path = path or []
    while len(tree) == 1:                      # one child: no call needed
        name, node = next(iter(tree.items()))
        path, tree = path + [name], children(node)
    if not tree:
        return path, None
    labels = {name: node.get("description", "") for name, node in tree.items()}
    out = classify(text, labels)
    if out["confidence"] < MIN_CONFIDENCE.get(depth, 0.35):
        return path, out                       # stop: hand the choice to a person
    path = path + [out["label"]]
    kids = children(tree[out["label"]])
    if not kids:
        return path, out
    return walk(text, kids, depth + 1, path)
# The top level is the tree itself: walk(text, children({"labels": TAXONOMY})).

A node with one child needs no call. The walk appends that child and moves down. classify-tree rejects a level with one label, and so does the array form of labels on classify. The object form does not, so skip the call yourself.

Tune the thresholds

Pick the numbers per level from real data, not from a guess.

Build a golden set of 50 to 200 real tickets with the full path labelled. Then score each level on its own and tune the thresholds per level.

Next