Taxonomy classification

A taxonomy is a tree of labels. The support taxonomy below has four top-level areas and four leaves under each of the first two. One flat classify call puts every leaf against every other leaf, so leaves from far-apart branches steal probability from each other.

Walk the tree instead. Each level is one classify call over the children of the winner from the level above.

The taxonomy

Give every label a description. The model reads "name: description" when a description exists, and a bare name otherwise. Descriptions improve accuracy, as Writing good statements and labels shows.

{
  "billing": {
    "description": "payments, invoices, charges, refunds and subscriptions",
    "children": {
      "duplicate_charge": "the customer was billed more than once for the same item",
      "refund_request": "the customer asks for money back",
      "invoice_question": "the customer asks about an invoice or receipt",
      "subscription_change": "the customer wants to upgrade, downgrade or cancel a plan"
    }
  },
  "shipping": {
    "description": "delivery, tracking, lost or damaged parcels",
    "children": {
      "lost_parcel": "the parcel is marked delivered but the customer never received it",
      "damaged_parcel": "the parcel arrived broken or with missing items",
      "late_delivery": "the parcel is still in transit and is past its promised date",
      "address_change": "the customer wants to change the delivery address"
    }
  },
  "technical": { "description": "bugs, errors, outages and login problems", "children": {} },
  "account": { "description": "profile, password, plan and team members", "children": {} }
}

Walk the tree

Each level is a separate call, so it re-sends the full text. A three-level walk makes three calls and sends the text three times, so it costs three times as much. You pay $0.04 per million input tokens and $0 per output token. See Pricing. One classify call took 1.18 s end to end in our measurement, so budget the depth.

A clean descent

The first call picks the branch. The second call picks the leaf inside that branch.

curl -s -X POST https://api.milliseconds.ai/v1/decision-machine-1/classify \
  -H "content-type: application/json" \
  -d '{
    "text": "Please cancel my Pro plan at the end of the current billing period. I do not need the extra seats any more.",
    "labels": {
      "billing": "payments, invoices, charges, refunds and subscriptions",
      "shipping": "delivery, tracking, lost or damaged parcels",
      "technical": "bugs, errors, outages and login problems",
      "account": "profile, password, plan and team members"
    }
  }'

Level 1 result:

{"label":"billing","probability":0.766,"confidence":0.607,"scores":{"billing":0.766,"shipping":0,"technical":0,"account":0.234}}

Level 2 sends the same text with the four children of billing:

{"label":"subscription_change","probability":0.814,"confidence":0.553,"scores":{"duplicate_charge":0.043,"refund_request":0.132,"invoice_question":0.011,"subscription_change":0.814}}

The walk returns billing > subscription_change. Level 1 confidence 0.607 clears the 0.60 bar in the code below. Level 2 confidence 0.553 clears the 0.45 bar.

When confidence drops

This ticket is clear at level 1 and ambiguous at level 2:

I was charged twice for my annual subscription on 3 March and the second charge has not been refunded.

{"label":"billing","probability":0.996,"confidence":0.98,"scores":{"billing":0.996,"shipping":0,"technical":0.004,"account":0}}

{"label":"refund_request","probability":0.434,"confidence":0.285,"scores":{"duplicate_charge":0.434,"refund_request":0.434,"invoice_question":0.131,"subscription_change":0}}

Two leaves tie at 0.434 and confidence reports 0.285. The ticket describes a duplicate charge and a refund request. Stop at billing and send the leaf choice to a person.

The walk in code

import requests
URL = "https://api.milliseconds.ai/v1/decision-machine-1/classify"
MIN_CONFIDENCE = {1: 0.60, 2: 0.45, 3: 0.35}   # example values: tune them
def classify(text, labels):
    r = requests.post(URL, json={"text": text, "labels": labels}, timeout=30)
    r.raise_for_status()
    return r.json()
def children(node):
    return {k: {"description": v} for k, v in (node.get("children") or {}).items()}
def walk(text, tree, depth=1, path=None):
    path = path or []
    while len(tree) == 1:                      # one child: no call needed
        name, node = next(iter(tree.items()))
        path, tree = path + [name], children(node)
    if not tree:
        return path, None
    labels = {name: node["description"] for name, node in tree.items()}
    out = classify(text, labels)
    if out["confidence"] < MIN_CONFIDENCE.get(depth, 0.35):
        return path, out                       # stop: hand the choice to a person
    path = path + [out["label"]]
    kids = children(tree[out["label"]])
    if not kids:
        return path, out
    return walk(text, kids, depth + 1, path)

A node with one child needs no call. The walk appends that child and moves down. The array form of labels also rejects a single label with invalid_request, so skip the call.

Tune the thresholds

Pick the numbers per level from real data, not from a guess.

Build a golden set of 50 to 200 real tickets with the full path labelled. Then score each level on its own and tune the thresholds per level.

Next