Batching | milliseconds.ai documentation

Every capability accepts a batch. You batch on two axes:

texts — the same question against many texts. Every capability supports it.
statements / questions — many questions against the same text. Only yes-no and answer support these.

The two axes combine. The envelope tells you which axis you used.

The batch keys

Key	Capabilities	Replaces	Limit
`texts`	all eight	`text`	1–32 items, each 1–20,000 characters
`statements`	`yes-no`	`statement`	1–32 items, each min 1 character
`questions`	`answer`	`question`	1–32 items, each min 1 character

text and texts are mutually exclusive, and so is image with texts. So are statement and statements, and question and questions. Send one key of each pair, on every capability. See Gotchas below.

Batch over texts

Replace text with texts. The response becomes {"results": [...]}, one entry per text, in input order.

curl -s -X POST https://api.milliseconds.ai/v1/decision-machine-1/classify \
  -H 'content-type: application/json' \
  -H 'authorization: Bearer test_sk-...' \
  -d '{
    "texts": [
      "I was charged twice for my subscription this month and support has not replied.",
      "My package still has not arrived after three weeks.",
      "I cannot log in to my account since yesterday."
    ],
    "labels": {
      "billing": "payments, invoices, charges and refunds",
      "shipping": "delivery, tracking and packages",
      "account": "login, passwords and profile settings"
    }
  }'

Response, one entry per text:

{
  "results": [
    {"label":"billing","probability":0.999,"confidence":0.99,"scores":{"billing":0.999,"shipping":0,"account":0.001}},
    {"label":"shipping","probability":0.988,"confidence":0.938,"scores":{"billing":0.011,"shipping":0.988,"account":0.001}},
    {"label":"account","probability":0.999,"confidence":0.994,"scores":{"billing":0.001,"shipping":0,"account":0.999}}
  ]
}

results[i] always matches texts[i]. Zip the two arrays by index. No id field exists.

Batch over statements or questions

yes-no takes statements, answer takes questions. Both return {"results": [...]} in input order. Each result echoes the statement or question it scored.

curl -s -X POST https://api.milliseconds.ai/v1/decision-machine-1/yes-no \
  -H 'content-type: application/json' \
  -H 'authorization: Bearer test_sk-...' \
  -d '{
    "text": "This is urgent, I need a refund today.",
    "statements": [
      "The customer expresses urgency.",
      "The customer is asking about shipping."
    ]
  }'

{
  "results": [
    {"statement":"The customer expresses urgency.","answer":true,"probability":1},
    {"statement":"The customer is asking about shipping.","answer":false,"probability":0}
  ]
}

This axis adds no inference call, only the length of the extra statements. All the statements go into one inference call. Two statements measured 0.49 s against 0.42 s for one statement. Three questions on answer measured 0.40 s, the same as one question.

Both axes at once

Combine texts with statements or questions. The envelope nests: an outer results per text, an inner results per statement or question.

curl -s -X POST https://api.milliseconds.ai/v1/decision-machine-1/answer \
  -H 'content-type: application/json' \
  -H 'authorization: Bearer test_sk-...' \
  -d '{
    "texts": [
      "John Ternus announced the iPhone Duo in Cupertino. It starts at $1,999.",
      "Sundar Pichai announced the Pixel 11 in Mountain View. It starts at $799."
    ],
    "questions": ["Who announced the product?", "How much does it cost?"]
  }'

Response:

{
  "results": [
    {"results":[
      {"question":"Who announced the product?","answer":"John Ternus","probability":0.998,"start":0,"end":11},
      {"question":"How much does it cost?","answer":"$1,999","probability":0.996,"start":64,"end":70}
    ]},
    {"results":[
      {"question":"Who announced the product?","answer":"Sundar Pichai","probability":0.998,"start":0,"end":13},
      {"question":"How much does it cost?","answer":"$799","probability":0.994,"start":68,"end":72}
    ]}
  ]
}

Read it as results[textIndex].results[questionIndex]. Offsets are relative to that text.

start and end index the text at the same position in texts. Never index them into a joined string.

The 32 limit

texts, statements and questions each cap at 32 items. A 33rd item fails validation before any inference runs.

{"error":{"code":"invalid_request","message":"texts: Too big: expected array to have <=32 items"}}

The status is 400. Split larger workloads into chunks of 32 in your own code. Each text still carries the 20,000 character limit.

A batch spends inference request units against your key type’s RPM limit. Test keys allow 30 units per minute; production limits depend on your tier.

One HTTP call with 24 texts consumes 24 units. Text counts multiply by statement or question counts: two texts with three statements consume six units.

Usage counts admitted units even if processing later fails. See Limits and rate limits.

How a batch spreads across inference slots

One scheduler leases each inference call to a free slot. Each slot runs one call at a time.

texts costs one inference call per text. The API fans them out and waits for all of them. A batch of 8 texts occupies up to 8 slots at once.
statements and questions share inference slots. They still multiply the inference request units used for RPM and usage.
Your batch shares the pool with everyone else. More texts means more parallel work, not slower work per text.

Measured evidence: extract over 2 texts took 0.44 s against 0.41 s for one text. yes-no over 2 texts × 2 statements took 0.49 s against 0.42 s for a single statement. The batch runs wide, not long.

When no slot frees in time, the call returns 529 with code overloaded and the message Every inference slot stayed busy. Retry with backoff. A large texts batch asks for many slots at once, so it meets this limit first. Retry the whole call with exponential backoff.

Cost and headers

Cost scales with everything you send, text plus statements, questions, labels, or schema, whatever the batch shape. The price is $0.04 per million input tokens and $0 per output token. See Pricing. Every capability response carries two headers:

Header	Value
`x-input-chars`	The number of input characters. For `texts`, the sum over all items.
`x-input-tokens`	The input tokens billed for this call.

The three-ticket classify batch above returned x-input-chars: 176 and x-input-tokens: 43.

Every request needs an API key. See Authentication.

Gotchas

Sending no input field returns 400

A body with no text, no texts and no image answers 400 with body: provide text, texts or image. Every capability enforces this, yes-no included. Read the message as “send exactly one of the three”.

A batch is one HTTP request, so one failure fails everything

There is no per-item error. A validation error rejects the whole body. A 502 runner_error or 529 overloaded ends the whole call. Keep batches small enough that a retry is cheap.

Input — how to shape text and the 20,000 character limit.
RAG passage filtering — one yes-no batch over retrieved passages.
Errors — the 529 backpressure signal and retry guidance.