Batching
Every capability accepts a batch. You batch on two axes:
texts— the same question against many texts. Every capability supports it.statements/questions— many questions against the same text. Onlyyes-noandanswersupport these.
The two axes combine. The envelope tells you which axis you used.
The batch keys
text and texts are mutually exclusive. So are statement and statements, and question and questions. Send one key of each pair. yes-no is the one exception. See Gotchas below.
Batch over texts
Replace text with texts. The response becomes {"results": [...]}, one entry per text, in input order.
Response, one entry per text:
results[i] always matches texts[i]. Zip the two arrays by index. No id field exists.
Batch over statements or questions
yes-no takes statements, answer takes questions. Both return {"results": [...]} in input order. Each result echoes the statement or question it scored.
This axis adds no inference call, only the length of the extra statements. All the statements go into one inference call. Two statements measured 1.12 s against 0.75–1.25 s for one statement. Three questions on answer measured 0.56 s, the same as one question.
Both axes at once
Combine texts with statements or questions. The envelope nests: an outer results per text, an inner results per statement or question.
Response:
Read it as results[textIndex].results[questionIndex]. Offsets are relative to that text.
start and end index the text at the same position in texts. Never index them into a joined string.
The 32 limit
texts, statements and questions each cap at 32 items. A 33rd item fails validation before any inference runs.
The status is 400. Split larger workloads into chunks of 32 in your own code. Each text still carries the 20,000 character limit.
How a batch spreads across inference slots
One scheduler leases each inference call to a free slot. Each slot runs one call at a time.
textscosts one inference call per text. The worker fans them out and waits for all of them. A batch of 8 texts occupies up to 8 slots at once.statementsandquestionscost nothing extra. Every statement becomes a label inside the same call.- Your batch shares the pool with everyone else. More texts means more parallel work, not slower work per text.
Measured evidence: extract over 2 texts took 1.03 s against 0.72 s for one text. yes-no over 2 texts × 2 statements took 1.08 s, inside the 0.75–1.25 s band of a single statement. The batch runs wide, not long.
When no slot frees in time, the call returns 529 with code overloaded and the message Every inference slot stayed busy. Retry with backoff. A large texts batch asks for many slots at once, so it meets this limit first. Retry the whole call with exponential backoff.
Cost and headers
Cost scales with everything you send, text plus statements, questions, labels, or schema, whatever the batch shape. The price is $0.04 per million input tokens and $0 per output token. See Pricing. Every capability response carries two headers:
The three-ticket classify batch above returned x-input-chars: 176 and x-input-tokens: 43.
No API key is required during the launch period.
Gotchas
Sending neither text nor texts reports a misleading message
Every capability except yes-no answers 400 with body: provide text or texts, not both. The message is wrong for that case. Read it as “send exactly one of the two”.
yes-no does not enforce the text / texts rule
yes-no applies its own statement / statements rule instead. It accepts both text and texts at once, uses text, and ignores texts. It also accepts neither, and returns {"results":[]} with x-input-chars: 0.
A batch is one HTTP request, so one failure fails everything
There is no per-item error. A validation error rejects the whole body. A 502 runner_error or 529 overloaded ends the whole call. Keep batches small enough that a retry is cheap.
Next
- Input — how to shape text and the 20,000 character limit.
- RAG passage filtering — one
yes-nobatch over retrieved passages. - Errors — the
529backpressure signal and retry guidance.