RAG passage filtering | decision-machine-1 documentation

Vector search returns the nearest passages, not the relevant ones. A top-10 retriever returns 10 passages whatever the question. The weak ones still enter your prompt and cost tokens. They also pull the generator toward the wrong answer.

This recipe adds one filter step between retrieval and generation. You send the passages as texts and one statement that describes relevance. You get one probability per passage, in input order.

The shape of the call

curl -X POST https://api.milliseconds.ai/v1/decision-machine-1/yes-no \
  -H "Content-Type: application/json" \
  -d '{
    "texts": [
      "The free tier includes 5 GB of storage and 100 GB of monthly bandwidth. Paid plans start at 100 GB of storage.",
      "Our offices are closed on 25 December and 1 January. Support tickets are answered on the next business day.",
      "Storage quotas are enforced per workspace. When a workspace passes its storage limit, uploads fail with a 507 response."
    ],
    "statement": "This passage states how much storage the free tier includes."
  }'

The real response, captured from https://api.milliseconds.ai:

{
  "results": [
    {"statement":"This passage states how much storage the free tier includes.","answer":true,"probability":1},
    {"statement":"This passage states how much storage the free tier includes.","answer":false,"probability":0.007},
    {"statement":"This passage states how much storage the free tier includes.","answer":false,"probability":0}
  ]
}

Passage 1 answers the question. Passage 2 is a calendar notice. Passage 3 shares the word “storage” and still scores 0. Embeddings reward topical overlap. This call rejects it. The response carried x-input-tokens: 82, the input tokens billed for this call.

No API key is required during the launch period.

Keep, flag, drop

One probability supports three actions. Pick the two boundaries per application, not per system.

Band	Action	Why
`probability >= 0.9`	Keep the passage	The passage carries the fact. Put it in the prompt.
`0.5 <= probability < 0.9`	Flag the passage	Partial support. Keep it last in the prompt, or hold it for a second retrieval pass.
`probability < 0.5`	Drop the passage	The passage is off-question. It only adds tokens.

A second live capture shows the middle band. The same statement over three shorter passages returned 1, 0.599 and 0.003. The 0.599 passage reads “Paid plans start at 100 GB of storage. The free tier gets a much smaller allowance.” It points at the answer without stating it.

Start with 0.5 and 0.9, then move both numbers with a labelled set. Tuning thresholds turns 50 to 200 real questions into precision and recall at each cut.

The filter in code

import requests
KEEP, FLAG = 0.9, 0.5
def filter_passages(question: str, passages: list[str]) -> dict:
    response = requests.post(
        "https://api.milliseconds.ai/v1/decision-machine-1/yes-no",
        json={
            "texts": passages,
            "statement": f"This passage contains the information needed to answer: {question}",
        },
        timeout=30,
    )
    scored = zip(passages, response.json()["results"])
    keep, flag = [], []
    for passage, result in scored:
        if result["probability"] >= KEEP:
            keep.append(passage)
        elif result["probability"] >= FLAG:
            flag.append(passage)
    return {"keep": keep, "flag": flag}

results arrives in the order of texts, so the index maps back to the passage. When keep and flag are both empty, answer “I do not know” instead of calling the generator.

Write the statement as a claim

The statement describes the passage, not the verdict. Build it from the question and keep the wording specific.

Good: This passage states how much storage the free tier includes.
Good: This passage contains the information needed to answer: what is the refund window?
Bad: relevant. The model reads the statement text, and one word describes nothing.

when_true and when_false hints can hurt here. A live run added two hints to the same statement and the same three passages:

when_true: The passage states the storage amount included in the free tier.
when_false: The passage is about another subject and does not state the free tier storage amount.

All three passages passed, at 1, 0.996 and 0.999. The hint pair normalizes the two scores against each other. The broad false-side hint matched nothing, so every passage won. Without hints the model scores the statement alone, and the separation returns.

Test the hints before you ship them. Bad hints turn a working filter into a filter that keeps everything. Writing good statements and labels covers the failure in detail.

Cost and limits

texts accepts up to 32 passages per call. Retrieve top 10 to top 30 and filter the whole set in one request.
The service runs one inference call per text and spreads the calls across its slots. A single yes-no call takes 0.75 s to 1.25 s. Batching explains how a batch spreads.
Each passage takes up to 20,000 characters. The service chunks a passage over 2,000 characters and keeps the highest chunk score. Long text and chunking covers the split.
Cost scales with the length of the passages and the statement you send, at $0.04 per million input tokens and $0 per output token. See Pricing. One statement across a batch of passages counts once per passage.
Write the statement in English even when the passages use another language. Languages covers non-English input.

Add a second statement with statements to score relevance and freshness in one pass. The response then nests as {"results":[{"results":[...]}]}, one outer entry per passage. Send only the flagged cases to a large model with Cascade to an LLM.

Yes / no — every field, hint rule, and limit.
Thresholds and confidence routing — pick the two boundaries per action.
Golden sets — the labelled questions you tune those boundaries with.