Recipes

Content moderation

Rate harm on a scale, flag the hard cases with yes-no, and send the uncertain middle to a person.

Moderation asks two different questions. How bad is this? is a scale. Does this break a named rule? is a yes or no. Ask them separately and you get two numbers you can act on.

This recipe runs rate for severity and one batched yes-no call for the hard flags. Every response below comes from a live call against https://api.milliseconds.ai.

QuestionCapabilityWhat you act on
How severe is this post?ratescore, level, confidence
Does it threaten, dox, or spam?yes-noOne probability per rule
Who reviews the rest?Your codeThe confidence bands below

Rate the severity

Write the scale as level descriptions, ordered low to high. A scale takes 2 to 10 levels. Numbers (“1”, “2”, “3”) carry no meaning for the model — describe each level.

curl -X POST https://api.milliseconds.ai/v1/decision-machine-1/rate \
  -H 'content-type: application/json' \
  -d '{
    "text": "Honestly this is the laziest article I have read all year. Did anyone proofread it?",
    "scale": [
      "Civil and harmless",
      "Rude or dismissive",
      "Insulting or demeaning toward a person or group",
      "Hateful or threatening"
    ]
  }'
{"score":1.509,"level":1,"confidence":0.463,"scores":[0,0.501,0.488,0.011]}

Read all three numbers. level 1 is the argmax, but levels 1 and 2 split almost evenly (0.501 against 0.488). score 1.509 sits between them, and confidence 0.463 reports the ambiguity. This post belongs in a review queue, not in an automatic action.

score is the probability-weighted position, Σ p_i · i. It ranges from 0 to scale.length − 1. level is the single most likely index. They disagree when the distribution spreads across levels, and that disagreement is your review signal.

Flag the hard cases

A scale smears a rule violation across levels. Name each rule as its own statement and send all of them in one yes-no call. The statements go into a single inference call, so the batch costs about the same as one statement.

curl -X POST https://api.milliseconds.ai/v1/decision-machine-1/yes-no \
  -H 'content-type: application/json' \
  -d '{
    "text": "You people are pathetic. I know where you live and I will make you regret posting this.",
    "statements": [
      "The post threatens violence against a person.",
      "The post attacks a person or group over race, religion, gender or origin.",
      "The post describes self-harm or suicidal intent.",
      "The post shares private personal information such as an address or phone number.",
      "The post is unsolicited advertising or spam."
    ]
  }'
{"results":[
  {"statement":"The post threatens violence against a person.","answer":true,"probability":1},
  {"statement":"The post attacks a person or group over race, religion, gender or origin.","answer":false,"probability":0.122},
  {"statement":"The post describes self-harm or suicidal intent.","answer":false,"probability":0.154},
  {"statement":"The post shares private personal information such as an address or phone number.","answer":false,"probability":0.243},
  {"statement":"The post is unsolicited advertising or spam.","answer":false,"probability":0.3}
]}

Each probability stands on its own. The five numbers do not sum to 1, so read every flag against its own threshold. Here the doxxing flag reaches 0.243 on “I know where you live” and still stays below any action band.

The same post scored {"score":2,"level":3,"confidence":0.208,"scores":[0,0.333,0.333,0.333]} on the severity scale — a three-way tie. The scale was unsure; the threat statement was not. This is why you run both calls.

The rude article comment returns the opposite picture. Its highest flag is spam at probability 0.141, and the threat flag drops to 0.008.

Write each statement as a case, never as a verdict. Hint text such as when_true: "yes" and when_false: "no" carries no meaning to the model. A measured probe flipped a correct answer: true (probability 1) to answer: false (probability 0.004) with that wording.

Route the item

Set one threshold per action, not one for the system. Removal needs more evidence than a warning label does. Raise the bar with the blast radius.

SignalBandAction
Self-harm flag above 0.5ActRoute to the crisis workflow
Any other hard flag above 0.9ActRemove and notify
score at or above 2.0 and confidence at or above 0.7ActRemove
score at or above 1.0 with confidence below 0.5ReviewHuman queue
score below 1.0 and every flag below 0.3ActPublish

Build the flags dict from the yes-no response, then route on both calls.

SELF_HARM = "The post describes self-harm or suicidal intent."
# flags = {r["statement"]: r["probability"] for r in body["results"]}
def route(rating: dict, flags: dict[str, float]) -> str:
    if flags[SELF_HARM] >= 0.5:
        return "crisis"
    if max(flags.values()) >= 0.9:
        return "remove"
    if rating["score"] >= 2.0 and rating["confidence"] >= 0.7:
        return "remove"
    if rating["score"] < 1.0 and max(flags.values()) < 0.3:
        return "publish"
    return "review"

Tune these numbers against your own labelled examples. The bands above are a starting shape, not a measured policy.

Batch a queue

Swap text for texts and keep the same scale. Send up to 32 posts. The results come back in input order, one entry per post — a compliment, the rude comment, and the threat:

{"results":[
  {"score":0,"level":0,"confidence":1,"scores":[1,0,0,0]},
  {"score":1.509,"level":1,"confidence":0.463,"scores":[0,0.501,0.488,0.011]},
  {"score":2,"level":3,"confidence":0.208,"scores":[0,0.333,0.333,0.333]}
]}

The clean post returns confidence 1 and score 0. Publish it without a human.

Operational notes

  • Each capability takes at most 32 texts, and yes-no at most 32 statements.
  • Batching texts costs one inference call per text. Batching statements does not.
  • Measured latency: rate with 4 levels 1.02 s, yes-no with 2 statements 1.12 s.
  • A text holds up to 20,000 characters. The API chunks text over 2,000 characters, and each level keeps its highest score over the chunks. A long thread scores on its worst part.
  • Write scale and statement text in English, even for non-English posts.
  • Handle overloaded (529) with backoff. Decide in advance whether a failed check holds the post or publishes it.

No API key is required during the launch period.

Next