Policies

A policy is where your line sits. Until you make one, every call is judged against the shipped defaults, which are a careful opinion but somebody else's: a dating app tolerates flirting and not harassment, a children's forum is the other way round, a marketplace cares about self-promotion above all of it.

Policies are created in your dashboard and named in the call:

curl https://toxicfilter.com/api/v1/text \
  -H "Authorization: Bearer tf_live_..." \
  -H "Content-Type: application/json" \
  -d '{ "content": "...", "surface": "listing", "policy": "marketplace" }'

When you create one you can say what it is for, such as a marketplace, a contact form or a job board. That fills the new policy with the rules that kind of business usually wants, and you edit them afterwards like any other rule. A short description of what your business does, at the top of the policy, lets the model tell what has nothing to do with it.

Leave policy out and the policy chosen for the call's project is used, or your default one when the project chose none. Name one that does not exist and the call is refused with 422 unknown_policy, never quietly answered under the default, because a typo would otherwise moderate your whole site under rules nobody chose and the only symptom would be verdicts that seem slightly off for a week.

Rules in the call, without a policy

You do not have to configure anything. Send rules with the request and those are the rules, for that call only:

curl https://toxicfilter.com/api/v1/text \
  -H "Authorization: Bearer tf_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "content": "...",
    "rules": { "thresholds": { "sexual": { "block": 0.7 } } }
  }'

Only what you send is acted on. No default policy is looked up and none of our numbers are laid underneath. In the call above, a category other than sexual can score 0.99 and the answer is still allow: it is reported in scores and signals, and it decides nothing.

That is the opposite of how a stored policy behaves, and deliberately so. A policy is sparse because what it does not mention should keep following our numbers and pick up our improvements. Rules in the call mean you have said what you want acted on, and inventing a line for a category you never mentioned would be a decision you did not ask for.

Those six keys are the whole of it, and a name we do not recognise inside any of them is a 422 unknown_rule naming it, not a rule that is quietly ignored. A toxity line, or a subject spelled the way your product spells it, would otherwise act on nothing and read exactly like a rule that works.

The answer says "policy": { "slug": "inline", "version": 0 }, and the stored record says the same. We do not keep the rules you sent, so the record cannot reproduce them: if a verdict has to be defensible six months from now, that is what a named, versioned policy is for.

Rules over a policy

Send policy and rules together and the rules are laid over that policy, for that call only. The call wins for what it names, and everything else stays as the policy has it:

{
  "content": "...",
  "policy": "comments",
  "rules": {
    "thresholds": { "spam": { "block": 0.6 } },
    "terms": { "block": ["casino online"] }
  }
}

The answer names the policy and says it was not applied as saved: "policy": { "slug": "comments", "version": 3, "overridden": true }. The stored record keeps the same three facts. Without policy, rules behave as above: they are the only rules, and nothing else is acted on.

In a batch the same applies on the envelope and per item. An item that names a policy or sends rules decides for itself; one that does neither takes the envelope's.

Why a policy at all, then

Both work, and neither is a fallback for the other. Rules in the call suit somebody who already knows what they want and does not want to configure anything. A policy earns its keep when the rules belong to the site rather than to one integration:

What does travel in the call is which policy to use and where the content sits: "policy": "listings" and "surface": "listing". Both are strings, both are optional, and between them you can vary the answer per request without carrying the rules around.

Text, images, and everything else

One policy covers every kind of call that names it: comments, images, names, addresses, signups, prompts, links and conversations. There is no separate policy for pictures.

It behaves sensibly across kinds because a category only fires where the content can produce it. A photograph can be sexual or violence and can never be gibberish; a link is judged on its shape and reaches no model at all. The lines you set are the lines for whatever can reach them.

If you want pictures judged differently from comments, that is a surface, not a second policy. surface is your own label, so send "surface": "photo" with your image calls and give that surface its own numbers:

{
  "photo":   { "sexual": { "review": 0.30, "block": 0.60 } },
  "comment": { "sexual": { "review": 0.60, "block": 0.90 } }
}

Switching a category off

Both of its numbers above 1. No score reaches them, so it never flags and never blocks. That is how the shipped defaults already stop toxicity and self_harm from blocking anything.

A switched-off category still scores, and still appears in scores and signals. That is deliberate: the finding is not hidden, it is not acted on. It also costs nothing to leave on, because the detectors run once per call regardless of how many categories you care about, and the price of a call does not depend on them.

Moving a line

Each of the fifteen categories has two numbers. review is where it starts being sent to a person, block is where it is refused outright, and the gap between them is where uncertainty is allowed to live. Closing that gap is how a moderation system starts eating real posts.

Anything above 1 never fires, because no score reaches it. That is not a trick, it is how the shipped defaults already stop toxicity and self_harm from ever blocking. One mechanism instead of a separate on/off switch that would behave differently in some corner nobody tested.

You only store what you changed. Categories you never touched keep following our defaults, so when we move one you get the improvement instead of a frozen copy of whatever the numbers were the day you signed up.

Per-surface overrides

The same sentence does not weigh the same everywhere. A link to a shop is the entire point of a marketplace listing and is spam in a support thread. Send surface with the call and a policy can carry a different line for it:

{
  "listing": { "spam":   { "review": 0.75, "block": 0.95 } },
  "support": { "policy": { "review": 1.01, "block": 1.01 } }
}

Anything not mentioned falls through to the policy's own numbers, and then to ours.

surface is your own label, with three exceptions: prompt, review and job each switch a check on that would be unusable on ordinary traffic. If you moderate reviews or job ads, send those exact strings, because nothing in the answer says a gated check did not run. The three, and what each one adds.

Your own words

Three lists, and they are the part most people came for: a competitor's name, a product that may not be advertised here, the phrase this particular community is done with.

Matches arrive as an ordinary signal, with the words that matched as evidence:

{
  "category": "policy",
  "score": 0.9,
  "detector": "policy_term",
  "reason": "Matches 1 term(s) on this account's block list.",
  "evidence": ["competitor-example"]
}

Your words go through the same normalisation as ours, which is most of the value here. Type competitor once and you also catch c0mpet1tor, сompetitor with a Cyrillic es, c o m p e t i t o r and competitors, which are precisely the spellings somebody working around your rule will reach for.

The Never flag list overrules our word list as well as yours. It has to: an allowlist that could not do that would be no use to a medical forum, a gun shop, or the town of Scunthorpe.

Reputation

Off by default, and a switch on the policy. With it on, an actor's own record on your account moves the line for them: a user with eight hundred clean messages gets a little more room, one with three blocks today gets a little less.

It is the one thing here about behaviour rather than content, so read what a record is not allowed to do before switching it on. In short: it never becomes a finding, never blocks anything on its own, never leaves your account, and is always reported in the answer.

Trying a policy before you mean it

The question before every threshold change is what would this have done to last week, and the only answers used to be "try it on live traffic and find out" or "guess". Point a policy at another one and both are judged, every call:

"policy":  { "slug": "live", "version": 7 },
"shadow":  { "slug": "strict-v2", "version": 2, "decision": "block" }

The live policy decides. The one on trial is recorded and does nothing: no different answer, no different billing, no webhook. Your dashboard then says, over thirty days of your real traffic: 1,240 calls, 38 would have been decided differently, and which way each of them went.

A trial never costs a model call. Whatever the model already said is reused as it is, and only the free detectors run again, and only when the two policies have different WORDS rather than different numbers. A feature that doubled the bill would make people choose between knowing what a change does and being able to afford to find out.

A trial pointing at a policy that has since been deleted is quietly dropped rather than failing the call.

Versions

Every save bumps the policy's version, and every answer says which one it was judged under:

"policy": { "slug": "marketplace", "version": 4 }

That pair is stored with the verdict too, so every decision says which rules judged it. We keep the policy, not its history: editing one moves its version on and the numbers it had before are not stored anywhere, so the version identifies a verdict's rules rather than letting you read them back.

It is also part of the cache key, which is the practical half: the moment you press Save, every answer reached under the old rules becomes unreachable and the next call is judged afresh.

Batches

A batch takes policy on the envelope and each item may override it. An async batch records what was asked for and resolves it again as each item is judged, so a backfill queued this morning and worked through this afternoon follows the rules that are in force when each item actually gets looked at, and every record says which version that was.