Bias in Moderation: What We Do to Detect and Correct It

Every moderation model inherits the biases of its training data. The question is not whether the bias exists, but whether the people running the model know about it, measure it, and work to reduce it. This post is a transparent look at how we audit our models at ToxicFilter: what we find, and what we do about it.

Where bias comes from

Three main sources, each one real:

1. Label bias

Who labels the training data decides what "toxic" means. A team of annotators from one region, one demographic, one set of cultural references, will produce a labelled corpus that reflects their norms, not the users the model will eventually judge.

2. Sampling bias

If the data you train on comes disproportionately from one platform, one language, one population, the model learns patterns specific to that source and misapplies them elsewhere. A model trained primarily on English-language Twitter does badly on, say, private messages in Korean.

3. Historical bias

Models trained on historical moderation decisions inherit the biases of the past. If a platform historically over-enforced against certain user groups, a model trained on those decisions will too, and call it objectivity.

What the specific biases look like

Bias in content moderation tends to cluster in predictable patterns. From our own audits and from published academic research:

• Identity mentions get flagged more often. Messages that mention "gay", "Muslim", "trans" get classified as toxic at higher rates than content-equivalent messages that do not mention those terms, because the training data saw these terms more often in attacks than in neutral discussion.
• African American Vernacular English gets flagged more often than white-mainstream English. A well-known finding going back to the Perspective API audits of 2019; we see it in our own data too.
• Reclaimed slurs are misclassified. A term used affectionately within an in-group is often flagged as hate speech.
• Non-English languages underperform. Languages with less training data (Catalan, Swahili, Tagalog) show systematically worse F1 than majority languages.
• Gender-coded harassment. Harassment targeting women is sometimes under-flagged because subtle misogynist patterns are less represented in toxicity datasets.

How we audit

Our quarterly model audit has three parts.

Identity-term sensitivity test

We run the model on matched pairs: the same sentence with and without an identity term. If classification flips (or confidence jumps meaningfully) when the identity term is added, that is a bias signal. We track this across dozens of identity dimensions and flag regressions.

Cross-demographic performance

We partition a labelled test set by linguistic markers that correlate with demographic groups (while avoiding any actual identifying data). We compute precision and recall per partition. Gaps are investigated.

Annotator disagreement audit

Our labelling protocol requires at least three annotators per sample for contested categories, with diverse backgrounds. Cases where annotators systematically disagree along demographic lines are surfaced as policy-ambiguous: they go to a human review flow and get used to train the model toward abstention rather than confident misclassification.

What we do about it

A few things that actually move the needle:

1. Counterfactual data augmentation

For every training example that mentions identity terms, we generate variants with different terms and ensure labels are consistent. This reduces the model's reliance on the terms themselves.

2. Diverse annotation teams

Annotator diversity is not window-dressing. It is a measurable quality input. Our label-consistency metrics improve when the annotator pool reflects the speaker communities in the data.

3. Calibration by subgroup

Once we detect a subgroup where the model is miscalibrated, we adjust the decision threshold for that subgroup to equalise error rates. This is not without trade-offs: it is a deliberate product decision about which kinds of fairness you want.

4. Abstention as an output

For high-stakes categories on ambiguous content, we train the model to output "uncertain" rather than force a binary classification. These go to human review, where nuance is appropriate.

5. Transparency reports

We publish per-language, per-category performance numbers on our public benchmark. If the model underperforms on Welsh compared to English, the docs say so. Customers deserve to know.

What customers can do

• Test on your own data. Benchmarks published by vendors are a starting point, not a guarantee. Run the model on a sample of your content; measure error rates; look for patterns.
• Avoid identity-term keyword rules layered on top of the model. They encode bias directly.
• Monitor appeal patterns. If users from certain regions, languages or demographics appeal disproportionately, that is a bias signal worth investigating.
• Feed back mistakes. Every false positive you catch and send back to the vendor is an opportunity to reduce bias for everyone.

The honest caveat

No moderation model is unbiased. The research community has shown this repeatedly. What a responsible vendor can do is measure the bias, reduce it where possible, and be transparent about what is left. Anyone claiming their model is bias-free is either not auditing, or not telling the truth. The rest of us have more work to do, and a responsibility to show it.