COPPA, GDPR and Moderation: Privacy When Analyzing User Content

Sending user content (messages, images, profiles) to a third-party moderation API is a data processing operation. That simple fact puts it squarely inside the scope of most data protection laws. This post walks through what COPPA, GDPR (and its UK equivalent) actually require when you integrate a moderation API, and where the usual pitfalls sit.

Is user content "personal data"?

Under GDPR: almost always yes. Even without a name attached, a message is tied to a user ID, an IP, a session. Context easily re-identifies. Under the EDPB's guidance, content like messages, photos and profile text qualifies as personal data.

Under COPPA: content produced by a child under 13 is protected "personal information" by default, regardless of whether it contains identifiable details.

Under the UK Age Appropriate Design Code: same direction of travel. Content from users likely to be under 18 is treated with elevated protection.

So: your moderation pipeline processes personal data. Plan accordingly.

GDPR basics you need to get right

1. Lawful basis

You need a lawful basis for processing. For moderation, the usual options are:

• Legitimate interest (Article 6(1)(f)). Keeping the platform safe is a recognised legitimate interest. Requires a documented Legitimate Interest Assessment (LIA).
• Legal obligation (Article 6(1)(c)), where a law requires you to moderate (e.g. DSA, UK Online Safety Act).
• Consent is rarely the right basis for moderation; users cannot meaningfully withhold consent from a safety feature.

Most platforms land on legitimate interest with a fallback to legal obligation where applicable. Your privacy notice should name the basis.

2. Data Processing Agreement (DPA)

Your moderation API provider is a processor; you are the controller. Article 28 requires a DPA. It should cover:

• Purpose and scope of processing.
• Data categories and subject categories.
• Duration of processing.
• Security measures.
• Sub-processors (and whether prior written consent is required for new ones).
• International transfer mechanism (see below).
• Audit rights.
• Data return / deletion on termination.

Any reputable moderation vendor (including ToxicFilter) provides a template DPA. Sign it before production traffic flows.

3. International transfers

If the moderation provider hosts inference outside the EEA, you need a transfer mechanism: Standard Contractual Clauses (SCCs), an adequacy decision (e.g. UK-EU, Japan), or Binding Corporate Rules. Post-Schrems II, SCCs require a Transfer Impact Assessment.

The cleanest path: use a provider with EU-resident inference. If that is not possible, make sure the SCCs are in the DPA and the TIA is on file.

4. Data subject rights

Users can request access, correction, erasure. For moderation logs, this gets interesting:

• Access: if a user asks what moderation decisions were made against them, you must provide them (Article 15).
• Erasure: moderation logs can usually be retained under legitimate interest (safety, legal defence). But the scope of retention must be proportionate.
• Automated decision-making (Article 22): users have the right to not be subject to solely automated decisions with legal or similarly significant effect. Moderation is often borderline here: account termination probably qualifies. The fix is human review on appeal.

5. Retention

Do not retain moderation data forever "just in case". Typical retention periods:

• Raw content passed to the moderation API: not retained at all, or retained briefly (hours to days) for debugging.
• Decision logs (category, score, action): 6 to 24 months, justified by safety and legal response needs.
• Samples used for model training: with explicit opt-in only, and anonymised where possible.

COPPA specifics

Under COPPA, platforms operating in the US and aimed at children under 13 (or with actual knowledge of users under 13) must:

• Provide notice of data collection practices, moderation processing included, to parents.
• Obtain verifiable parental consent before collecting personal information.
• Minimise data collection to what is reasonably necessary.
• Allow parents to review and delete child data.
• Establish reasonable procedures for data security and confidentiality.

Moderation processing must be included in your COPPA disclosures. The FTC has fined multiple apps for moderation vendors that were not disclosed. "It is a technical detail" is not a defense.

The three practical questions to ask every vendor

1. "Will our user content be used to train your models?" (Hope for: no, or only with our explicit opt-in.)
2. "Where is the inference performed, and can we pin it to a region?" (Hope for: yes, at least for EU.)
3. "How long do you retain request data, and how do I verify deletion?" (Hope for: defined short window, documented deletion process.)

Any vendor that dodges these is not ready for regulated use cases. Any vendor that answers them clearly and contractually is a vendor worth considering, even if they cost more than the one that handwaves the questions.

The overarching principle

Data protection regulators do not expect perfection. They expect deliberation. A documented decision ("we use provider X because of Y, with these safeguards") is defensible. A default integration ("we copy-pasted the example") is not. Treat moderation's data protection posture like any other piece of production infrastructure: designed, reviewed, documented.