Moderation for Marketplaces: Protecting Reviews and User-to-User Messages

In a marketplace, moderation is not a "trust & safety feature". It is the product. Once buyers stop trusting the reviews or sellers stop feeling safe in their inbox, the GMV chart rolls over and never comes back.

The two surfaces where moderation matters most in marketplaces are reviews (public, permanent, SEO-visible) and user-to-user messages (private, high-frequency, where harassment lives). They need different approaches.

Reviews: the public, permanent problem

A review is a piece of user-generated content that your SEO team relies on, your buyers make decisions from, and your sellers sometimes sue you over. The moderation bar is high.

Threats you will see

• Fake positives: sellers inflating their own ratings, possibly through "review rings".
• Fake negatives: competitors or disgruntled buyers tanking a seller's reputation with fabricated reviews.
• Retaliation: a seller leaving a harsh counter-review after getting a bad one.
• Off-topic rants: personal attacks, politics, content unrelated to the product.
• PII leakage: reviews that include phone numbers, addresses, order IDs.
• Spam and link injection: "check out my store at" in a review body.

What the model needs to check

Classic text moderation catches the spam and toxicity. For authenticity you need additional signals that a moderation API alone cannot provide:

• Account age and purchase history of the reviewer.
• IP and device correlation across reviews on the same seller.
• Linguistic fingerprint overlap (fake review rings often use similar phrasing).
• Timing clusters: 20 five-star reviews in 30 minutes is a signal.

ToxicFilter handles the content classification side. The fraud signals come from your own platform data. Combine them in a review scoring pipeline before you publish.

A sensible review workflow

1. Buyer submits a review.
2. Content moderation runs synchronously, so obvious toxicity or spam is rejected with a reason the buyer can act on.
3. If the content passes, a fraud score is computed from platform signals.
4. Low fraud risk → published immediately.
5. Medium fraud risk → published but marked pending verification (not counted toward seller rating until cleared).
6. High fraud risk → held for human review.

User-to-user messages: the private, high-frequency problem

DMs are where the worst behaviour lives. Buyers get harassed, sellers get extorted, both sides encounter sexually inappropriate contact. Privacy expectations are real: you cannot show DMs to moderators the way you can show public reviews.

The usual threats

• Harassment after a failed transaction.
• Off-platform coercion: "give me your WhatsApp and I'll pay you outside the platform".
• Scams: fake payment confirmations, overpayment schemes, phishing links.
• Sexual harassment especially toward service providers.

A privacy-respecting moderation flow

The pattern most mature marketplaces converge on:

1. All DMs pass through automated classification (never human eyes by default).
2. Confirmed high-severity messages trigger automated action: warning, rate-limit, temporary block.
3. Only messages that trigger specific categories (harassment, safety threats, illegal content) are escalated to humans, and only then with the explicit user action "Report this message".
4. Users can consent to sharing conversation history when they report. Without consent, the reviewer sees only the reported message plus a short context window.

This preserves privacy-by-default while giving you the tools to act decisively when something goes wrong.

Off-platform solicitation is its own category

Marketplaces have a unique adversarial pattern: users trying to take the transaction off-platform to avoid fees. Detection is a specific skill: phone numbers, messenger handles, specific phrases ("pay me on X", "my personal email is..."). A generic toxicity model will miss all of this. You need either a marketplace-specific model (what we train for our marketplace customers) or a combination of regex + classification.

Metrics that actually matter

DAU and GMV will not tell you if your moderation is working. Better signals:

• Report-to-message ratio: how often users manually report per 1000 messages. Should trend down.
• Re-contact rate after report: does the reported user come back and do it again? If yes, your enforcement is too soft.
• Review authenticity rate: percentage of reviews that pass fraud scoring. Volatile, but directional.
• New-seller retention: new sellers who get harassed in week one do not stick around. This is a moderation metric, even though nobody labels it as one.

Marketplace moderation is one of the highest-leverage investments a platform can make. Buyers who feel safe buy more. Sellers who feel safe list more. Both compounding, for as long as you keep doing it.