Dating apps
Profiles, bios, photos and messages, checked as they are written. ToxicFilter reads the same person across a conversation, so it sees the sweet talk turn into a plane ticket, the WhatsApp number in a bio and an adult pushing somebody who said they are fifteen to another app, and it tells you why in a sentence.
First the compliments, then "I don't use this app much, let's talk on WhatsApp", then a story nobody can check, then the money: a ticket, a customs fee, a frozen account, or an investment platform an uncle taught them about. Each message alone is somebody being keen. The order is the scam.
A phone number or a chat handle in the bio, put there to take people off your app before they have matched. Sometimes it is an eager user; often it is the first step of a scam or of selling something your app does not allow.
A fifteen-year-old who lied at signup, and an adult who answers by asking for secrecy and a move to Snapchat. A stated age is not a finding, and treating it as one would accuse children of being children. What matters is what the other person does next.
Dating is adults flirting, so a filter that refuses anything suggestive empties the app. An unsolicited explicit picture or a stream of insults after somebody says no are a different thing, and they arrive in private messages nobody else will ever report.
Judged right now by the engine the API uses, under the template for this kind of platform, with no model involved.
A romance scam
POST /v1/conversation
Hi! Your photos from Lisbon are lovely.
Thank you! I am an engineer on an oil rig, so I travel a lot. I have fallen for you, my love. Let's talk on WhatsApp, I don't use this app much.
My love, my account is frozen and I need you to pay for my ticket so I can come and see you.
Affection and a move to WhatsApp are how people who like each other talk. With an oil rig nobody can check, the first two messages are already held for review; money asked for by the same person completes the shape, and it is refused.
{
"decision": "block",
"flagged": [
"scam",
"spam"
],
"scores": {
"scam": 0.9,
"spam": 0.45
},
"signals": [
{
"category": "spam",
"score": 0.45,
"reason": "Proposes continuing outside the platform.",
"evidence": [
"talk on whatsapp",
"lets talk on whatsapp"
]
},
{
"category": "scam",
"score": 0.9,
"reason": "Reads like a romance scam: affection, a move to another app, a request for money and a story nobody can check, from the same person.",
"evidence": [
"my love",
"i have fallen for you",
"lets talk on whatsapp",
"talk on whatsapp",
"i dont use this app much",
"pay for my ticket",
"my account is frozen",
"oil rig"
]
}
],
"model": {
"read": false
},
"took_ms": 30
}
An approach to somebody who said they are 15
POST /v1/conversation
hi im 15 is that ok
age is just a number. add me on snapchat and dont tell your parents
The fifteen-year-old's own words are never a finding. What is refused is the other person asking for secrecy and a move to another app after reading them, and it needs a person now.
{
"decision": "block",
"flagged": [
"minor_safety"
],
"scores": {
"minor_safety": 0.9
},
"signals": [
{
"category": "minor_safety",
"score": 0.9,
"reason": "Somebody in this conversation has said they are 15. The other participant asked for it to be kept secret, asked to move to another app. This needs a person now.",
"evidence": []
}
],
"model": {
"read": false
},
"took_ms": 11
}
A WhatsApp number in a bio
POST /v1/text
Hiking on Sundays, terrible at karaoke. Add me on WhatsApp: +44 7700 900456
A phone number alone would be held for review. Pointing people to a private chat app is what the dating template refuses, because that is how a profile takes people off your app before they have matched.
{
"decision": "block",
"flagged": [
"spam",
"personal_data"
],
"scores": {
"spam": 0.65,
"personal_data": 0.5
},
"signals": [
{
"category": "personal_data",
"score": 0.5,
"reason": "Contains what looks like a phone number.",
"evidence": [
"447•••••••56"
]
},
{
"category": "spam",
"score": 0.65,
"reason": "Points the reader to a private chat app, which is where a deal leaves the platform.",
"evidence": [
"WhatsApp: +44 7700 900456"
]
}
],
"model": {
"read": false
},
"took_ms": 10
}
A number after a first date
POST /v1/conversation
That was such a fun date!
It really was. Text me on WhatsApp, it's easier than this app: +44 7700 900321
Held, not refused: two people who met may want to swap numbers, and a person or your own rule decides. Suggesting WhatsApp without the number is held too, as a move off the app.
{
"decision": "review",
"flagged": [
"personal_data",
"spam"
],
"scores": {
"personal_data": 0.5,
"spam": 0.45
},
"signals": [
{
"category": "personal_data",
"score": 0.5,
"reason": "Contains what looks like a phone number.",
"evidence": [
"447•••••••21"
]
},
{
"category": "spam",
"score": 0.45,
"reason": "Proposes continuing outside the platform.",
"evidence": [
"text me on whatsapp"
]
}
],
"model": {
"read": false
},
"took_ms": 7
}
An honest opener
POST /v1/text
Hi Sam! I saw you like hiking too. Have you done the coastal path near Brighton? I am still looking for a good one.
Nearly every message looks like this. It is settled by the free checks in about a millisecond, and nothing is paid for.
{
"decision": "allow",
"flagged": [],
"signals": [],
"model": {
"read": false
},
"took_ms": 7
}Judged right now by the engine the API uses, under the template for this kind of platform, with no model involved.
A romance scam
POST /v1/conversation
Hi! Your photos from Lisbon are lovely.
Thank you! I am an engineer on an oil rig, so I travel a lot. I have fallen for you, my love. Let's talk on WhatsApp, I don't use this app much.
My love, my account is frozen and I need you to pay for my ticket so I can come and see you.
Affection and a move to WhatsApp are how people who like each other talk. With an oil rig nobody can check, the first two messages are already held for review; money asked for by the same person completes the shape, and it is refused.
{
"decision": "block",
"flagged": [
"scam",
"spam"
],
"scores": {
"scam": 0.9,
"spam": 0.45
},
"signals": [
{
"category": "spam",
"score": 0.45,
"reason": "Proposes continuing outside the platform.",
"evidence": [
"talk on whatsapp",
"lets talk on whatsapp"
]
},
{
"category": "scam",
"score": 0.9,
"reason": "Reads like a romance scam: affection, a move to another app, a request for money and a story nobody can check, from the same person.",
"evidence": [
"my love",
"i have fallen for you",
"lets talk on whatsapp",
"talk on whatsapp",
"i dont use this app much",
"pay for my ticket",
"my account is frozen",
"oil rig"
]
}
],
"model": {
"read": false
},
"took_ms": 30
}
An approach to somebody who said they are 15
POST /v1/conversation
hi im 15 is that ok
age is just a number. add me on snapchat and dont tell your parents
The fifteen-year-old's own words are never a finding. What is refused is the other person asking for secrecy and a move to another app after reading them, and it needs a person now.
{
"decision": "block",
"flagged": [
"minor_safety"
],
"scores": {
"minor_safety": 0.9
},
"signals": [
{
"category": "minor_safety",
"score": 0.9,
"reason": "Somebody in this conversation has said they are 15. The other participant asked for it to be kept secret, asked to move to another app. This needs a person now.",
"evidence": []
}
],
"model": {
"read": false
},
"took_ms": 11
}
A WhatsApp number in a bio
POST /v1/text
Hiking on Sundays, terrible at karaoke. Add me on WhatsApp: +44 7700 900456
A phone number alone would be held for review. Pointing people to a private chat app is what the dating template refuses, because that is how a profile takes people off your app before they have matched.
{
"decision": "block",
"flagged": [
"spam",
"personal_data"
],
"scores": {
"spam": 0.65,
"personal_data": 0.5
},
"signals": [
{
"category": "personal_data",
"score": 0.5,
"reason": "Contains what looks like a phone number.",
"evidence": [
"447•••••••56"
]
},
{
"category": "spam",
"score": 0.65,
"reason": "Points the reader to a private chat app, which is where a deal leaves the platform.",
"evidence": [
"WhatsApp: +44 7700 900456"
]
}
],
"model": {
"read": false
},
"took_ms": 10
}
A number after a first date
POST /v1/conversation
That was such a fun date!
It really was. Text me on WhatsApp, it's easier than this app: +44 7700 900321
Held, not refused: two people who met may want to swap numbers, and a person or your own rule decides. Suggesting WhatsApp without the number is held too, as a move off the app.
{
"decision": "review",
"flagged": [
"personal_data",
"spam"
],
"scores": {
"personal_data": 0.5,
"spam": 0.45
},
"signals": [
{
"category": "personal_data",
"score": 0.5,
"reason": "Contains what looks like a phone number.",
"evidence": [
"447•••••••21"
]
},
{
"category": "spam",
"score": 0.45,
"reason": "Proposes continuing outside the platform.",
"evidence": [
"text me on whatsapp"
]
}
],
"model": {
"read": false
},
"took_ms": 7
}
An honest opener
POST /v1/text
Hi Sam! I saw you like hiking too. Have you done the coastal path near Brighton? I am still looking for a good one.
Nearly every message looks like this. It is settled by the free checks in about a millisecond, and nothing is paid for.
{
"decision": "allow",
"flagged": [],
"signals": [],
"model": {
"read": false
},
"took_ms": 7
}Each a score of its own, with its own lines, and the reason in a sentence whenever one acts.
Pick it when you create a policy and these rules are written for you, ready to edit. Everything it does not mention keeps following our defaults.
Set by the template
Send the text with where it will appear, and act on the decision.
curl https://toxicfilter.com/api/v1/text \
-H "Authorization: Bearer $TOXICFILTER_KEY" \
-d content="Hi! Your photos from Lisbon are lovely." \
-d surface=profileWhat to check, when to check it and how to set the rules, for profiles, photos and private messages.
Call the API before a profile goes live and before a message is delivered, and act on one of three answers: deliver it, hold it for a person, or refuse it. Almost every message on a dating app is somebody saying hello, and the free checks settle that in about a millisecond, so nobody waits for a reply because of moderation. The model only reads what they leave open, and pictures. What lands in review waits in a queue, in your dashboard or through the API, and a signed webhook tells your app when somebody approves it.
A romance scam is never one message. It starts as affection, moves to WhatsApp or Telegram, adds a story nobody can check (an oil rig, a military mission) and ends in money: a ticket, a customs fee, a hospital bill, or a trading platform. Send the thread to /v1/conversation and ToxicFilter reads the same person across it. Affection and a move to another app are not a romance scam on their own, because that is how people who like each other talk: the dating template only holds the move for a look, as it holds any invitation to leave the app. Money or an investment pitch beside them is refused under the dating template, and the reason names what was found.
A bio that says "add me on WhatsApp" with a number is a profile working for another app. ToxicFilter reports the number as personal_data and the pointer to a chat app as spam. The dating template holds a number alone for review and refuses the pair. If you would rather publish the bio without the number, ask for redact and the answer carries the same text with it masked.
Somebody under eighteen saying their age is not a violation by them, and ToxicFilter never treats it as one. What it reads is the other participant: secrecy, a move to another app, photos, gifts, isolation. After a stated age under eighteen, any of those is refused under minor_safety with a reason that says what happened, never what anybody is. Without a stated age, two of them together are held for review on the dating template. The instant checks read these shapes in eight languages, a hit is a person's job immediately, and with effort: high the model reads the whole conversation for what the wording leaves to context.
Sexual content in pictures is judged by the model, which reads the image and any text written in it in the same call. The dating template holds sexual content for a look at 0.60 and refuses it at 0.85: higher than a forum's lines, because flirting is what the app is for. Repeat uploads of a picture already refused are recognised by their fingerprint, and if a file declares that it was generated by AI, that comes back as a fact for you to use as you see fit.
Start from the dating template, which lowers the lines for scams, spam, disguised text and approaches to minors, and raises them for sexual content. Then try any change as a second policy running in parallel with the one in force: both verdicts are computed on your own traffic, only one acts, and the dashboard shows where they disagree before you switch.
Send the conversation to /v1/conversation with an author on each message. ToxicFilter reads the same person across it for five families: affection, a move to another app, a story nobody can check, a request for money and an investment pitch. Affection and a move are how real couples talk and never make the shape, on their own or together (the dating template still holds a move to another app for a look, as it holds any invitation to leave the app); money or an investment from the same person is what makes the shape. Somebody asking "is this a scam?" is never read as running one.
It reports a phone number or an email as personal data and a pointer to a private chat app as spam. Under the dating template a number alone is held for review and a number with a chat app is refused. With redact, the answer carries the same text with the number masked, so the rest of the bio can be published.
It looks at what another participant does after somebody states an age under eighteen: asking for secrecy, moving to another app, asking for photos, offering gifts, isolating them. That is refused under the dating template. Without a stated age, two of those together are held for review, because the template lowers that line. A child stating their own age is never a finding.
Pictures are read by the model, which scores sexual content alongside the other categories and reads any text in the picture as well. The dating template holds sexual content for review at 0.60 and refuses it at 0.85, because flirting is the point of the app. A picture already refused on your app is recognised by its fingerprint without paying for a reading again.
Only when the file says so. ToxicFilter reads Content Credentials, the IPTC source type and the settings generators write into the file, and reports them as a fact, never as a reason to refuse. A screenshot or a re-save strips all of it, so no answer means the file does not say, not that the photo is real.
In the EU, yes: article 17 of the Digital Services Act requires a statement of reasons for every removal or restriction, whatever the size of the platform. ToxicFilter can write it with every block, in the user's language, and take their appeal.
2,000 credits a month on the free plan, no card. Enough to send a week of your own content and see what it says about it.