How We Tell the Difference Between Flirting and Harassment in Dating Apps
How do you detect sexual harassment in an environment that is supposed to be sexually charged?
With more and more of us finding love online these days, it is becoming more and more important for dating apps to answer to this question. If an app becomes known as an unsafe place then revenue falls — fast.
In this post, we’ll illustrate how we solved this problem for one of the largest dating apps in the world.
Community Context
“I love those curves.”
Said within an art gallery forum? Most likely not sexual.
Said within a dating app chat? Most likely sexual.
Said on a gaming forum? Hmmmm. Maybe sexual?
First, we matched our definition of sexual to the company’s definition (we do this with all our customers for every type of behavior they want to detect and respond to — calibrating is important). Some dating apps allow more forward behaviors, while some discourage it. So, how sexual is defined in the app influences all that follows.
In this case, our customer’s definition of sexual wouldn’t have labeled this message as so.
Conversation Context
“I love those curves.”
The only way to tell whether this is a flirtatious response to a picture shared between two willing adults, or a part of something more sinister, is to look at the whole conversation, from start to present time.
Let’s take a look:
Hey, I found your profile and wanted to say hi.
Hi
Looks like you like to play poker. I do too.
Nice. Yeah, I like poker.
Would you be interested in playing strip poker on video with me?? :-)”
Maybe. I’d like to get to know you better before tho.
…
Hey, thought you might like this pic of me. I’ve been working on my abs.
You look hot.
Wanna send me some sexy photos of you?
Sure.
I love those curves.
Thanks. You’re sexy.
…
Hey, you awake? I’m thinking of you.
I’d like to kiss you.
Why no response? I thought we were on to something?
Hey
Remember this photo? You look really sexy here. I’d like to kiss you.
No response?
Hey, are you ignoring me?
Does your employer know you have these sexy pictures? I’m pretty sure they’d be shocked…
Hey. I was gone for a few days. What are you talking about? This is getting weird.
…
Hey, I edited your photo a bit … I’m pretty good with photoshop. Like??
What? What is this?
It’s just how I think about us. I’d like to kiss you.
It is easy for a person to see this exchange is going south — fast. But, you can’t hire thousands of moderators to sift through every exchange in the hopes of catching bad behavior.
And keyword-based moderation systems? They’d fail miserably here because no individual word is flaggable. Remember, this is a dating app where saying things like “sexy” and “strip” and “kiss” are appropriate and typically welcomed!
So, our second step was to roll out our Sexual Harassment Aspect Model. All of our Aspect Models (we have them for Hate Speech, Radicalization, Sexual Grooming — basically, all the complex behaviors that are impossible to catch with keywords) use cutting edge natural language processing capabilities to examine a conversation in real-time and warn when it is turning sour — without causing delays felt on the user side.
Simply put, Aspect Models are able to sense problems that other traditional (e.g. keyword-based) systems cannot because they take a holistic look at an exchange. Why does it matter that we can sense these problems and even sense them early on? Easy: it means:
- Sexual harassment can be detected before significant problems develop
- Sexual grooming can be caught before a child is coerced into sharing sexual materials
- Radicalization can be found before someone is brainwashed into hate
Even More Context
Given the description above, you may think Aspect Models deliver the One-Two Punch on sexual harassment.
While they are extraordinarily powerful, it is imperative that the signals we deliver to our customers are absolutely trustworthy. Only then can our customers build automated workflows that manage incidents appropriately and deliver only the most sensitive cases to people (i.e. allowing our customers to safeguard and scale their moderation teams).
So, we layered on:
- a user reputation system where scores are constantly updated to reflect the latest behavior app-wide,
- and, meta data analysis because timestamps can reveal a lot! So can app-wide actions taken by the people involved.
“It’s a Learning Computer”
We never miss a chance to land a Terminator 2 reference and, in this case, the reference is perfect.
Finally, we designed a retraining process that constantly ingests data from the app (again, while maintaining privacy). So, we’re completely abreast of changes to the community dynamic and see changes to slang a mile away.
Putting it All Together
So, for our dating app customer we gave them the power to detect sexual harassment by
- Aligning our system’s definitions to theirs
- Rolling out our Sexual Harassment Aspect Model to evaluate conversations
- Adding user rep scores and meta data to the mix
- Standing up a retraining process that evolves with the community
This whole package was deployed over one month and results were immediate. If this sounds like something your dating app needs, let’s talk.
In Closing, A Note on Privacy
Whenever we share what we do for dating apps with our friends and family their immediate reaction is, “Wait, you’re looking at my conversations on dating apps?!”
Nope. No one is looking at your conversations.
Our technology eliminates the need for moderators to read conversations except in the most sensitive of situations. So, the moderators aren’t peeping.
Further, our technology doesn’t know the personal details of conversations because all personally identifying data is removed before being analyzed.
