Breeze Looks to Fix its Discriminatory AI
Dutch dating app Breeze harnesses AI to suggest singles to one another. However, the platform realised that its algorithm was putting people with a darker skin colour at a disadvantage. They’ve now taken the first step to putting it right.
Thomas Crul, a matchmaking researcher and data analyst at Breeze, explained the series of events in a recent LinkedIn post.
“At Breeze…I discovered a possible problem within our matchmaking system: in short, we suspect that the learning algorithm reinforces human prejudices, putting users of non-Dutch backgrounds at a disadvantage”, he shared.
“Because of the seriousness and sensitivity of this possible problem”, the platform took this issue to the Netherlands Institute for Human Rights. Breeze wanted to hear the organisation’s perspective on how it should respond.
The situation is complicated as “implementing levelling measures” to reduce discriminatory outcomes could be seen as a “preferential policy” that is not allowed, Crul highlighted.
The Netherlands Institute for Human Rights ruled on the situation, saying:
“Measures that Breeze must take to prevent indirect discrimination are not regarded by the Board as a form of preferential policy. This would not give users a privileged position, but would only compensate them for a threat of subordination. After all, users with a dark skin color or non-Dutch origin would then not be introduced relatively more often, but “only” as often as the other users.”
Now that this first step has been completed, the next question is how can the algorithm be fixed? Both the Institute and Crul highlight that remediating any discrimination would require knowing personal details about the user’s background, opening up a GDPR can of worms.
The situation is no doubt complicated and full of regulatory questions. While we wait to see how Breeze continues to address this issue, it is clear that they have done the right thing by flagging up its own discriminatory algorithms and beginning the journey to more equitable outcomes.
Thomas Crul shares that “if you as a reader have expertise and / or experience on these topics and would like to think along with us about a solution, please contact me!”. You can find his LinkedIn profile here.
