I’ve been a Pinterest user for a long time. I have boards going back years, spanning past interests (art deco weddings) and more recent ones (rubber duck-themed first birthday parties). When I log into the site, I get served up a slate of relevant recommendations—pins featuring colorful images of baby clothes alongside pins of hearty Instant Pot recipes. With each click, the recommendations get more specific. Click on one chicken soup recipe, and other varieties appear. Click on a pin of rubber duck cake pops, and duck cupcakes and a duck-shaped cheese plate quickly populate beneath the header “More like this.”
These are welcome, innocuous suggestions. And they keep me clicking.
But when a recent disinformation research project led me to a Pinterest board of anti-Islamic memes, one night of clicking through those pins—created by fake personas affiliated with the Internet Research Agency—turned my feed upside down. My babies-and-recipes experience morphed into a strange mish-mash of videos of Dinesh D’Souza, a controversial right-wing commentator, and Russian-language craft projects.
Recommendation engines are everywhere, and while my Pinterest feed’s transformation was rapid and pronounced, it is hardly an anomaly. BuzzFeed recently reported that Facebook Groups nudge people toward conspiratorial content, creating a built-in audience for spammers and propagandists. Follow one ISIS sympathizer on Twitter, and several others will appear under the “Who to follow” banner. And sociology professor Zeynep Tufekci dubbed YouTube “the Great Radicalizer” in a recent New York Times op-ed: “It seems as if you are never ‘hard core’ enough for YouTube’s recommendation algorithm,” she wrote. “It promotes, recommends and disseminates videos in a manner that appears to constantly up the stakes.”
Today, recommendation engines are perhaps the biggest threat to societal cohesion on the internet—and, as a result, one of the biggest threats to societal cohesion in the offline world, too. The recommendation engines we engage with are broken in ways that have grave consequences: amplified conspiracy theories, gamified news, nonsense infiltrating mainstream discourse, misinformed voters. Recommendation engines have become The Great Polarizer.
Ironically, the conversation about recommendation engines, and the curatorial power of social giants, is also highly polarized. A creator showed up at YouTube’s offices with a gun last week, outraged that the platform had demonetized and downranked some of the videos on her channel. This, she felt, was censorship. It isn’t, but the Twitter conversation around the shooting clearly illustrated the simmering tensions over how platforms navigate content : there are those who hold an absolutist view on free speech and believe any moderation is censorship, and there are those who believe that moderation is necessary to facilitate norms that respect the experience of the community.
As the consequences of curatorial decisions grow more dire, we need to ask: Can we make the internet’s recommendation engines more ethical? And if so, how?
Finding a solution begins with understanding how these systems work, since they are doing precisely what they’re designed to do. Recommendation engines generally function in two ways. The first is a content-based system. The engine asks, is this content similar to other content that this user has previously liked? If you binge-watched two seasons of, say, Law and Order, Netflix’s reco engine will probably decide that you’ll like the other seventeen, and that procedural crime dramas in general are a good fit. The second kind of filtering is what’s called a collaborative filtering system. That engine asks, what can I determine about this user, and what do similar people like? These systems can be effective even before you’ve given the engine any feedback through your actions. If you sign up for Twitter and your phone indicates you’re in Chicago, the initial “Who To Follow” suggestions will feature popular Chicago sports team and other accounts that people in your geographical area like. Recommender systems learn; as you reinforce by clicking and liking, they will serve you things based on your clicks, likes, and searches—and those of people similar to their ever-more-sophisticated profile of you. This is why my foray onto an anti-Islamic Pinterest board created by Russian trolls led to weeks of being served far-right videos and Russian-language craft pins; it was content that had been enjoyed by others who had spent time with those pins.
Now imagine that a user is interested in content more extreme than Law and Order and Chicago sports. What then? The Pinterest algorithms don’t register a difference between suggesting duckie balloons and serving up extremist propaganda; the Twitter system doesn’t recognize that it’s encouraging people to follow additional extremist accounts, and Facebook’s Groups engine doesn’t understand why directing conspiracy theorists to new conspiracy communities is possibly a bad idea. The systems don’t actually understand the content, they just return what they predict will keep us clicking. That’s because their primary function is to help achieve one or two specific key performance indicators (KPIs) chosen by the company. We manage what we can measure. It’s much easier to measure time on site or monthly average user stats than to quantify the outcomes of serving users conspiratorial or fraudulent content. And when this complexity is combined with the overhead of managing outraged people who feel that moderating content violates free speech, it’s easy to see why the companies default to the hands-off approach.
But it isn’t actually hands-off—there is no First Amendment right to amplification—and the algorithm is already deciding what you see. Content-based recommendation systems and collaborative filtering are never neutral; they are always ranking one video, pin, or group against another when they’re deciding what to show you. They’re opinionated and influential, though not in the simplistic or partisan way that some critics contend. And as extreme, polarizing, and sensational content continues to rise to the top, it’s increasingly obvious that curatorial algorithms need to be tempered with additional oversight, and reweighted to consider what they’re serving up.
Some of this work is already underway. Project Redirect, an effort by Google Jigsaw, redirects certain types of users who are searching YouTube for terrorist videos—people who appear to be motivated by more than mere curiosity. Rather than offer up more violent content, the approach of that recommendation system is to do the opposite—it points users to content intended to de-radicalize them. This project has been underway around violent extremism for a few years, which means that YouTube has been aware of the conceptual problem, and the amount of power their recommender systems wield, for some time now. It makes their decision to address the problem in other areas by redirecting users to Wikipedia for fact-checking even more baffling.
Guillaume Chaslot, a former YouTube recommendation engine architect and now independent researcher, has written extensively about the problem of YouTube serving up conspiratorial and radicalizing content—fiction outperforming reality, as he put it in The Guardian. “People have been talking about these problems for years,” he said. “The surveys, Wikipedia, and additional raters are just going to make certain problems less visible. But it won’t impact the main problem—that YouTube’s algorithm is pushing users in a direction they might not want.” Giving people more control over what their algorithmic feed serves up is one potential solution. Twitter, for example, created a filter that enables users to avoid content from low-quality accounts. Not everyone uses it, but the option exists.
In the past, companies have spontaneously cracked down on content related to suicide, pro-anorexia, payday lending, and bitcoin scams. Sensitive topics are often dealt with via ad-hoc moderation decisions in response to a public outcry. Simple keyword bans are often overbroad, and lack the nuance to understand if an account, Group, or Pin is discussing a volatile topic, or promoting it. Reactive moderation often leads to outcries about censorship.
Platforms need to transparently, thoughtfully, and deliberately take ownership of this issue. Perhaps that involves creating a visible list of “Do Not Amplify” topics in line with the platform’s values. Perhaps it’s a more nuanced approach: inclusion in recommendation systems is based on a quality indicator derived from a combination of signals about the content, the way it’s disseminated (are bots involved?), and the authenticity of the channel, group, or voice behind it. Platforms can decide to allow Pizzagate content to exist on their site while simultaneously deciding not to algorithmically amplify or proactively proffer it to users.
Ultimately, we’re talking about choice architecture, a term for the way that information or products are presented to people in a manner that takes into account individual or societal welfare while preserving consumer choice. The presentation of choices has an impact on what people choose, and social networks' recommender systems are a key component of that presentation; they are already curating the set of options. This is the idea behind the “nudge”—do you put the apples or the potato chips front and center on the school lunch line?
The need to rethink the ethics of recommendation engines is only growing more urgent as curatorial systems and AI crop up in increasingly more sensitive places: local and national governments are using similar algorithms to determine who makes bail, who receives subsidies, and which neighborhoods need policing. As algorithms amass more power and responsibility in our everyday lives, we need to create the frameworks to rigorously hold them accountable—that means prioritizing ethics over profit.