Paul, who has 16 million mostly teen subscribers to his YouTube channel, removed the video from YouTube 24 hours later amid a furious backlash. It was still long enough for the footage to receive 6m views and a spot on YouTube’s coveted list of trending videos.
The answer was a slew of videos of men mocking distraught teenage fans of Logan Paul, followed by CCTV footage of children stealing things and, a few clicks later, a video of children having their teeth pulled out with bizarre, homemade contraptions.
I had cleared my history, deleted my cookies, and opened a private browser to be sure YouTube was not personalising recommendations. This was the algorithm taking me on a journey of its own volition, and it culminated with a video of two boys, aged about five or six, punching and kicking one another.
“I’m going to post it on YouTube,” said a teenage girl, who sounded like she might be an older sibling. “Turn around an punch the heck out of that little boy.” They scuffled for several minutes until one had knocked the other’s tooth out.
There are 1.5 billion YouTube users in the world, which is more than the number of households that own televisions. What they watch is shaped by this algorithm, which skims and ranks billions of videos to identify 20 “up next” clips that are both relevant to a previous video and most likely, statistically speaking, to keep a person hooked on their screen.
Company insiders tell me the algorithm is the single most important engine of YouTube’s growth. In one of the few public explanations of how the formula works – an academic paper that sketches the algorithm’s deep neural networks, crunching a vast pool of data about videos and the people who watch them – YouTube engineers describe it as one of the “largest scale and most sophisticated industrial recommendation systems in existence”.
The algorithm does not appear to be optimising for what is truthful, or balanced, or healthy for democracy — Guillaume Chaslot, an ex-Google engineer
Lately, it has also become one of the most controversial. The algorithm has been found to be promoting conspiracy theories about the Las Vegas mass shooting and incentivising, through recommendations, a thriving subculture that targets children with disturbing content such as cartoons in which the British children’s character Peppa Pig eats her father or drinks bleach.
Lewd and violent videos have been algorithmically served up to toddlers watching YouTube Kids, a dedicated app for children. One YouTube creator who was banned from making advertising revenues from his strange videos – which featured his children receiving flu shots, removing earwax, and crying over dead pets – told a reporter he had only been responding to the demands of Google’s algorithm. “That’s what got us out there and popular,” he said. “We learned to fuel it and do whatever it took to please the algorithm.”
Google has responded to these controversies in a process akin to Whack-A-Mole: expanding the army of human moderators, removing offensive YouTube videos identified by journalists and de-monetising the channels that create them. But none of those moves has diminished a growing concern that something has gone profoundly awry with the artificial intelligence powering YouTube.
Yet one stone has so far been largely unturned. Much has been written about Facebook and Twitter’s impact on politics, but in recent months academics have speculated that YouTube’s algorithms may have been instrumental in fuelling disinformation during the 2016 presidential election. “YouTube is the most overlooked story of 2016,” Zeynep Tufekci, a widely respected sociologist and technology critic, tweeted back in October. “Its search and recommender algorithms are misinformation engines.”
If YouTube’s recommendation algorithm really has evolved to promote more disturbing content, how did that happen? And what is it doing to our politics?
‘Like reality, but distorted’
Those are not easy questions to answer. Like all big tech companies, YouTube does not allow us to see the algorithms that shape our lives. They are secret formulas, proprietary software, and only select engineers are entrusted to work on the algorithm. Guillaume Chaslot, a 36-year-old French computer programmer with a PhD in artificial intelligence, was one of those engineers.
During the three years he worked at Google, he was placed for several months with a team of YouTube engineers working on the recommendation system. The experience led him to conclude that the priorities YouTube gives its algorithms are dangerously skewed.
“YouTube is something that looks like reality, but it is distorted to make you spend more time online,” he tells me when we meet in Berkeley, California. “The recommendation algorithm is not optimising for what is truthful, or balanced, or healthy for democracy.”
Chaslot explains that the algorithm never stays the same. It is constantly changing the weight it gives to different signals: the viewing patterns of a user, for example, or the length of time a video is watched before someone clicks away.
The engineers he worked with were responsible for continuously experimenting with new formulas that would increase advertising revenues by extending the amount of time people watched videos. “Watch time was the priority,” he recalls. “Everything else was considered a distraction.”
Chaslot was fired by Google in 2013, ostensibly over performance issues. He insists he was let go after agitating for change within the company, using his personal time to team up with like-minded engineers to propose changes that could diversify the content people see.
He was especially worried about the distortions that might result from a simplistic focus on showing people videos they found irresistible, creating filter bubbles, for example, that only show people content that reinforces their existing view of the world. Chaslot said none of his proposed fixes were taken up by his managers. “There are many ways YouTube can change its algorithms to suppress fake news and improve the quality and diversity of videos people see,” he says. “I tried to change YouTube from the inside but it didn’t work.”
YouTube told me that its recommendation system had evolved since Chaslot worked at the company and now “goes beyond optimising for watchtime”. The company said that in 2016 it started taking into account user “satisfaction”, by using surveys, for example, or looking at how many “likes” a video received, to “ensure people were satisfied with what they were viewing”. YouTube added that additional changes had been implemented in 2017 to improve the news content surfaced in searches and recommendations and discourage the promotion of videos containing “inflammatory religious or supremacist” content.
It did not say why Google, which acquired YouTube in 2006, waited over a decade to make those changes. Chaslot believes such changes are mostly cosmetic, and have failed to fundamentally alter some disturbing biases that have evolved in the algorithm. In the summer of 2016, he built a computer program to investigate.
The software Chaslot wrote was designed to provide the world’s first window into YouTube’s opaque recommendation engine. The program simulates the behaviour of a user who starts on one video and then follows the chain of recommended videos – much as I did after watching the Logan Paul video – tracking data along the way.
It finds videos through a word search, selecting a “seed” video to begin with, and recording several layers of videos that YouTube recommends in the “up next” column. It does so with no viewing history, ensuring the videos being detected are YouTube’s generic recommendations, rather than videos personalised to a user. And it repeats the process thousands of times, accumulating layers of data about YouTube recommendations to build up a picture of the algorithm’s preferences.
Over the last 18 months, Chaslot has used the program to explore bias in YouTube content promoted during the French, British and German elections, global warming and mass shootings, and published his findings on his website, Algotransparency.com. Each study finds something different, but the research suggests YouTube systematically amplifies videos that are divisive, sensational and conspiratorial.
When his program found a seed video by searching the query “who is Michelle Obama?” and then followed the chain of “up next” suggestions, for example, most of the recommended videos said she “is a man”. More than 80% of the YouTube-recommended videos about the pope detected by his program described the Catholic leader as “evil”, “satanic”, or “the anti-Christ”. There were literally millions of videos uploaded to YouTube to satiate the algorithm’s appetite for content claiming the earth is flat. “On YouTube, fiction is outperforming reality,” Chaslot says.
He believes one of the most shocking examples was detected by his program in the run-up to the 2016 presidential election. As he observed in a short, largely unnoticed blogpost published after Donald Trump was elected, the impact of YouTube’s recommendation algorithm was not neutral during the presidential race: it was pushing videos that were, in the main, helpful to Trump and damaging to Hillary Clinton. “It was strange,” he explains to me. “Wherever you started, whether it was from a Trump search or a Clinton search, the recommendation algorithm was much more likely to push you in a pro-Trump direction.”
Trump won the electoral college as a result of 80,000 votes spread across three swing states. There were more than 150 million YouTube users in the US. The videos contained in Chaslot’s database of YouTube-recommended election videos were watched, in total, more than 3bn times before the vote in November 2016.
Even a small bias in the videos would have been significant. “Algorithms that shape the content we see can have a lot of impact, particularly on people who have not made up their mind,” says Luciano Floridi, a professor at the University of Oxford’s Digital Ethics Lab, who studies the ethics of artificial intelligence. “Gentle, implicit, quiet nudging can over time edge us toward choices we might not have otherwise made.”
Promoting conspiracy theories
Chaslot sent me a database of more YouTube-recommended videos his program identified in the three months leading up to the presidential election. It contained more than 8,000 videos – all of them detected by his program appearing “up next” on 12 dates between August and November 2016, after equal numbers of searches for “Trump” and “Clinton”.
It was not a comprehensive set of videos and it may not have been a perfectly representative sample. But it was, Chaslot said, a previously unseen dataset of what YouTube was recommending to people interested in content about the candidates – one snapshot, in other words, of the algorithm’s preferences.
Jonathan Albright, research director at the Tow Center for Digital Journalism, who reviewed the code used by Chaslot, says it is a relatively straightforward piece of software and a reputable methodology. “This research captured the apparent direction of YouTube’s political ecosystem,” he says. “That has not been done before.”
I spent weeks watching, sorting and categorising the trove of videos with Erin McCormick, an investigative reporter and expert in database analysis. From the start, we were stunned by how many extreme and conspiratorial videos had been recommended, and the fact that almost all of them appeared to be directed against Clinton.
Some of the videos YouTube was recommending were the sort we had expected to see: broadcasts of presidential debates, TV news clips, Saturday Night Live sketches. There were also videos of speeches by the two candidates – although, we found, the database contained far more YouTube-recommended speeches by Trump than Clinton.
But what was most compelling was how often Chaslot’s software detected anti-Clinton conspiracy videos appearing “up next” beside other videos.
There were dozens of clips stating Clinton had had a mental breakdown, reporting she had syphilis or Parkinson’s disease, accusing her of having secret sexual relationships, including with Yoko Ono. Many were even darker, fabricating the contents of WikiLeaks disclosures to make unfounded claims, accusing Clinton of involvement in murders or connecting her to satanic and paedophilic cults.
One video that Chaslot’s data indicated was pushed particularly hard by YouTube’s algorithm was a bizarre, one-hour film claiming Trump’s rise was predicted in Isaiah 45. Another was entitled: “BREAKING: VIDEO SHOWING BILL CLINTON RAPING 13 YR-OLD WILL PLUNGE RACE INTO CHAOS ANONYMOUS CLAIMS”. The recommendation engine appeared to have been particularly helpful to the Alex Jones Channel, which broadcasts far-right conspiracy theories under the Infowars brand.
There were too many videos in the database for us to watch them all, so we focused on 1,000 of the top-recommended videos. We sifted through them one by one to determine whether the content was likely to have benefited Trump or Clinton. Just over a third of the videos were either unrelated to the election or contained content that was broadly neutral or even-handed. Of the remaining 643 videos, 551 were videos favouring Trump, while only only 92 favoured the Clinton campaign.
The sample we had looked at suggested Chaslot’s conclusion was correct: YouTube was six times more likely to recommend videos that aided Trump than his adversary. YouTube presumably never programmed its algorithm to benefit one candidate over another. But based on this evidence, at least, that is exactly what happened.
‘Leading people down hateful rabbit holes’
“We have a great deal of respect for the Guardian as a news outlet and institution,” a YouTube spokesperson emailed me after I forwarded them our findings. “We strongly disagree, however, with the methodology, data and, most importantly, the conclusions made in their research.”
The spokesperson added: “Our search and recommendation systems reflect what people search for, the number of videos available, and the videos people choose to watch on YouTube. That’s not a bias towards any particular candidate; that is a reflection of viewer interest.”
It was a curious response. YouTube seemed to be saying that its algorithm was a neutral mirror of the desires of the people who use it – if we don’t like what it does, we have ourselves to blame. How does YouTube interpret “viewer interest” – and aren’t “the videos people choose to watch” influenced by what the company shows them?
Offered the choice, we may instinctively click on a video of a dead man in a Japanese forest, or a fake news clip claiming Bill Clinton raped a 13-year-old. But are those in-the-moment impulses really a reflect of the content we want to be fed?
Tufekci, the sociologist who several months ago warned about the impact YouTube may have had on the election, tells me YouTube’s recommendation system has probably figured out that edgy and hateful content is engaging. “This is a bit like an autopilot cafeteria in a school that has figured out children have sweet teeth, and also like fatty and salty foods,” she says. “So you make a line offering such food, automatically loading the next plate as soon as the bag of chips or candy in front of the young person has been consumed.”
Once that gets normalised, however, what is fractionally more edgy or bizarre becomes, Tufekci says, novel and interesting. “So the food gets higher and higher in sugar, fat and salt – natural human cravings – while the videos recommended and auto-played by YouTube get more and more bizarre or hateful.”
But why would a bias toward ever more weird or divisive videos benefit one candidate over another? That depends on the candidates. Trump’s campaign was nothing if not weird and divisive. Tufekci points to studies showing that “field of misinformation” largely tilted anti-Clinton before the election. “Fake news providers,” she says, “found that fake anti-Clinton material played much better with the pro-Trump base than did fake anti-Trump material with the pro-Clinton base.”
She adds: “The question before us is the ethics of leading people down hateful rabbit holes full of misinformation and lies at scale just because it works to increase the time people spend on the site – and it does work.”
Tufekci was one of several academics I shared our research with. Philip Howard, a professor at the Oxford Internet Institute, who has studied how disinformation spread during the election, was another. He questions whether a further factor might have been at play. “This is important research because it seems to be the first systematic look into how YouTube may have been manipulated,” he says, raising the possibility that the algorithm was gamed as part of the same propaganda campaigns that flourished on Twitter and Facebook.
In testimony to the House intelligence committee, investigating Russian interference in the election, Google’s general counsel, Kent Walker, played down the degree to which Moscow’s propaganda efforts infiltrated YouTube. The company’s internal investigation had only identified 18 YouTube channels and 1,100 videos suspected of being linked to Russia’s disinformation campaign, he told the committee in December – and generally the videos had relatively small numbers of views. He added: “We believe that the activity we found was limited because of various safeguards that we had in place in advance of the 2016 election, and the fact that Google’s products didn’t lend themselves to the kind of micro-targeting or viral dissemination that these actors seemed to prefer.”
Walker made no mention of YouTube recommendations. Correspondence made public just last week, however, reveals that Senator Mark Warner, the ranking Democrat on the intelligence committee, later wrote to the company about the algorithm, which he said seemed “particularly susceptible to foreign influence”. The senator demanded to know what the company was specifically doing to prevent a “malign incursion” of YouTube’s recommendation system. Walker, in his written reply, offered few specifics, but said YouTube had “a sophisticated spam and security breach detection system to identify anomalous behavior and malignant incursions”.
Tristan Harris, a former Google insider turned tech whistleblower, likes to describe Facebook as a “living, breathing crime scene for what happened in the 2016 election” that federal investigators have no access to. The same might be said of YouTube. About half the videos Chaslot’s program detected being recommended during the election have now vanished from YouTube – many of them taken down by their creators. Chaslot has always thought this suspicious. These were videos with titles such as “Must Watch!! Hillary Clinton tried to ban this video”, watched millions of times before they disappeared. “Why would someone take down a video that has been viewed millions of times?” he asks.
I located a copy of “This Video Will Get Donald Trump Elected”, a viral sensation that was watched more than 10m times before it vanished from YouTube. It was a benign-seeming montage of historical footage of Trump, accompanied by soft piano music. But when I played the video in slow motion, I saw that it contained weird flashes of Miley Cyrus licking a mirror. It seemed an amateurish and bizarre attempt at inserting subliminal, sexualised imagery. But it underscored how little oversight we have over anyone who might want to use YouTube to influence public opinion on a vast scale.
I shared the entire database of 8,000 YouTube-recommended videos with John Kelly, the chief executive of the commercial analytics firm Graphika, which has been tracking political disinformation campaigns. He ran the list against his own database of Twitter accounts active during the election, and concluded many of the videos appeared to have been pushed by networks of Twitter sock puppets and bots controlled by pro-Trump digital consultants with “a presumably unsolicited assist” from Russia.
“I don’t have smoking-gun proof of who logged in to control those accounts,” he says. “But judging from the history of what we’ve seen those accounts doing before, and the characteristics of how they tweet and interconnect, they are assembled and controlled by someone – someone whose job was to elect Trump.”
Chaslot and some of the academics I spoke to felt this social media activity was significant. YouTube’s algorithm may have developed its biases organically, but could it also have been nudged into spreading those videos even further? “If a video starts skyrocketing, there’s no question YouTube’s algorithm is going to start pushing it,” Albright says.
YouTube did not deny that social media propaganda might have influenced its recommendations, but played down the likelihood, stressing its system “does not optimise” for traffic from Twitter or Facebook. “It appears as if the Guardian is attempting to shoehorn research, data and their conclusions into a common narrative about the role of technology in last year’s election,” the spokesperson added. “The reality of how our systems work, however, simply don’t support this premise.”
After the Senate’s correspondence with Google over possible Russian interference with YouTube’s recommendation algorithm was made public last week, YouTube sent me a new statement. It emphasised changes it made in 2017 to discourage the recommendation system from promoting some types of problematic content. “We appreciate the Guardian’s work to shine a spotlight on this challenging issue,” it added. “We know there is more to do here and we’re looking forward to making more announcements in the months ahead.”
With its flashy graphics and slick-haired anchor, the Next News Network has the appearances of a credible news channel. But behind the facade is a dubious operation that recycles stories harvested from far-right publications, fake news sites and Russian media outlets.
The channel is run by anchor Gary Franchi, once a leading proponent of a conspiracy that claimed the US government was creating concentration camps for its citizens. It was the Next News Network that broadcast the fabricated claims about Bill Clinton raping a teenager, although Franchi insists he is not a fake news producer. (He tells me he prefers to see his channel as “commentating on conservative news and opinion”.)
In the months leading up to the election, the Next News Network turned into a factory of anti-Clinton news and opinion, producing dozens of videos a day and reaching an audience comparable to that of MSNBC’s YouTube channel.
Chaslot’s research indicated Franchi’s success could largely be credited to YouTube’s algorithms, which consistently amplified his videos to be played “up next”. YouTube had sharply dismissed Chaslot’s research.
I contacted Franchi to see who was right. He sent me screen grabs of the private data given to people who upload YouTube videos, including a breakdown of how their audiences found their clips. The largest source of traffic to the Bill Clinton rape video, which was viewed 2.4m times in the month leading up to the election, was YouTube recommendations.
The same was true of all but one of the videos Franchi sent me data for. A typical example was a Next News Network video entitled “WHOA! HILLARY THINKS CAMERA’S OFF… SENDS SHOCK MESSAGE TO TRUMP” in which Franchi, pointing to a tiny movement of Clinton’s lips during the a TV debate, claims she says “fuck you” to her presidential rival. The data Franchi shared revealed in the month leading up to the election, 73% of the traffic to the video – amounting to 1.2m of its views – was due to YouTube recommendations. External traffic accounted for only 3% of the views.
Franchi is a professional who makes a living from his channel, but many of the other creators of anti-Clinton videos I spoke to were amateur sleuths or part-time conspiracy theorists. Typically, they might receive a few hundred views on their videos, so they were shocked when their anti-Clinton videos started to receive millions of views, as if they were being pushed by an invisible force.
In every case, the largest source of traffic – the invisible force – came from the clips appearing in the “up next” column. William Ramsey, an occult investigator from southern California who made “Irrefutable Proof: Hillary Clinton Has a Seizure Disorder!”, shared screen grabs that showed the recommendation algorithm pushed his video even after YouTube had emailed him to say it violated its guidelines. Ramsey’s data showed the video was watched 2.4m times by US-based users before election day. “For a nobody like me, that’s a lot,” he says. “Enough to sway the election, right?”
Daniel Alexander Cannon, a conspiracy theorist from South Carolina, tells me: “Every video I put out about the Clintons, YouTube would push it through the roof.” His best-performing clip was a video titled “Hillary and Bill Clinton ‘The 10 Photos You Must See’”, essentially a slideshow of appalling (and seemingly doctored) images of the Clintons with voiceover in which Cannon speculates on their health. It has been seen 3.7m times on YouTube, and 2.9m of those views, Cannon said, came from “up next” recommendations.
Chaslot has put a spotlight on a trove of anti-Clinton conspiracy videos that had been hidden in the shadows – unless, that is, you were one of the the millions YouTube served them to. But his research also does something more important: revealing how thoroughly our lives are now mediated by artificial intelligence.
Less than a generation ago, the way voters viewed their politicians was largely shaped by tens of thousands of newspaper editors, journalists and TV executives. Today, the invisible codes behind the big technology platforms have become the new kingmakers.
They pluck from obscurity people like Dave Todeschini, a retired IBM engineer who, “let off steam” during the election by recording himself opining on Clinton’s supposed involvement in paedophilia, child sacrifice and cannibalism. “It was crazy, it was nuts,” he said of the avalanche of traffic to his YouTube channel, which by election day had more than 2m views.
“Breaking news,” he announced in one of his last dispatches before the vote: the FBI, he said, had just found graphic images of Clinton and her aide in “sexually compromising positions” with a teenager. “It seems to me, with Bill Clinton’s trips to paedophile island a number of times, that what we have here is nothing short of the Clinton paedophile ring,” he declared.
Todeschini sits in his darkened living room in New Jersey, staring into his smartphone. “I’ll tell you what: the rabbit hole just got a couple of yards deeper.”
Contact the author: firstname.lastname@example.org.
A full description of the methodology Chaslot used to detect YouTube recommendations (and an explanation of how the Guardian analysed them) is available here.
‘Our minds can be hijacked’: the tech insiders who fear a smartphone dystopia
Google, Twitter and Facebook workers who helped make technology so addictive are disconnecting themselves from the internet. Paul Lewis reports on the Silicon Valley refuseniks alarmed by a race for human attention
By Paul Lewis in San Francisco,
Justin Rosenstein had tweaked his laptop’s operating system to block Reddit, banned himself from Snapchat, which he compares to heroin, and imposed limits on his use of Facebook. But even that wasn’t enough. In August, the 34-year-old tech executive took a more radical step to restrict his use of social media and other addictive technologies.
Rosenstein purchased a new iPhone and instructed his assistant to set up a parental-control feature to prevent him from downloading any apps.
He was particularly aware of the allure of Facebook “likes”, which he describes as “bright dings of pseudo-pleasure” that can be as hollow as they are seductive. And Rosenstein should know: he was the Facebook engineer who created the “like” button in the first place.
A decade after he stayed up all night coding a prototype of what was then called an “awesome” button, Rosenstein belongs to a small but growing band of Silicon Valley heretics who complain about the rise of the so-called “attention economy”: an internet shaped around the demands of an advertising economy.
These refuseniks are rarely founders or chief executives, who have little incentive to deviate from the mantra that their companies are making the world a better place. Instead, they tend to have worked a rung or two down the corporate ladder: designers, engineers and product managers who, like Rosenstein, several years ago put in place the building blocks of a digital world from which they are now trying to disentangle themselves. “It is very common,” Rosenstein says, “for humans to develop things with the best of intentions and for them to have unintended, negative consequences.”
Rosenstein, who also helped create Gchat during a stint at Google, and now leads a San Francisco-based company that improves office productivity, appears most concerned about the psychological effects on people who, research shows, touch, swipe or tap their phone 2,617 times a day.
There is growing concern that as well as addicting users, technology is contributing toward so-called “continuous partial attention”, severely limiting people’s ability to focus, and possibly lowering IQ. One recent study showed that the mere presence of smartphones damages cognitive capacity – even when the device is turned off. “Everyone is distracted,” Rosenstein says. “All of the time.”
Drawing a straight line between addiction to social media and political earthquakes like Brexit and the rise of Donald Trump, they contend that digital forces have completely upended the political system and, left unchecked, could even render democracy as we know it obsolete.
In 2007, Rosenstein was one of a small group of Facebook employees who decided to create a path of least resistance – a single click – to “send little bits of positivity” across the platform. Facebook’s “like” feature was, Rosenstein says, “wildly” successful: engagement soared as people enjoyed the short-term boost they got from giving or receiving social affirmation, while Facebook harvested valuable data about the preferences of users that could be sold to advertisers. The idea was soon copied by Twitter, with its heart-shaped “likes” (previously star-shaped “favourites”), Instagram, and countless other apps and websites.
It was Rosenstein’s colleague, Leah Pearlman, then a product manager at Facebook and on the team that created the Facebook “like”, who announced the feature in a 2009 blogpost. Now 35 and an illustrator, Pearlman confirmed via email that she, too, has grown disaffected with Facebook “likes” and other addictive feedback loops. She has installed a web browser plug-in to eradicate her Facebook news feed, and hired a social media manager to monitor her Facebook page so that she doesn’t have to.
“One reason I think it is particularly important for us to talk about this now is that we may be the last generation that can remember life before,” Rosenstein says. It may or may not be relevant that Rosenstein, Pearlman and most of the tech insiders questioning today’s attention economy are in their 30s, members of the last generation that can remember a world in which telephones were plugged into walls.
It is revealing that many of these younger technologists are weaning themselves off their own products, sending their children to elite Silicon Valley schools where iPhones, iPads and even laptops are banned. They appear to be abiding by a Biggie Smalls lyric from their own youth about the perils of dealing crack cocaine: never get high on your own supply.
One morning in April this year, designers, programmers and tech entrepreneurs from across the world gathered at a conference centre on the shore of the San Francisco Bay. They had each paid up to $1,700 to learn how to manipulate people into habitual use of their products, on a course curated by conference organiser Nir Eyal.
Eyal, 39, the author of Hooked: How to Build Habit-Forming Products, has spent several years consulting for the tech industry, teaching techniques he developed by closely studying how the Silicon Valley giants operate.
“The technologies we use have turned into compulsions, if not full-fledged addictions,” Eyal writes. “It’s the impulse to check a message notification. It’s the pull to visit YouTube, Facebook, or Twitter for just a few minutes, only to find yourself still tapping and scrolling an hour later.” None of this is an accident, he writes. It is all “just as their designers intended”.
He explains the subtle psychological tricks that can be used to make people develop habits, such as varying the rewards people receive to create “a craving”, or exploiting negative emotions that can act as “triggers”. “Feelings of boredom, loneliness, frustration, confusion and indecisiveness often instigate a slight pain or irritation and prompt an almost instantaneous and often mindless action to quell the negative sensation,” Eyal writes.
Attendees of the 2017 Habit Summit might have been surprised when Eyal walked on stage to announce that this year’s keynote speech was about “something a little different”. He wanted to address the growing concern that technological manipulation was somehow harmful or immoral. He told his audience that they should be careful not to abuse persuasive design, and wary of crossing a line into coercion.
But he was defensive of the techniques he teaches, and dismissive of those who compare tech addiction to drugs. “We’re not freebasing Facebook and injecting Instagram here,” he said. He flashed up a slide of a shelf filled with sugary baked goods. “Just as we shouldn’t blame the baker for making such delicious treats, we can’t blame tech makers for making their products so good we want to use them,” he said. “Of course that’s what tech companies will do. And frankly: do we want it any other way?”
Finally, Eyal confided the lengths he goes to protect his own family. He has installed in his house an outlet timer connected to a router that cuts off access to the internet at a set time every day. “The idea is to remember that we are not powerless,” he said. “We are in control.”
But are we? If the people who built these technologies are taking such radical steps to wean themselves free, can the rest of us reasonably be expected to exercise our free will?
Not according to Tristan Harris, a 33-year-old former Google employee turned vocal critic of the tech industry. “All of us are jacked into this system,” he says. “All of our minds can be hijacked. Our choices are not as free as we think they are.”
Harris, who has been branded “the closest thing Silicon Valley has to a conscience”, insists that billions of people have little choice over whether they use these now ubiquitous technologies, and are largely unaware of the invisible ways in which a small number of people in Silicon Valley are shaping their lives.
A graduate of Stanford University, Harris studied under BJ Fogg, a behavioural psychologist revered in tech circles for mastering the ways technological design can be used to persuade people. Many of his students, including Eyal, have gone on to prosperous careers in Silicon Valley.
Harris is the student who went rogue; a whistleblower of sorts, he is lifting the curtain on the vast powers accumulated by technology companies and the ways they are using that influence. “A handful of people, working at a handful of technology companies, through their choices will steer what a billion people are thinking today,” he said at a recent TED talk in Vancouver.
“I don’t know a more urgent problem than this,” Harris says. “It’s changing our democracy, and it’s changing our ability to have the conversations and relationships that we want with each other.” Harris went public – giving talks, writing papers, meeting lawmakers and campaigning for reform after three years struggling to effect change inside Google’s Mountain View headquarters.
It all began in 2013, when he was working as a product manager at Google, and circulated a thought-provoking memo, A Call To Minimise Distraction & Respect Users’ Attention, to 10 close colleagues. It struck a chord, spreading to some 5,000 Google employees, including senior executives who rewarded Harris with an impressive-sounding new job: he was to be Google’s in-house design ethicist and product philosopher.
Looking back, Harris sees that he was promoted into a marginal role. “I didn’t have a social support structure at all,” he says. Still, he adds: “I got to sit in a corner and think and read and understand.”
He explored how LinkedIn exploits a need for social reciprocity to widen its network; how YouTube and Netflix autoplay videos and next episodes, depriving users of a choice about whether or not they want to keep watching; how Snapchat created its addictive Snapstreaks feature, encouraging near-constant communication between its mostly teenage users.
I have two kids and I regret every minute that I’m not paying attention to them because my smartphone has sucked me in. — Loren Brichter, app designer
The techniques these companies use are not always generic: they can be algorithmically tailored to each person. An internal Facebook report leaked this year, for example, revealed that the company can identify when teens feel “insecure”, “worthless” and “need a confidence boost”. Such granular information, Harris adds, is “a perfect model of what buttons you can push in a particular person”.
Tech companies can exploit such vulnerabilities to keep people hooked; manipulating, for example, when people receive “likes” for their posts, ensuring they arrive when an individual is likely to feel vulnerable, or in need of approval, or maybe just bored. And the very same techniques can be sold to the highest bidder. “There’s no ethics,” he says. A company paying Facebook to use its levers of persuasion could be a car business targeting tailored advertisements to different types of users who want a new vehicle. Or it could be a Moscow-based troll farm seeking to turn voters in a swing county in Wisconsin.
Harris believes that tech companies never deliberately set out to make their products addictive. They were responding to the incentives of an advertising economy, experimenting with techniques that might capture people’s attention, even stumbling across highly effective design by accident.
A friend at Facebook told Harris that designers initially decided the notification icon, which alerts people to new activity such as “friend requests” or “likes”, should be blue. It fit Facebook’s style and, the thinking went, would appear “subtle and innocuous”. “But no one used it,” Harris says. “Then they switched it to red and of course everyone used it.”
That red icon is now everywhere. When smartphone users glance at their phones, dozens or hundreds of times a day, they are confronted with small red dots beside their apps, pleading to be tapped. “Red is a trigger colour,” Harris says. “That’s why it is used as an alarm signal.”
The most seductive design, Harris explains, exploits the same psychological susceptibility that makes gambling so compulsive: variable rewards. When we tap those apps with red icons, we don’t know whether we’ll discover an interesting email, an avalanche of “likes”, or nothing at all. It is the possibility of disappointment that makes it so compulsive.
It’s this that explains how the pull-to-refresh mechanism, whereby users swipe down, pause and wait to see what content appears, rapidly became one of the most addictive and ubiquitous design features in modern technology. “Each time you’re swiping down, it’s like a slot machine,” Harris says. “You don’t know what’s coming next. Sometimes it’s a beautiful photo. Sometimes it’s just an ad.”
The designer who created the pull-to-refresh mechanism, first used to update Twitter feeds, is Loren Brichter, widely admired in the app-building community for his sleek and intuitive designs.
Now 32, Brichter says he never intended the design to be addictive – but would not dispute the slot machine comparison. “I agree 100%,” he says. “I have two kids now and I regret every minute that I’m not paying attention to them because my smartphone has sucked me in.”
Brichter created the feature in 2009 for Tweetie, his startup, mainly because he could not find anywhere to fit the “refresh” button on his app. Holding and dragging down the feed to update seemed at the time nothing more than a “cute and clever” fix. Twitter acquired Tweetie the following year, integrating pull-to-refresh into its own app.
Since then the design has become one of the most widely emulated features in apps; the downward-pull action is, for hundreds of millions of people, as intuitive as scratching an itch.
Brichter says he is puzzled by the longevity of the feature. In an era of push notification technology, apps can automatically update content without being nudged by the user. “It could easily retire,” he says. Instead it appears to serve a psychological function: after all, slot machines would be far less addictive if gamblers didn’t get to pull the lever themselves. Brichter prefers another comparison: that it is like the redundant “close door” button in some elevators with automatically closing doors. “People just like to push it.”
All of which has left Brichter, who has put his design work on the backburner while he focuses on building a house in New Jersey, questioning his legacy. “I’ve spent many hours and weeks and months and years thinking about whether anything I’ve done has made a net positive impact on society or humanity at all,” he says. He has blocked certain websites, turned off push notifications, restricted his use of the Telegram app to message only with his wife and two close friends, and tried to wean himself off Twitter. “I still waste time on it,” he confesses, “just reading stupid news I already know about.” He charges his phone in the kitchen, plugging it in at 7pm and not touching it until the next morning.
“Smartphones are useful tools,” he says. “But they’re addictive. Pull-to-refresh is addictive. Twitter is addictive. These are not good things. When I was working on them, it was not something I was mature enough to think about. I’m not saying I’m mature now, but I’m a little bit more mature, and I regret the downsides.”
Not everyone in his field appears racked with guilt. The two inventors listed on Apple’s patent for “managing notification connections and displaying icon badges” are Justin Santamaria and Chris Marcellino. Both were in their early 20s when they were hired by Apple to work on the iPhone. As engineers, they worked on the behind-the-scenes plumbing for push-notification technology, introduced in 2009 to enable real-time alerts and updates to hundreds of thousands of third-party app developers. It was a revolutionary change, providing the infrastructure for so many experiences that now form a part of people’s daily lives, from ordering an Uber to making a Skype call to receiving breaking news updates.
But notification technology also enabled a hundred unsolicited interruptions into millions of lives, accelerating the arms race for people’s attention. Santamaria, 36, who now runs a startup after a stint as the head of mobile at Airbnb, says the technology he developed at Apple was not “inherently good or bad”. “This is a larger discussion for society,” he says. “Is it OK to shut off my phone when I leave work? Is it OK if I don’t get right back to you? Is it OK that I’m not ‘liking’ everything that goes through my Instagram screen?”
His then colleague, Marcellino, agrees. “Honestly, at no point was I sitting there thinking: let’s hook people,” he says. “It was all about the positives: these apps connect people, they have all these uses – ESPN telling you the game has ended, or WhatsApp giving you a message for free from your family member in Iran who doesn’t have a message plan.”
A few years ago Marcellino, 33, left the Bay Area, and is now in the final stages of retraining to be a neurosurgeon. He stresses he is no expert on addiction, but says he has picked up enough in his medical training to know that technologies can affect the same neurological pathways as gambling and drug use. “These are the same circuits that make people seek out food, comfort, heat, sex,” he says.
All of it, he says, is reward-based behaviour that activates the brain’s dopamine pathways. He sometimes finds himself clicking on the red icons beside his apps “to make them go away”, but is conflicted about the ethics of exploiting people’s psychological vulnerabilities. “It is not inherently evil to bring people back to your product,” he says. “It’s capitalism.”
That, perhaps, is the problem. Roger McNamee, a venture capitalist who benefited from hugely profitable investments in Google and Facebook, has grown disenchanted with both companies, arguing that their early missions have been distorted by the fortunes they have been able to earn through advertising.
He identifies the advent of the smartphone as a turning point, raising the stakes in an arms race for people’s attention. “Facebook and Google assert with merit that they are giving users what they want,” McNamee says. “The same can be said about tobacco companies and drug dealers.”
That would be a remarkable assertion for any early investor in Silicon Valley’s most profitable behemoths. But McNamee, 61, is more than an arms-length money man. Once an adviser to Mark Zuckerberg, 10 years ago McNamee introduced the Facebook CEO to his friend, Sheryl Sandberg, then a Google executive who had overseen the company’s advertising efforts. Sandberg, of course, became chief operating officer at Facebook, transforming the social network into another advertising heavyweight.
McNamee chooses his words carefully. “The people who run Facebook and Google are good people, whose well-intentioned strategies have led to horrific unintended consequences,” he says. “The problem is that there is nothing the companies can do to address the harm unless they abandon their current advertising models.”
But how can Google and Facebook be forced to abandon the business models that have transformed them into two of the most profitable companies on the planet?
McNamee believes the companies he invested in should be subjected to greater regulation, including new anti-monopoly rules. In Washington, there is growing appetite, on both sides of the political divide, to rein in Silicon Valley. But McNamee worries the behemoths he helped build may already be too big to curtail. “The EU recently penalised Google $2.42bn for anti-monopoly violations, and Google’s shareholders just shrugged,” he says.
Rosenstein, the Facebook “like” co-creator, believes there may be a case for state regulation of “psychologically manipulative advertising”, saying the moral impetus is comparable to taking action against fossil fuel or tobacco companies. “If we only care about profit maximisation,” he says, “we will go rapidly into dystopia.”
James Williams does not believe talk of dystopia is far-fetched. The ex-Google strategist who built the metrics system for the company’s global search advertising business, he has had a front-row view of an industry he describes as the “largest, most standardised and most centralised form of attentional control in human history”.
Williams, 35, left Google last year, and is on the cusp of completing a PhD at Oxford University exploring the ethics of persuasive design. It is a journey that has led him to question whether democracy can survive the new technological age.
He says his epiphany came a few years ago, when he noticed he was surrounded by technology that was inhibiting him from concentrating on the things he wanted to focus on. “It was that kind of individual, existential realisation: what’s going on?” he says. “Isn’t technology supposed to be doing the complete opposite of this?”
That discomfort was compounded during a moment at work, when he glanced at one of Google’s dashboards, a multicoloured display showing how much of people’s attention the company had commandeered for advertisers. “I realised: this is literally a million people that we’ve sort of nudged or persuaded to do this thing that they weren’t going to otherwise do,” he recalls.
He embarked on several years of independent research, much of it conducted while working part-time at Google. About 18 months in, he saw the Google memo circulated by Harris and the pair became allies, struggling to bring about change from within.
Williams and Harris left Google around the same time, and co-founded an advocacy group, Time Well Spent, that seeks to build public momentum for a change in the way big tech companies think about design. Williams finds it hard to comprehend why this issue is not “on the front page of every newspaper every day.
“Eighty-seven percent of people wake up and go to sleep with their smartphones,” he says. The entire world now has a new prism through which to understand politics, and Williams worries the consequences are profound.
The same forces that led tech firms to hook users with design tricks, he says, also encourage those companies to depict the world in a way that makes for compulsive, irresistible viewing. “The attention economy incentivises the design of technologies that grab our attention,” he says. “In so doing, it privileges our impulses over our intentions.”
That means privileging what is sensational over what is nuanced, appealing to emotion, anger and outrage. The news media is increasingly working in service to tech companies, Williams adds, and must play by the rules of the attention economy to “sensationalise, bait and entertain in order to survive”.
In the wake of Donald Trump’s stunning electoral victory, many were quick to question the role of so-called “fake news” on Facebook, Russian-created Twitter bots or the data-centric targeting efforts that companies such as Cambridge Analytica used to sway voters. But Williams sees those factors as symptoms of a deeper problem.
It is not just shady or bad actors who were exploiting the internet to change public opinion. The attention economy itself is set up to promote a phenomenon like Trump, who is masterly at grabbing and retaining the attention of supporters and critics alike, often by exploiting or creating outrage.
Williams was making this case before the president was elected. In a blog published a month before the US election, Williams sounded the alarm bell on an issue he argued was a “far more consequential question” than whether Trump reached the White House. The reality TV star’s campaign, he said, had heralded a watershed in which “the new, digitally supercharged dynamics of the attention economy have finally crossed a threshold and become manifest in the political realm”.
Williams saw a similar dynamic unfold months earlier, during the Brexit campaign, when the attention economy appeared to him biased in favour of the emotional, identity-based case for the UK leaving the European Union. He stresses these dynamics are by no means isolated to the political right: they also play a role, he believes, in the unexpected popularity of leftwing politicians such as Bernie Sanders and Jeremy Corbyn, and the frequent outbreaks of internet outrage over issues that ignite fury among progressives.
All of which, Williams says, is not only distorting the way we view politics but, over time, may be changing the way we think, making us less rational and more impulsive. “We’ve habituated ourselves into a perpetual cognitive style of outrage, by internalising the dynamics of the medium,” he says.
It is against this political backdrop that Williams argues the fixation in recent years with the surveillance state fictionalised by George Orwell may have been misplaced. It was another English science fiction writer, Aldous Huxley, who provided the more prescient observation when he warned that Orwellian-style coercion was less of a threat to democracy than the more subtle power of psychological manipulation, and “man’s almost infinite appetite for distractions”.
Since the US election, Williams has explored another dimension to today’s brave new world. If the attention economy erodes our ability to remember, to reason, to make decisions for ourselves – faculties that are essential to self-governance – what hope is there for democracy itself?
“The dynamics of the attention economy are structurally set up to undermine the human will,” he says. “If politics is an expression of our human will, on individual and collective levels, then the attention economy is directly undermining the assumptions that democracy rests on.” If Apple, Facebook, Google, Twitter, Instagram and Snapchat are gradually chipping away at our ability to control our own minds, could there come a point, I ask, at which democracy no longer functions?
“Will we be able to recognise it, if and when it happens?” Williams replies. “And if we can’t, then how do we know it hasn’t happened already?”
Mark Zuckerberg marked his return from paternity leave Thursday with a concerted effort to put lipstick on the pig of Facebook’s role in swaying the 2016 presidential election. In a Facebook live address from an earth-toned, glass-walled office, the chief executive laid out a series of steps the company will take to “protect election integrity and make sure that Facebook is a force for good in democracy”.
This proactive approach to a growing public relations problem is par for the course for Facebook. The company has a tendency to respond to negative press, and with US lawmakers making noise about the $100,000 in Facebook ads purchased by a Russian influence operation during the election, Zuckerberg may hope that he can pre-empt regulation.
But the problem for Zuckerberg is not just that pigs don’t look good in lipstick. The problem is that more and more people are waking up to the fact that Facebook is less little piggy than it is out-of-control Tyrannosaurus Rex whose creator thought he was building a fun and profitable theme park until it was too late.
Facebook did not grow into a $500bn business whose CEO’s statements have become this century’s fireside chats by being “a force for good in democracy”. Nor was it by “making the world more open and connected” (its first mission statement) and “bringing the world closer together” (its new mission statement).
Facebook grew to its current size, influence and wealth by selling advertisements. And it sold those advertisements by convincing users to provide it with incredibly intimate information about our lives so that advertisers could in turn use that information to convince us to do things.
Some advertisers want us to buy things. Some want us to attend events. And some want us to vote a certain way.
Facebook makes all that advertising cheap and easy and astonishingly profitable by cutting out the sales staff who, in a different kind of company, at a different time, would have looked at an advertisement before it ran.
So Facebook’s systems didn’t fail when they allowed shadowy actors connected to the Russian government to purchase ads targeting American voters with messages about divisive social and political issues. They worked. “There was nothing necessarily noteworthy at the time about a foreign actor running an ad involving a social issue,” Facebook’s vice-president of policy and communications, Elliot Schrage, wrote of the Russian ads in a blogpost.
Shortly after the election, Zuckerberg attempted to shrug off the impact of Facebook in Americans’ voting decision, calling the notion that fake news swayed voters a “pretty crazy idea”. But the CEO of an advertising company can’t afford to denigrate the power of advertising too much; his revenues depend on his actual paying customers believing that Facebook can influence behavior.
Instead, Zuckerberg has constructed for himself a worldview in which every good outcome (such as 2 million people registering to vote thanks to Facebook’s advertisements) was intentional, and every bad outcome was just a mistake that can be addressed by tweaking the existing systems.
If Facebook actually wanted to restrict the ability of foreign powers to interfere with democratic elections, the obvious solution would be to implement strict standards for political ads and employ human staff to enforce them. Such a system would introduce significant friction to the company’s self-service system, however, so in the grand tradition of self-justification, Zuckerberg has chosen to pass off expediency as principle.
“We don’t check what people say before they say it, and frankly, I don’t think our society should want us to,” he said. “Freedom means you don’t have to ask permission first, and that by default you can say what you want.”
That may be true for an individual’s right to express herself, but if Facebook’s targeted, pay-per-click ad campaigns were actually about freedom, wouldn’t they be free?
Instead, the company will continue to rely on an apolitical set of “community standards” that are ill-equipped to address the complex moral and political quandaries faced by the world’s most powerful publisher, and attempt to improve artificial intelligence tools that are nowhere near intelligent enough to prevent bone-headed errors like transforming a screenshot of a rape threat into an ad for Instagram.
This should frighten us all.
Just this week, Facebook admitted that it had designated a Rohingya insurgent group as a “dangerous organization” and instructed its moderators to delete any content by or in praise of the group. That decision, which Facebook said was made by its counter-terrorism team, made sense within the strictures of Facebook’s algorithmic approach to applying “community standards”. The group, the Arakan Rohingya Salvation Army, has engaged in violent acts against Myanmar’s security forces, so Facebook will not allow it to use its powerful communications tools.
Of course, Myanmar’s military is itself engaged in what the top United Nations human rights official has described as a “textbook example of ethnic cleansing”, but attempting to apply a set of community standards to 2 billion users across the globe doesn’t leave a lot of room for nuance.
The obvious historical analogy is to imagine how Facebook would have treated Nelson Mandela and his fellow freedom fighters when they took up arms against apartheid South Africa. My guess does not make for a pretty picture. The revolution will not comply with Facebook’s community standards. But if it has enough money, it may just be allowed to purchase a Facebook ad campaign.