Now, just over a month later, we have a better idea at how many takedowns have actually happened as a result of the policy. Brian Fishman, Director of Counterterrorism and Dangerous Organizations at Facebook, said the company had identified more than 300 organizations, and taken down thousands of pages and groups as a result of the updated rules. Fishman didn’t disclose exactly which groups the takedowns were associated with.

I know folks are also curious about the new Militarized Social Movements policy announced on Aug 19. Bottom line: we’ve identified more than 300 groups under this policy and removed more than 6500 Groups/Pages btw 8/19 and 9/15.

— Brian Fishman (@brianfishman) September 30, 2020

The policy came under scrutiny last month after Facebook failed to remove an event page that called for an armed response to protests in Kenosha, Wisconsin, despite several users reporting it. Mark Zuckerberg later called it an “operational mistake.”

Fishman also detailed Facebook’s actions against the far-right extremist group Proud Boys, after the group got renewed attention following Donald Trump’s comments during the presidential debate. He said Facebook had seen “an uptick” in memes and other content related to the Proud Boys, but said “much of this content” was condemning the group and Trump’s remarks. Facebook banned the Proud Boys in 2018, and has recently taken down other accounts associated with it, Fishman said.

Facebook groups, especially ones that dabble in health-related topics, have long been problematic for the company. Groups dedicated to anti-vaccine conspiracy theories, for example, have also been linked to QAnon and COVID-19 disinformation — often by the company’s own algorithmic suggestions. Mark Zuckerberg recently said the company won’t take down anti-vaccine posts the way it does with COVID-19 misinformation.

Speaking of QAnon, Facebook says it’s taking an additional step to keep groups associated with the conspiracy theory from spreading by “reducing their content in News Feed.” The company previously removed hundreds of groups associated with the movement, but hasn’t rooted out its presence entirely.

Facebook

Finally, Facebook will now archive groups that no longer have an active admin. “In the coming weeks, we’ll begin archiving groups that have been without an admin for some time,” Facebook writes. In the future, the company will recommend admin roles to members of groups without one before archiving.

Facebook notes that it penalizes groups that repeatedly share false claims that are debunked by its fact checkers and that it’s removed more than a million groups in the last year for repeat offenses or otherwise breaking its rules. 

But critics have long said that Facebook doesn’t do enough to police groups on its platform, which have been linked to disinformation, harassment and threats of violence. The company came under fire last month after it failed to remove a Wisconsin militia group that organized an armed response to protests in Kenosha until the day after a deadly shooting. And a number of Facebook groups have been credited with hampering the emergency response to devastating wildfires in Oregon after spreading baseless conspiracy theories about how the fires were started. Facebook eventually began removing these claims after emergency responders begged people to stop sharing the rumors.

Twitter has banned the accounts of far-right extremist group the Oath Keepers and its founder, Stewart Rhodes. According to the Southern Poverty Law Center, the group is one of the US’s largest anti-government movements. It claims to have tens of thousands of members, many of whom identify as former law enforcement and military. The group recently tweeted there would be “open warfare against the Marxist insurrectionists (its shorthand for Black Lives Matter activists) by election night.”

A spokesperson for Twitter told Engadget it banned accounts associated with the Oath Keepers for violating its policies on violent extremist groups. “There is no place on Twitter for terrorist organizations or violent extremist groups and individuals who affiliate with and promote their illicit activities,” the company says on the policy page. “We examine a group’s activities both on and off Twitter to determine whether they engage in and/or promote violence against civilians to advance a political, religious and/or social cause.”

QAnon’s reach has exploded since the start of the coronavirus pandemic, and Facebook’s own algorithms, which recommended QAnon content on Facebook and Instagram, have been blamed for helping fuel its rise. And the company has faced heavy criticism for not doing more to prevent QAnon content from going viral on its platform. Facebook’s actions comes weeks after Twitter banned the group and nearly two years after Reddit booted QAnon off its platform.

Facebook’s approach is slightly different: Instead of trying to root out the conspiracy entirely, the company is taking steps to prevent it from continuing to go viral on its services. Facebook says it will prevent pages, groups and Instagram accounts from appearing in its algorithmic recommendations, reduce their distribution in News Feed and rank accounts lower in search results. The company notes that on Instagram it has “temporarily removed” the “related hashtags” feature, which has been a major source QAnon recommendations.

In addition to QAnon, the new rules will also apply to “offline anarchist groups that support violent acts amidst protests” — an apparent reference to “antifa” — and “US-based militia organizations,” the company said. These groups “have demonstrated significant risks to public safety,” Facebook said, even if they “do not meet the rigorous criteria to be designated as a dangerous organization and banned from having any presence on our platform.”

What’s less clear is just how effective these steps will be. NBC News recently reported that the largest QAnon groups have millions of members (it’s so far unclear how many members were in the groups Facebook took down), and QAnon memes have seeped into other conspiracy theory groups on the platform. Facebook also has a mixed track record when it comes to dealing with conspiracy theories. For example, the company said last year it would limit the visibility of anti-vaccine misinformation, but these groups have continued to find new ways to evade these efforts.

How QAnon spread

That QAnon, a far right conspiracy theory that originated on 4Chan in 2017, could gain so much prominence might sound ridiculous. Proponents of QAnon believe the government is run by a ring of pedophiles/child-eating cannibals who will ultimately be brought down by Donald Trump. QANon’s favorite targets are somewhat predictable: the Clintons, George Soros and Bill Gates frequently star in their memes. But Hollywood celebrities like Tom Hanks, Chrissy Tiegien and Oprah also play a central role in the conspiracy. 

The conspiracy theory may have started on the darker corners of the internet, but it flourished on social media. Facebook and other platforms have played a central role in fueling the growth of QAnon, and the coronavirus pandemic has only accelerated its spread. 

“It is certainly more mainstream than it’s ever been before, says Katy Byron who runs the digital literacy initiative MediaWise, which teaches people how to spot misinformation and disinformation online. “You can find support for the theories that feed the QAnon beast on any given day, it’s not that hard. It is hard to bat down.”

QAnon is also unique compared with other conspiracy theories in that it’s constantly evolving, says Mick West, a conspiracy theory researcher and author of Escaping the Rabbit Hole, a book about debunking conspiracy theories. “It’s very up to the minute, whereas most conspiracy theories are about things that happened in the past, like JFK or 9/11,” West says. “QAnon is day-to-day, everything is happening on a daily basis. It’s like this wave is constantly going forward and constantly changing and shifting.”

It’s perhaps unsurprising, then, that the coronavirus pandemic presented a unique opportunity for QAnon to spread its theories about the dangers posed by the global elite. Even before widespread lockdowns and shelter in place orders, QAnon took “a significant interest in several aspects of the coronavirus outbreak,” according to researchers at Graphika. “The community was highly engaged on the topic of the outbreak even in its early stages,” noting that #coronavirus was the fourth most popular hashtag among QAnon supporters in the month of February. 

Besides conspiracies about the Chinese origins of the virus, there were also “attempts to incorporate the coronavirus outbreak into the wider QAnon structure using the hashtag #StormIsHere,” Graphika writes. (The “storm” refers to a widespread belief that there will be a global reckoning when there are mass arrests of the supposed pedophiles running the government.)

As coronavirus conspiracy theories continued to spread, so did interest in QAnon. Data from Google Search trends shows that searches for QAnon spiked in mid-March around the start of widespread lockdowns and shelter in place orders. QAnon’s Wikipedia page saw similar spikes during the same period, as noted by Mother Jones. 

Google Trends

It’s not clear exactly why interest in QAnon skyrocketed at the start of the pandemic.  It’s possible that many people, seeking out alternative explanations on the complex cause of the pandemic, were exposed to QAnon content for the first time. A simpler explanation is that people had much more free time on their hands and QAnon inadvertently found itself a much more captive audience than it might normally have. Facebook reported in March that it was struggling with “unprecedented” demand for its services.

Regardless, the interest translated to new social media followers. Some of the most popular QAnon Facebook groups and Instagram accounts saw a huge influx of followers that coincided with lockdown orders, according to The Wall Street Journal. 

At the same time, other groups that regularly traffic in conspiracy theories, such as anti-vaccination groups, seized on coronavirus conspiracy theories in March and April. Many of these have become intermingled with QAnon, often helped by Facebook’s recommendation algorithms. (Facebook says it removes accounts that break its rules from recommendations.)

As Engadget reported in May, Instagram’s algorithm pushes followers of anti-vaccine conspiracies to QAnon pages and hashtags. The same has happened in Facebook groups, where QAnon has an even bigger presence. There are thousands of QAnon groups with millions of members, according to data obtained by NBC News. While on Instagram, the most popular QAnon pages have hundreds of thousands of followers each. The hashtag #WWG1WGA, a reference to a popular QAnon mantra, has more than 1 million posts on the app.

The (sort of) crackdowns

It’s not just Facebook, either. QAnon has also had significant influence on Twitter, which recently announced a widespread crackdown on the movement in order to prevent “offline harm.” Even TikTok has tried to limit the visibility of QAnon content in its app. 

But unlike Twitter and TikTok, Facebook has done little to slow QAnon’s momentum. The company has removed a handful of accounts for breaking its rules against platform manipulation and took down a 200,000-member group for breaking harassment and hate speech policies. But the company has so far declined to take any broader action, despite hinting that it may do so.

“We take action against accounts, Groups, and Pages tied to QAnon that break our rules,” a Facebook spokesperson said in a statement. “Just last week, we removed a large Group with QAnon affiliations for violating our content policies, and removed a network of accounts for violating our policies against coordinated inauthentic behavior. We have teams assessing our policies against QAnon and are currently exploring additional actions we can take.” 

Meanwhile, QAnon has attached itself to issues with a more “mainstream” appeal. Recently, #SavetheChildren has become a rallying cry for QAnon believers, who have inundated Facebook groups and Instagram pages with memes about supposed child trafficking. 

Stephanie Keith / reuters

Though human trafficking is a real and serious problem, QAnon believers’ obsession has twisted the issue to fit the broader Q conspiracy. Perhaps the clearest example of this was when rumors started spreading that furniture retailer Wayfair was using expensive industrial cabinets as a front for selling actual children. 

According to BuzzFeed News, it originated on Reddit, but quickly took off on Instagram thanks, in part, to lifestyle influencers. Fact checkers quickly debunked the hoax, but not before it went so viral that Polaris, the group behind the National Human Trafficking hotline, had to issue a press release begging people to stop reporting the bogus theory because it was making it more difficult for them to help actual victims. 

This underscores what experts warn is the most worrying part of QAnon: that it has been linked to real-world harm. As the conspiracy theory has gained steam, its followers have been involved in real-world violence. The FBI said last year the group posed a domestic terror threat.

But even if most QAnon supporters never turn to violence, the effects could still be damaging, says West, the conspiracy theory researcher and author. For individuals, believing in conspiracy theories like QAnon can lead to social isolation or even financial distress, if they are duped into supporting the causes financially. Distrust of the medical establishment can have health consequences. 

But these beliefs can also be bad for society when they become widespread,  West says, pointing to Greene’s recent election win.

“You’ve got someone who believes in complete nonsense. And she’s going to be making decisions based on that belief, and people make voting decisions based on their belief.  if a false belief becomes widespread enough, it essentially corrupts all of society because it starts affecting the way people make decisions.”

Facebook just took down one of the larger targets in its crackdown on QAnon conspiracy theories. The social network told Reuters that it had removed Official Q/Qanon, a group with almost 200,000 members, after multiple posts reportedly violated policies against harassment, hate, and potentially harmful misinformation. The shutdown took effect August 4th, Facebook said, but wasn’t disclosed until August 6th (and publicized on the 7th).

The company added that it was keeping an eye on other QAnon groups on the site.

The company isn’t removing the videos using the hashtags, however. Followers will still see the affected clips in their feeds, and they could theoretically appear in the For You feed of suggested videos.

We’ve asked TikTok if it can comment further on its decision.

The approach isn’t as strict as with rivals that completely removed some accounts and pages. Even so, it could significantly reduce exposure to QAnon videos and discourage theory backers from creating material. TikTok is already fighting some forms of misinformation on its platform, but this is an acknowledgment that there’s still more work to do.

If RSA’s attendees were even able to score an Airbnb, it was second to the tech companies who’d, for years, packed employees into expensive rentals that once were on the normal-person market. Companies whose fat salaries also pushed rents out of reach for locals. Both had ensured a steady flow of evictions among artists, writers, musicians, teachers, sex workers, people of color, the elderly, and restaurant workers. Or they became part of San Francisco’s thousands upon thousands of homeless (like the grocery cashiers and pizza servers I knew living in cars).

This was February, yet I was already too aware of COVID-19’s contagion to brave going to the RSA conference. My best friend, a hacker visiting for conference-related meetings, felt the same way. Instead, we went to Haight-Ashbury, essentially where I grew up, loving the gritty contrast of Haight street punks posing for Japanese tourists under the Ben and Jerry’s sign on that iconic corner of colorful Victorians. 

At Japantown’s mall, she cautioned me to keep my phone clean with sterile wipes; while there we saw a man in a mask have a coughing fit that drove people away from him like dish soap dripped into a pan of oily water. She avoided RSA too, but caught covid when she got home. And in the following five months the world would come to a screeching halt and over half a million would be dead with no end in sight.

RSA Conference 2020 added 40,000 faces to our downtown of glittering towers and their corporate tenants’ technological promises of a better future, but that was a nominal blur for San Francisco tourism. We barely felt it. Yet the conference is a crystalline moment for me. I can pinpoint the day I began self-quarantining by the publication of my February 28 Bad Password column, Coronavirus bursts Big Tech’s bubble.

That column, like many of the Bad Passwords we’ve done here over the past five years, reads now like something that was published from the future, recognizing we were at the tipping point of the pandemic and cautioning the violent contractions to come.

Elon Musk is not a fan

Steve Nesius / Reuters

Like the rest of the world right now, Bad Password is going on a pandemic-induced hiatus. Shining a light on tech’s monsters and hypocrites has been our jam for five years, and there’s been plenty of greed, data dealing, security chicanery, discrimination, misinformation, and recklessness to go around. When Bad Password started, infosec slang was finally becoming everyday terminology. No one understood yet what a skid was (most still don’t) but I no longer had to explain what a dox was. 

Right out of the starting gate we surfaced a National Network to End Domestic Violence (NNEDV) report that showed Facebook is the epicenter of abuse for over 23 million women — with the site’s “real names” policy at the heart of it. At the time, Facebook was targeting LGBTQ users and outing their names. After that column, I was “real named” by Facebook, who stalled their response to my attorneys, after which my account remained in Facebook jail for one reason or another. I don’t miss it.

Bad Password’s next big fun-time was when Hacking Team, arguably a  spyware-for-dictators company, got royally, publicly pwned. In How spyware peddler Hacking Team was publicly dismantled we examined what the hack revealed: a country-by-country rundown of who Hacking Team had done deadly deeds for. I cross-referenced Hacking Team’s client work with human rights reports on digital abuses by date and place, then worked with a team to make an interactive map — it was later used as a case study.

When Oculus Rift founder (and alt-right shitpost financier extraordinaire), Palmer Luckey, pivoted into pitching LIDAR tech to hunt immigrants, we took it apart brick by brick. Luckey’s response made my colleagues envious by decrying it as “fake news.” Then there was the time we documented Elon Musk’s PR lackeys calling Pulitzer-prize winning investigative outlet Reveal an “extremist organization” for reporting on Tesla factory safety issues. And when that Bad Password was directly cited to Musk on Twitter, he famously responded with a call for a journalist rating system. Elon really wanted to leave me a bad Yelp review.

When FOSTA passed, we explained why this was a horrible defining moment for every internet user, and not just for sex workers. When revisiting it, we found it left a very real body count behind — and that particular Bad Password is cited in academic articles on the topic. This heralded the great internet war on sex we’re suffering through, and with a sobering post-FOSTA terror we explained exactly how sex censorship killed the internet we love.

We did what Bad Password loves to do, which is show you the hypocrisy of a techie thing, shine a humorous spotlight on the greedy opportunists, and find the human thread to engender empathy (while seeking a strong positive to pull us forward when we can). I entered an alternative 1995 universe to take Rudy Giuliani, cybersecurity expert, down several notches. While others took the WikiLeaks bait hook, line, and sinker, we diagrammed exactly how Julian Assange was actually pushing propaganda. We hated Ajit Pai before it was cool. We also got to do one of the most thorough and painfully humorous takedowns of Uber’s toxic techbro culture you may ever read.

Bad Password also reveled in exposing the lies, dirty dealings, and anti-sex crusades of all those alleged “anti trafficking” orgs that love policing sex on the internet (and off). We also did one of the most referenced investigations on PayPal, Square and big banking’s war on the sex industry.

Where the past meets the present, before the disastrous 2016 election, we said yes, you should absolutely be worried about election hacking. And Bad Password did something many had hoped someone, somewhere would do: We drew a direct line between IBM working with Nazi Germany and tech companies working directly with ICE.

New by me, in which Facebook literally gives me a “both sides” comment about InfoWars https://t.co/N4F8C7H9vy

— Violet Blue® (@violetblue) July 13, 2018

And yeah. It’s still all Facebook’s fault. I mean, they’ve raised what, at least an entire generation on firmly defended Holocaust denial. So here we are.

Locked indoors during a global pandemic, re-reading the Bad Password about Apps and gadgets for the ‘Blade Runner’ future we didn’t ask for. Watching Georgia’s election-hacking Brian Kemp rescind all local mask mandates while masks have become mandatory in France (and other countries). Wondering if we can somehow hack our way out of all this.

Rats fleeing the ship they sunk

Justin Sullivan via Getty Images

It has been five months since RSA Conference 2020. I feel like I emerged from my apartment to a San Francisco brutally ransacked by tech charlatans and venture capital Bioshock villains.

The tech buses are gone. The unbelievable traffic of Ubers, Lyfts, and Teslas has vanished. The Facebook, Google, Salesforce, and massive tech properties are vacant. The Twitter building is empty and could remain that way permanently. Tech employees have moved out in droves: one in ten city renters have broken their leases and moved out; others, like a house of Google employees who live near me, plan to be gone by the end of the year.

Vultures linger to see if we have anything left to bleed. Like Grubhub ignoring delivery fee caps and hiking fees on coronavirus-crippled restaurants yet again and Airbnb asking guests to “donate” money to their hosts.

Airbnbs across the city have no bookings. Zero. The (reviled) one on my block has nothing booked through at least 2021. It sits dark with the power shut off. I can see its back garden has turned brown, dry and dead. At least a third of the apartments and houses around me are vacant; I’ve watched them move out.

Airbnbs linger on Craiglist as fully furnished apartments where they sit and gradually become discounted, then include all utilities and wifi, then offer the first month free. SF Craigslist, where rents are absolutely schizophrenic, veering drunkenly from 1990 levels ($1400/month) to tech boom heights ($5K and up). The Craigslist “free” section overflows with designer furniture and high-end household items. More often, these spoils of the pandemic get dumped on the sidewalk in haste.

I can tell you for a fact that we won’t miss those people with more money than sense, whose businesses were so plainly naive and fraudulent, whose lack of empathy was a trait cherished as aspirational, and whose solidarity was predicated on the exclusion, use, and degradation of others. San Francisco had become a performative playground for sheltered college grads who wrote racist algorithms, who enforced “real names” policies on our LGBTQ communities, and whose companies leeched hate and deadly misinformation into our collective bloodstreams until eventually the world as we know it stopped.

Yet tech’s impact on my hometown, its invasive services no one wanted and human-unfriendly gig economy (as well as its economic crushing of the poor and disenfranchised), now combined with COVID-19 has delivered a one-two punch bringing us to our knees.

Where once we had localized areas of homeless encampments, they now sprawl block after block. Think “Skid Row,” but evenly distributed. Upper Haight, the neighborhood of my youth, looks like a post-nuclear blast zone town in Fallout 4, or Fallout 76. Five businesses on Haight closed permanently in the last week alone. Some blocks have two or three businesses remaining. Everything is boarded up. So many people have gone missing recently that my Haight Street friends and I wonder if it’s coronavirus, or a serial killer.

The smell of Ben and Jerry’s ice cream is gone. For now.

Hold back the night

bluejayphoto via Getty Images

While Big Tech had been unconcerned with the outcomes of their privacy abuses, held a blatant disregard for user security, and were unwilling to believe their tools would be used to livestream massacres, Bad Password tirelessly documented, raised the alarm, and worked its hardest to shine a light into the dark. Our attitude here has never been “You should have known better.” Instead, it has been “the powerful people making decisions for the rest of us knew better, but did nothing.”

Our plan is for Bad Password to return. Our hope is that when we do, the tech forces that got us into much of this mess (and certainly made it worse) will decide that enough people have died to justify excising anti-science propaganda, banning hate groups and Holocaust denial, and will own up to their catastrophic failures at being responsible, ethical, just, and compassionate participants in the world around them. 

The days ahead feel dark now, but whatever comes next is in our hands. Let the unhappy techies keep their internet of shit garbage while we repurpose their devices and designs to reveal monsters, to document abuses that should never be repeated, and to take care of one another. 

Where we go from here, is forward.

We will permanently suspend accounts Tweeting about these topics that we know are engaged in violations of our multi-account policy, coordinating abuse around individual victims, or are attempting to evade a previous suspension — something we’ve seen more of in recent weeks.

— Twitter Safety (@TwitterSafety) July 22, 2020

“We will permanently suspend accounts Tweeting about these topics that we know are engaged in violations of our multi-account policy, coordinating abuse around individual victims, or are attempting to evade a previous suspension — something we’ve seen more of in recent weeks,” the company wrote in a statement. 

These actions will be rolled out comprehensively this week. We will continue to review this activity across our service and update our rules and enforcement approach again if necessary.

— Twitter Safety (@TwitterSafety) July 22, 2020

Twitter also said it would take steps to limit the reach of QAnon-related content throughout its service. The company will block accounts and content associated with conspiracy theory from trends and recommendations and will “work to ensure we’re not highlighting this activity in search and conversations.” Twitter will also prevent users from sharing URLs linked to QAnon, though Twitter didn’t elaborate on which sites or forums might be impacted. 

Twitter’s actions are, notably, much more aggressive than Facebook’s stance toward QAnon. Though Facebook banned a small subset of QAnon accounts this year for breaking its rules around coordinated inauthentic behavior, proponents of the conspiracy theory remain quite active on Facebook and Instagram. The company also continues to surface groups, pages and accounts associated with the conspiracy theory in its recommendations. 

Instagram’s ‘rabbit hole’ problem

Like Facebook, Instagram doesn’t ban anti-vaccine content, though the company claims it has attempted to make it less visible to users. The company blocks some hashtags and says it tries to make anti-vaccine content harder to find in public areas of the app, like Explore. Yet accounts promoting conspiracy theories and inaccurate information about vaccines dominate the app’s search results. 

When you search the word “vaccine” on Instagram, the app recommends dozens of anti-vaccine accounts in its top results. Accounts with names such as “Vaccines_revealed,” “Vaccinesuncovered,” “vaccines_kill_” “vaccinesaregenocide_” and “say_no_to_bill_gates_vaccine” are front and center. While some of these accounts are popular, with nearly 100,000 followers, others have only a few hundred. Yet Instagram’s algorithm consistently recommends these accounts — and not one verified health organization — as the most relevant accounts for the search term “vaccine.” 

Accounts promoting conspiracy theories and inaccurate information about vaccines dominate search results.

Some of these accounts are meant to sow fear — many are aimed at parents — and post clearly spurious claims like “vaccines are causing autism rates to skyrocket”. Many have pivoted to posting conspiracy theories about Bill Gates and the coronavirus pandemic.

Instagram’s recommendation algorithm also pushes users toward accounts spreading conspiracy theories, including those about vaccines and COVID-19. 

I made a new Instagram account, searched “vaccine,” and then followed a few of the top results mentioned above. Within seconds, the app began suggesting I follow more anti-vaccine pages and other accounts peddling conspiracy theories, including QAnon. This isn’t a new phenomenon, either. Vice noted last year that Instagram’s follow suggestions could easily lead users down an anti-vax rabbit hole. The company said at the time it would look into it, but it doesn’t appear much has changed. 

Not only do the suggestions still appear, these recommendations are now pushing users toward other fringe conspiracy theories. I only had to follow four anti-vaccine accounts before Instagram began recommending popular QAnon pages, one of which prominently linked to the Plandemic documentary Facebook and others have struggled to successfully banish from their platform. A couple days later, the app sent push notifications recommending I follow two more QAnon pages.

An Instagram spokesperson reiterated that the company aims to make misinformation about vaccines harder to find in public areas of the app.

Searching for specific hashtags can also lead users into a rabbit hole of conspiracy theories.

Searching for #vaccine prompts you to first visit the CDC’s website and contains relatively sanitized results, but Instagram’s hashtag search recommends other “related” search terms that are less filtered, including #vaccineinjuryadvocate and #vaççineskillandinjure. (Using the cedilla character instead of a “c” is a common tactic used by anti-vaccine advocates in order to evade detection, as Coda reported last year.)

Screenshot / Instagram

And when you look at search results for these recommended hashtags, like #vaccineinjuryadvocate, Instagram further suggests more hashtags associated with various other conspiracy theories, including coronavirus conspiracy theories: #plandemic, #governmentconspiracy, #populationcontrol, and #scamdemic. (Instagram has since blocked search results for #plandemic, which had more than 26,000 posts, according to the app.)

Screenshot / Instagram

Instagram’s algorithm recommending hashtags associated with conspiracy theories isn’t just limited to vaccines either. Search #5G and the app surfaces “related hashtags” like #fuckbillgates #billgatesisevil #chemtrails and #coronahoax. Other seemingly innocuous suggestions, like #5Gtowers, also lead to conspiracy theories like #projectbluebeam #markofthebeast #epsteindidntkillhimself. 

Misinformation on Instagram

None of these are new issues for Instagram, but the photo-sharing app’s misinformation problem has often avoided the same scrutiny that’s been applied to Facebook. When company officials testified in front of Congress, they downplayed Instagram’s role in spreading Russian disinformation. The Senate Intelligence Committee’s subsequent report found that Instagram ”was the most effective tool used by the IRA.” 

The problem, according to those who study it, is that misinformation on Instagram often takes the form of memes and other images that are harder for the company’s systems to detect and can be more difficult for the company’s human reviewers to parse. And while Instagram is building out new systems to address this, images can be a much more effective conduit for bad actors, says Paul Barrett, the deputy director of NYU’s Stern Center for Business and Human Rights.

“Disinformation, which would include anti-vaxxer material, is increasingly a visual game. This is not something that’s done exclusively or even primarily anymore via big blasts of text,” Barrett said. “Visual material makes it easy to digest, and something that’s not going to seem threatening or overbearing. And I think as a result that makes Instagram appealing.” 

Yet Instagram has been much slower to deal with its misinformation problem than Facebook. The photo-sharing app didn’t implement any fact-checking efforts until last May — nearly three years after Facebook began debunking posts with outside fact-checkers. And the app has only recently moved to make debunked posts less visible in users’ feeds.

Instagram

And though Instagram, like Facebook, has prioritized coronavirus misinformation it considers “harmful,” the company doesn’t apparently consider anti-vaccine content, which researchers have linked to measles outbreaks and other instances of actual harm, to be as urgent a problem as some coronavirus conspiracies. 

“We’re prioritizing reviewing certain types of content, like child safety, suicide and self injury, terrorism and harmful misinformation related to COVID, to make sure that we’re handling the most dangerous issues,” Mark Zuckerberg said during a call with reporters to discuss the company’s content moderation efforts this week. 

When asked whether the company was prioritizing anti-vaccine content given its links to coronavirus misinformation, Facebook’s VP of integrity, Guy Rosen, said, “Health-related harm is something that’s very much top of mind and very much something that we want to prioritize.”

An Instagram spokesperson told Engadget the company doesn’t bar anti-vaccination content, but noted it has removed some posts with misinformation in response to a deadly measles outbreak in Samoa and a polio resurgence in Pakistan. Officials in both countries have blamed misinformation for rising anti-vaccination sentiment.

In most cases, though, the company doesn’t act to remove such content entirely, attempting to make it less visible or adding “false information” labels when the content has been debunked by fact-checkers.

But fact-checking might not be enough, according to Barrett. “Facebook is so outmatched by the scale of the problem, it’s almost a little naive to assume that fact-checking is — even if it’s done vigorously — that you’re going to be able to catch a substantial majority of false information that’s being posted on a continuous basis,” Barrett says. “When you’re talking about billions of posts a day, even if you have Facebook’s 60 fact-checking organizations around the world, a lot of stuff is going to slip by them.”