Facebook had a problem on its hands. People were making posts that got caught in the company’s automated moderation system or were taken down by its human moderators. The problem wasn’t that the moderators, human or otherwise, were wrong to take down the posts. No, the problem was that the people behind the posts were famous or noteworthy, and the company didn’t want a PR mess on its hands.

So Facebook came up with a program called XCheck, or cross check, which in many instances became a de facto whitelist. Over the years, XCheck has allowed celebrities, politicians, athletes, activists, journalists, and even the owners of “animal influencers” like “Doug the Pug” to post whatever they want, with few to no consequences for violating the company’s rules.

“For a select few members of our community, we are not enforcing our policies and standards,” reads an internal Facebook report published as part of a Wall Street Journal investigation. “Unlike the rest of our community, these people can violate our standards without any consequences.”

“Few” must be a relative term at Facebook, as at least 5.8 million people were enrolled in the program as of last year, many of them with significant followings. That means a large number of influential people are allowed to post largely unchecked on Facebook and Instagram.

“PR fires”

The social media giant acknowledged that the program existed for politicians in 2019, but details of XCheck’s extent and mismanagement are new. Facebook is apparently aware of XCheck’s problems, based on excerpts from the report, but the company has struggled to fix the issues. Before making any changes to XCheck, the program’s product manager wrote in a report, “we have to balance that with business risk.”

XCheck was initially designed to keep “PR fires” to a minimum. When typical users post something that’s flagged by moderation algorithms or taken down by human moderators, they can file a report with Facebook. If they don’t win their appeal, there’s not much else they can do.

But when prominent users are moderated, they can air their grievances with their followers, creating a potential public relations headache. If the users are politicians, they might call for increased regulation of the platform. So rather than treat these elite users the same as everyone else, Facebook reportedly allows them to post whatever they like. If a post is flagged by an algorithm, it’s sent to a different set of moderators, which the WSJ says are “better-trained, full-time employees,” for review.

Yet as the list of users has grown, XCheck moderation teams apparently haven’t been able to keep pace. “We currently review less than 10% of XChecked content,” one document stated. It’s gotten so bad that elite users have been reportedly able to publish everything from disinformation to threats of violence, revenge porn, and more, allowing posts that violate Facebook’s policies to nonetheless appear on the feeds of thousands to millions of people.

VIP treatment

Even in cases where the content is ultimately removed, Facebook has treated VIP users differently from others. In the documents reviewed by the WSJ, the case of Neymar, the Brazilian soccer player, stands out. In 2019, he posted a video to his Facebook and Instagram accounts that contained nude photos of a woman who had accused him of rape. He claimed that she was extorting him.

For normal users, posting “nonconsensual intimate imagery” triggers a straightforward response—it is immediately deleted, and the person’s account is deactivated. Instead, Neymar’s video remained up for more than a day. Regular moderators couldn’t touch it, and by the time it was taken down by the XCheck team, 56 million people had seen it. The video was reposted 6,000 times, and many commenters bullied and harassed the woman. Neymar denied the rape allegations, and no charges were filed. But despite posting what Facebook itself called “revenge porn,” Neymar’s account was not deleted.

“After escalating the case to leadership, we decided to leave Neymar’s accounts active, a departure from our usual ‘one strike’ profile disable policy,” a Facebook review said.

“Tradeoffs”

“We know our enforcement is not perfect, and there are tradeoffs between speed and accuracy,” Facebook spokesperson Andy Stone said in a Twitter thread. “We have new teams, new resources, and an overhaul of the process that is an existing work-stream at Facebook.”

Some of the documents reviewed by the WSJ came from a person seeking federal whistleblower protection. The person reportedly turned over the same documents to the Security and Exchange Commission and to Congress.

As recently as March, users were being added to the XCheck lists, expanding a program that had already grown too unwieldy for Facebook to adequately manage. The company has reportedly set a goal to enforce “high severity” rule violations by the first half of this year. In the meantime, Facebook plans to assume “good intent” on the part of high-profile users in the program and take an approach of “innocent until proven guilty,” a product manager on Facebook’s Mistakes Prevention Team wrote.

“We do not have systems built out to do that extra diligence for all integrity actions that can occur for a VIP,” she wrote.

https://arstechnica.com/?p=1794212