Amazon is announcing a one-year moratorium on allowing law enforcement to use its controversial Rekognition facial recognition platform, the e-commerce giant said on Wednesday.

The news comes just two days after IBM said it would no longer offer, develop, or research facial recognition technology, citing potential human rights and privacy abuses and research indicating facial recognition tech, despite the advances provided by artificial intelligence, remains biased along lines of age, gender, race, and ethnicity.

Much of the foundational work showing the flaws of modern facial recognition tech with regard to racial bias is thanks to Joy Buolamwini, a researcher at the MIT Media Lab, and Timnit Gebru, a member at Microsoft Research. Buolamwini and Gebry co-authored a widely cited 2018 paper that found error rates for facial recognition systems from major tech companies, including IBM and Microsoft, for identifying darker-skinned individuals were dozens of percentage points higher than when identifying white-skinned individuals. The issues lie in part with the data sets used to train the systems, which can be overwhelmingly male and white, according to a report from The New York Times.

In a separate 2019 study, Buolamwini and co-author Deborah Raji analyzed Rekognition and found that Amazon’s system too had significant issues identifying the gender of darker-skinned individuals, as well as mistaking darker-skinned women for men. The system worked with a near-zero error rate when analyzing images of lighter-skinned people, the study found.

Amazon tried to undermine the findings, but Buolamwini posted a lengthy and detailed response to Medium, in which she says, “Amazon’s approach thus far has been one of denial, deflection, and delay. We cannot rely on Amazon to police itself or provide unregulated and unproven technology to police or government agencies.” Her and Raji’s findings were later backed up by a group of dozens of AI researchers who penned an open letter saying Rekognition was flawed and should not be in the hands of law enforcement.

Amazon did not give a concrete reason for the decision beyond calling for federal regulation of the tech, although the company says it will continue providing the software to rights organizations dedicated to missing and exploited children and combating human trafficking. The unspoken context here of course is the death of George Floyd, a black man killed by former Minnesota police officers, and ongoing protests around the US and the globe against racism and systemic police brutality.

It seems as if Amazon decided police cannot be trusted to use the technology responsibly, although the company has never disclosed just how many police departments do actually use the tech. As of last summer, it appeared like only two departments— one in Oregon and one in Florida — were actively using Rekognition, and Orlando has since stopped. A much more widely used facial recognition system is that of Clearview AI, a secretive company now facing down a number of privacy lawsuits after scraping social media sites for photos and building a more than 3 billion-photo database it sells to law enforcement.

In a statement given to The Verge, Clearview AI CEO Hoan Ton-That doubled down on the technology as an effective law enforcement tool. “While Amazon, Google, and IBM have decided to exit the marketplace, Clearview AI believes in the mission of responsibly used facial recognition to protect children, victims of financial fraud and other crimes that afflict our communities,” he said. Ton-That says Clearview’s technology “actually works,” but that facial recognition is “not intended to be used as a surveillance tool relating to protests or under any other circumstances.”

Beyond studies calling into question its effectiveness, Amazon has faced constant criticism over the years for selling access to Rekognition to police department from activists, civil rights organizations like the ACLU, and lawmakers, all of which have cited concerns about the lack of oversight into how the tech is used in investigations and potential built-in bias that makes it unreliable and ripe for discrimination and other abuses.

Even after employees voiced concern about the tech in 2018, Amazon’s cloud chief Andrew Jassy said the company would continue to sell it to police. Only through media reports and activists, as well as the work of researchers like Buolamwini, highlighting the pitfalls of police use of facial recognition tech like Rekognition have departments begun discontinuing contracts with Amazon.

Here’s Amazon’s full note on the one-year ban:

We’re implementing a one-year moratorium on police use of Amazon’s facial recognition technology. We will continue to allow organizations like Thorn, the International Center for Missing and Exploited Children, and Marinus Analytics to use Amazon Rekognition to help rescue human trafficking victims and reunite missing children with their families.

We’ve advocated that governments should put in place stronger regulations to govern the ethical use of facial recognition technology, and in recent days, Congress appears ready to take on this challenge. We hope this one-year moratorium might give Congress enough time to implement appropriate rules, and we stand ready to help if requested.

Update June 10th, 7:17PM ET: Added additional information around studies finding evidence of racial bias in Amazon Rekognition and other facial recognition systems.

Update June 10th, 8:43PM ET: Added statement from facial recognition firm Clearview AI.

https://www.theverge.com/2020/6/10/21287101/amazon-rekognition-facial-recognition-police-ban-one-year-ai-racial-bias