How well can algorithms recognize your masked face?

Facial-recognition algorithms from Los Angeles startup TrueFace are good enough that the US Air Force uses them to speed security checks at base entrances. But CEO Shaun Moore says he’s facing a new question: How good is TrueFace’s technology when people are wearing face masks?

“It’s something we don’t know yet because it’s not been deployed in that environment,” Moore says. His engineers are testing their technology on masked faces and are hurriedly gathering images of masked faces to tune their machine-learning algorithms for pandemic times.

Facial recognition has become more widespread and accurate in recent years, as an artificial intelligence technology called deep learning made computers much better at interpreting images. Governments and private companies use facial recognition to identify people at workplaces, schools, and airports, among other places, although some algorithms perform less well on women and people with darker skin tones. Now the facial-recognition industry is trying to adapt to a world where many people keep their faces covered to avoid spreading disease.

Facial-recognition experts say that algorithms are generally less accurate when a face is obscured, whether by an obstacle, a camera angle, or a mask, because there’s less information available to make comparisons. “When you have fewer than 100,000 people in the database, you will not feel the difference,” says Alexander Khanin, CEO and cofounder of VisionLabs, a startup based in Amsterdam. With 1 million people, he says, accuracy will be noticeably reduced and the system may need adjustment, depending on how it’s being used.

Some vendors and users of facial recognition say the technology works well enough on masked faces. “We can identify a person wearing a balaclava, or a medical mask and a hat covering the forehead,” says Artem Kuharenko, founder of NtechLab, a Russian company whose technology is deployed on 150,000 cameras in Moscow. He says that the company has experience with face masks through contracts in southeast Asia, where masks are worn to curb colds and flu. US Customs and Border Protection, which uses facial recognition on travelers boarding international flights at US airports, says its technology can identify masked faces.

But Anil Jain, a professor at Michigan State University who works on facial recognition and biometrics, says such claims can’t be easily verified. “Companies can quote internal numbers, but we don’t have a trusted database or evaluation to check that yet,” says. “There’s no third-party validation.”

A US government lab at the National Institute of Standards and Technology that functions as the world’s arbiter on the accuracy of facial-recognition algorithms hopes to provide that external validation—but is being held up by the same pandemic that prompted the project.

Patrick Grother, a computer scientist who leads NIST’s facial-recognition testing program, says his group is preparing tests to quantify how accurately algorithms identify people wearing masks. NIST plans to digitally add masks to its existing stockpile of photos and test algorithms previously submitted to a test that involves checking whether one photo matches another, similar to the job of a border guard checking passports. Later, it will invite companies to submit new algorithms tuned for face masks. But Grother says the timing of the project is uncertain, because NIST has reduced staffing due to the Covid-19 crisis.

Chinese and Russian companies tend to dominate NIST’s widely watched leaderboards for facial-recognition accuracy. Lighter privacy rules and wider acceptance of surveillance make it easier for those companies to gather the data and operational experience needed to improve facial-recognition algorithms. This year, companies from China and Russia were first to claim their products are ready for a world of half-covered faces.

Early in March, China’s SenseTime, which became the world’s most valuable AI startup in part through providing face recognition to companies and government agencies, said it had upgraded its product for controlling access to buildings and workplaces to work with face masks. The software attends to facial features left uncovered, such as eyes, eyebrows, and the bridge of the nose, a spokesperson said. The US restricted sales to SenseTime and other Chinese AI companies last year for allegedly supplying technology used to oppress Uighur Muslims in China’s northwest.

China’s facial-recognition vendors faced the challenge of identifying masked faces earlier, and more broadly, because the country is both the origin of the novel coronavirus and the world’s most developed market for facial recognition. Chinese citizens can use their faces to pay in stores or use ATMs, while government agencies tap the technology to pluck persons of interest from streets and crowds.

Reports from China of the systems’ effectiveness with masks are mixed. One Beijing resident told WIRED she appreciated the convenience of not having to remove her mask to use Alipay, China’s leading mobile payments network, which has updated its facial-recognition system. But Daniel Sun, a Gartner analyst also in Beijing, says he has had to step out of crowds to pull down his mask to use facial recognition for payments. Still, he believes facial recognition will continue to grow in usage, perhaps helped by interest in more hygienic, touch-free transactions. “I don’t think Covid-19 will stop the increase in usage of this technology in China,” Sun says.

The Japanese conglomerate NEC, which provides the facial-recognition technology used by Customs and Border Patrol at US airports, is cautious in discussing the tech’s capability against masked faces. Benji Hutchinson, a vice president with NEC’s US division, says the company’s labs in Japan that develop its algorithms have always tested on face masks because they are commonly worn during flu seasons in Asia. But the company has launched new rounds of testing now that masks are set to become the norm. “Masks are nothing new to us, but that doesn’t mean it’s all perfect,” Hutchinson says. He says the company is advising customers, such as CBP, to make their own decisions about the technology for now.

Although international passengers are currently rare, a CBP spokesperson said it is still using facial recognition at more than two dozen US airports and that the technology works with masks. “CBP’s biometric facial-comparison technology can match travelers wearing masks to photos from their travel documents,” the spokesperson said.

CBP’s system checks faces of passengers at the departure gate against “faceprints” from photos the Department of Homeland Security has on file for people listed on that flight. Although the agency says passengers are always free to opt out, some people have found it difficult to do so. CBP says that if its technology fails on a person wearing a face mask, they may keep it on while a person manually checks their passport.

Will Knight contributed reporting.

This story originally appeared on wired.com.

https://arstechnica.com/?p=1672592