In response to internal and external protest and expressions of revulsion, Google yesterday announced it would stop contracting with the Department of Defense to provide artificial intelligence technology to the US military. Google’s move is a rare instance in which one of the Big Five artificial intelligence firms in the United States takes an action to undercut its own profits in service of its professed ethical commitments. Other companies, including Amazon, must follow suit, and stop selling technology to human rights violators, including police departments.
Last month, the ACLU and a coalition of over forty civil rights, civil liberties, religious, and community groups sent a public letter to Amazon demanding that the company stop selling its face surveillance software to law enforcement and other government agencies. Since at least November 2016, Amazon has sold the software, called Rekognition, to police departments in Oregon and Florida, and aggressively marketed it to others. In a promotional video for Rekognition, Amazon brags that the technology can “detect, track, and analyze people and objects in video.” The company advertises that, using Rekognition, law enforcement officers can “track persons of interest from a collection of tens of millions of faces.”
The coalition letter to Amazon comes on the heels of disclosures about law enforcement use of Amazon’s technology, obtained after the ACLU Foundations of California, the ACLU of Oregon, and the ACLU of Florida submitted public records requests to learn more about police departments’ deployments of Rekognition. The records show that the Washington County, Oregon sheriff’s office has already compiled a face surveillance-ready database of at least 300,000 mugshots of arrestees. The records also reveal that Amazon employees have been working closely with police and sheriffs departments to guide their use of the technology. The Seattle-based company even offered the Orlando Police Department free consulting services. As a result, Amazon’s Rekognition face surveillance technology is now intertwined with that city’s infrastructure. As a presenter at an Amazon conference in Seoul explained, “There are cameras all over [Orlando]. Authorized cameras are streaming the data to Kinesis video stream…We analyze that data in real time and search against the collection of faces that they have. Maybe they want to know if the mayor of the city is in a place, or there are persons of interest they want to track.” Subtract “Black Lives Matter activist” for “Mayor” and you’ve got the makings of a turnkey totalitarian system of control. Chilling stuff.
In our coalition letter to Amazon, the ACLU and other groups warn that Rekognition “is a powerful surveillance system readily available to violate rights and target communities of color.” With its “people of interest” monitoring and “person tracking” features, Rekognition could easily enable law enforcement agencies to track groups of people who may not be committing any crimes, but may nonetheless be of interest to law enforcement agencies, including undocumented immigrants, Black activists involved with protest movements to counter police violence, and journalists critical of the police or city government. These dangers are compounded by the race-based discrepancies in face surveillance accuracy rates. A recent study from a researcher at the M.I.T. Media Lab confirms that face surveillance technologies produce substantially less accurate results for people of color and women. Whereas the technology was correct 99% of the time when comparing photos of white men, it was incorrect for almost 35% of the photos of women of color. The error rate was highest for women with the darkest skin.
In response to concerns about the racial justice and human rights implications of the technology, boosters of Rekognition maintain that it is merely meant to increase government efficiency. In response to press inquiries about their use of the tech, public information officer for the Washington County Sheriff’s Office Jeff Talbot sought to clarify that the county government was aware of their use of software. He further emphasized the focus on more efficient policing by pointing out that the booking photos in their database are already publicly available; the technology just helps them sift through them much more quickly. “Our goal is to inform the public about the work we’re doing to solve crimes. It is not mass surveillance or untargeted surveillance,” he said. For its part, Amazon released a statement in which it noted that, “Our quality of life would be much worse today if we outlawed new technology because some people could choose to abuse the technology. Imagine if customers couldn’t buy a computer because it was possible to use that computer for illegal purposes?”
Both these positions fall flat. Defenders say the tech merely makes law enforcement practices more efficient, but this defense ignores a fundamental question: Should we use technology like AI to make racially discriminatory or oppressive systems (like United States police practices) more efficient? After all, we know that people of color are more likely to be arrested than white people, even when crime rates are comparable. Therefore the Washington County mugshot database likely contains more people of color than whites, per capita, even if white people and people of color commit crimes at similar rates. Making this system more efficient could have the impact of exacerbating racial disparities in policing, meaning “efficiency” is not value neutral.
Other companies make face surveillance technology, so why are the ACLU and its partners targeting Amazon? As Alvaro Bedoya, executive director at the Center on Privacy and Technology at the Georgetown University Law Center, argues, “The idea that a massive and highly resourced company like Amazon has moved decisively into this space could mark a sea change for this technology.” Indeed, Amazon is offering its face surveillance software at extremely low prices, making it easy for even resource-strapped police departments to participate. The commercial viability of Amazon’s system and its low price create the conditions for widespread law enforcement use of Rekognition, which means widespread opportunities for abuse, misuse, and racially disparate enforcement. According to the documents obtained by the ACLU affiliates, law enforcement agencies in California and Arizona as well as several fusion centers have already reached out to Washington County about their experience with Rekognition. And Amazon has tried to facilitate this kind of communication with other agencies as well as a body camera manufacturer.
Currently, there are no state or federal laws restricting or limiting law enforcement’s use of face surveillance technology. Police departments nationwide continue to struggle with racially disparate enforcement, and even racially disparate violence against Black, Native, and Latinx people. As our coalition letter charges, “Rekognition is primed for abuse in the hands of governments.” Amazon should follow Google’s lead, and stop selling Rekognition to governments. If we want to live in a free and open society, the alternative is simply unacceptable.
This blog was co-authored by Iqra Asghar and Kade Crockford