Enlarged government powers and new information-gathering programs have swelled the databases that are a cornerstone of surveillance in the age of Total Information Awareness. The secret mining and sharing of data can have life-changing consequences for ordinary people, who have no way to know about it or challenge it.

For instance, the Automated Targeting System (ATS) relies on a 5.3 billion-record government database to detect “indicators of possible illegal or other activities” and assign a numerical “terrorist risk profile” to everyone who enters or leaves the United States. That numerical rating is supposed to be kept on file for forty years and can be shared with state, local and foreign governments and used in hiring and contracting decisions.

From the error-ridden terrorist watch list at the National Counterterrorism Center that contains over a million names to the repository of nearly two billion records collected through wiretaps, through hundreds of thousands of National Security Letters or purchased from commercial data brokers and housed at the FBI’s National Branch Analysis Center, we are put at risk by the absence of quality controls to ensure the accuracy of data and by the “echo chamber” effect that ensues when that data is widely shared.

How many government data mining programs are there? The Government Accountability Office in 2004 counted 199 programs that combed through data to detect links or patterns indicating suspicious activity, with names like Automated Detection, Identification and Tracking of Deceptive Terrorist Activity. When some programs were shut down or delayed because of privacy concerns, new ones appeared on the drawing board.

In 2007, the FBI's System to Assess Risk (STAR) was made public. Early in 2008, it was reported that the NSA and the FBI were working together to mine huge volumes of domestic emails, Internet searches, bank transfers, travel and telephone records for “social-network analysis” purposes. The NSA intercepts and stores some 1.7 billion e-mails and other communications every day. The secret $1 billion program was weaving a wide net of associations to find out who might be connected – however remotely – to a suspicious person.

“Advanced technology” will do the work of uncovering threats – that was the response of the FBI to the 2009 report by the Justice Department’s Inspector General that it had fewer translators that it did a few years ago and a backlog of 47,000 hours of wiretaps and at least 7.2 million unreviewed electronic files.

But will it? An exhaustive National Research Council study conducted in 2008 concluded that hunting terrorists through data mining will not work and will lead to ordinary, law abiding citizens and businesses being wrongly treated as suspects. The report was given scant attention by the media, and the nation remains largely unaware of the surveillance state being erected in the shadows.