Review
Shortlisted for the Cundill History Prize • A Kirkus Best Nonfiction Book of the Year
“It may be the most presciently timed book ever written. . . . It's the love story between America and its secrets.” —Jon Stewart
“Connelly has defined an existential crisis: the suppression of American history. . . . The Declassification Engine makes the case that the culture of secrecy diminishes democracy. And it has now become a culture of destruction as well.” —Tim Weiner, The New York Times Book Review
“Fascinating and urgent. . . . If you believe in the founding principles of the American form of government, then the stakes could scarcely be higher.” —Patrick Radden Keefe, Foreign Affairs
“Harrowing. . . . Connelly’s book unearths disturbing tales. . . . Readers will doubtless look to The Declassification Engine to make sense of the classified files that are now in the news. Yet to insist on the timeliness of Connelly’s research may be to miss its most powerful lesson. There is a much sadder story detailed in the pages of The Declassification Engine—a story about the existential threat that secrecy poses to civic knowledge.” —The Washington Post
“Connelly has written a gripping and sobering account of the exponential increase in government secrets. He persuasively argues that the United States needs a new strategy to handle classified material, demonstrating that both our national security and the health of our democracy are at stake.” —The Christian Science Monitor
“A brilliant, deeply unsettling look at the history and inner workings of ‘the dark state.’ The number of things that truly must be kept secret is small. The vast amount of information classified by the government is simply a means of wielding enormous power without real oversight. Again and again, Connelly reveals, secrecy has been used to hide mistakes, avoid embarrassment, cover up incompetence, and mislead the public. At a time when federal agencies are increasingly classifying or destroying documents with historical significance, this book could not be more important. An inscription at the entrance to the National Archives says it best: ‘Eternal vigilance is the price of liberty.’ ” —Eric Schlosser, New York Times best-selling author of Command and Control
“In The Declassification Engine, Matthew Connelly provides an incisive, unexpected account of the history and practice of official secrecy, offering a glimpse into a world that truly exists in the shadows. By showing the corrosive effects of state secrecy, he successfully makes the case for a different attitude to public information.” —Anne Applebaum, New York Times best-selling author of Twilight of Democracy
“The Declassification Engine is an outstanding expose of the secrecy-industrial complex that is suffocating our democracy. Matthew Connelly describes in vivid detail how the dark state became rooted in our national-security institutions and provides common-sense prescriptions for restoring transparency.” —Craig Whitlock, New York Times best-selling author of The Afghanistan Papers
“A profoundly important work of scholarship, one that addresses core questions about American democracy and the challenges to the nation’s venerable tradition of open government. Connelly’s findings are deeply troubling but also hopeful, showing us how data science can be used to help us better understand the past and thereby point the path to a more enlightened future.” —Fredrik Logevall, Pulitzer Prize-winning author of Embers of War
“Matthew Connelly has played three essential roles in the struggle against government secrecy: advocate, archive-maker, and historian. In The Declassification Engine, he combines all three into an unforgettable account, one that is full of fresh and startling revelations that demonstrate how much of our own history has been kept hidden from us.” —Nicholas Lemann, author of Transaction Man: The Rise of the Deal and the Decline of the American Dream
“This is an absorbing account of the evolution of government secrecy, and an insightful exploration of the relationship between transparency, accountability, and self-government. At a moment when democratic renewal seems absolutely urgent, Connelly’s fascinating study could hardly be more relevant.” —Jameel Jaffer, director of the Knight First Amendment Institute at Columbia University, and former director of the ACLU’s Center for Democracy
“An impassioned indictment of America’s culture of official secrecy. . . . Compelling.” —Washington Independent Review of Books
“The U.S. government is hopelessly awash in secret information, and this gripping history describes how we got that way and lays out the dismal consequences. . . . [Connelly] delivers a wild, page-turning ride packed with intelligence mistakes, embarrassing decisions, expensive failed weapons programs, and bizarre research that has ranged from the silly to the murderous. . . . Yet more evidence, brilliantly delivered, of the extent of the U.S. government’s dysfunction.” —Kirkus Reviews (starred)
“What [Connelly] discovered was unnerving: a highly fallible, exorbitantly expensive (over $18 billion annually, by Connelly's estimate), virtually uncontrollable [classification] system that ultimately renders its administrators unaccountable to the American taxpayers funding it. . . . One hopes this book will generate serious discussion of the issue.” —Booklist
About the Author
MATTHEW CONNELLY is a professor of international and global history at Columbia University, codirector of its social science institute, and the principal investigator at History Lab, a project to apply data science to the problem of preserving the public record and accelerating its release. He received his BA from Columbia and his PhD from Yale. His previous publications include A Diplomatic Revolution: Algeria’s Fight for Independence and the Origins of the Post–Cold War Era and Fatal Misconception: The Struggle to Control World Population.
Excerpt. © Reprinted by permission. All rights reserved.
PREFACE: Should This Book Be Legal?
There I was, sitting at a massive conference table inside a multibillion-dollar foundation, staring at the wood-paneled walls. I was facing a battery of high-powered attorneys, including the former general counsel to the National Security Agency, and another who had been chief of the Major Crimes Unit at the U.S. Attorney’s Office in the Southern District of New York. The foundation was paying each of them about a thousand dollars an hour to determine whether I could be prosecuted under the Espionage Act.
I am a history professor, and my only offense had been to apply for a research grant. I proposed to team up with data scientists at Columbia University to investigate the exponential growth in government secrecy. Earlier that year, in 2013, officials reported that they had classified information more than ninetyfive million times over the preceding twelve months, or three times every second. Every time one of these officials decided that some transcript, or e-mail, or PowerPoint presentation was “confidential,” “secret,” or “top secret,” it became subject to elaborate protocols to ensure safe handling. No one without a security clearance would see these records until, decades from now, other government officials decided disclosure no longer endangered national security. The cost of keeping all these secrets was growing year by year, covering everything from retinal scanners to barbed-wire fencing to personnel training programs, and already totaled well over eleven billion dollars. But so, too, were the number and size of data breaches and leaks. At the same time, archivists were overwhelmed by the challenge of managing just the first generation of classified electronic records, dating to the 1970s. Charged with identifying and preserving the subset of public records with enduring historical significance but with no increase in staff or any new technology, they were recommending the deletion of hundreds of thousands of State Department cables, memoranda, and reports, sight unseen. The costs in terms of democratic accountability were incalculable and included the loss of public confidence in political institutions, the proliferation of conspiracy theories, and the increasing difficulty historians would have in reconstructing what our leaders do under the cloak of secrecy.
We wanted to assemble a database of declassified documents and use algorithms to reveal patterns and anomalies in the way bureaucrats decide what information must be kept secret and what information can be released. To what extent were these decisions balanced and rule-based, as official spokesmen have long claimed? Were they consistent with federal laws and executive orders requiring the preservation of public records, and prompt disclosure when possible? Were the exceptions so numerous as to prove the existence of unwritten rules that really served the interests of a “deep state”? Or was the whole system so dysfunctional as to be random and inexplicable, as other critics insist?
We were trying to determine whether we could reverse engineer these processes, and develop technology that could help identify truly sensitive information. If we assembled millions of documents in databases, and harnessed the power of high-performance computing clusters, it might be possible to train algorithms to look for sensitive records requiring the closest scrutiny and accelerate the release of everything else. The promise was to make the crucial but dysfunctional declassification process more equitable and far more efficient. We had begun to call it a “declassification engine,” and if someone did not start building and testing prototypes, the exponential increase in government secrets—more and more of them consisting of data rather than paper documents—might make it impossible for public officials to meet their own legal responsibilities to maximize transparency. Even if we failed to get the government to adopt this kind of technology, testing these tools and techniques would reveal gaps and distortions in the public record, whether from official secrecy or archival destruction.
The lawyers in front of me started to discuss the worst-case scenarios, and the officers of the foundation grew visibly uncomfortable. What if my team was able to reveal the identity of covert operatives? What if we uncovered information that would help someone build a nuclear weapon? If the foundation gave us the money, their lawyers warned that the foundation staff might be prosecuted for aiding and abetting a criminal conspiracy. Why, the most senior program officer asked, should they help us build “a tool that is purpose-built to break the law”? The only one who did not seem nervous was the former ACLU lawyer whom Columbia had hired to represent us. He had argued cases before the Supreme Court. He had defended people who published schematics of nuclear weapons—and won. He had shown how any successful prosecution required proving that someone had possession of actual classified information. How could the government go after scholars doing research on declassified documents?
The ex–government lawyers pointed out that we were not just academics making educated guesses about state secrets—not when we were using high-performance computers and sophisticated algorithms. True, no journalist, no historian, can absorb hundreds of thousands of documents, analyze all of the words in them, instantly recall every one, and rank each according to one or multiple criteria. But scientists and engineers can turn millions of documents into billions of data points and use machine learning—or teaching a computer to teach itself—to detect patterns and make predictions. We agree with these predictions every time we watch a movie Netflix recommends, or buy a book that Amazon suggests. If we threw enough data at the problem of parsing redacted documents—the ones in which government officials have covered up the parts they do not want us to see— couldn’t these techniques “recommend” the words most likely to be hiding behind the black boxes, which presumably were hidden for good reason?
