February 28, 2017

Lawyers needed for forecasting, director of national intelligence program says

Keeping our nation safe is an increasingly complicated task, and decision makers at the 17 U.S. national security agencies must rely on emerging technologies to improve and refine operations. 

Enter Jason Matheny, director of the Intelligence Advanced Research Projects Activity, under the Office of the Director of National Intelligence. Often referred to as the spy agencies’ technology incubator, IARPA was founded in 2006 and focuses on the intersection between data analytics and human judgment.

IARPA funds research projects in more than 500 organizations globally and in 12 countries. “We work with the best scientists and engineers … from universities, colleges, small businesses, large businesses, national labs and think tanks. We pay them to work on our hardest problems,” Matheny said.

But IARPA researchers need lawyers’ help, Matheny said Feb. 23 at an ABA Standing Committee on Law and National Security breakfast.

“This is a group that has an unusual intersection to both a dedication to service and a dedication to the life of the mind and its applications to important problems but also a sense of humor, which is sometimes rare in national security circles.”

Matheny said the work of IARPA often involves problems that blur the lines between technology, national security, law and policy – which is where lawyers come in.

Two studies that Matheny said lawyers are particularly well-suited for and offer a chance to add to the body of forecasting knowledge:

  • Hybrid Forecasting Competition, which aims to improve forecasting by combining human judgment with machine models;

  • CREATE – Crowdsourcing Evidence, Argumentation, Thinking and Evaluation – which aims to give analysts a clearer understanding of conflicting evidence, knowledge gaps and degrees of uncertainty. 

He urged lawyers in the room to participate in either or both by logging in at IARPA.gov and Good Judgment Open (gjopen.com). “We’ll need thousands of participants for these two programs if we’re going to make progress,” Matheny said.

One of the most exciting aspects of IARPA and a defining characteristic is that it looks at proposals from anyone, from hobbyists to large companies and academic labs. “In a way we’ve outsourced Q Branch,” he said, referencing the fictional research and development division of the British Secret Service from James Bond novels and films.

Matheny said IARPA funds the toughest of the tough problems that other agencies are not equipped for – everything from quantum computing research, superconducting computing, machine learning and things like speech recognition and facial recognition, machine translation, video analysis and the development of new nuclear sensors, biological sensors and chemical sensors.

They also work in “soft sciences,” such as cognitive psychology, sociology and political science, because many of the big problems in the intelligence community are fundamentally “problems of human judgment,” he said.

Over the last several years, IARPA has held forecasting tournaments, in which research teams try to predict real-world events such as cyberattacks, disease outbreaks and technology milestones before they happen. “IARPA plays the role of referee,” Matheny said.

What they’ve learned as a result of these tournaments is that the best forecasters are those with high fluid intelligence or high critical-thinking ability. These individuals are deeply self-critical, often doubting their judgments and continually revising their judgments when presented with new information. The best critical thinkers also doggedly seek out information that contradicted their existing beliefs, which is “unusual and precious,” he said.

These critical thinkers also tend to assign more weight to judgments that others might have discounted, which makes them “constructive contrarians.” The best forecasts were obtained by averaging the top 1 percent of the best critical thinkers, which can be traced to the Condorcet legal theory – that correct decision-making improves when more are added to the deciding entity, Matheny said.