From a great article by Robert Jervis, professor of international politics at Columbia University:
The problem isn’t usually – or at least isn’t only – too little information, but too much, most of it ambiguous, contradictory, or misleading. The blackboard is filled with dots, many of them false, and they can be connected in innumerable ways. Only with hindsight does the correct pattern leap out at us, and to fix what “broke” the last time around only guarantees you have solved yesterday’s problem.
Far more important, and useful, is to address the flaws in how we interpret and use the intelligence that we already gather. Intelligence analysts are human beings, and many of their failures follow from intuitive ways of thinking that, while allowing the human mind to cut through reams of confusing information, often end up misleading us. This isn’t a problem that occurs only with spying. It is central to how we make sense of our everyday lives, and how we reach decisions based on the imperfect information we have in our hands. And the best way to fix it is to craft policies, institutions, and analytical habits that can compensate for our very understandable flaws.
[...]
The first and most important tendency is that our minds are prone to see patterns and meaning in our world quite quickly, and then tend to ignore information that might disprove them. Premature cognitive closure, to use the phrase employed by psychologists, lies behind many intelligence failures.
[...]
Second, people pay more attention to visible information than to information generated by an absence. In a famous Arthur Conan Doyle story, it took the extraordinary skill of Sherlock Holmes to see that an important clue in the case was a dog not barking. The equivalent, in the intelligence world, is information that should be there but is not.
[...]
Third, conclusions often rest on assumptions that are not readily testable, and may even be immune to disproof.
I’ll add a fourth — ignoring threat intelligence all together or treating it as taboo. This may take several forms: ”it’s beyond our control”, “we don’t have good data”, “it’s too hard to quantify”, “we aren’t paid for guess-work”, “we rely on vendors for that”, “everybody knows what the threats are”, “if we bring it up, we will get too many questions we can’t answer”, or other excuses. (See Josh Corman’s post on the folly of relying on security vendors for your threat intelligence. Vendors only have incentive to inform you about threats they can mitigate.)
If you want a good methodology for threat intelligence, look at Intel’s. It was adapted for use by the Information Technology Sector Coordinating Council in their risk assessment for critical IT industry infrastructure.
As good as it is, it could even be better if they had some systematic methods to actively seek out contradictory information and contrary hypotheses about threats. One simple way to do this is to create a “Mental Model Red Team” whose primary job is to disprove everything you think you know, or at least generate and validate contrary hypotheses. (For social and cultural reasons, you should probably rotate your staff through this team rather than keeping the team membership fixed.) Formal methods exist, including “Analysis of Competing Hypotheses” (slides). (I’m in the process of evaluating a tool for this called SHEBA. I hope to have a demo read for Mini-metricon, something like this.) Another possible method is prediction markets, but I’ve never seen them used for this purpose.
What You’ve Said