These came across the SIRA mailing list. They were so good, I had to share:
http://eight2late.wordpress.com/2009/10/06/on-the-limitations-of-scoring-methods-for-risk-analysis/
Thanks to Kevin Riggins for finding them and pointing them out.
Archive for the 'Science of Risk Management' Category
In comments to my “Why I Don’t Like CRISC” article, Oliver writes:
CobIT allows to segregate what is called IT in analysable parts. Different Risk models apply to those parts. e.g. Information Security, Architecture, Project management. In certain areas the risk models are more mature (Infosec / Project Management) and in certain they are not (software distribution). That is for the risk modelling (sic) part.
Oliver: I’m very glad that others in our industry are preaching the concept of model selection & fit. And because you’ve demonstrated that at least you believe this is an important aspect of IRM, I’m ready to believe what you’re saying there. But before I do so, I spent a good deal of time in Missouri, so I need you to show me:
- Define “mature” – what makes a mature information risk model? In fact, show me the industry standards for gauging model maturity, so that I can examine different models, similarly.
- Show me, oh please show me, an information risk model that has even been tested (publicly) for repeatability and accuracy, more or less been shown to provide repeatability and accuracy to a measurable degree of confidence.
Now my thought is that you can’t have a mature risk model without having a measurable notion of repeatability (two analysts with the same data and same model go into separate rooms and come out with reasonably similar results) and accuracy (model outcomes have been tested to be correct some degree of the time). Maybe I’m not subscribing to the right scientific journals out there, but I’ve yet to see the data sets and the published models or model maturity tests for IRM.
For risk identification and KRIs (note to readers: I’m assuming Oliver means Key Risk Indicator – a useful but loaded phrase itself), an internal control framework which is based on cobit allows an adequate and comprehensive net of indicators for risk assessment based on operational performance.
You’re assertion is that COBIT’ is proven to be an “adequate” and “comprehensive” internal control framework. Can you show me evidence of this? What documentation for this has ISACA released? How was it proven? Where’s the study? How did they seek to falsify COBIT’s adequacy and comprehension? How was comprehensive measured? At what point was it shown that more COBIT effort decidedly into the realm of diminishing returns?
If you think that “some things can’t be measured” will prove your thesis, you don’t know Risk Management at all.
I never said that, and due to the fact that I’ve taught courses based on Hubbard’s “How To Measure Anything” to risk analysts, I’m going to offer that you don’t know me well enough to come to any conclusion about my knowledge around Information Risk Management.
What I’m saying is that ISACA, COBIT, and RiskIT aren’t mature enough to certify practitioners in a meaningful manner – where “maturity” is an ability to consistently, repeatably, and accurately show a change in risk using ISACA’s own documentation. If you can’t show me how COBIT measurably (again, where the concept of measurement requires known accuracy and repeatability – just drilling the point home, here) modifies exposure to risk or capability to manage risk in these ways, I don’t think ISACA is ready to say that we, as an industry, are more than isolated alchemists trying to find our own, individual ways to turn lead into gold. To carry the analogy, the attestation that CRISC would provide has nothing to do with knowledge of chemistry, but everything to do with the alchemists ability to repeat a known means of trying to turn lead into gold.
There is no mathematical voodoo to model a risk exposure which is 100% correct.
We’re in agreement about modeling risk exposure. To paraphrase Jaynes (poorly), probabilistic models are hypothesis and therefore we should expect (hope!) for them to be frequently falsified. In addition – just to complete the picture for you, Oliver, I’m also on record as stating that arriving at a state of knowledge for capability to manage risk is similarly difficult (and this is the whole crux of the COBIT/RISKIT/CRISC request for proof – understanding capability in a measurable way is a key dependency to understanding exposure, and therefore, ISACA is silly for trying to certify that someone can discuss exposure if they can’t even show me how COBIT reduces risk) .
You have to keep the purpose in mind and also use professional judgment based on your experience (which CRISC by the way tries to attestate)
Fascinating, so CRISC tries to provide clear evidence that an individuals experience and professional judgment is of some quality? My whole point in this series is that any individual with experience in information risk management should know enough to know that a certification around Information Risk Analysis and management is goofy. As for documenting an individual’s professional judgment skills, I’d love to see how the test does that in a rational manner.
You fight against an attestation which takes into full consideration your own challenge.
Nope. Not even close. You have no CLUE what I stand for. I’m all for good attestation. As I said the other day:
(…I’d argue that IRM shouldn’t be part of an MIS course load, rather it’s own tract with heavier influences from probability theory, history of science, complexity theory, economics, and epidemiology than, say, Engineering, Computer Science or MIS.)
My position is that given the difficult nature of risk analysis (as I’m saying above), there’s no way CRISC can attest to any competency around Information Risk Analysis, and if ISACA can’t show me how COBIT changes exposure or capability in a measurably way, then CRISC can’t possibly even attest to competency around Information Risk Management. Maybe it can serve as a RiskIT test, sure and I’m fine with that. From the same blog post as my quote above:
IRM is not (just one) “process”. Now obviously certain risk management standards (document a simple) process. In my opinion, most risk management standards are nothing BUT a re-iteration of a Plan/Do/Check/Act process. And just to be clear, I have no problems if you want to go get certified in FAIR or OCTAVE or Blahdity-Blah – I’m all for that. That shows that you’ve studied a document and can regurgitate the contents of that document, presumably on demand, and within the specific subjective perspective of those who taught you.And similarly if ISACA wants to “certify” that someone can take their RiskIT document and be a domain expert at it, groovy. Just don’t call that person “Certified in Risk and Information Systems Control™” because they’re not. They’re “Certified in our expanded P/D/C/A cycle that is yet another myopic way to list a bajillion risk scenarios in a manner you can’t possibly address before the Sun exhausts it’s supply of helium.” “TM”
I’ll state it again, if they want to change the certification’s title and meaning to simply state that an individual can do the above for RiskIT – have a day, good on you. Just don’t expect me to believe that this certification means that the individual knows anything about information risk analysis, or risk analysis in general.
For your consideration, two articles in today’s New York Times. First, “How to Remind a Parent of the Baby in the Car?:”
INFANTS or young children left inside a vehicle can die of hyperthermia in a few hours, even when the temperature outside is not especially hot. It is a tragedy that kills about 30 children a year, according to the National Safety Council.
…
Janette Fennell is the founder and president of KidsAndCars.org, a safety advocacy group based in Leawood, Kan., that focuses on issues involving children and automobiles. In a telephone interview, Ms. Fennell made her view clear, saying she believed that carmakers must develop reminder devices to warn drivers if a child is left behind.
Second, “The Hard Sell on Salt:”
High blood pressure is rising among adults and children. Government health experts estimate that deep cuts in salt consumption could save 150,000 lives a year.
Bets on which problem is “addressed” first are encouraged in the comments.
If you are developing or using security metrics, it’s inevitable that you’ll have to deal with the dimension of time. It’s harder than it looks and I’ve seen many people make mistakes with it, and in doing so, rendering their overall metrics faulty or worse. The problems often start with our basic concepts and how we use words.
"Time flies like an arrow, but fruit flies like bananas" -- Groucho Marx
“Data” tells you about the past
“Data” is the output of some observation or measurement process. If your data is about some states of the world, then by definition your data lives in the past. You did your measurements or your experiments, generated your data, and then time passed as you assess it, report it, and act on it. Thus, your data is reporting on history. Only by acts of inference can you connect your data with the present state of the world or the future state.
In the physical sciences and engineering, they can safely assume that the system under study is the same over time — past, present, and future. This is called the ergodic hypothesis. In statistics, the underlying stochastic process is treated as stationary. This makes it possible to extrapolate the past into the present and future using regression and other techniques.
There are people in the security metrics community that only want to operate on data. They view anything that is not the result of empirical measurement is pure speculation or a dangerously-seductive “model”. (See Models are Distracting, and Measurement over Models) Being an engineer myself, I’m all in favor of empirical data, measurment, and experiments. But I contend that we will never get to measures of “security” or “risk” through empirical data alone. Our systems are non-stationary and non-ergodic.
“Security” is a judgement about the present
If we start with the simple high-level question: “Am I secure?”, it becomes clear that any measurement of security must relate to the present time (or possibly a retrospective view on a previous time, i.e. past perfect tense, or prospective view on a future time, i.e. “will I be secure?”). I call it a “judgement” because security depends on the threats you are facing. (I play a historically-realistic computer game with my son, called Total War, that includes features that allow you invest in offensive and defensive capabilities. How much to invest and how fast to invest depends on who you are facing. A wooden pallisade will be an adequate defense against peasants and spear militia, but hopelessly inadequate against onagers and trebuchets, backed by armored cavalry!)
Thus, you can measure anything and everything you want about security, generating tons of data, and in the end you will have to make a judgement: “Am I secure?” — or are my security provisions adequate given the threats we face? Seen this way, your data is really just evidence that is used in this judgement (and inference) process. What I mean by this is that I don’t think you can simply calculate your way from ground-truth data to any overall security metrics. There will always be a judgement or inference step(s).
Why? Because we must account for events, circumstances, and scenarios that haven’t happened yet, or happen so rarely that we have no relevant data, or are beyond the reach of measurements. (Afterall, the miscreants often do their best to hide their actions.) On top of this, the security landscape changes rapidly and occasionally dramatically. Our judgement about security must factor in these changes, to the best of our knowledge. Finally, our judgement about “are we secure?” is predicated on our risk tolerence. But what is “risk”?
“Risk” is a cost of the future, brought to the present
This is the economist’s definition of risk, where “cost” here means downside cash flows that are beyond some threshold of expectation or variability. Those costs become “risk” when you can account for them in present dollars using some discounting and insurance method. (This says nothing about the “insurability” of the risk, only about the theoretical possibility of accounting for risk in present dollars by some reasonable method. The “insurance method” might be diversification, hedging, self-insurance, risk pooling, contingent contracts, or traditional insurance.)
This parallels Peter Drucker’s characterization of profit: “Profit is … needed to pay for attainment of the objectives of the business. Profit is a condition of survival. It is the cost of the future. The cost of staying in business.” [emphasis added] Ontologically, “profit” and “risk” are in the same category, which is why it makes sense to measure “risk-adjusted return” and the like.
From the viewpoint of risk, what you have spent in the past is irrelevant (“sunk costs”). All rational decisions are based on future cash flows and options. The only value of the past is if it helps you predict or forecast the future. Thus, you can’t reach a final judgement about security in the present if you don’t also have some useful estimate of risk in the future. If the answer to “Am I secure?” is “Yes”, then the implication is that you can live with the risk associated with this level of security. By “useful”, I mean sufficiently discriminating to inform the judgement — “bigger than a breadbox, smaller than a house”.
This is where information security deviates from reliability engineering. In the latter, the ergodic hypothesis holds and the dynamics are sufficiently “tame” to permit statistical data analysis for inference and forecasting. Even when there are “humans in the loop”, their behavioral tendencies can often be characterized by stable probability distributions. In information security, we are dealing with adaptive, intelligent, strategic players — not only miscreants, but also “ancillary players” like end-users, auditors, supply chain partners, and so on. This makes risk estimation a ”wicked problem“. But is it hopeless?
Estimating risk may be hard, but not impossible
Plenty of smart security people contend that quantitative risk estimation is impossible or infeasible in principle. Proving or disproving this assertion would take heavy-duty theoretical analysis (and I may do it some day). But for now consider two extreme situations.
Think of security and risk as a black-box process that generates a continuous stream of cash flows in time (i.e. total spending on security and losses in that time period). At one extreme, the output is a stationary function or stochastic process. This is the relm that Nicholas Taleb called “Medicoristan“, since the data stream is well-behaved enough that nothing very surprising happens. With enough historical data and enough data analysis, I think we’d all agree that risk estimation is feasable with current methods.
At the other extreme, the output is generated by a strategic agent (inside the box) whose sole purpose is to screw up our risk estimation process. Let’s call this Descartes’ Demon, after Rene Descartes, who introduces a skeptical scenario called the deceiving demon argument to challenge our beliefs that an external world exists; in particular, it raises the possibility that some sort of malicious, demonic non-God, has “employed all his energies in order to deceive me”. If Descartes’ Demon can maintain history of the output and also has information about our risk estimation process, he can mimic any output pattern and change those patterns arbitrarily to defeat any estimation process we might apply. (This is more extreme than Taleb’s “Extremestan” in terms of defying estimation or prediction.) In this case, I believe it could be proved that estimation is impossible (or undecidable or infeasable from a computation point of view).
Some people might argue that information security is exactly in this latter extreme situation, but I don’t think so. The reason is that all the players have much stronger motives and forcing functions than to subvert the risk estimation processes. Bad guys want to make money or cause harm. End users want to avoid hassles and minimize effort and get their job done. Managers want to manage their business while avoiding negative repercussions. All of these factors add some elements of predictability and understandability.
But it may only be possible to factor all of these in through the use of models and simulations that represent our best knowledge, our best estimates, and our best beliefs about how they all relate to each other and the overall results.
The marriage of data, security, and risk = social learning processes
Putting this all together, we need to gather a lot of empirical data to understand relationships, patterns, and dependencies. But to measure security we need to add inference and judgement processes that extend our data into the present, given the threat landscape we believe we are facing. But to make a judgement about security and make decisions about alternative security postures, we need a useful estimate of risk to decide how much security is enough. To tie these all together over time requires effective social learning processes, including model validation through experiments and data analysis. Likewise, risk estimation and security judgement processes tell us what data we need to collect and how to analyze it.
Whether you agree with this framework or not, you should make explicit and consistent definitions of the time dimension relative to your metrics.
One of the reasons I like climate studies is because the world of the climate scientist is not dissimilar to ours. Their data is frought with uncertainty, it has gaps, and it might be kind of important (regardless of your stance of anthropomorphic global warming, I think we can all agree that when the climate changes, crazy things can happen).
Recently, the mainstream press has begun to pick up on this, and trying to explain what science is doing. One such example is this Times (UK) story called
Scientists Need The Guts To Say, “I Don’t Know”
In it, the author (David Spiegelhalter – Professor of the Public Understanding of Risk at the University of Cambridge) discusses uncertainty in past (and forward) looking predictions. Yes, it’s worth noting that the science of prediction applies to all three states of time: past, present, and future.
As a security professional, I always encourage the representation of uncertainty. Depending on the audience, I’ll represent uncertainty technically, or at a high level with words like “back of the napkin, very rough, a lot of unknowns, fairly certain, pretty good idea…” I’ve found that as long as they are properly qualified, demonstrations of risk with high degrees of uncertainty are not unuseful.
HEY, YOU GOT YOUR VISIBILITY INTO MY UNCERTAINTY!!! AND YOU GOT YOUR UNCERTAINTY IN MY VISIBILITY!!!
They really *are* two great tastes that taste great together….
One of the great reasons for the IT Risk management/Security team to communicate uncertainty (esp. to others with money) is that if you say “here’s what we think but we’re not sure “, you can then tell the business owner “and if you give me $funding we can decrease that uncertainty by gaining visibility into $whatever”. If they decline, they’re accepting both the risk and the probability that you’re wrong. But if they’re uncomfortable with the uncertainty, now you have a pretty good qualitative way of knowing that their tolerance for this level of risk is pretty low, and you might even be able to skip right past the “buy more visibility” step above and move right into “of course, we can just spend $Y and take care of the whole thing, visibility, risk reduction and all….”
Similarly, if you, the security manager, keep getting risk analyses back that have significant uncertainty in them – you know that these are areas where you really don’t have much control. They may represent reasons or opportunities to strengthen policies, processes, capabilities (w00t everybody goes to training in Cancun!) and so forth.
So while it’s also the enemy of accuracy, uncertainty can also be your friend.
One last note, having to do with uncertainty; in the article the author uses the Taleb definition of “Black Swan”. Again, calling a rare event a “Black Swan” is a misnomer. Rarity in frequency is only one aspect of what the concept of Black Swan represents. A much better definition of a Black Swan is “an occurance which is not representable at all given our prior distributions. Certainly, even after before Prof. Spiegelhalter corrected the model for double yoked eggs – the occurance of 6 is not a true Black Swan. We could have run MCMC sims until our computers melted into hot lumps of toxic waste and various occurrences of double yoked eggs would/could have been represented.
Some time back, a friend of mine said “Alex, I like the concept of Risk Management, but it’s a little like the United Nations – Good in concept, horrible in execution”.
Recently, a couple of folks have been talking about how security should just be a “diligence” function, that is, we should just prove that we’re doing best efforts and that should be enough. Now conceptually, I love the idea that we can prove our “compliance” or diligence and get a get out of jail free card when an incident happens. I always think it’s lame when good CISO’s get canned because they got “unlucky”.
Unfortunately, if risk management is infeasible, I’ve been thinking that the concept of Due Diligence Security is complete fantasy. To carry the analogy, if Risk Management is the United Nations, then Due Diligence Security is the Justice League of Superfriends. With He-Man. And the animated Beatles from Yellow Submarine. That live in the forrest with the Keebler elves and the Ewoks and where the glowing ghosts of Anakin, Obi-Wan and Yoda perform the “Chub-Chub” song with the glowing ghosts of John Lennon and George Harrison. That sort of fantasy.
DUE DILIGENCE BASED SECURITY IS AN ARGUMENT FROM IGNORANCE
Here’s the rub – lets say an incident happens. Due Diligence only matters when there’s a court case, really. And in most western courts of law these days, there’s still this concept of innocent until proven guilty. This concept is known as the argument from ignorance in logic and it is known as a logical fallacy.
Now arguments from ignorance are known as logical fallacies thanks to the epistemological notion of falsification. Paraphrasing Hume paraphrasing John Stuart Mill – we cannot prove “all swans are white” simply because we’ve observed all white swans - BUT the observation of a single black swan is enough to prove that “not all swans are white”. This matters in a court of law, as your ability to prove Due Diligence as a defendant will be a function your ability to prove all swans white – all systems compliant. But the prosecution only has to show a single black swan to prove that you are NOT diligent.
Sir Karl Popper says, “Good luck with that, Mr. CISO”.
The result is this – the CISO, in my humble opinion, will be in a worse condition because we have a really poor ability to control the expansion of sensitive information throughout the complex systems (network, system, people, organization) for which they are responsible. Let me put it this way: If information (and specifically, sensitive information) operates like a gas, automatically expanding to where it’s not controlled – then how can we possibly hope that the CISO can control the “escape” or leakage of information 100% of the time with no exceptions? And a solitary exception in a forensic investigation becomes our black swan.
And therefore… When it comes to proving Due Diligence in the court of law – Security *screws* the CISO. Big Time.
For some years, I’ve been following the world of academic and industrial research on information security, especially interdisciplinary research. There is wide-spread agreement on what needs to be done:
- A Roadmap for Cybersecurity Research, by DHS
- National Cyber Security Research and Development Challenges , by the I3P
- Toward a Safer and More Secure Cyberspace, National Academies
- Report to the President on Cyber Security: A Crisis of Prioritization , by PITAC
- Ensuring (and Insuring?) Critical Information Infrastructure Protection, 2005 Rueschlikon Conference on Information Policy
- Four Grand Challenges in Trustworthy Computing , Computing Research Association Conference, 2003
- Others
But no one seems to be able to mobilize any signficant research into solutions. It’s been very frustrating to see so much talk and so little action.
This reminds me of the quote by Mark Twain, shown at left, which is the inspiration for the title of this post.
The latest iteration of this was a panel at RSA: “The role of research in industry and government“. SC Magazine summarized the discussion this way:
A disconnect between the primary research sectors and a lack of appropriate funding in each is leading to decreased technological progress, exposing a huge gap in security that is happily being exploited by cybercriminals.
(read on for a diagnosis and two proposed solutions…)
Yesterday, I offered up a little challenge to suggest that we aren’t ready for a certification around understanding information risk. Today I want to mention why I think this CRISCy stuff is dangerous.
What if how we’re approaching the subject is wrong? What if it’s mostly wrong and horribly expensive?
I’m going to offer that we’re still too early on to know the answers to these questions (an offer that if correct, would also serve to prove my point yesterday about CRISC). But if it turns out that we are doing things incorrectly (and really, what’s the probability that we are doing risk management correctly) – does something like CRISC make it easier or more difficult to change to something more effective?
Obviously, you don’t have to have a degree in Organizational Behavior to identify the problem here. If our approach to risk management is wrong, then CRISC is only going to serve to ensure that we are set in our incorrect ways.
Now where this should *really* upset you, my dear reader, is if you subscribe to various theories about how sciences progress. If you believe that sciences progress by sporadic, somewhat instantaneous little revolutions – then we’re totally screwing ourselves by creating a bureaucracy that makes it more difficult for the next revolution to take place. And believe me, as I’ve found out over the past 4 years, creating that revolution in risk management is hard enough already.
From a great article by Robert Jervis, professor of international politics at Columbia University:
The problem isn’t usually – or at least isn’t only – too little information, but too much, most of it ambiguous, contradictory, or misleading. The blackboard is filled with dots, many of them false, and they can be connected in innumerable ways. Only with hindsight does the correct pattern leap out at us, and to fix what “broke” the last time around only guarantees you have solved yesterday’s problem.
Far more important, and useful, is to address the flaws in how we interpret and use the intelligence that we already gather. Intelligence analysts are human beings, and many of their failures follow from intuitive ways of thinking that, while allowing the human mind to cut through reams of confusing information, often end up misleading us. This isn’t a problem that occurs only with spying. It is central to how we make sense of our everyday lives, and how we reach decisions based on the imperfect information we have in our hands. And the best way to fix it is to craft policies, institutions, and analytical habits that can compensate for our very understandable flaws.
[...]
The first and most important tendency is that our minds are prone to see patterns and meaning in our world quite quickly, and then tend to ignore information that might disprove them. Premature cognitive closure, to use the phrase employed by psychologists, lies behind many intelligence failures.
[...]
Second, people pay more attention to visible information than to information generated by an absence. In a famous Arthur Conan Doyle story, it took the extraordinary skill of Sherlock Holmes to see that an important clue in the case was a dog not barking. The equivalent, in the intelligence world, is information that should be there but is not.
[...]
Third, conclusions often rest on assumptions that are not readily testable, and may even be immune to disproof.
I’ll add a fourth — ignoring threat intelligence all together or treating it as taboo. This may take several forms: ”it’s beyond our control”, “we don’t have good data”, “it’s too hard to quantify”, “we aren’t paid for guess-work”, “we rely on vendors for that”, “everybody knows what the threats are”, “if we bring it up, we will get too many questions we can’t answer”, or other excuses. (See Josh Corman’s post on the folly of relying on security vendors for your threat intelligence. Vendors only have incentive to inform you about threats they can mitigate.)
If you want a good methodology for threat intelligence, look at Intel’s. It was adapted for use by the Information Technology Sector Coordinating Council in their risk assessment for critical IT industry infrastructure.
As good as it is, it could even be better if they had some systematic methods to actively seek out contradictory information and contrary hypotheses about threats. One simple way to do this is to create a “Mental Model Red Team” whose primary job is to disprove everything you think you know, or at least generate and validate contrary hypotheses. (For social and cultural reasons, you should probably rotate your staff through this team rather than keeping the team membership fixed.) Formal methods exist, including “Analysis of Competing Hypotheses” (slides). (I’m in the process of evaluating a tool for this called SHEBA. I hope to have a demo read for Mini-metricon, something like this.) Another possible method is prediction markets, but I’ve never seen them used for this purpose.
Longtime readers know that I’m not the biggest fan of GRC as it is “practiced” today. I believe G & C are subservient to risk management. So let me offer you this statement to chew on:
“A metric for Governance is only useful inasmuch as it describes an ability to manage risk”
True or False, why, and what are the implications if true or false.
Please discuss.
#newschoolsecurity
What You’ve Said