Tag Archive for 'risk management'

ISACA CRISC – A Faith-Based Initiative? Or, I Didn’t Expect The Spanish Inquisition

Published by alex on July 2, 2010 in Science of Risk Management. 1 Comment Tags: , , , , , .

In comments to my “Why I Don’t Like CRISC” article, Oliver writes:

CobIT allows to segregate what is called IT in analysable parts.  Different Risk models apply to those parts. e.g. Information Security, Architecture, Project management. In certain areas the risk models are more mature (Infosec / Project Management) and in certain they are not (software distribution). That is for the risk modelling (sic) part.
Oliver:  I’m very glad that others in our industry are preaching the concept of  model selection & fit.  And because you’ve demonstrated that at least you believe this is an important aspect of IRM, I’m ready to believe what you’re saying there.  But before I do so, I spent a good deal of time in Missouri, so I need you to show me:
  1. Define “mature” – what makes a mature information risk model?  In fact, show me the industry standards for gauging model maturity, so that I can examine different models, similarly.
  2. Show me, oh please show me, an information risk model that has even been tested (publicly) for repeatability and accuracy, more or less been shown to provide repeatability and accuracy to a measurable degree of confidence.
Now my thought is that you can’t have a mature risk model without having a measurable notion of repeatability (two analysts with the same data and same model go into separate rooms and come out with reasonably similar results) and accuracy (model outcomes have been tested to be correct some degree of the time).  Maybe I’m not subscribing to the right scientific journals out there, but I’ve yet to see the data sets and the published models or model maturity tests for IRM.
For risk identification and KRIs (note to readers:  I’m assuming Oliver means Key Risk Indicator – a useful but loaded phrase itself), an internal control framework which is based on cobit allows an adequate and comprehensive net of indicators for risk assessment based on operational performance.
You’re assertion is that COBIT’ is proven to be an “adequate” and “comprehensive” internal control framework.  Can you show me evidence of this?  What documentation for this has ISACA released?  How was it proven?  Where’s the study?  How did they seek to falsify COBIT’s adequacy and comprehension?  How was comprehensive measured?  At what point was it shown that more COBIT effort decidedly into the realm of diminishing returns?
If you think that “some things can’t be measured” will prove your thesis, you don’t know Risk Management at all.
I never said that, and due to the fact that I’ve taught courses based on Hubbard’s “How To Measure Anything” to risk analysts, I’m going to offer that you don’t know me well enough to come to any conclusion about my knowledge around Information Risk Management.
What I’m saying is that ISACA, COBIT, and RiskIT aren’t mature enough to certify practitioners in a meaningful manner – where “maturity” is an ability to consistently, repeatably, and accurately show a change in risk using ISACA’s own documentation.  If you can’t show me how COBIT measurably (again, where the concept of measurement requires known accuracy and repeatability – just drilling the point home, here) modifies exposure to risk or capability to manage risk in these ways, I don’t think ISACA is ready to say that we, as an industry, are more than isolated alchemists trying to find our own, individual ways to turn lead into gold.  To carry the analogy, the attestation that CRISC would provide has nothing to do with knowledge of chemistry, but everything to do with the alchemists ability to repeat a known means of trying to turn lead into gold.
There is no mathematical voodoo to model a risk exposure which is 100% correct.
We’re in agreement about modeling risk exposure.  To paraphrase Jaynes (poorly), probabilistic models are hypothesis and therefore we should expect (hope!) for them to be frequently falsified.  In addition – just to complete the picture for you, Oliver, I’m also on record as stating that arriving at a state of knowledge for capability to manage risk is similarly difficult  (and this is the whole crux of the COBIT/RISKIT/CRISC request for proof – understanding capability in a measurable way is a key dependency to understanding exposure, and therefore, ISACA is silly for trying to certify that someone can discuss exposure if they can’t even show me how COBIT reduces risk) .
You have to keep the purpose in mind and also use professional judgment based on your experience (which CRISC by the way tries to attestate)
Fascinating, so CRISC tries to provide clear evidence that an individuals experience and professional judgment is of some quality?  My whole point in this series is that any individual with experience in information risk management should know enough to know that a certification around Information Risk Analysis and management is goofy.  As for documenting an individual’s professional judgment skills, I’d love to see how the test does that in a rational manner.
You fight against an attestation which takes into full consideration your own challenge.
Nope.  Not even close.  You have no CLUE what I stand for.  I’m all for good attestation.  As I said the other day:
(…I’d argue that IRM shouldn’t be part of an MIS course load, rather it’s own tract with heavier influences from probability theory, history of science, complexity theory, economics, and epidemiology than, say, Engineering, Computer Science or MIS.)
My position is that given the difficult nature of risk analysis (as I’m saying above), there’s no way CRISC can attest to any competency around Information Risk Analysis, and if ISACA can’t show me how COBIT changes exposure or capability in a measurably way, then CRISC can’t possibly even attest to competency around Information Risk Management.  Maybe it can serve as a RiskIT test, sure and I’m fine with that.  From the same blog post as my quote above:
IRM is not (just one) “process”. Now obviously certain risk management standards (document a simple) process. In my opinion, most risk management standards are nothing BUT a re-iteration of a Plan/Do/Check/Act process. And just to be clear, I have no problems if you want to go get certified in FAIR or OCTAVE or Blahdity-Blah – I’m all for that.  That shows that you’ve studied a document and can regurgitate the contents of that document, presumably on demand, and within the specific subjective perspective of those who taught you.
And similarly if ISACA wants to “certify” that someone can take their RiskIT document and be a domain expert at it, groovy.  Just don’t call that person “Certified in Risk and Information Systems Control™” because they’re not.  They’re “Certified in our expanded P/D/C/A cycle that is yet another myopic way to list a bajillion risk scenarios in a manner you can’t possibly address before the Sun exhausts it’s supply of helium.” “TM”
I’ll state it again, if they want to change the certification’s title and meaning to simply state that an individual can do the above for RiskIT – have a day, good on you. Just don’t expect me to believe that this certification means that the individual knows anything about information risk analysis, or risk analysis in general.

CRISC -O

Published by alex on June 24, 2010 in careers. 3 Comments Tags: , , , .
PREFACE:  You might interpret this blog post as being negative about risk management here, dear readers.  Don’t. This isn’t a diatrabe against IRM, only why “certification” around information risk is a really, really silly idea.
Apparently, my blog about why I don’t like the idea of CRISC has long-term stickiness.  Just today, Philip writes in the comments:
Lets be PROACTIVE instead of critical. I would love to hear about what CAN be a better job practice and skill set that is needed. I am working on both the commercial and Department of Defense and develop programs for training and coaching the skills from MBA to IT Audit and all of technical security for our Certification of Information Assurance Workforce and conduct all the CISM/CISA training and review courses for ISACA in both commercial and military environments. I have worked on Risk Management for years at ERM as well as IT Security/Risk, and A common theme in all of this is RISK MANAGEMENT. When I discuss the Value of IT with MBA students or discuss CMMI with MIS students or development houses, or discuss why ITIL/Cobit or other discuss with business managers what will keep them from reaching their goals and objectives, it is ALL risk management put into a different taxonomy that that particular audience can understand.
I have not been impressed with the current Risk Management certifications that are available. I did participate in the job task analysis of ISACA (which is a VERY positive thing about how ISACA keeps their certifications) more aligned to practice. It is also not perfect, but I think it is a start. If we contribute instead of just complain, it can get better, or we can create something better. What can be better?
So Alex I welcome a personal dialog with you or others on what and how we can do it better. I can host a web conference and invite all who want to participate (upto 100 attendee capacity).
I’ll take you up on that offer, Philip.  Unfortunately, it’s going to be a very short Webex, because the answer is simple, “you can’t do risk certification better because you shouldn’t be doing it in the first place.”
That was kind of the point of my blog posts.
Just to be clear:
In IT I’m sort of seeing 2 types of certifications:
  1. Process based certifications (I can admin a checkpoint firewall, or active directory or what not)
  2. Domain knowledge based certifications (CISA, CISM)
The problems with a risk management certification are legion.  But to highlight a few in the context of Certifying individuals:
A).  Information Risk Management is not an “applied” practice of two domains.  CISM, CISA, and similar certs are mainly, you know how to X – now apply it to InfoSec.  IRM, done with more than a casual hand wave towards following a process because you have to, is much more complex than these, requiring more than just mashing up, say, “management” and “security”, or “auditing” and “security”.
(In fact, I’d argue that IRM shouldn’t be part of an MIS course load, rather it’s own tract with heavier influences from probability theory, history of science, complexity theory, economics, and epidemiology than, say, Engineering, Computer Science or MIS.)
B).  IRM is not a “process”. Now obviously certain risk management standards are a process. In my opinion, most risk management standards are nothing BUT a re-iteration of a Plan/Do/Check/Act process. And just to be clear, I have no problems if you want to go get certified in FAIR or OCTAVE or Blahdity-Blah – I’m all for that.  That shows that you’ve studied a document and can regurgitate the contents of that document, presumably on demand, and within the specific subjective perspective of those who taught you.
And similarly if ISACA wants to “certify” that someone can take their RiskIT document and be a domain expert at it, groovy.  Just don’t call that person “Certified in Risk and Information Systems Control™” because they’re not.  They’re “Certified in our expanded P/D/C/A cycle that is yet another myopic way to list a bajillion risk scenarios in a manner you can’t possibly address before the Sun exhausts it’s supply of helium.” “TM”
RE-ITERATING THE POINT
Look, as my challenge to quantify the impact of risk reduction of a COBIT program suggests, IRM is more than these standards.
And I gotta be clear here, you’ve hit a pet peeve of mine, the whole “Let’s be PROACTIVE” thing.  First, criticism and dis-proof is part of the natural evolution of ideas.  To act like it isn’t is kinda bogus.  And like I said above, you’re assuming that there is something we should be doing about individual certification instead of CRISC – but THERE ISN’T ANY ALTERNATE, AND THERE SHOULD’NT BE.  You’re saying, “let’s verify people can ride their Unicorns properly into Chernobyl” and assuming I’m saying, you know, “maybe we shouldn’t ride Unicorns”.  I’m not.  I’m saying “we shouldn’t go to Chernobyl regardless of the means of transportation”.
And in terms of what we CAN do, well in my eyes – that’s SOIRA.  Now don’t get me wrong, as best as I understood Jay’s vision, it’s not a specific destination, it’s just a destination that isn’t Chernobyl.  I don’t know where it is going yet Phil, but I’m optimistic that Kevin, Jay, John, and Chris are pretty capable of figuring it out, and doing so because of passion, not because they want to sell more memberships, course materials, or certifications.  Either way, I’m just along for the ride, interested in driving when others get tired and playing a few mix tapes along the way.

Why I’m Skeptical of “Due Diligence” Based Security

Published by alex on March 17, 2010 in Doing it Differently, Science of Risk Management and argument. 10 Comments Tags: , , , .

Some time back, a friend of mine said “Alex, I like the concept of Risk Management, but it’s a little like the United Nations – Good in concept, horrible in execution”.

Recently, a couple of folks have been talking about how security should just be a “diligence” function, that is, we should just prove that we’re doing best efforts and that should be enough.  Now conceptually, I love the idea that we can prove our “compliance” or diligence and get a get out of jail free card when an incident happens.  I always think it’s lame when good CISO’s get canned because they got “unlucky”.

Unfortunately, if risk management is infeasible, I’ve been thinking that the concept of Due Diligence Security is complete fantasy.  To carry the analogy, if Risk Management is the United Nations, then Due Diligence Security is the Justice League of Superfriends.  With He-Man.  And the animated Beatles from Yellow Submarine.  That live in the forrest with the Keebler elves and the Ewoks and where the glowing ghosts of Anakin, Obi-Wan and Yoda perform the “Chub-Chub” song with the glowing ghosts of John Lennon and George Harrison. That sort of fantasy.

DUE DILIGENCE BASED SECURITY IS AN ARGUMENT FROM IGNORANCE

Here’s the rub – lets say an incident happens.  Due Diligence only matters when there’s a court case, really.  And in most western courts of law these days, there’s still this concept of innocent until proven guilty.  This concept is known as the argument from ignorance in logic and it is known as a logical fallacy.

Now arguments from ignorance are known as logical fallacies thanks to the epistemological notion of falsification.  Paraphrasing Hume paraphrasing John Stuart Mill – we cannot prove “all swans are white” simply because we’ve observed all white swans -  BUT the observation of a single black swan is enough to prove that “not all swans are white”.   This matters in a court of law, as your ability to prove Due Diligence as a defendant will be a function your ability to prove all swans white – all systems compliant.  But the prosecution only has to show a single black swan to prove that you are NOT diligent.

Sir Karl Popper says, “Good luck with that, Mr. CISO”.

IT’S A TRAP!!!

The result is this – the CISO, in my humble opinion, will be in a worse condition because we have a really poor ability to control the expansion of sensitive information throughout the complex systems (network, system, people, organization) for which they are responsible.  Let me put it this way:  If information (and specifically, sensitive information) operates like a gas, automatically expanding to where it’s not controlled – then how can we possibly hope that the CISO can control the “escape” or leakage of information 100% of the time with no exceptions?  And a solitary exception in a forensic investigation becomes our black swan.

And therefore…   When it comes to proving Due Diligence in the court of law  – Security *screws* the CISO.  Big Time.

For Blog/Twitter Conversation: Can You Defend “GRC”?

Published by alex on December 15, 2009 in Doing it Differently, Science of Risk Management and argument. 15 Comments Tags: , , , , , , , .

Longtime readers know that I’m not the biggest fan of GRC as it is “practiced” today.  I believe G & C are subservient to risk management. So let me offer you this statement to chew on:

“A metric for Governance is only useful inasmuch as it describes an ability to manage risk”

True or False, why, and what are the implications if true or false.

Please discuss.

#newschoolsecurity

Can quantitative risk estimation serve as a guide for every-day policy decisions?

Published by Russell on December 5, 2009 in Science of Risk Management. 7 Comments Tags: , , , .

[Update: The main purpose of this post is to present and demonstrate a method of risk estimation and quantification to support practical policy decision.  The email password policy is just a simplistic case to facilitate the debate.  I also modified the blog post title and the text below to make it clear that this method is aimed to support quantitative risk estimation.]

Our favorite colliquist, Anton Chuvakin, posted a provocative challenge in his blog post “Is Risk Just Too Risky?” :

What is the risk-driven, correct frequency of changing my email password?

<crickets…. silence… more silence>

Yes, we all can quote that “PCI DSS says 90 days” or “whatever regulation says 30 days”, but what does risk say? What actuarial information we need – if we are to define risk through probability of loss? What info about my email usage? Value of information stored there? Frequency of attacks on other similar email accounts? Chances of attack success? My approach to protecting the password? My personal password reuse “policy?” Anything else? On a related note, maybe this is simpler: what is my risk [of having the account compromised] if I change the password every 30 days, 90 days, 300 days?

So, any idea how to go about it?

This little experiment might well show us that “risk-based security” is an awesome thing – but not one achievable in this world today… [emphasis in original]

I wanted to blog about this, but hadn’t collected enough specifics.  Now I can, thanks to the blog conversation by David Mortman, Rich Mogull,  Chris Popper, and “Steve”, we have some smart/experienced people providing the needed detail.

Below, I offer a method for reasoning in order to estimate relative risk of alternatives that is compatible with quantitative risk analysis management, but doesn’t require massive amounts of risk calculations.  I use the conversation by Mortman, et. al. as an example of this method in action (armchair-style).

Continue reading ‘Can quantitative risk estimation serve as a guide for every-day policy decisions?’

For Those Not In The US (or even if you are)

Published by alex on November 26, 2009 in Data Analysis, Science of Risk Management and metrics. 0 Comments Tags: , .

I’d like to wish US readers a happy Thanksgiving. For those outside of the US, I thought this would be a nice little post for today: A pointer to an article in the Financial Times,

“Baseball’s love of statistics is taking over football“

Those who indulge my passion for analysis and for sport know that I love baseball and love how the “Moneyball” approach challenged decades of dogma in the national pastime with scientific analysis.  Today’s financial times discusses how Chelsea (“The Blues” – UK football team) collaborates with the Boston Red Sox (the most superficial bandwagon team ever in baseball) on decision making and analytics.

Go Blues

Best lines:

“Mike Forde, Chelsea’s performance director, visits the US often. “The first time I went to the Red Sox,” he says of the Boston baseball team, “I sat there for eight hours, in a room with no windows, only flipcharts. I walked out of there saying, ‘Wow, that is one of the most insightful conversations on sport I have ever had.’ It was not: ‘What are you doing here? You do not know anything about our sport.’ That was totally irrelevant. It was: ‘How do you make decisions on players? What information do you use? How do we approach the same problems?’”

and:

“Forde sees his task as “risk management”.

Huh.

Rich Mogull’s Divine Assumptions

Published by alex on November 13, 2009 in Science of Risk Management. 0 Comments Tags: , , , .

Our friend Rich Mogull has an interesting post up on his blog called “Always Assume“.  In it, he offers that “assumption” is part of a normal scenario building process, something that is fairly inescapable when making business decisions.  And he offers a simple, pragmatic process for assumptions which is mainly scenario development, justification, and action.    Rich’s process looks like this:

  1. Assumption
  2. Reasoning: The basis for the assumption.
  3. Indicators: Specific cues that indicate whether the assumption is accurate or if there’s a problem in that area.
  4. Controls: The security/recovery/safety controls to mitigate the issue.

Nothing earth shattering here.  And like much of Rich’s work, there is an elegance, almost a minimalism to what he offers.

JUST BECAUSE I CAN’T LEAVE WELL ENOUGH ALONE….

What immediately struck me was how similar Rich’s assumption was to a little something I like to call “scientific method”.  In scientific method, we essentially have (the following shamelessly pasted from Wikipedia):

So if we were to add to Rich’s assumption process above, we’d simply add the “experiments” bits up there.  If we’re building controls in like Rich’s examples in his blog post, we might try a “test” that “penetrates” those controls (or, as I believe Richard Bejtlich smartly tries to get us to say, perform “Adversary Simulation”).

Also, though it will probably sour his stomach a bit, we’d also probably want to make Rich’s assumption steps a hamster-wheel-of-pain(TM) by suggesting that since every so often, the threat landscape will change which will challenge our assumptions/conclusions/hypothesis and so re-testing is necessary.

IF I HAD ANY INDICATION…

Rich does have a certain “informality” around his evidence “indication” step that I’d like to build upon.  Let me offer that when discussing probability of failure in a complex IT system, there are only four basic categories of information indicators we need to consider in Information Assurance/Security/Risk Management/Protection/Whatever.  There might be evidences around:

  • Assets (the things we want to protect and their state)
  • Threats (the things that want to harm our assets and their state)
  • Controls (the things that resist the threats and their state)
  • Impacts (the things that will happen if we are unable to resist the threat)

And if you’re going to look for clues to suggest whether there might be a problem, look no further than these basic categories for evidence.  If you’d like, you can build structure around what “state” means for each category and further develop taxonomies and metrics and whatnot.  That’s the fun bits and I’ll let you be creative rather than write too much this morning.

Note that where these categories applied to Assumption may break down is in discussing management capabilities (are we operating well enough and so forth).  Rich’s assumptive process (must.resist.urge.to.make.acronym – RAP) can certainly be used here, I’m just not sure if there wouldn’t be a better taxonomy of indicators.

The Cost of a Near-Miss Data Breach

Published by Russell on October 6, 2009 in Science of Risk Management. 6 Comments Tags: , , .
Jerry escapes death, but is it cost-free? (Image from tomandjerryonline.com)

Jerry escapes death, but is it cost-free?

If one of your security metrics is Data Breach Cost, what is the cost of a near miss incident? This seemingly simple question gets at the heart of security metrics problem.

Consider the gleeful Jerry Mouse in this cartoon. Tom the Cat has just missed in his attempt to swat Jerry and turn him into mouse meat. Is there any cost to Jerry for this near miss? Is Jerry’s cost any different than if he was running with Tom no where in sight?

By “near miss” I mean a security incident or sequence of incidents that could have resulted in a severe data breach (think TJX or Heartland), but somehow didn’t succeed. Let’s call the specific near-miss event “NM” for short. For sake of argument, let’s assume that the lack of attack success was due to dumb luck or attacker mistakes, not due to brilliant defenses or detection. Let’s say that you only discover NM long after the events took place. For simplicity let’s assume that discovering NM doesn’t result in any extraordinary costs, meaning that out-of-pocket costs are the same just before and immediately after NM. Finally, assume that your expected cost of a successful large-scale data breach is on the order of tens of millions, with the worst case being hundreds of millions of dollars.

How much does NM cost?  The realist answer is “zero”.  (Most engineers are realists, by disposition and training.)  There is a saying in street basketball that expresses the realist philosophy about losses and associated costs: “No blood, no foul”.  If you ask your accountants to pour over the spending and budget reports, they will probably agree. Case closed, right?

Not so fast….

Continue reading ‘The Cost of a Near-Miss Data Breach’

Is risk management too complicated and subtle for InfoSec?

Published by Russell on September 13, 2009 in Science of Risk Management and Uncategorized. 1 Comment Tags: , , .

In his blog, Luther Martin has been advising caution about the use of risk assessment and risk management methods for information security. In many posts, he’s out-right skeptical, and seems to be advising against it.  Here’s the most recent example, from the post “The two-envelope problem in risk management” [emphasis added]:

Does it make sense to never change your information security strategy? That’s a possible consequence of the so-called two-envelope paradox. This is a problem in probability theory that has confused students of probability theory for over 50 years.

[explanation of the problem and how it applies to InfoSec investments]

The bottom line is probably that probability is a complicated and subtle concept, which means that risk management, which relies on it, also is.

Luther and I agree on his bottom-line statement.  This stuff can be complicated and subtle.  It is easy for college-educated professionals and executives to make mistakes that lead to erroneous conclusions.  In fact, Luther’s own blog post is a case study in how easy it is to make mistakes.  

Luther uses the “Two Envelopes Problem” in probability theory as an example where Bayesian (subjectivist) probability methods seem to break down in paradox.  Here’s the problem and paradox, leaving out the math for now  (details at the end):

The player is given two indistinguishable envelopes, each of which contains a positive sum of money. One envelope contains twice as much as the other. The player may select one envelope and keep whatever amount it contains, but upon selection, is offered the possibility to take the other envelope instead ["switch"]. 

[...Bayesian analysis, leading to the decision to "switch", and then "switch" again, ad infinitum...]

  • As it seems more rational to open just any envelope than to swap indefinitely, the player is left with a paradox.

    • Following Wikipedia, Luther points out that ”There’s a problem with this argument, of course, but it’s fairly subtle. Even specialists in probability theory don’t agree what the problem actually is,…”.  Luther then applies Two Envelopes Problem structure to an analogous problem in information security (I’ll call it “Two Technologies”):

    Now let’s suppose that we can’t find a flaw in the above argument and we apply it to our information security strategy. Let’s suppose that we have some initial set of technology, policies and procedures that end up giving us some exposure to risk that we’ll denote R, and if we change to a different set of technology, policies and procedures, we might either increase the risk to 2R or decrease it to R/2. If we apply the same reasoning that we applied above, we find that it never pays to change, because the alternative always has a greater than the risk than what we have now. This clearly doesn’t make sense, but it’s what you might get if you do a risk analysis that isn’t as careful as it could be.

    This seems OK on the surface, but Luther has made a mistake in framing the Two Technologies problem.  Simply put, his Two Technologies problem is not structured the same as the Two Envelopes problem.  Therefore Bayesian formulas for the Two Envelopes does not fit his Two Technologies problem.  Even if it did, a proper framing of the Bayesian analysis avoids the paradox. (For math details, see below).

    So, yes, probability and risk management can be complicated and subtle.  But Luther’s use of the Two Envelopes Problem and his attempt to construct the same problem in InfoSec only supports this conclusion through the “covert channel” of his own mistakes.   

    Is Luther implying that we shouldn’t even try risk assessment and management because it’s beyond our ken?  “Yes” seems to be the answer, if I understand his posts in the “risk” category.   What alternative to risk management is Luther proposing?  In another post, he suggest that we should just use trial-and-error to muddle through:

    In the absence of reliable risk information, a similar approach to information security may be the best that we can do – just try different things and see which works the best. You might call this approach “experimental security.” There may be no better approach.

    If someone can prove that risk assessment/management for InfoSec is impossible in principle, then Luther would be right.  But I know of no such proof of impossibility.  Just because we currently find risk management to be “complicated and subtle” doesn’t mean we should dump it.  We do need tools and frameworks that are up to the job and sufficiently usable to help ordinary people avoid the paradoxes and ratholes in the analysis.   As our academic colleagues would say: “further research is needed”.

    (For mathematical details, read on…)

    Continue reading ‘Is risk management too complicated and subtle for InfoSec?’