The Blog Inspired By The Book
Hi,
I’m very interested right now in finding the quality of risk analysis as it relates to operational security. If you’re a risk analyst, a security executive, or operational security analyst, would you mind taking a one question survey? It’s on SurveyMonkey, here: http://www.surveymonkey.com/s/GCSXZ2Q”
adapted from the t-shirt seen in the anton corbijn work here.
With all apologies to both
Paul Morely and Katherine Hamnett.
And that’s about all I have to say on the subject.
This GAO Report is a good overall summary of the state of Federal cyber security R&D and why it’s not getting more traction. Their recommendations (p22) aren’t earth-shaking:
“…we are recommending that the Director of the Office of Science and Technology Policy, in conjunction with the national Cybersecurity Coordinator, direct the Subcommittee on Networking and Information Technology Research and Development to exercise its leadership responsibilities…”
We could paraphrase this by quoting Spike Lee’s movie title: “Do the right thing.”
The only problem with this is recommendation is that NITRD’s Cyber Security and Information Assurance Working Group has specifically defined it’s role as facilitator, not a leader (p15). Wishing that they would take the lead won’t make it so.
In comments to my “Why I Don’t Like CRISC” article, Oliver writes:
CobIT allows to segregate what is called IT in analysable parts. Different Risk models apply to those parts. e.g. Information Security, Architecture, Project management. In certain areas the risk models are more mature (Infosec / Project Management) and in certain they are not (software distribution). That is for the risk modelling (sic) part.
For risk identification and KRIs (note to readers: I’m assuming Oliver means Key Risk Indicator – a useful but loaded phrase itself), an internal control framework which is based on cobit allows an adequate and comprehensive net of indicators for risk assessment based on operational performance.
If you think that “some things can’t be measured” will prove your thesis, you don’t know Risk Management at all.
There is no mathematical voodoo to model a risk exposure which is 100% correct.
You have to keep the purpose in mind and also use professional judgment based on your experience (which CRISC by the way tries to attestate)
You fight against an attestation which takes into full consideration your own challenge.
(…I’d argue that IRM shouldn’t be part of an MIS course load, rather it’s own tract with heavier influences from probability theory, history of science, complexity theory, economics, and epidemiology than, say, Engineering, Computer Science or MIS.)
IRM is not (just one) “process”. Now obviously certain risk management standards (document a simple) process. In my opinion, most risk management standards are nothing BUT a re-iteration of a Plan/Do/Check/Act process. And just to be clear, I have no problems if you want to go get certified in FAIR or OCTAVE or Blahdity-Blah – I’m all for that. That shows that you’ve studied a document and can regurgitate the contents of that document, presumably on demand, and within the specific subjective perspective of those who taught you.And similarly if ISACA wants to “certify” that someone can take their RiskIT document and be a domain expert at it, groovy. Just don’t call that person “Certified in Risk and Information Systems Control™” because they’re not. They’re “Certified in our expanded P/D/C/A cycle that is yet another myopic way to list a bajillion risk scenarios in a manner you can’t possibly address before the Sun exhausts it’s supply of helium.” “TM”
When opining on security in “the cloud” we, as an industry, speak very much in terms of real and imagined threat actions. And that’s a good thing: trying to anticipate security issues is a natural, prudent task. In Lori McVittie’s blog article, “Risk is not a Synonym for “Lack of Security”, she brings up an example of one of these imagined cloud security issues, a “jailbreak” of the hypervisor.
And that made me think a bit because essentially, we want to discuss, we want to know, “what will happen not if, but when vulnerabilities in the hypervisor, or cloud api, or some new technology bit of cloud computing actually are discovered and/or begin to be exploited?”
Now in baseball, sabermetricians (those who do big-time baseball analysis) break the game down into three true outcomes – walk, homerun, strikeout. They are called the three true outcomes because they supposedly are the only events that do not involve the defensive team (other than the pitcher and catcher). A player like Mark McGwire, Adam Dunn, or if you’re old-school, Rob Deer are described as three true outcome players because their statistics over-represent those outcomes vs. other probable in-play events (like a double or reach via error or what have you). Rumor has it that Verizon’s Wade “swing for the fences” Baker was a three true outcome player in college.
In Vulnerability research and discovery, I see three true outcomes, as well. Obviously the reality isn’t that an outcome is always one of the three, just as in baseball there are plenty of Ichiro Suzuki or David Eckstein-type players whose play does not lend itself to a propensity towards the three true baseball outcomes. But for the purposes of discussing cloud risk, I see that any new category of vulnerability can be said to be:
1.) Fundamentally unfixable (RFID, for example). If this level of vulnerability is discovered, cloud computing might devolve back into just hosting or SaaS circa 2001 until something is re-engineered.
2.) Addressed but continues to linger due to some fundamental architecture problem that, for reasons of backwards compatibility, cannot be significantly “fixed”, but rather involves patches over and over again on a constant basis.
3.) Addressed and re-engineered so that the probability of similar, future events is dramatically reduced.
Now about a year ago, I said that for the CISO, the move to the cloud was the act of gracefully giving up control. The good news is that in terms of which of the three we see in the future, the CISO really has no ability to affect which outcome will exist. But how cloud security issues are addressed long-term, via technology, information disclosure (actually the entropy of information disclosure), and contract language (how technology and entropy are addressed as part of the SLA) – these are the aspects of control the CISO must seek to impact. Impact that is both contractual and on her part, procedural – expecting one of the three outcomes to eventually happen.
The technology and information disclosure aspects of that contract with the cloud provider, well, they simply must focus on the ability to detect and respond quickly. Chances are your cloud farm won’t be among the first compromised when an eventual exploit occurs. But, especially if the value of what you’ve moved to the cloud has significance to the broader threat community, that doesn’t mean (as Lori says) you simply accept the risk. Developing contingency plans based on the three true outcomes can help assure that the CISO can at least cope with the cloud transition.
Just to pile on a bit….
You ever hear someone say something, and all of the sudden you realize that you’ve been trying to say exactly that, in exactly that manner, but hadn’t been so succinct or elegant at it? That someone much smarter than you had already thought about the subject a whole lot and there’s actually formal studies and definitions and so forth, and you feel dumb because there’s no way you could have actually googled for the subject in that way, but there it is on wikipedia in black and white? Happens to me all the time.
Was reading the following from the NYT this morning on the Dunning-Kruger effect, and had a little bit of synchronicity when I realized that my entire problem with certifying people about risk and controls has to do with exactly this subject. My issue with ISACA and CRISC is that because I know that there’s so much that I don’t know, and indeed – that we, the infosec industry does not know, and so in my mind if we wanted to rationally, ethically “certify” someone about risk and controls as a domain expert, about the only thing we can do is to test that they are aware of their (and our) limitations. To do otherwise seems rather irrational to me.
TAKE ALEX’S “APPARENTLY OK” RISK PROFESSIONAL EXAM:
A dozen years or so ago, I was PM for a firewall product that had to go through certification. The certification involved several different tests, but at the time the key to certification was simply that your firewall would not pass packets when set in default-deny state. And it cost a lot to certify. This upset me to no end, mainly because I’d have rather spent the $50k certification cost on building a new GUI.
Knowing my frustration, one of the engineering team printed out Marcus Ranum’s “Apparently OK” firewall certification and taped to one of our boxes and congratulated me on getting certification.
With that spirit in mind, and with all apologies to Marcus, let me present Alex Hutton’s Apparently OK Risk Professional Certification Exam! Because frankly, the problem isn’t with “using risk management” – no I’m still a very big proponent of that. The problem is that risk analysis is steeped in critical thinking, and not identifying uncertainty is, well, less than professional in my opinion.
THE ALEX HUTTON “APPARENTLY OK” RISK PROFESSIONAL EXAM
Dear Prospective Risk Professional,
Congratulations on deciding to enter the exciting field of Information Risk Management! Your journey will be confusing, frustrating, and if you get off on performing sisyphean tasks, rewarding.
To achieve a state of “APPARENTLY OK” we ask that you take the following exam. The exam is one question, and you have three (3) minutes to answer it.
For all the risk assessment methodologies inventoried by ENISA (http://rm-inv.enisa.europa.eu/rm_ra_methods.html) and for RiskIT, please tell us the how the assessment methodology is fundamentally incapable of delivering the results claimed.
BONUS: Doing so in a total of two sentences.
There are multiple right answers.
Good Luck!
Alex’s posts on Posts on CRISC are, according to Google, is more authoritative than the CRISC site itself:
Not that it matters. CRISC is proving itself irrelevant by failing to make anyone care. By way of comparison, I googled a few other certifications for the audit and security world, then threw in the Certified Public Accountant (CPA) for good measure.
Needless to say, CPA crushed the audit and security certs with ?30,700,000 Google hits. CISM & CISA had 15,400,000 and 15,000,000, respectively. The CISSP showed a not-disrespectable 9,390,000.
Then we got into what I will kindly call the “add-on” certs, even though they are frequently intended to be extensions or specialist certifications. I chose the ISSAP and ISSMP, the post-CISSP Architecture & Security Management certifications from ISC^2. ISSAP had 181,000 hits, ISSMP had only 69,000 hits, making it the only certification I checked that fared worse than CRISC.
Now that the data is out of they way, I can get to the real question.
Does no one care about CRISC because no one cares about yet-another-super-specialized-certification? And/Or does no one care about CRISC because no one cares about risk assessment?
Well, given that googling “Risk Assessment” (in quotes) got me 12,400,000 hits, I’m going to go with yes on the first question and no on the second.
Now, combining Alex’s CRISC-O post with something Nick Selby said in a conversation he and I had a while back, “You can’t manage a risk you don’t understand,” then all a Risk Assessment Certification can even potentially do is imply that the holder knows how to follow a process–which I would argue is the least intellectually challenging and valuable part of any knowledge work activity.
Personally, I care a great deal about Risk Assessment, both as an interesting intellectual problem and also as a tool for solving real-world problems, even if I generally lack the time to do it right. I certainly don’t have time to get certified as a Risk Assessor, nor do I feel the need. Given my opinion that certifications are just a signalling mechanism in the hiring process, that should come as a surprise to no one.
Lets be PROACTIVE instead of critical. I would love to hear about what CAN be a better job practice and skill set that is needed. I am working on both the commercial and Department of Defense and develop programs for training and coaching the skills from MBA to IT Audit and all of technical security for our Certification of Information Assurance Workforce and conduct all the CISM/CISA training and review courses for ISACA in both commercial and military environments. I have worked on Risk Management for years at ERM as well as IT Security/Risk, and A common theme in all of this is RISK MANAGEMENT. When I discuss the Value of IT with MBA students or discuss CMMI with MIS students or development houses, or discuss why ITIL/Cobit or other discuss with business managers what will keep them from reaching their goals and objectives, it is ALL risk management put into a different taxonomy that that particular audience can understand.I have not been impressed with the current Risk Management certifications that are available. I did participate in the job task analysis of ISACA (which is a VERY positive thing about how ISACA keeps their certifications) more aligned to practice. It is also not perfect, but I think it is a start. If we contribute instead of just complain, it can get better, or we can create something better. What can be better?So Alex I welcome a personal dialog with you or others on what and how we can do it better. I can host a web conference and invite all who want to participate (upto 100 attendee capacity).
Lurnene Grenier has a post up on the Google/Microsoft vunlerability disclosure topic. I commented on the SourceFire blog (couldn’t get the reminder from Zdnet about my password, and frankly I’m kind of surprised I already had an account – so I didn’t post there), but thought it was worth discussing my comments here a bit because I think we can see a difference between evidence-based risk management or New School Security and expert opinion. I’m not trying to rip on Lurene here, far from it. But disclosure is such a crazy topic for our industry that I think we should look to back up all the logical assertions we make.
For example, Lurnene says:
“when a vulnerability becomes public it is no longer as useful for serious attackers”
I have to ask, do we have data set to support this claim? What Lurnene is saying makes sense, right? Bad guys like to use special toys for various reasons, not the least of which is our inability to Prevent or Detect those. But to really test this hypothesis, we’d actually need to have a rational scale for describing threat capability and then match a frequency component for particular vulnerability/exploit uses for that population – and then compare that frequency component to a data set that describes known 0day use in data breaches.
Again from the SourceFire Blog:
“The companies with high-value data that are regularly attacked are able to proactively protect themselves.”
My 7th grade science teacher would toss this out on its ear as a hypothesis. And I’m not just picking on Lurnene here, we (the security industry) do this all the time – making statements without enough definition around our loaded terms like “high-value data”, “regularly attacked” and “proactive protection”. As I say in my comments, my experience (small sample size warning) is that there isn’t necessarily always a correlation between “high-value data” (where high-value data is financial, medical, trade secret, or government/defense data) and ability/willingness to create “proactive protection”.
Even more frustrating is when he says “these companies patch within some time frame”. Not faulting Lurnene here, just really lamenting that there isn’t a public data store where I can see “some time frame” and compare against data for “uptick” of attacks.
YOU DOWN WITH APT? (YEAH YOU KNOW ME)
“The loss due to a 20+ company exploit spree such as “Aurora” is significantly greater than the monetary loss due to low-end compromises which can be cleaned with off the shelf anti-virus tools.”
Reviewing the data sets I have at my disposal, I’m seeing:
1.) I don’t have a good estimate for hard (or soft) costs for “Aurora”, though I suppose I would accept a “high” qualitative label.
and
2.) data supporting that breaches of significant value are predominately caused by tools that are not able to be “cleaned with off the shelf anti-virus tools.” Rather, I’m seeing data that supports the notion that the for the significant portion of data breaches, the effort to prevent could have been classified as “simple and cheap” (source: VZ DBIR).
Finally, I’ll add my own editorial point here, just so Lurene can rip me back 
I think I would have difficulty asserting that we should *only* care about ” large corporations, government, and military targets with the goals of industrial espionage and military superiority.” Off the top of my head, I can think of hundreds of millions of records exposed by data breaches that came from organizations we might say are in the “SMB market”.
BOTTOM LINE: Defending the faith might be a lot easier if there were data to support the defense.
Using a dish full of marshmallows. We’re doing this with my oldest kids, and while I was reading up on it, I had to laugh out loud at the following:
…now you have what you need to measure the speed of light. You just need to know a very fundamental equation of physics:
Speed of a Wave (c) = Frequency (f) x Wavelength (L)
The distance between the melted sections of the marshmallow is in fact L/2, because there are two nodes for each wave (see animation). So if you have measured 6cm and your oven operates at 2450 MHz, then your measured speed of light is (0.12 x 2450,000,000) 294,000,000 metres per second.
The agreed value of the speed of light through a vacuum is 299,792,458 metres per second. See how accurately you can measure it? what could you do to make the experiment better, and thus get a closer answer?
IMHO, we need more published security metrics (and risk analytics) that don’t worry about those few million meters per second, and focus rather on the cleverness of using marshmallows and microwaves.
What You’ve Said