These came across the SIRA mailing list. They were so good, I had to share:
http://eight2late.wordpress.com/2009/10/06/on-the-limitations-of-scoring-methods-for-risk-analysis/
Thanks to Kevin Riggins for finding them and pointing them out.
Author Archive for alex
Last we learned, Peter Coffee was Director of Platform Research for salesforce.com. He also blogs on their corporate weblog, CloudBlog, a blog that promises “Insights on the Future of Cloud Computing”.
He has a post up from last week that called “Private Clouds, Flat Earths, and Unicorns” within which he tries to “bust some myths” about Cloud Security. Now most of the time, corporate blogs are almost always light on content, but at least most of the time they are banal, emasculated vagaries that can neither be too stupid nor particularly insightful. Well, not this one.
This blog post is so dangerously filled with complete and utter cowpie that I could not but respond here – it is fundamentally contrary to the concerns James Arlen and I wrote for the IRM chapter of the CSA document.
It is a completely Cloud-tarded article.
WHAT? A SAAS PROVIDER RAILING AGAINST PRIVATE CLOUDS?!?! I CAN’T BELIEVE IT!!!
First, Peter makes the assertion that when IT managers are surveyed and state a strong preference for “private clouds” and that this is a desire, not their choice, and subsequently ridicules IT managers for having that desire. His claim is that:
“To have physical possession of the data, you also have to own and maintain the data storage hardware and software.”
This is a dangerous assertion for a cloud provider to make. The semantic game he’s playing here is around the word “possession”. I’ll agree that it is impossible to have physical possession without ownership of the infrastructure. Fine. But this is NOT what security and responsibility (both ethical and legal responsibility) in cloud computing is about. Cloud Risk Management is not about possession, it is about custodianship. If I put credit card numbers in your (his) cloud, I am still, in the eyes of class action lawsuits at least, it’s custodian. Looking at all forms of loss associated with a data breach, there is no pure, 100% transfer of risk. And *this* is the problem IT managers face.
COFFEE’S NAIVE IMPACT MODELING
A desire for security (more below) operations stems solely from the fact that risk is not completely transferred. What do I mean? If my bank puts my banking data in Salesforce.com and there’s a breach, yes – I’m going to be mad at Salesforce. But I’m going to pull my money (and it’s earning potential) out of my bank because some Cloudtard over there decided that this was a good idea.
In other words (use ISO 27005, FAIR, VERIS, whatever) Primary Loss Factors are only half the impact model. And maybe the Cloud Provider owns some of that loss, maybe they don’t. Secondary Factors are still borne solely by those that use the cloud provider.
BEING GOOD AT OPSEC IS A CRITICAL COMPONENT IN RISK MANAGEMENT
“If you have operational control of the infrastructure you have to employ and supervise a “team of experts” who spend most of their time waiting to respond to critical but rare events.”
One has to but wonder how often Peter talks to his own OpSec guys. I imagine (sincerely hope) that the CIRT team at Salesforce absolutely cringed when they read this. Indeed, my sympathies go out to you, Your Supreme Galactic Director Of Platform just called you a mostly useless operating expense, and you may have a little internal selling to do.
That aside, anyone who has heard me speak for the last year knows the stress I put on understanding your OpSec capabilities as a component of managing risk. I won’t rehash those presentations (they’re online – use The Google) but both understanding the Cloud Providers capabilities and your ability to interact with them – that is the crux of managing risk in the cloud.
ILLOGICAL CLOUD POSITIVISM
“Security in cloud services can be constructed, maintained and operated at levels that are far beyond what’s cost-effective for almost any individual company or organization.”
The irony of this quote, of course, is that his linked supporting evidence is SalesForce’s statement of ISO 27001 compliance. ’nuff said.
But in regards to the actual argument, the answer is: “maybe”. If InfoSec is the study of the collision of multiple complex systems, and the cloud provider has a much more complex system to manage than you do on your own, the answer must be “maybe”. To support such a statement we must identify models that are informative about cost to secure per $. I haven’t seen much research around this, and certainly Coffee doesn’t link to any.
COMPLIANCE ISN’T A “ROLES” AUDIT
“The discipline and clarity of service invocations in true cloud environments can greatly aid the control of access, and the auditability of actions, that are dauntingly expensive and complex to achieve in traditional IT settings.”
I honestly have no idea what this means. I take it that he’s saying that proof of access control and audit evidence is easier to gain just by casually tossing an email in the direction of your SaaS buddy. But in terms of regulatory compliance (PCI DSS, HIPAA, OTHER ALPHABET SOUP) my experience is vastly different.
HEY, TECHNOLOGY IS TECHNOLOGY, RIGHT?
“Customization and integration of cloud services are neither intrinsically better nor inherently worse than the capabilities of an on-premise stack.”
This, of course, viewed in the context of the top two points Coffee makes, for the purposes of risk management, are statements with contradictory implications. Let me some up what Peter says to me when I read this article:
“The cloud is easy to use. The cloud is easier to secure. The cloud is friendly to auditors.” and “The cloud is neither better nor worse than securing data on site.”
I’M STILL OK WITH THE CLOUD
With my years of SMB F.I. audit/PenTest experience, I don’t disagree that cloud computing may make sense from a risk management standpoint. I’ve been appalled at times about the one guy in the middle of nowhere Midwest whose in charge of $200 million in credit union/pension fund assets. The other thing to remember is that in fact, in many ways these SMB FI folks have been offloading financial machinations to partners for years – they are already “cloud users” by some informal lose definition. But it’s an awfully cavalier attitude Mr. Coffee takes here with us making these assertions, esp. with no real supporting evidence. Welcome to the NewSchool, Peter Coffee. In God We Trust, everyone else brings data.
Interesting interactive data app from the Wall Street Journal about your privacy online and what various websites track/know about you.
Full disclosure, our site uses Mint for traffic analytics.
Cisco has their security report up – find it here. My favorite part? ”The Artichoke of Attack”
Hey, just so you all know, SOIRA is having our lunch (or breakfast) Al-Desko Webex. This month we have the pleasure of watching Chris Hayes show how to use quantitative risk analysis for real, pragmatic business purposes. It’s going to be seriously useful.
Join SOIRA here: http://groups.google.com/group/InfoRiskSociety?hl=en for the invite.
First, thanks to everyone who took the unscientific, perhaps poorly worded survey. I appreciate you taking time to help out. I especially appreciate the feedback from the person who took the time to write in:
“Learn the proper definition of “Control Systems” as in, Distributed Control Systems or Industrial Control systems. These are the places that need real security, not some bullshit enterprise network.”
You, sir or madam, are chock full of rock and roll. Thanks for cheering me up.
Next, the results were:
Daily = 6
a few times a month = 2
a few times a quarter = 1
less than a few times a quarter = 10
never = 43
and the chart looks something like this:
UPDATE: Jeff Lowder asked me to clarify this a bit. I’ll start by re-iterating that this was a not really a proper survey, but akin to asking a handful of friends (the survey existence was announced here, on twitter, to a couple of security – centric mailing lists). As such, don’t get all bent out of shape about it.
I was interested in the question – “how often does GRC analysis impact actual OpSec?” and decided that a frequency of interaction would be a pretty good bellwether. The question (and results with proper caveats) were part of the presentation Allison Miller and I gave at Black Hat. More on that presentation in a while, btw.
Hi,
I’m very interested right now in finding the quality of risk analysis as it relates to operational security. If you’re a risk analyst, a security executive, or operational security analyst, would you mind taking a one question survey? It’s on SurveyMonkey, here: http://www.surveymonkey.com/s/GCSXZ2Q”
adapted from the t-shirt seen in the anton corbijn work here.
With all apologies to both
Paul Morely and Katherine Hamnett.
And that’s about all I have to say on the subject.
In comments to my “Why I Don’t Like CRISC” article, Oliver writes:
CobIT allows to segregate what is called IT in analysable parts. Different Risk models apply to those parts. e.g. Information Security, Architecture, Project management. In certain areas the risk models are more mature (Infosec / Project Management) and in certain they are not (software distribution). That is for the risk modelling (sic) part.
Oliver: I’m very glad that others in our industry are preaching the concept of model selection & fit. And because you’ve demonstrated that at least you believe this is an important aspect of IRM, I’m ready to believe what you’re saying there. But before I do so, I spent a good deal of time in Missouri, so I need you to show me:
- Define “mature” – what makes a mature information risk model? In fact, show me the industry standards for gauging model maturity, so that I can examine different models, similarly.
- Show me, oh please show me, an information risk model that has even been tested (publicly) for repeatability and accuracy, more or less been shown to provide repeatability and accuracy to a measurable degree of confidence.
Now my thought is that you can’t have a mature risk model without having a measurable notion of repeatability (two analysts with the same data and same model go into separate rooms and come out with reasonably similar results) and accuracy (model outcomes have been tested to be correct some degree of the time). Maybe I’m not subscribing to the right scientific journals out there, but I’ve yet to see the data sets and the published models or model maturity tests for IRM.
For risk identification and KRIs (note to readers: I’m assuming Oliver means Key Risk Indicator – a useful but loaded phrase itself), an internal control framework which is based on cobit allows an adequate and comprehensive net of indicators for risk assessment based on operational performance.
You’re assertion is that COBIT’ is proven to be an “adequate” and “comprehensive” internal control framework. Can you show me evidence of this? What documentation for this has ISACA released? How was it proven? Where’s the study? How did they seek to falsify COBIT’s adequacy and comprehension? How was comprehensive measured? At what point was it shown that more COBIT effort decidedly into the realm of diminishing returns?
If you think that “some things can’t be measured” will prove your thesis, you don’t know Risk Management at all.
I never said that, and due to the fact that I’ve taught courses based on Hubbard’s “How To Measure Anything” to risk analysts, I’m going to offer that you don’t know me well enough to come to any conclusion about my knowledge around Information Risk Management.
What I’m saying is that ISACA, COBIT, and RiskIT aren’t mature enough to certify practitioners in a meaningful manner – where “maturity” is an ability to consistently, repeatably, and accurately show a change in risk using ISACA’s own documentation. If you can’t show me how COBIT measurably (again, where the concept of measurement requires known accuracy and repeatability – just drilling the point home, here) modifies exposure to risk or capability to manage risk in these ways, I don’t think ISACA is ready to say that we, as an industry, are more than isolated alchemists trying to find our own, individual ways to turn lead into gold. To carry the analogy, the attestation that CRISC would provide has nothing to do with knowledge of chemistry, but everything to do with the alchemists ability to repeat a known means of trying to turn lead into gold.
There is no mathematical voodoo to model a risk exposure which is 100% correct.
We’re in agreement about modeling risk exposure. To paraphrase Jaynes (poorly), probabilistic models are hypothesis and therefore we should expect (hope!) for them to be frequently falsified. In addition – just to complete the picture for you, Oliver, I’m also on record as stating that arriving at a state of knowledge for capability to manage risk is similarly difficult (and this is the whole crux of the COBIT/RISKIT/CRISC request for proof – understanding capability in a measurable way is a key dependency to understanding exposure, and therefore, ISACA is silly for trying to certify that someone can discuss exposure if they can’t even show me how COBIT reduces risk) .
You have to keep the purpose in mind and also use professional judgment based on your experience (which CRISC by the way tries to attestate)
Fascinating, so CRISC tries to provide clear evidence that an individuals experience and professional judgment is of some quality? My whole point in this series is that any individual with experience in information risk management should know enough to know that a certification around Information Risk Analysis and management is goofy. As for documenting an individual’s professional judgment skills, I’d love to see how the test does that in a rational manner.
You fight against an attestation which takes into full consideration your own challenge.
Nope. Not even close. You have no CLUE what I stand for. I’m all for good attestation. As I said the other day:
(…I’d argue that IRM shouldn’t be part of an MIS course load, rather it’s own tract with heavier influences from probability theory, history of science, complexity theory, economics, and epidemiology than, say, Engineering, Computer Science or MIS.)
My position is that given the difficult nature of risk analysis (as I’m saying above), there’s no way CRISC can attest to any competency around Information Risk Analysis, and if ISACA can’t show me how COBIT changes exposure or capability in a measurably way, then CRISC can’t possibly even attest to competency around Information Risk Management. Maybe it can serve as a RiskIT test, sure and I’m fine with that. From the same blog post as my quote above:
IRM is not (just one) “process”. Now obviously certain risk management standards (document a simple) process. In my opinion, most risk management standards are nothing BUT a re-iteration of a Plan/Do/Check/Act process. And just to be clear, I have no problems if you want to go get certified in FAIR or OCTAVE or Blahdity-Blah – I’m all for that. That shows that you’ve studied a document and can regurgitate the contents of that document, presumably on demand, and within the specific subjective perspective of those who taught you.And similarly if ISACA wants to “certify” that someone can take their RiskIT document and be a domain expert at it, groovy. Just don’t call that person “Certified in Risk and Information Systems Control™” because they’re not. They’re “Certified in our expanded P/D/C/A cycle that is yet another myopic way to list a bajillion risk scenarios in a manner you can’t possibly address before the Sun exhausts it’s supply of helium.” “TM”
I’ll state it again, if they want to change the certification’s title and meaning to simply state that an individual can do the above for RiskIT – have a day, good on you. Just don’t expect me to believe that this certification means that the individual knows anything about information risk analysis, or risk analysis in general.
When opining on security in “the cloud” we, as an industry, speak very much in terms of real and imagined threat actions. And that’s a good thing: trying to anticipate security issues is a natural, prudent task. In Lori McVittie’s blog article, “Risk is not a Synonym for “Lack of Security”, she brings up an example of one of these imagined cloud security issues, a “jailbreak” of the hypervisor.
And that made me think a bit because essentially, we want to discuss, we want to know, “what will happen not if, but when vulnerabilities in the hypervisor, or cloud api, or some new technology bit of cloud computing actually are discovered and/or begin to be exploited?”
Now in baseball, sabermetricians (those who do big-time baseball analysis) break the game down into three true outcomes – walk, homerun, strikeout. They are called the three true outcomes because they supposedly are the only events that do not involve the defensive team (other than the pitcher and catcher). A player like Mark McGwire, Adam Dunn, or if you’re old-school, Rob Deer are described as three true outcome players because their statistics over-represent those outcomes vs. other probable in-play events (like a double or reach via error or what have you). Rumor has it that Verizon’s Wade “swing for the fences” Baker was a three true outcome player in college.
In Vulnerability research and discovery, I see three true outcomes, as well. Obviously the reality isn’t that an outcome is always one of the three, just as in baseball there are plenty of Ichiro Suzuki or David Eckstein-type players whose play does not lend itself to a propensity towards the three true baseball outcomes. But for the purposes of discussing cloud risk, I see that any new category of vulnerability can be said to be:
1.) Fundamentally unfixable (RFID, for example). If this level of vulnerability is discovered, cloud computing might devolve back into just hosting or SaaS circa 2001 until something is re-engineered.
2.) Addressed but continues to linger due to some fundamental architecture problem that, for reasons of backwards compatibility, cannot be significantly “fixed”, but rather involves patches over and over again on a constant basis.
3.) Addressed and re-engineered so that the probability of similar, future events is dramatically reduced.
Now about a year ago, I said that for the CISO, the move to the cloud was the act of gracefully giving up control. The good news is that in terms of which of the three we see in the future, the CISO really has no ability to affect which outcome will exist. But how cloud security issues are addressed long-term, via technology, information disclosure (actually the entropy of information disclosure), and contract language (how technology and entropy are addressed as part of the SLA) – these are the aspects of control the CISO must seek to impact. Impact that is both contractual and on her part, procedural – expecting one of the three outcomes to eventually happen.
The technology and information disclosure aspects of that contract with the cloud provider, well, they simply must focus on the ability to detect and respond quickly. Chances are your cloud farm won’t be among the first compromised when an eventual exploit occurs. But, especially if the value of what you’ve moved to the cloud has significance to the broader threat community, that doesn’t mean (as Lori says) you simply accept the risk. Developing contingency plans based on the three true outcomes can help assure that the CISO can at least cope with the cloud transition.
What You’ve Said