In “New rules for big data,” the Economist seems to advocate for more disclosure of security problems:
The benefits of information security—protecting computer systems and networks—are inherently invisible: if threats have been averted, things work as normal. That means it often gets neglected. One way to deal with that is to disclose more information. A pioneering law in California in 2003 required companies to notify people if a security breach had compromised their personal information, which pushed companies to invest more in prevention. The model has been adopted in other states and could be used more widely.
In addition, regulators could require large companies to undergo an annual information-security audit by an accredited third party, similar to financial audits for listed companies. Information about vulnerabilities would be kept confidential, but it could be used by firms to improve their practices and handed to regulators if problems arose. It could even be a requirement for insurance coverage, allowing a market for information security to emerge.
In December, Andy Jaquith and I had a fun conversation about info security with Bill Brenner listening in. The transcript is at “Meeting of the Minds,” and the audio is here.
The New School approach to information security promotes the idea that we can make better security decisions if we can measure the effectiveness of alternatives. Critics argue that so much of information security is unmeasurable, especially factors that shape risk, that quantitative approaches are futile. In my opinion, that is just a critique of our current methods and instruments, not any proof of ultimate feasability. What we need is major innovations in metrics, instrumentation, and such.
We can take inspiration from other fields. Consider this innovation in statistical value management in baseball, a.k.a. the ”Moneyball” approach:
Evaluating fielding is baseball’s hardest math. There are just too many unknowns in a play. How much ground did Jeter cover? How fast was the ball moving? In essence: How unlikely was it that he’d catch the ball? [...]
Sportvision’s FieldFX camera system records the action while object-recognition software identifies each fielder and runner, as well as the ball. After a play, the system spits out data for every movement: the trajectory of the ball, how far the fielder ran, and so on. “After an amazing catch by an outfielder, we can compare his speed and route to the ball with our database and show the TV audience that this player performed so well that 80 percent of the league couldn’t have made that catch,” says Ryan Zander, Sportvision’s manager of baseball products. That information, he says, will allow a much more quantitative measure of exactly what is an error.
I’m at The Open Group Security Forum this week in Seattle, speaking about risk and stuff. Adam gave a great talk about Security: From Art to Science. One recurring theme all week was the need to borrow from disciplines outside of Comp Sci and Engineering. When we think about the data owner and their decisions regarding “guns vs. butter” – I’d be willing to bet that utility theory and decision theory have plenty of wonderful bits of experience and knowledge we should be familiar with.
You may have some offenses which are no-appeal firing offenses. If you do, those need to be told to employees at the moment of hiring and then the rules need to be enforced.
So I’m curious. What are such offenses in an IT environment? Does anyone out there have a clear written list? I’m not looking for “violations of this policy may lead to consequences up to and including termination.” I’m looking for a list of things that will get you fired, like suggesting your secretary’s job is dependent on sex. I want clear measurable statements like “IT staff will be canned if they don’t change the default password within 7 days of deployment of any IT system or device.”
Absent such up front guidance, we can’t go making statements like that and expect to have any credibility.
Generally, mission capability runs 20% higher than availability, but availability is hidden on new stuff, while shouted about on older stuff, because there would be severe embarrassment if you considered that 40% of the brand new V-22 were not available (okay 60% available sounds much better, buy a car which is broke 40% of the time, how good does the warranty service need to be?).
The Navy and GAO are not sure which metrics to use. One of the reasons that US quality fell in the 70’s was avoiding measuring the hard things [that] gets you in trouble; a weakness of the DoD acquisition process. But the spending is more important than meaningful results.
Missing mission capable suggests that basic reliability and maintenance performance are not part of V-22 repertoire. Quality may not have been affordable during the long development cycle, and the savings are now costing in added support and lost use of the V-22
And as one commenter notes, the problem is even more fundamental than poor quality–the Osprey “cannot do a lot of what it is replacing: HH 53 and HH 46.” I would pretty much guarantee that no one is measuring the number of missions that are not performed by the Osprey but which could have been by the helicopters it replaced.
Metrics are powerful tools, but they can be as much a force for evil as a force for good. Choosing the easy-to-gather metrics or the metrics that make the thing being measured look better may play well in Slide-Deck-Land, but it doesn’t change the fact that there is still a reality lurking underneath there which isn’t going away just because someone refuses to measure it.
What people choose to measure can tell you a lot about both their competence and their motivations. Ignore it at your peril.
So last night the family and I sat down and watched a little TV together for the first time in ages. We happened to settle on the X-Games on ESPN, purely because they were showing a sport that I can only describe as Artistic Snowmobile Jumping. Basically, these guys get on snowmobiles, jump them in the air flip around and stuff, and then a panel of judges score their efforts. I suppose the criteria is like ice skating or gymnastics where they score creativity and technique and so forth… If you haven’t seen this sport, here’s a little youtube video of what it’s like:
So we’re watching this sport on ESPN, and after a while I’m noticing a couple of things about the scores. First, they’re using a 100 point scale, and all the scores are coming in between 85 and 92. Fine, I suppose they’re summing up a number of elements.
Then this one rider scores an 88.3. Point Three. Seriously, what judge decides to go decimal? You know, a 100 point scale isn’t good enough, I really need the precision of that tenth of a point to determine if the member of “Team Slednecks” is that much better than the “Red Bull Rockstars” or whatever.
Their judgment was based on wishful thinking rather than on sound calculation of probabilities; for the usual thing among men, is when they want something, they will, without any reflection, leave that to hope; which they will employ the full force of reasoning in rejecting what they find unpalatable.
What You’ve Said