There’s also a very strong cultural bias in the engineering and hard science communities. To put it bluntly, “solutions” = technical and technological. This makes it hard or impossible to propose research projects that have no technological product, but instead aim to produce a social or economic system or process. To engineers, this feels too squishy and “not real science”.

There’s a big difference between being a “nay-sayer” and a “skeptic”. At worst, the nay-sayer is against any effort to solve the research problems because they believe that such effort is futile or wasteful or counter-productive. IMHO, that approach is not distinguishable from a position based on defeatism and cynicism. Dig deeper, and you’ll find both arrogance and fear.

In contrast, the skeptical view is very compatible with real scientific progress, including revolutionary breakthroughs. The skeptical view simply demands the highest standards of proof, experimental tests, critical thinking, challenging evaluations, etc.

I’m both skeptical and also hopeful regarding this line of research. I go back and forth between being optimistic and pessimistic. But we’ll never know unless we give it a serious try.

I think that the research makes sense. But I also think that the results so far are close to zero. Almost all organization wide risk assessments I’ve seen are not so far from pure fiction and wishful thinking.

Actually, I’m interested on the next step, the decision-making process. Like when you mention a model to estimate aggregate risk. Interesting, but it’s hard to see how we could base decisions on that. Is this aggregate risk too low or too high? Even with some benchmarking in place for that, if we don’t try to forecast it’s hard to say how our actions will impact those numbers. It’s useful to ensure that you are doing the minimal, fighting Today’s main threats and working on the most exploited vulnerabilities, but we still don’t have anything that help us to decide on medium and long term. And that’s only for which controls to implement and class of vulnerabilities to fix, when we include the decision class of which systems/environments/LOBs to address inside the organization, it becomes even harder, specially because those benchmarks will not help at all.

Ok, I confess I’m one of the “nay-sayers” , but that’s not only for the sake of saying that everybody is wrong and the world is going to end because everything now is working over port 80 (I’m already sounding like a security rockstar). I’m curious too see defense strategies and decision-making models that won’t rely on those models. Honestly, except from some quite naive approaches, haven’t seen any so far.

If the “risk management” approach (a.k.a. Moneyball applied to InfoSec) was about forecasts based on statistical analysis of history, I would totally agree with you, not only for the reasons you mention but also others.

It’s not just the Black Swan phenomena. Sadly, Taleb’s popular book did a very poor job of explaining the random process behind Black Swan phenomena. You have to go to his technical papers, and also technical and academic papers of other people. So it’s very common for people to equate “Black Swan” with any process that defies statistical analysis, forecasting, and prediction. To understand how mistaken that is, consider that this category includes four completely different processes:

* Non-stationary random processes
* Higher-order correlation structures
* Self-referential processes (aka scale-free, chaotic)
* Strategic game process (actors who might mimic randomness when they choose)

Back to your main points…

Because information security risk is not driven by the same class of processes as physical and skill variations, and because of non-stationarity, fast evolution of threats and technology, etc., that forces us to use inference and analysis methods that pick up where the traditional methods leave off.

To some people in InfoSec, this is absurd, silly, or hopeless. Many of the nay-sayers regarding risk measurement and management hold this position.

However, to some of us, this is an exciting research challenge. Yes, “research” in the sense of theory and academia and first principles.

This also means that no amount of traditional statistical crunching on piles of data will ever arrive at aggregate risk metrics that satisfy minimum thresholds of reliability and credibility.

So… contrary to Adam’s post, models do matter, especially when you are trying to put all security metrics together to estimate aggregate risk.

Here at Newschool I’ve posted on the new methods and approaches that I think will help get us to the Promised Land. One thing for sure — to get there, we won’t be attempting to forecast. Instead, we will aim to generate meaningful signals for action, especially in the form of incentives.

Take, for instance, the “80 percent of the league couldn’t have made that catch” thing. Thinking on the nice work from Nassim Taleb (the Black Swan guy), people (and so outfielders) physical attributes are usually only slightly different. Checking the past features from league outfielders should not give you enough information to say something like that, specially considering the interval between the games and the constant training for the athletes. It’s too much conclusion based on past data that don’t have a direct causality relation with the event you are trying to predict.

That is also common on security. With the speed of changes and complexity of IT systems, constant changes of user behaviour due to those new systems (social networks?), it is extremely hard to produce a decent forecast of future events based on past data. Why would all the data about the exploitation of OS and web servers vulnerabilities from the past decade be useful to determine exploitation trends of browser vulnerabilities or XSS on social network websites?

We should be a little more skeptical on our ability to forecast events, specially security incidents.