What my critique is trying to do is hold all of us to a higher standard. This report was presented as a large-scale study based on both attack and vulnerability data. Nearly all of the blogs and online articles that echoed this report mentioned it’s quantitative basis.

The New School manifesto is that it’s not good enough anymore to rely on informed opinion and generally-held beliefs about top security risks, etc. Instead, what we need are robust data analysis, field studies, experiments, and simulations. We need to test hypotheses in a way they can be confirmed or discredited . We need to know more about the limits of our knowledge, unknown-unknowns, and what we need to learn next to improve our understanding. Most of all, we need critical tests that challenge our conventional wisdom.

At the very least, any study should make clear which conclusions are based on solid data analysis and which are based on informed opinion, collective wisdom, or hand-waving.

Yes, some of my critique points were relatively small or cosmetic. But even the cosmetic flaws can get in the way of people making good use of this study, and increase the chances that they will get confused or misinterpret the results.

Lastly, the authors of this report should have been writing for another audience: other organizations who might want to replicate the study internally. Just think how much more compelling the results would be if 5, 10, or 20 large organizations did the same study internally and then published the results in a comparable manner?

- Many of the vulnerabilities these days are in applications such as Adobe Acrobat Reader and Flash Player.
- There have been lots of zero day vulnerabilities reported this year. I can think of half a dozen in the last few months alone.
- Web application attacks are on the rise and are one of the primary threats on the Internet today.
- Patch management is a time consuming and challenging task that many companies do poorly. This in turn can lead to compromised systems due to the number of malicious web sites and malicious spam email.

We can argue about whether 60% of attacks are against web applications or if 80% of vulnerabilities are SQL Injection or XSS flaws. But it seems to me that the broader picture being painted here is in line with what I see every day in my work, research, and reading.

They’ve quoted Gartner there. What’s concerning is that Gartner are known for 80/20 results, so this result is suspicious…

What we need are checks and balances and transparency. If readers know where the data comes from, has some basis for confidence that the data was collected and recorded reliably, and that the data analysis methods are well-suited to the goals, then we can have some confidence in the outcomes, especially if the study is reproducable or if it faces competing studies.

If for-profit organizations know that they will be facing this sort of scrutiny, they will do better studies. Maybe not perfect or ideal, but at least fairly good.

If, instead, we take a “leave it to the government” approach, we will be depriving ourselves of the resources, creativity, and initiative of the private sector. I’d rather see positive economic incentives for private sector organization to do more and more of this type of study, especially to collaborate with others. IMHO that is the best way to get the richest set of information and insights into a very complex, fast-changing world.

This phenomenon is precisely why we have government-sponsored agencies such as Centers for Disease Control and Prevention, and the World Health Organization, doing surveillance for disease prevalence. Imagine if our best data about the H1N1 pandemic came from Roche, based on a geographic breakdown of quarterly sales of Tamiflu. It would be intolerable for public health, and it should be intolerable for public cyberhealth.

The U.S. government already has collection points at all the right locations to provide authoritative measurements, but they’re secret anti-terrorism projects, and inter-agency rivalries (or is it intra-agency now that they’re all within DHS?) prevent them from giving anyone the data needed to “connect the dots” into a credible picture.

I did see a couple of online articles that did add some information, but only in the form of interviews of people involved in the study or the bosses at SANS.

This pattern of uncritical acceptance of these studies makes me skeptical when ever I hear the phrase “best practices” — as if the practices have been carefully evaluated against alternatives using some objective performance standards. Instead, I wonder how many “best practices” are just this: “I heard it from a guy… who heard it from a guy… who heard it from another guy…”

I can see parallels in lead-up to the Iraq War, esp. the “Iraq Dossier” that was “sexed up”, and the rest. When you have self-interested parties producing reports of “fact” that promote their interests and agenda, we should all be very skeptical.

They are useless for exactly the reason you have mentioned: because there is no sound methodology for discovering, characterizing, and reporting “trends” over any long (>3 days, say) time period.

What these reports will tell you that in the past 12 months the community as a whole saw a shift to…the stuff you have been dealing with over the past 12 months. What I need to know from a trending perspective is what response trends appeared at different organizations. Did they spend more or less? Did they hire more people with more security certifications? Did incidents cost more?

None of this stuff is actionable.