Archive for the 'data' Category

Alex on Science and Risk Management

Published by adam on June 17, 2010 in Data Analysis, argument and data. 1 Comment

Alex Hutton has an excellent post on his work blog:

Jim Tiller of British Telecom has published a blog post called “Risk Appetite, Counting Security Calories Won’t Help”. I’d like to discuss Jim’s blog post because I think it shows a difference in perspectives between our organizations. I’d also like to counter a few of the assertions he makes because I find these to be misunderstandings that are common in our industry.

“Anyone who knows me or has subjected themselves to my writings knows I have some uneasiness with today’s role of risk. It’s not the process, but more of how there is so much focus on risk as if it were a science – but it’s not. Not even close.”

Let me begin my rebuttal by first arguing that risk management, at its basis, is at least ”scientific work”. What I mean by that is elegantly summed up by Eliezer Yudkowsky on the Less Wrong blog. To use Eliezer’s words, I’ll offer that scientific work is “the reporting of the likelihood ratios for any popular hypotheses.”

You should go read “Risk Appetite: Counting Risk Calories is All You Can Do“.

Counterpoint: There is demand for security innovation

Published by adam on March 23, 2010 in argument and data. 1 Comment

Over in the Securosis blog, Rich Mogull wrote a post “There is No Market for Security Innovation.”

Rich is right that there’s currently no market, but that doesn’t mean there’s no demand. I think there are a couple of inhibitors to the market, but the key one is that transaction costs are kept high by a lack of data about outcomes. Every one of the startups selling you a product will claim that it blocks “APT” and “Data loss” but none of them have compelling data about efficacy. None of us have great, broad data about what problems lead to breaches, and none of us have data about what solutions products effectively prevent those problems. None of us have data about how often the products are deployed and managed effectively.

So when the salespeople come in with their “$204 per record” and compliance demands and all the rest, there’s no good way to distinguish between it, and as a result, the market is a slog for both real innovation and snake-oil.

If someone could innovate to address these problems, say by collecting and analyzing data about what really happens inside a company, they might have a business.

More broadly, for a market to function, there needs to be supply which exists in plenty, and demand, which exists, and a way to link them. And there’s the chasm.

I’ll also point out that we discussed innovation a bit on pages 126-127 of The New School, where we opine that much security needs to be integrated into your infrastructure and thus will be purchased from larger vendors.

Data void: False Positives

Published by Russell on March 10, 2010 in Data Analysis, data, measurement and metrics. 2 Comments

There’s a good post at Gartner pointing out the lack of data reported by vendors or customers regarding the false positive rates for anti-spam solutions.  

Although Gartner customers almost never complain about false positive rates, I wonder if false positives are under estimated. End users rarely complain about false positives, but they are very vocal reporting Spam in their inbox. Box Sentry (www.boxsentry.com) recently did a tests in a number of organizations and found the false positive rate in some organizations using popular anti-spam tools was as high as 13% of legitimate emails. The largest proportion of false positives in their study was legitimate person-to-person traffic.  While it could be that these organizations have over-tuned their systems to block more Spam at the expense of quarantining more legit email, the reality was the email administrators had no idea they had such a high false positive rate because they never checked.  Have you? 

Going further, it would be very valuable to estimate the cost of false positives.

As I’ve discussed in a previous post, this is just another instance of a general problem in the security industry.  You can’t do rational analysis of effectiveness, cost-effectiveness, risk, and the rest without some estimate of false positive rates and their costs.

Everybody Should Be Doing Something about InfoSec Research

Published by adam on March 10, 2010 in Doing it Differently, argument and data. 8 Comments

Previously, Russell wrote “Everybody complains about lack of information security research, but nobody does anything about it.”

In that post, he argues for a model where

Ideally, this program should be “idea capitalists”, knowing some people and ideas won’t payoff but others will be huge winners. One thing for sure — we shouldn’t focus this program only on people who have been “officially” annointed by some hierarchy, some certification program, or by credentials alone.

I agree that a focus on those anointed won’t help, but that doesn’t mean it’s easy to set up such an institution.

The trouble with the approach is that we have such institutions (*ARPA, venture capital) and they’ve all failed for institutional reasons. However high their aspirations, such organizations over time get flack from their funders over their failures, their bizarre and newsworthy ideas and the organizations become conservative. They trend towards “proven entrepreneurs” and incrementalism. The “Pioneer Fellows” idea does not overcome this structural issue. (There is an argument that the MacArthur genius grants overcome it. I’m not aware of any research into the relative importance of work done before and after such grants, but I have my suspicions, prejudices and best practices.)

Of course, I might be wrong. If you have a spare million bucks, please set this up, and we can see how it goes. An experiment, if you will.

Experiments are a big part of why Andrew and I focused on free availability of data. With data, those with ideas can test them. There will be a scrum of entrepreneurial types analyzing the data. Fascinating stuff will emerge from that chaos. With evidence, they will go to the extant ‘big return’ organizations and get funding. Or they’ll work for big companies and shift product directions.

That is, the issue in infosec is not a lack of interesting ideas, it’s the trouble in testing them without data. We need data to test ideas and figure out how they impact outcomes.

Krebs on Cyber vs Physical Crooks

Published by adam on March 9, 2010 in breach laws, data and government. 0 Comments

In addition, while traditional bank robbers are limited to the amount of money they can physically carry from the scene of the crime, cyber thieves have a seemingly limitless supply of accomplices to help them haul the loot, by hiring so-called money mules to carry the cash for them.

I can’t help but notice one other important distinction between these two types of bank crimes: The federal government sure publishes a lot more information about physical bank robberies that it makes available about online stick-ups.

Go read “Cyber Crooks Leave Traditional Bank Robbers in the Dust” by Brian Krebs. Then ask why we sweep these crimes under the rug.

Symantec State of Security 2010 Report Out

Published by alex on February 22, 2010 in Data Analysis and data. 0 Comments Tags: , , , .

http://www.symantec.com/content/en/us/about/presskits/SES_report_Feb2010.pdf

Thanks to big yellow for not making us register!  Oh, and Adam thanks you for not using pie charts…

Open Security Foundation Looking for Advisors

Published by adam on February 13, 2010 in data. 0 Comments

Open Security Foundation – Advisory Board – Call for Nominations:

The Open Security Foundation (OSF) is an internationally recognized 501(c)(3) non-profit public organization seeking senior leaders capable of providing broad-based perspective on information security, business management and fundraising to volunteer for an Advisory Board. The Advisory Board will provide insight and guidance when developing future plans, an open forum for reviewing community feedback and a broader view when prioritizing potential new services.

I figure readers of this blog should be interested in helping drive open data sources.

Data Not Assertions

Published by David Mortman on December 10, 2009 in data. 12 Comments

There have already been a ton of posts out there about the Verizon DBIR Supplement that came out yesterday, so I’m not going to dive into the details, but I wanted to highlight this quick discussion from twitter yesterday that really sums of the value of the supplement and similar reports:

georgevhulme: I’m glad we have data to refute the “insiders conduct 80% of all attacks” mantra that has been repeated, ad nauseum for at least a decade

adamshostack: @alexhutton @georgevhulme yeah, but… Data, not assertions

This is so awesome, I can barely stand it. We’re actually starting to be able to make data based decisions as opposed to just asserting something is true because we believe it on faith or like the way it sounds.

“Data, not assertions” really sums up so much of what I was trying to get at in the the discussion on securosis last week about password changing time frames. Read the comments over there. It really shows how far we have yet to go.

“80 Percent of Cyber Attacks Preventable”

Published by David Mortman on November 18, 2009 in data. 4 Comments

Threatlevel (aka 27B/6) reported yesterday that Richard Schaeffer, the NSA’s information assurance director testified to the Senate Senate Judiciary Subcommittee on Terrorism, Technology and Homeland Security on the issue of computer based attacks.

If network administrators simply instituted proper configuration policies and conducted good network monitoring, about 80 percent of commonly known cyber attacks could be prevented, a Senate committee heard Tuesday.

The remark was made by Richard Schaeffer, the NSA’s information assurance director, who added that simply adhering to already known best practices would sufficiently raise the security bar so that attackers would have to take more risks to breach a network, “thereby raising [their] risk of detection.”

I’m really curious however on what data Director Schaeffer is basing his testimony on. Is it the DBIR? Another open set of breach data or is it based on data gathered by the NSA? Regardless, it’s great to see more folks talking about what the Verizon DBIR report told us and what we’ve known anecdotally for a long time; which is, we still aren’t even close to doing the basics well.

The article then goes on to tell us:

A 2009 Price Waterhouse Cooper study on global information security found that 47 percent of companies are reducing or deferring their information security budgets, despite the growing dangers of cyber incursions.

The thing is, as we’ve learned from the Verizon study, most of the found issues were due to failing at doing the basics, like not removing default passwords, not revoking accounts when employees leave and misconfigurations. Even in the case of patching, the vast majority of holes exploited had patches available for over a year and 100% had patches available for over 6 months. This is not the stuff of big budgets and sexy technology, but rather about having solid, repeatable and auditable processes, in other words, serious operational discipline. Budget cuts might actually be a good thing because it will force organizations to focus on the people and process portions of security rather then the technology. It’d be really cool to if PWC were to track correlation of budgets to breaches within their survey groups, then we’d have some actual data on potential optimal spend levels.

Botnet Research

Published by David Mortman on October 6, 2009 in Data Analysis and data. 0 Comments

Rob Lemos has a new article up on the MIT Technology Review, about some researchers from UC Santa Barbara who spent several months studying the Mebroot Botnet. They found some fascinating stuff and I’m looking forward to reading the paper when it’s finally published. While the vast majority of infected machines were Windows based (64% XP, 23% Vista), 6.4% were running either OS X Tiger or Leopard, demonstrating yet again that just because you have a Mac doesn’t mean you are safe. More interesting to me was:

The researchers also discovered that nearly 70 percent of those redirected by Mebroot–as classified by Internet address–were vulnerable to one of almost 40 vulnerabilities regularly used by the most popular infection toolkits designed to compromise computer systems. About half that number were vulnerable to the six specific vulnerabilities used by the Mebroot toolkit.

The research suggests that users need to update more often, says UCSB’s Vigna.

Unfortunately, until the paper comes out we won’t know which vulnerabilities were being used and how old they are. Hopefully, that will be explained further as it would be really interesting to see how this data compares with what Verizon found in their research.