The Blog Inspired By The Book
This is cool. Visualization of relative storage capacities in terms of media and format.
Notice that it goes all the way back into pre-digital forms, a subtle tweak that I’ll bet a lot of people miss on first inspection. Too bad, too, since the ability to seamlessly compare seemingly-different things is a valuable skill when explaining risk and security issues.
Just saw where Symantec has released their 2010 Security Trends to watch. Now not to pick on Symantec (I’m guilty of the same mess in the past myself over on my old blog) but usually these sorts of prognostication lists are full of the same horse@!@#$. For example:
8. Mac and Mobile Malware Will Increase
In 2009, Macs and smartphones will be targeted more by malware authors. As Mac and smartphones continue to increase in popularity in 2010,
“More” is a fuzzy, useless prediction. We have a fairly benign “DNS Changer” thing on the Mac. And that’s about it (source: an informal and utterly unscientific poll of College Security Admins I did on Twitter). Does “more” = something else you have to be looking at naughty pr0n and give admin rights to be taken advantage of? Or does it mean something that will cause us all to actually *use* anti-malware on the Mac? We don’t know. But all the author needs is “more = another”, and they’re right. Bleh.
TURN THE BEAT AROUND
So this year, let me challenge you to make a change. If you think that there’s going to be a “trend” or “something” to watch for in 2010, let’s see you put your money where your mouth is and be specific.
By specific, I mean go ahead, play weatherman and add an 1-100& “chance” that it’ll happen. What I’ll do here on the NewSchool blog is collect these, and then we’ll do an ad-hoc sort of “Alex + Brier Score” model on the foretelling this time next year and we’ll see who does a good job. Yep, it’s a challenge – if you think you’re good/important/wise enough to make a prediction for next year, then you don’t mind if we hold you accountable, right?
Score Rules/ Model:
1.) We’ll use Wikipedia’s Brier Score example as the basis for our Model:
Suppose it is required to give a probability P forecast of a binary event – such as a forecast of rain. The forecast issued says that there is a probability P that the event will occur. Let X = 1 if the event occurs and X = 0 if it doesn’t. Then the Brier score is given by:
- If you forecast 100% (P = 1) and there is at least 1 mm of rain in the bucket, your Brier Score is 0, or “perfect”.
- If you forecast 100% P and there is no rain in the bucket, your Brier Score is 1, or “awful”.
- If you forecast 70% P and there is at least 1 mm of rain in the bucket, your Brier Score is (0.70-1)^2 = 0.09, or “not too shabby”.
- If you forecast 30% P and there is at least 1 mm of rain in the bucket, your Brier Score is (0.30-1)^2 = 0.49, or “needs work”.
- If you hedge your forecast with a 50% P and whether or not there is at least 1 mm of rain in the bucket, your Brier Score is 0.25, or “no courage”.
Then I’ll poll NewSchool bloggers to see if the prognostication was “lame” (i.e. the sun will shine at some point in 2010). I’ll use an ad-hoc completely stupid 1-10 scoring system where 1=lame and 10=gutsy, multiplying the Brier score by the “Alex” score to come up with the final score for the prediction.
Just to make this even more fun, in addition, we’ll also gather a “% cowardly prognostication” metric. The losers will be given the “Brave Sir Robin” award for soiling their armor in the face of a cute little bunny.
“Of the thousands of cases that we’ve investigated, the public knows about a handful,” said Shawn Henry, assistant director for the Federal Bureau of Investigation’s Cyber Division. “There are million-dollar cases that nobody knows about.”
…
“Keeping your head in the sand on filing a report means that the bad guys are out there hitting the next guy, and the next guy after that,” Henry said.
You mean we need to share data and learn from each other’s mistakes? Sounds good to me!
Quoting Reuters, “Cyber breaches are a closely kept secret.”
I’d like to wish US readers a happy Thanksgiving. For those outside of the US, I thought this would be a nice little post for today: A pointer to an article in the Financial Times,
“Baseball’s love of statistics is taking over football“
Those who indulge my passion for analysis and for sport know that I love baseball and love how the “Moneyball” approach challenged decades of dogma in the national pastime with scientific analysis. Today’s financial times discusses how Chelsea (“The Blues” – UK football team) collaborates with the Boston Red Sox (the most superficial bandwagon team ever in baseball) on decision making and analytics.
Best lines:
“Mike Forde, Chelsea’s performance director, visits the US often. “The first time I went to the Red Sox,” he says of the Boston baseball team, “I sat there for eight hours, in a room with no windows, only flipcharts. I walked out of there saying, ‘Wow, that is one of the most insightful conversations on sport I have ever had.’ It was not: ‘What are you doing here? You do not know anything about our sport.’ That was totally irrelevant. It was: ‘How do you make decisions on players? What information do you use? How do we approach the same problems?’”
and:
“Forde sees his task as “risk management”.
Huh.
Great post today over on SecureThinking about a customer who used a very limited signature set for their IDS.
Truth of the matter was that our customer knew exactly what he was doing. He only wanted to see a handful of signatures that were generic and could indicate that “something” was amiss that REALLY needed to be looked at. Not that something was a quasi attack or could be successful if only that OS was running this configuration of application X — just the nuts and bolts fundamentals of good ‘ole fashion network monitoring. His SNORT’s ran fast, faster than any other IDS of the same hardware investment, because pattern matching was reduced to a handful of rules.
I’m a huge fan of this sort of setup and something that I’ve promoted within the companies I’ve worked with. Why bother looking for something you know you aren’t vulnerable to either because you’ve patched it, configured around it or don’t have that issue at all? Furthermore, if you have signatures installed that you don’t care about, you are just creating noise that is hiding the stuff you really care about.
This does assume that you have a certain level of maturity and actually have the asset, patch and configuration management issues more or less under control. If you don’t, then this like many other problems remain intractable.
If you have a disciplined mature organization, you can largely, if not completely (depends on how complex your company is) move to only uses signatures to tell you when something out of the ordinary is going on and it doesn’t take a complex piece of software, such as Cisco Mars or Maltego to warn you. Instead, you configure just signatures for things like too many of certain classes of events coming from a certain machine:
Error 404: A client has requested something from my webserver that it does not have, or does not have at the location some client was looking for. When a high number of distinct web servers report 404 to a single client host, that host is not up to any good.
Or use of IP space you should never see on your internal network:
DARKNET: There was some IP traffic (ICMP/TCP/UDP doesn’t matter) from an RFC1918 (private) host that we didn’t allocate, or just don’t know about. This is the equivalent of the Police “running” a license plate, and the response coming back “not in system.” How many police would consider that a routine false positive and let the driver go without further questioning?
Alternately, you can look for events such as machines serving up DHCP who shouldn’t be or the sudden appearance of web servers on subnets that didn’t have them in the past.
I like to call this sort of configuration, “Signature Based Anomaly Detection.” It’s not fancy and it’s not complex, but it will tell you when something weird is going on. It may turn out to be a security issue, a misconfigured machine or someone violating change control, but regardless, it’s a great way to actually make your IDS useful and not just something you have to do because an auditor says you have to.
I’m starting on an academic-oriented research project and I’m looking for collaborators, contributors, reviewers, etc.
The topic is the arms race between attackers and defenders from the perspective of innovation rates and “evolutionary success” – the Red Queen problem (running just to stand still). Here’s a sample research question: “can bureaucracies (defenders) keep up with a decentralized black market (attackers)?”, and similar. Answering these research questions would have policy implications on the effectiveness of regulation/mandates vs. incentive-based approaches, R&D policy, etc.
Sail shells from Borneo shaped by a Red Queen arms race with their main predator (a slug of the genus Atopos)
I want to focus primarily on theoretical models, but I’m also keen on grounding them in reality. If I can present some empirical data on the rate of innovation for various players as calibration, that would be superb.
On the theory side, I will be drawing from Evolutionary Ecology (host-parasite co-evolution, adaptive landscapes), Political Economy (models of *real* arms races), Computational Social Science (agent-based models, genetic algorithms, evolutionary game theory), and Economic-Engineering models of innovation and organization learning (risk/reward, optimal investment, etc.). I will also draw on “computable economics” that attempts to measure the information processing/learning capabilities of central planning vs. markets, etc.
Regarding empirical data, I would be interested in any of the following:
Of course, this list is extremely broad. I’m all in favor of narrowing down to a particular security domain and ecosystem. Please make suggestions! Pointers to existing empirical reports are most welcome! Please email me privately (russell.thomas A-T meritology D-O-T com) if you are interested in collaborating or contributing in any way. Ideally, I’d like to have a paper ready to submit to WEIS, in Feb. Grad students welcome!
The Wall Street Journal and 103 hundreds of other news outlets have published articles about the stolen/leaked email files from the Hadley Climate Center University of East Anglia (UEA) Climate Research Unit, in the UK. The blogs are going nuts. Sadly, there is no critical investigation or reporting about the credibility of the leaked email files. Instead, all the news outlets are all caught up in the debate over whether this proves that the Global Warming science is a con job and conspiracy. (A sampling of the more moderate reports: Washington Post, Associated Press, and Christian Science Monitor. The blogs and tweets are more rabid: e.g. proof that “Al Gore lied!”)
Everyone is treating these stolen/leaked documents as real and undoctored, without any real evidence. I couldn’t find any critical/questioning articles when I did a web search. To this, I can only repeat Homer Simpson’s exclamation when he is hit in the face with (his own) stupidity: “D’oh!!“.
For example, the WSJ blog stated that the emails were confirmed as “genuine” by the Director of the breached organization, but a close reading of the source news article shows that the Director only states that the files “appear” to be from his organization. Hadley Climate Center UEA Climate Research Unit hasn’t actually had a chance to review the posted files or even investigate the breach.
Also, no one has questioned the claim that this was the act of “hackers”. The WSJ blog called them “Russian Black Hats” based on the report that the ZIP file first appeared on an FTP server hosted in Russia. Ridiculous! It is easy for anyone located anywhere to upload files to an FTP server with a Russian domain name.
I did find a few security bloggers commenting on this incident, e.g. Graham Cluley, and they are more reserved about the implications of this incident, given the lack of real information. Hopefully, more security experts will speak out on this in the coming days.
Now a rant for the uncritical news organizations and bloggers:
NEWSFLASH — Anyone who has the motives and skills to steal private documents and to upload them on a Russian FTP server in order to generate a public scandal also has the motives and skills to “doctor” those documents . DO NOT trust their content until it is proven genuine!
This is news/publicity incident is just more evidence of wide-spread misunderstanding about trust and credibility regarding online information, and also misunderstandings about nature of security breaches, Black Hats vs. White Hats, etc. This is another case of the meme: “If it’s on the Internet, it must be true”. Sadly, the “echo chamber” of free Internet news media and “advocacy journalism” only makes it worse. Takeaway: This is yet another call-to-arms to security experts to provide evidence-based analyisis that educates the broad public and the institutions that serve them.
[Update -- Corrected the name of the breached organization]
[Update 2: See Comment #2 below for additional "connect the dots" that make the insider attack most plausible, not a "Russian Black Hat".]
More is not always better. This is especially true for screening and detection systems.
False positives can be very costly in a sneaky way. For example, they can cause users, administrators, or managers to go around or turn off the detection/protection mechanism. Here are a few publicized examples of false positives in information security:
We need to be able to steer away from policies, designs, or controls where the detection/prevention costs are greater than the benefits. No security measurement or management program can be considered complete unless it includes assessment for the likely costs of false positives.
We can learn lessons from recent pronouncements from public health organizations: one on mammograms for breast cancer screening, and the other on pap tests for cervical cancer screening. Both are a result of statistical analysis of the total costs and total benefits of testing. Both reports recommend less frequent and/or later testing in most cases, basically because the cost of frequent testing (including false positives) exceeds the benefits in risk reduction. Here are quotes from summary articles:
On Mammograms: “While many women do not think a screening test can be harmful, medical experts say the risks are real. A test can trigger unnecessary further tests, like biopsies, that can create extreme anxiety. And mammograms can find cancers that grow so slowly that they never would be noticed in a woman’s lifetime, resulting in unnecessary treatment. Over all, the report says, the modest benefit of mammograms — reducing the breast cancer death rate by 15 percent — must be weighed against the harms. And those harms loom larger for women in their 40s, who are 60 percent more likely to experience them than women 50 and older but are less likely to have breast cancer, skewing the risk-benefit equation.” [emphasis added]
On Pap testing: “The tradition of doing a Pap test every year has not been supported by recent scientific evidence,” Alan G. Waxman, MD, of the University of New Mexico in Albuquerque, said in a statement. “A review of the evidence to date shows that screening at less frequent intervals prevents cervical cancer just as well, has decreased costs, and avoids unnecessary interventions that could be harmful.” [emphasis added]
Similar conclusions have been reached regarding other medical screening tests, including colonoscopy, PSA test (for prostate cancer), chest X-ray (lung cancer screening for smokers), and full body scan (for everything!). In nearly all of these situations, the forces that were promoting more frequent and earlier testing were ignoring or downplaying the consequences of false positives.
If only the information security community had as much well-organized data and well-controlled tests and experiments as our public health brethren, we would be able to make better informed decisions based on evidence and not prevalent beliefs. This is the direction we need to go.
[Update: Here's a good article from the Wall Street Journal on the cost aspects of risk/benefit analysis in these cases. Great quote: "Americans feel that in health care, more is always better and more means better outcomes," she said. "That's just not true, but it's counterintuitive to a lot of people."]
[Update 2: Bruce Schneier has a good post on the significance of false positives in evaluating detection mechanisms. In the second half of the post, he gives a fairly clear example of how even a "high quality" detection system (= very low false-positive rate) can still yield poor results when the underlying phenomena are very rare, even if you have huge piles of data. Great line: "It's a needle-in-a-haystack problem, and throwing more hay on the pile doesn't make that problem any easier."]
Cormac Herley at Microsoft Research has done us all a favor and released a paper So Long, And No Thanks for the Externalities: The Rational Rejection of Security Advice by Users which opens its abstract with:
It is often suggested that users are hopelessly lazy and unmotivated on security questions. They chose weak passwords, ignore security warnings, and are oblivious to certi cates errors. We argue that users’ rejection of the security advice they receive is entirely rational from an economic perspective.
And you know it’s going to be good when they write:
Thus we find that most security advice simply offers a poor cost-benefit tradeoff to users and is rejected. Security advice is a daily burden, applied to the whole population, while an upper bound on the benefit is the harm suffered by the fraction that become victims annually. When that fraction is small, designing security advice that is beneficial is very hard. For example, it makes little sense to burden all users with a daily task to spare 0.01% of them a modest annual pain.
People are not stupid. They make what we, as relative experts on the topic of security, perceive to be bad decisions, but this paper argues that their behavior is rational.
[W]e argue for a third view, which is that users’ rejection of the security advice they receive is entirely rational from an economic viewpoint. The advice o ers to shield them from the direct costs of attacks, but burdens them with increased indirect costs, or externalities. Since the direct costs are generally small relative to the indirect ones they reject this bargain. Since victimization is rare, and imposes a one-time cost, while security advice applies to everyone and is an ongoing cost, the burden ends up being larger than that caused by the ill it addresses.
The paper provides both a good and accessible overview of externalities and rational behavior using spam as an example.
For example, Kanich et al. [32] document a campaign of 350 million spam messages sent for $2731 worth of sales made. If 1% of the spam made it into in-boxes, and each message in an inbox absorbed 2 seconds of the recipient’s time this represents 1944 hours of user
time wasted, or $28188 at twice the US minimum wage of $7.25 per hour.
Coincidentally, we get a little over 300 million spam messages into our corporate email gateways every month, which means that I can compare the cost-per-delete-click (at $7.25/hour) against the cost of our corporate spam filtering contract without having to do any real math. Since we pay about $50,000/month for filtering. That means that we’re getting a pretty good deal, since our white-collar employees cost over $14/hour.
That’s just time that would be spent seeing and deleting the message, don’t forget. Fourteen Dollars per hour completely ignores the cost of attention disruption (much more than two seconds) and the Direct Losses, either because I cannot quantify, which causes the entire argument to appear specious in the eyes of Senior Leadership, or I am not at liberty to disclose enough detail to pass the “cannot quantify” test.
They then go on to document in fairly accessible models why password complexity, anti-phishing awareness, and SSL Errors are cost-inefficient, and get into a favorite topic of mine, the difficulty of defining security losses or the benefit from adding safeguards at the end-user level. This section should be mandatory reading for any security person who attempts to talk to non-security people about the topic–i.e. all of us.
What’s missing from the paper, though, is the next logical step of analysis, the appropriate Risk Management strategy in response to the information presented. Hopefully that will be the follow-on paper, because as it was, it felt like a bit of a cliff-hanger to me. All of the discussion assumes that mitigation is the only option. This may feel right from a Security perspective, but it’s probably not the correct risk management decision.
To manage the risk in these cases, though, I see a strong argument for risk transfer. High-Impact, Low-Likelihood events are best managed by aggregating the risk into a pool and spreading the cost across the pool, i.e. buying insurance against these losses. If you could buy anti-phishing insurance for $1/person/year (which, realistically, is multiples of what it could cost if 200 million people all bought in) rather than throwing large, uncoordinated piles of money at ineffective awareness training or technical countermeasures which will probably be out-innovated by the attackers in hours or days, why wouldn’t you?
Why have anti-virus vendors not thought of this? If your AV vendor said they would also insure you against Direct Losses (having your bank account cleaned out) for your $50/year subscription, would that differentiate them enough to win your business?
By all means, we should continue to work on the challenges of improving the security experience and reducing the risk of using computers. More accurately, though, we should be reducing the amount that must be experienced by users at all to improve security of their information and transactions.
Threatlevel (aka 27B/6) reported yesterday that Richard Schaeffer, the NSA’s information assurance director testified to the Senate Senate Judiciary Subcommittee on Terrorism, Technology and Homeland Security on the issue of computer based attacks.
If network administrators simply instituted proper configuration policies and conducted good network monitoring, about 80 percent of commonly known cyber attacks could be prevented, a Senate committee heard Tuesday.
The remark was made by Richard Schaeffer, the NSA’s information assurance director, who added that simply adhering to already known best practices would sufficiently raise the security bar so that attackers would have to take more risks to breach a network, “thereby raising [their] risk of detection.”
I’m really curious however on what data Director Schaeffer is basing his testimony on. Is it the DBIR? Another open set of breach data or is it based on data gathered by the NSA? Regardless, it’s great to see more folks talking about what the Verizon DBIR report told us and what we’ve known anecdotally for a long time; which is, we still aren’t even close to doing the basics well.
The article then goes on to tell us:
A 2009 Price Waterhouse Cooper study on global information security found that 47 percent of companies are reducing or deferring their information security budgets, despite the growing dangers of cyber incursions.
The thing is, as we’ve learned from the Verizon study, most of the found issues were due to failing at doing the basics, like not removing default passwords, not revoking accounts when employees leave and misconfigurations. Even in the case of patching, the vast majority of holes exploited had patches available for over a year and 100% had patches available for over 6 months. This is not the stuff of big budgets and sexy technology, but rather about having solid, repeatable and auditable processes, in other words, serious operational discipline. Budget cuts might actually be a good thing because it will force organizations to focus on the people and process portions of security rather then the technology. It’d be really cool to if PWC were to track correlation of budgets to breaches within their survey groups, then we’d have some actual data on potential optimal spend levels.
What You’ve Said