The New School of Information Security » measurement

Aviation Safety

adam — Wed, 25 Jan 2012 16:06:00 +0000

The past 10 years have been the best in the country’s aviation history with 153 fatalities. That’s two deaths for every 100 million passengers on commercial flights, according to an Associated Press analysis of government accident data.

The improvement is remarkable. Just a decade earlier, at the time the safest, passengers were 10 times as likely to die when flying on an American plane. The risk of death was even greater during the start of the jet age, with 1,696 people dying — 133 out of every 100 million passengers — from 1962 to 1971. The figures exclude acts of terrorism.

…
There are a number of reasons for the improvements.

The industry has learned from the past. New planes and engines are designed with prior mistakes in mind. Investigations of accidents have led to changes in procedures to ensure the same missteps don’t occur again.
Better sharing of information. New databases allow pilots, airlines, plane manufactures and regulators to track incidents and near misses. Computers pick up subtle trends. For instance, a particular runway might have a higher rate of aborted landings when there is fog. Regulators noticing this could improve lighting and add more time between landings.

(“It’s never been safer to fly; deaths at record low“, AP, link to Seattle PI version.)

Well, it seems there’s nothing for information security to learn here. Move along.

Kudos to Ponemon

adam — Mon, 23 Jan 2012 15:59:03 +0000

In the past, we have has some decidedly critical words for the Ponemon Institute reports, such as “A critique of Ponemon Institute methodology for “churn”” or “Another critique of Ponemon’s method for estimating ‘cost of data breach’“. And to be honest, I’d become sufficiently frustrated that I’d focused my time on other things.

So I’d like to now draw attention to a post by Patrick Florer, “Some Thoughts about PERT and other distributions“, in which he says:

What follows are the results of an attempt to answer this question using a small data set extracted from a Ponemon Institute report called “Compliance Cost Associated with the Storage of Unstructured Information”, sponsored by Novell and published in May, 2011. I selected this report because, starting on page 14, all of the raw data are presented in tabular format. As an aside, this is the first report I have come across that publishes the raw data – please take note, Verizon, if you are reading this!

So I simply wanted to offer kudos to the Ponemon Institute for doing this.

I haven’t yet had a chance to dig into the report, but felt that given our past critiques I should take note of a very positive step.

The New School of Security Predictions

adam — Wed, 21 Dec 2011 16:30:06 +0000

Bill Brenner started it with “Stop them before they predict again!:”

My inbox has been getting hammered with 2012 vendor security predictions since Halloween. They all pretty much state the obvious:

Mobile malware is gonna be a big deal
Social networking will continue to be riddled with security holes
Technologies A, B and C will be dead
Microsoft will release a lot of security patches
Data security breaches will continue to get more expensive

Looking at the predictions I got this time last year for 2011, I found that any of them could be repackaged as 2012 predictions and nobody would know the difference. Here are some examples from the Zscaler Labs Research Team…

Jack Daniel followed up with “The Pandering Pentagram of Prognostication :”

The five points of the pentagram represent the key elements of “good” predictions, get them all and your prediction will land in the center of the pentagram, assuring a center brain shot to your victim. I mean reader. Whatever.

The five elements are outlined below, miss even one and your prediction may be off target and you will fail to hit your target.

Your prediction must be self-serving.
Your prediction must suck up to your customers, prospects, or others whose favor you are trying to win…

I’ll respond with a prediction that 90% of 2012 infosec predictions will contain no numbers and no dates. If someone selects a group of 10 or more predictors (say, bloggers in SBN, or 2011 BlackHat speakers with blogs) and proves me wrong, I’ll donate $100 to a charity of your

Both Bill and Jack are helping the community by pointing out the “best practices in predictions” so that people can recognize them for the self-serving (ad-serving) linkbait that most of them are.

To get something positive out of this, I encourage everyone to ask anyone who sends you predictions about the lack of underlying data.

The One Where David Lacey’s Article On Risk Makes Us All Stupider

alex — Fri, 25 Nov 2011 17:08:37 +0000

In possibly the worst article on risk assessment I’ve seen in a while, David Lacey of Computerworld gives us the “Six Myth’s Of Risk Assessment.” This article is so patently bad, so heinously wrong, that it stuck in my caw enough to write this blog post. So let’s discuss why Mr. Lacey has no clue what he’s writing about, shall we?

First Mr. Lacey writes:

1. Risk assessment is objective and repeatable

It is neither. Assessments are made by human beings on incomplete information with varying degrees of knowledge, bias and opinion. Groupthink will distort attempts to even this out through group sessions. Attitudes, priorities and awareness of risks also change over time, as do the threats themselves. So be suspicious of any assessments that appear to be consistent, as this might mask a lack of effort, challenge or review.

Sounds reasonable, no? Except it’s not alltogether true. Yes, if you’re doing idiotic RCSA of Inherent – Control = Residual, it’s probably as such, but those assessments aren’t the totality of current state.

“Objective” is such a loaded word. And if you use it with me, I’m going to wonder if you know what you’re talking about. Objectivity / Subjectivity is a spectrum, not a binary, and so for him to say that risk assessment isn’t “objective” is an “of course!” Just like there is no “secure” there is no “objective.”

But Lacey’s misunderstanding of the term aside, let’s address the real question: “Can we deal with the subjectivity in assessment?” The answer is a resounding “yes” if your model formalizes the factors that create risk and logically represents how they combine to create something with units of measure. And not only will the right model and methods handle the subjectivity to a degree that is acceptable, you can know that you’ve arrived at something usable when assessment results become “blindly repeatable.” And yes, Virginia, there are risk analysis methods that create consistently repeatable results for information security.

2. Security controls should be determined by a risk assessment

Not quite. A consideration of risks helps, but all decisions should be based on the richest set of information available, not just on the output of a risk assessment, which is essentially a highly crude reduction of a complex situation to a handful of sentences and a few numbers plucked out of the air. Risk assessment is a decision support aid, not a decision making tool. It helps you to justify your recommendations.

So the key here is “richest set of information available” – if your risk analysis leaves out key or “rich” information, it’s pretty much crap. Your model doesn’t fit, your hypothesis is false, start over. If you think that this is a trivial matter for him to not understand, I’ll offer that this is kind of the foundation of modern science. And mind you, this guy was supposedly a big deal with BS7799. Really.

4. Risk assessment prevents you spending too much money on security

Not in practice. Aside from one or two areas in the military field where ridiculous amounts of money were spent on unnecessary high end solutions (and they always followed a risk assessment), I’ve never encountered an information system that had too much security. In fact the only area I’ve seen excessive spending on security is on the risk assessment itself. Good security professionals have a natural instinct on where to spend the money. Non-professionals lack the knowledge to conduct an effective risk assessment.

This “myth” basically made me physically ill. This statement “I’ve never encountered an information system that had too much security” made me laugh so hard I keeled over and hurt my knee in the process by slamming it on the little wheel thing on my chair.

Obviously Mr. Lacey never worked for one of my previous employers that forced 7 or so (known) endpoint security applications on every Windows laptop. Of course you can have too much !@#%ing security! It happens all the !@#%ing time. We overspend where frequency and impact ( <- hey, risk!) don’t justify the spend. If I had a nickel for every time I saw this in practice, I’d be a 1%er.

But more to the point, this phrase (never too much security) makes several assumptions about security that are patently false. But let me focus on this one: This statement implies that threats are randomly motivated. You see, if a threat has targeted motivation (like IP or $) then they don’t care about systems that offer no value in data or in privilege escalation. Thus, you can spend too much on protecting assets that offer no or limited value to a threat agent.

5. Risk assessment encourages enterprises to implement security

No, it generally operates the other way around. Risk assessment means not having to do security. You just decide that the risk is low and acceptable. This enables organisations to ignore security risks and still pass a compliance audit. Smart companies (like investment banks) can exploit this phenomenon to operate outside prudent limits.

I honestly have no idea what he’s saying here. Seriously, this makes no sense. Let me explain. Risk assessment outcomes are neutral states of knowledge. They may feed a state of wisdom decision around budget, compliance, and acceptance (addressing or transferring, too) but this is a logically separate task.

If it’s a totally separate decision process to deal with the risk, and he cannot recognize this is a separate modeling construct, these statements should be highly alarming to the reader. It screams “THIS MAN IS AUTHORIZED BY A MAJOR MEDIA OUTLET TO SPEAK AS AN SME ON RISK AND HE IS VERY, VERY CONFUSED!!!!”

Then there is that whole thing at the end where he calls companies that address this process illogically as “smart.” Deviously clever, I’ll give you, but not smart.

6. We should aspire to build a “risk culture” across our enterprises

Whatever that means it sounds sinister to me. Any culture built on fear is an unhealthy one. Risks are part of the territory of everyday business. Managers should be encouraged to take risks within safe limits set by their management.

So by the time I got to this “myth” my mind was literally buzzing with anger. But then Mr. Lacey tops us off with this beauty. This statement is so contradictory to his past “myth” assertions, is so bizarrely out of line with his last statement in any sort of deductive sense, that one has to wonder if David Lacey isn’t actually an information security surrealist or post-modernist who rejects ration, logic, and formality outright for the sake of random, disconnected and downright silly approaches to risk and security management. Because that’s the only way this statement could possibly make sense. And I’m not talking “pro” or “con” for risk culture here, I’m just talking about how his mind could possibly conceptually balance the concept that an “enterprise risk culture” sounds sinister vs. “Managers should be encouraged to take risks within safe limits set by their management” and even “I’ve never encountered an information system that had too much security.”

(Mind blown – throws up hands in the air, screams AAAAAAAAAAAAAAAAAHHhHHHHHHHHHHHHH at the top of his lungs and runs down the hall of work as if on fire)

See? Surrealism is the only possible explanation.

Of course, if he was an information security surrealist, this might explain BS7799.

AT&T Hack Attempt

adam — Tue, 22 Nov 2011 17:41:04 +0000

First, good on AT&T for telling people that there’s been an attempt to hack their account. (My copy of the letter that was sent is after the break.) I’m curious what we can learn by discussing the attack.

An AT&T spokesperson told Fox News that “Fewer than 1 percent of customers were targeted.”

I’m currently aware of 3 other folks in the security industry who’ve gotten these. Can someone recommend a good embeddable polling software that I might use to see what the prevalence is on the biased audience that reads this blog?

Dear ADAM SHOSTACK:

In our ongoing effort to provide you with the best privacy protections possible, AT&T regularly monitors the security of our online services such as MyATT online account management. The purpose of this letter is to advise you that we recently detected an organized and systematic attempt to obtain information on a number of AT&T customer accounts, including yours. We have not yet determined the source or intent of the attempt to gather information, but we are continuing to investigate.

We do not believe that the perpetrators of this attack obtained access to your online account or any of the information contained in that account. Use of your mobile device or other AT&T services has not been affected by this incident.

Customer privacy and data security are top priorities for AT&T. Because there may be an increased risk of fraudulent attempts to access your account information, you should be cautious about efforts to obtain sensitive information through email (“phishing”) or text messages (“smishing”). You can find out more about phishing, smishing, and protecting your online account information below, and at att.com/safety.

We appreciate having you as a customer and if you have questions or you need assistance, please contact AT&T Customer Care by dialing 611 on your AT&T wireless device or calling 1.800.331.0500.

Sincerely,

AT&T Online Services

Ref Code: F3221

Measures available to you to help ensure that all of your online accounts
and your Internet access devices are better protected.

Be aware of the risks to any online account and the steps you can take to reduce your exposure to them. “Phishing,” also known as “brand spoofing” or “carding,” is a trick used to gather financial information and password data using fake emails and websites. Scammers using phishing techniques send consumers email messages that appear to be from well-known companies. These messages usually contain links to web pages that have been disguised to look nearly identical to legitimate companies’ sites and request customers to enter sensitive information including financial and password data. “Smishing” is a technique similar to phishing; it uses cell phone text messages to get customers to divulge personal information. Scammers using smishing techniques may send a text message with a link to a website which requests sensitive information. More commonly, “smishers” will include a phone number in the text message that connects to an automated voice response system to capture sensitive information.

Some of the measures you can take online and on your Internet access device to help protect yourself include:

• Be aware that email and text message headers can be easily forged, so the posing sender may not be the real sender.

• Avoid providing or filling out forms via email because the data is likely to be unsecured.

• Realize that Internet scammers can create realistic forgeries of websites, so avoid clicking on links in an unsolicited email or text message. If appropriate, go directly to the company’s website to investigate the validity of the communication.

Additional recommended measures to protect your Internet access devices include:

• Scan all computers with an up-to-date anti-virus program.

• Use an up-to-date anti-spyware (anti-malware) application, as some malware is not detected by anti-virus scans.

• Ensure your operating system has been updated with all the recommended security updates from the operating system provider.

• Check all your online accounts regularly for unauthorized activity.

• Review trusted online sources for information on protecting your computer and mobile devices.

If you have any questions about how AT&T collects, uses and protects your personal information as a customer, please visit our Privacy Policy.

© 2011 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. Subsidiaries and affiliates of AT&T Inc. provide products and services under the AT&T brand.

Lean Startups & the New School

adam — Tue, 20 Sep 2011 16:10:54 +0000

On Friday, I watched Eric Ries talk about his new Lean Startup book, and wanted to talk about how it might relate to security.

Ries concieves as startups as businesses operating under conditions of high uncertainty, which includes things you might not think of as startups. In fact, he thinks that startups are everywhere, even inside of large businesses. You can agree or not, but suspend skepticism for a moment. He also says that startups are really about management and good decision making under conditions of high uncertainty.

He tells the story of IMVU, a startup he founded to make 3d avatars as a plugin instant messenger systems. He walked through a bunch of why they’d made the decisions they had, and then said every single thing he’d said was wrong. He said that the key was to learn the lessons faster to focus in on the right thing–that in that case, they could have saved 6 months by just putting up a download page and seeing if anyone wants to download the client. They wouldn’t have even needed a 404 page, because no one ever clicked the download button.

The key lesson he takes from that is to look for ways to learn faster, and to focus on pivoting towards good business choices. Ries defines a pivot as one turn through a cycle of “build, measure, learn:”

Ries jokes about how we talk about “learning a lot” when we fail. But we usually fail to structure our activities so that we’ll learn useful things. And so under conditions of high uncertainty, we should do things that we think will succeed, but if they don’t, we can learn from them. And we should do them as quickly as possible, so if we learn we’re not successful, we can try something else. We can pivot.

I want to focus on how that might apply to information security. In security, we have lots of ideas, and we’ve built lots of things. We start to hit a wall when we get to measurement. How much of what we built changed things (I’m jumping to the assumption that someone wanted what you built enough to deploy it. That’s a risky assumption and one Ries pushes against with good reason.) When we get to measuring, we want data on how much your widget changed things. And that’s hard. The threat environment changes over time. Maybe all the APTs were on vacation last week. Maybe all your protestors were off Occupying Wall Street. Maybe you deployed the technology in a week when someone dropped 34 0days on your SCADA system. There are a lot of external factors that can be hard to see, and so the data can be thin.

That thin data is something that can be addressed. When doctors study new drugs, there’s likely going to be variation in how people eat, how they exercise, how well they sleep, and all sorts of things. So they study lots of people, and can learn by comparing one group to another group. The bigger the study, the less likely that some strange property of the participants is changing the outcome.

But in information security, we keep our activities and our outcomes secret. We could tell you, but first we’d have to spout cliches. We can’t possibly tell you what brand of firewall we have, it might help attackers who don’t know how to use netcat. And we certainly can’t tell you how attackers got in, we have to wait for them to tell you on Pastebin.

And so we don’t learn. We don’t pivot. What can we do about that?

We can look at the many, many people who have announced breaches, and see that they didn’t really suffer. We can look at work like Sensepost has offered up at BlackHat, showing that our technology deployments can be discovered by participation on tech support forums.

We can look to measure our current activities, and see if we can test them or learn from them.

Or we can keep doing what we’re doing, and hope our best practices make themselves better.

Diginotar Quantitative Analysis (“Black Tulip”)

adam — Tue, 13 Sep 2011 15:12:05 +0000

Following the Diginotar breach, FOX-IT has released analysis and a nifty video showing OCSP requests.

As a result, lots of people are quoting a number of “300,000″.

Cem Paya has a good analysis of what the OCSP numbers mean, what biases might be introduced at “DigiNotar: surveying the damage with OCSP.”

To their credit, FoxIt tried to investigate the extent of the damage by monitoring OCSP logs for users checking on the status of the forged Google certificate. There is a neat YouTube video showing the geographic distribution of locations around the world over time. Unfortunately while this half-baked attempt at forensics makes for great visualization, it presents a very limited picture of impacted users.

Digitar and Fox-IT released enough that a dedicated secondary analyst like Cem can see methodological flaws in what they did. What else could we learn if we had more of the raw observations? When I read the report, I noticed the claim “A number of malicious/hacker software tools was found. These vary from commonly used tools such a the famous Cain & Abel tool to tailor made software.” This claim mixes analysis and observation. The observation is that there was software with which the analyst was not familiar. It may be that it was a perl script or other code that can be easily skimmed to see that it was “tailor made.” It may be that it was just something re-compiled to not match a hash. We don’t know. Similarly, the report claims (4.1) “In at least one script, fingerprints from the hacker are left on purpose, which were also found in the Comodo breach investigation of March 2011.” Really? On purpose? Perhaps the fingerprints were inserted as a matter of dis-information. Perhaps the Fox-IT analyst called the intruder on the phone, and he owned up to it. We don’t know.

I want to be clear that I don’t mean to be picking on Fox-IT here. My understanding is that the report they prepped came out incredibly quickly, and kudos to them for that. I’ve cherry picked two areas where I can ask for better editing, but I’m very aware that that editing comes at a cost in timeliness.

Cem’s article is very much worth reading, as is the Fox-IT report. But Cem’s analysis helps illustrate a theme of the New School, which is that we need diverse perspectives and analysis brought to bear on each report. The more data we see, the more we can learn from it. No single analysis will tell us everything we might learn. (I made a similar point here.)

I am left with a question for Cem, which I would have added to his post, but couldn’t comment there. My question is, having given all that thought to all the biases, what do you think is the probably true number (or range) of affected people?

Sex, Lies & Cybercrime Surveys: Getting to Action

adam — Thu, 23 Jun 2011 16:19:57 +0000

My colleagues Dinei Florencio and Cormac Herley have a new paper out, “Sex, Lies and Cyber-crime Surveys.”

Our assessment of the quality of cyber-crime surveys is harsh: they are so compromised and biased that no faith whatever can be placed in their findings. We are not alone in this judgement. Most research teams who have looked at the survey data on cyber-crime have reached similarly negative conclusions.

In the book, Andrew and I wrote “today’s security surveys have too many flaws to be useful as sources of evidence.” Dinei and Cormac were kind enough to cite that, saving me the trouble of looking it up.

I wanted to try here to carve out, perhaps, a small exception. I think of surveys as coming in two main types: surveys of things people know, and surveys of what they think. Both have the potential to be useful (although read the paper for a long list of ways in which they can be problematic.)

So there’s surveys of things people know. For example, what’s your budget, or how many people do you employ? There are people in an organization who know those things, and, starved as we are for knowledge, perhaps they would be useful to know. So maybe a survey makes sense.

But how many people Microsoft employs in security probably doesn’t matter to you. And the average of how many people Boeing, State Farm, Microsoft, Archer Daniels Midland, and Johnson & Johnson employ in security is even less useful. (Neighbors on the Fortune 500 list.) So even in the space that we might want to defend surveys, they’re not that useful.

So our desire for surveys is really evidence of how starved we are for data about outcomes and data about efficacy. We’re like the drunk looking for keys under the lamppost, not because we think the keys are there, but because there’s at least a little light.

So next time someone shows you a survey, don’t even bother to ask them what action they expect you to take, or what decision they expect you to alter, or ask them why you should accept what it says as acceptable arguments for that choice.

Rather, ask them to see the section titled “How we overcame the issues that Dinei and Cormac talked about.” It’ll save everyone a bunch of time.

A Few Data Points

Chandler — Thu, 19 May 2011 19:47:12 +0000

First, for those who might have missed it, Google has released Google Refine, a free tool for cleaning dirty data sets. It allows you to pull in disparate data, then organize and clean it for consistency.

Next, some interesting thoughts on how “anonymized” data sets aren’t, and some thoughts on the implications of this from a risk perspective. None of this is groundbreaking, but it’s nice to see some sane thinking about two facts that aren’t going away, no matter how much people might like them: that data will continue to be accumulated and that it will be shared with varying levels of consideration for the risks of doing so.

Finally, yet another real-world example of risk homeostasis at work: People who take vitamins make poorer health decisions in other areas. Based on the number of times I’ve been asked questions along the lines of, “I don’t need to worry about x because I’ve {patched|installed anti-virus|switched to Apple|etc.}, right?” I’d say this still holds true for computing, too.

Now if you’ll excuse me, I have to go clean my anonymized data set so I can share it far and wide, which is OK since I’m going to encrypt it before I send it, right?

Just Because YOU Think Your Clients Are Too Busy and/or Stupid Doesn’t Mean Everyone Else Is

alex — Tue, 01 Mar 2011 15:24:22 +0000

Mike Rothman’s “Firestarter” on “Risk Metrics are Crap“.

It’s very difficult to argue with a poorly constructed argument. Especially when I have no idea what a “risk metric” is. But best as I can tell, Mike’s position is that unless you are smart and/or have strong resources allocated to your InfoSec team, things like metrics, GRC, application security, SEIM, and a “host of other security processes or technologies.” are “science projects…”

Meaning, I suppose, that they provide no “pragmatic” use to security departments (whatever pragmatic means).

Problem is, for many folks, metrics, risk management, appsec and other “security processes or technologies” can and do have significant value. In fact, in terms of managing a large, disparate enterprise, the data gathering process for risk analysis alone can be more valuable than the result (experience contrary to what Rothman writes in comments: “the value of the (assessment) benefit is outweighed by the cost of gathering the data.”).

That said, it’s a shame that his argument is poorly constructed because, by in large, I have to agree that there’s plenty of poopy risk statements to pick on. As I said in my CSO Magazine interview (shameless self promotion) and in my RSA Risk Management Smackdown panel – there have been times when I’ve counseled an organization to put off making risk statements until their visibility into their environment is much better. In the public record you should be able to find past statements where I say it’s better for a small business to focus resources away from risk assessment when the required assessment was a bureaucratic quest rather than a quest for knowledge or wisdom.

Bottom Line, for risk and metrics, Rothman shouldn’t generalize for the sake of sensationalism and marketing. And he probably shouldn’t be doing that for other security processes and technologies beyond risk analysis and security management, too. But as a consulting firm, what Securosis could/should be doing is giving people help – helping them recognize their organizational maturity, and helping them understand what resource allocation is or isn’t appropriate at various levels of maturity. Just a thought.