Ries concieves as startups as businesses operating under conditions of high uncertainty, which includes things you might not think of as startups. In fact, he thinks that startups are everywhere, even inside of large businesses. You can agree or not, but suspend skepticism for a moment. He also says that startups are really about management and good decision making under conditions of high uncertainty.

He tells the story of IMVU, a startup he founded to make 3d avatars as a plugin instant messenger systems. He walked through a bunch of why they’d made the decisions they had, and then said every single thing he’d said was wrong. He said that the key was to learn the lessons faster to focus in on the right thing–that in that case, they could have saved 6 months by just putting up a download page and seeing if anyone wants to download the client. They wouldn’t have even needed a 404 page, because no one ever clicked the download button.

The key lesson he takes from that is to look for ways to learn faster, and to focus on pivoting towards good business choices. Ries defines a pivot as one turn through a cycle of “build, measure, learn:”

Ries jokes about how we talk about “learning a lot” when we fail. But we usually fail to structure our activities so that we’ll learn useful things. And so under conditions of high uncertainty, we should do things that we think will succeed, but if they don’t, we can learn from them. And we should do them as quickly as possible, so if we learn we’re not successful, we can try something else. We can pivot.

I want to focus on how that might apply to information security. In security, we have lots of ideas, and we’ve built lots of things. We start to hit a wall when we get to measurement. How much of what we built changed things (I’m jumping to the assumption that someone wanted what you built enough to deploy it. That’s a risky assumption and one Ries pushes against with good reason.) When we get to measuring, we want data on how much your widget changed things. And that’s hard. The threat environment changes over time. Maybe all the APTs were on vacation last week. Maybe all your protestors were off Occupying Wall Street. Maybe you deployed the technology in a week when someone dropped 34 0days on your SCADA system. There are a lot of external factors that can be hard to see, and so the data can be thin.

That thin data is something that can be addressed. When doctors study new drugs, there’s likely going to be variation in how people eat, how they exercise, how well they sleep, and all sorts of things. So they study lots of people, and can learn by comparing one group to another group. The bigger the study, the less likely that some strange property of the participants is changing the outcome.

But in information security, we keep our activities and our outcomes secret. We could tell you, but first we’d have to spout cliches. We can’t possibly tell you what brand of firewall we have, it might help attackers who don’t know how to use netcat. And we certainly can’t tell you how attackers got in, we have to wait for them to tell you on Pastebin.

And so we don’t learn. We don’t pivot. What can we do about that?

We can look at the many, many people who have announced breaches, and see that they didn’t really suffer. We can look at work like Sensepost has offered up at BlackHat, showing that our technology deployments can be discovered by participation on tech support forums.

We can look to measure our current activities, and see if we can test them or learn from them.

Or we can keep doing what we’re doing, and hope our best practices make themselves better.

LulzSec is running around pummelling some of the world’s most powerful organisations into the ground… for laughs! For lulz! For shits and giggles! Surely that tells you what you need to know about computer security: there isn’t any.

And I have to admit, I’m taking a certain amount of pleasure in watching LulzSec. Whoever’s doing it are actually entertaining, when they’re not breaking the law. And even sometimes when they are. But at those times, they’re hurting folks, so it’s a little harder to chortle along.

Now Patrick’s argument is in the close, and I don’t want to ruin it, but I will:

So why do we like LulzSec?

“I told you so.”

That’s why.

The essence of this argument is that we in security have been telling management for a long time that things are broken, and we’ve been ignored. We poor, selfless martyrs. If only we’d been given the budget, we would have implemented a COBIT ISO27001 best practices program of making users leap through flaming hoops before they got their job done, and none of this would ever have happened. We here in the business of defending our organizations would love to have been effective, except we weren’t, and now we’re mother-freaking cheering a bunch of kids who can’t even spell LOL? Really? I told you so? Is that the best that we as a community will do?

Apparently.

We’re being out-communicated by folks who can’t spell.

Why are we being out-communicated? Because we expect management to learn to understand us, rather than framing problems in terms that matter to them. We come in talking about 0days, whale pharts, cross-site request jacking and a whole alphabet soup of things whose impact to the business are so crystal clear obvious that they go without saying.

And why are we being out-communicated? Because every time there’s a breach, we cover it up. We claim it wasn’t so bad. Or maybe that the poor, hapless American citizen will get tired of hearing about the breaches. And so we’re left with the Lulz crowd breaking and entering for shits and giggles to demonstrate that there are challenges in making things secure.

I don’t mean to sound like a broken record, but maybe we should start talking openly about breaches instead. Maybe then, we’d get somewhere without needing to see Sony, PBS, and Infraguard attacked. Heck, maybe if we talked about breaches, one or more of those organizations would have learned from the pain of others.

Nah.

Let’s just wait for “the world’s leaders in high-quality entertainment at your expense” to let us say I told you so.

It sure is easier than admitting our communications were sub-par.

[Thanks for the many good comments! I've written a follow-up post on the topic of communication, "Communicating with Executives for more than Lulz."]

I’m glad to see how the debate is maturing, and how no one bothered with some of the silly arguments we’ve heard in the past.

]]> http://newschoolsecurity.com/2011/05/representative-bono-mack-on-the-sony-hack/feed/ 0 http://newschoolsecurity.com/2011/01/a-critique-of-ponemon-institute-methodology-for-churn/ http://newschoolsecurity.com/2011/01/a-critique-of-ponemon-institute-methodology-for-churn/#comments Tue, 25 Jan 2011 16:56:10 +0000 adam http://newschoolsecurity.com/?p=2028 Both Dissent and George Hulme took issue with my post Thursday, and pointed to the Ponemon U.S. Cost of a Data Breach Study, which says:

Average abnormal churn rates across all incidents in the study were slightly higher than last year (from 3.6 percent in 2008 to 3.7 percent in 2009), which was measured by the loss of customers who were directly affected by the data breach event (i.e., typically those receiving notification). The industries with the highest churn rate were pharmaceuticals, communications and healthcare (all at 6 percent), followed by financial services and services (both at 5 percent.)

Some comments:

• 126 of the hundreds of organizations that suffered a breach were selected (no word on how) to receive a survey. 45 responded, which might be a decent response rate, but we need to know how the 126 were selected from the set of breached entities.
• We don’t understand the baseline for customer churn. What is normal turnover? Is it the median for the last 3 years for that company? The mean for the sector last year? If we knew how normal turnover was defined, and its variance, then we could ask questions about what abnormal means. Is it the difference between management estimates and prior years? Is it the difference between a standard deviation above the mean for the sector for the past 3 years and the observed?
• Most importantly, it’s not an actual measure of customer churn. The report states that it measured not actual customer loss, but the results of a survey that asked for:
  

  The estimated number of customers who will most likely terminate their relationship as a result of the breach incident. The incremental loss is abnormal turnover attributable to the breach incident. This number is an annual percentage, which is based on estimates provided by management during the benchmark interview process. [Emphasis added.]

The report has other issues, and I encourage readers to examine its claims and evidence closely. I encourage this in general, it’s not a comment unique to the Ponemon report. Some examples from a number of additional surveys, that George Hulme raised in argment in this blog post:

Briefly, the CMO council found concern about security, not any knowledge of breaches. Forrester showed that some folks are scared to shop online, which means brand doesn’t matter, or they’d shop online from trusted brands. Javelin reports 40% of consumers reporting that their relationship “changed,” and 30% reporting a choice to not purchase from the organization again. Which is at odds with even the most ‘consumer-concerned’ estimates from Ponemon, and is aligned with the idea that surveys are hard to do well.

]]> http://newschoolsecurity.com/2011/01/a-critique-of-ponemon-institute-methodology-for-churn/feed/ 7 http://newschoolsecurity.com/2011/01/requests-for-a-proof-of-non-existence/ http://newschoolsecurity.com/2011/01/requests-for-a-proof-of-non-existence/#comments Mon, 24 Jan 2011 16:17:10 +0000 adam http://newschoolsecurity.com/?p=2024 So before I respond to some of the questions that my “A day of reckoning” post raises, let me say a few things. First, proving that a breach has no impact on brand is impossible, in the same way that proving the non-existence of god or black swans is impossible. It will always be possible for a new breach to impact a brand.

Second, and far more importantly, I’m not the one making the surprising claim, or bringing it to the marketing department. If you are making a surprising claim, the responsibility to back it up lies on you. Ideally, someone’s going to produce a convincing and predictive theory of brand costs that works across a defined subset of the thousands of breaches in the DataLossDB or DBIR. Until they do, there are still lots and lots of breaches that have minimal effect on stock price and very little on overall brand.

Finally, the marketing department owns branding, in the same way that IT owns operational roll-outs. You need to convince them in the same way you need to convince IT to roll out a new IDS or development to implement an SDL. Information security people don’t own questions about brand any more than legal does. If you want to influence the folks who write for “the CMO site,” you’re going to have to bring data. In other words, your argument is going to have to resonate with the business leaders who think that a guy picking his nose and posting the video to YouTube is far more likely to hurt their brand.

I’ll have more on the Ponemon report & other reports cited by George Hulme here shortly.

Let me call your attention to this as a turning point for a trend. Those of us in the New School have been saying this for several years, but the idea that a breach is unlikely to kill your organization is spreading, because it’s backed by data.

That’s a good thing for folks who are in the New School, but not so good for others. If you’ve been spreading FUD (even with the best of intentions), you’re going to face some harsh questions.

By regularly making claims which turn out to be false, people undermine their credibility. If you’re one of those people, expect questions from those outside security who’ve heard you make the claim. The questions will start with the claim of brand damage, but they might not end there. They’ll continue into other areas where neither the questioner or you have any data. If you make good calls in the absence of data, then that’s ok. Leaders always make calls with insufficient data. What’s important is that they’re good calls. And talking about brand damage no longer looks like a good call, an ok call, or even a defensible call. It’s something that should have stopped years ago. If you’re still doing it, you’re creating problems for yourself.

Even worse, you’re creating problems for security professionals in general. There’s a very real problem with our community spreading fear, and even those of us who have been pushing back against it have to deal with the perception that our community thrives on FUD.

If you’ve been making this claim, your best move is to start repudiating it. Get ahead of the curve before it hits you. Or polish up your resume. Maybe better to do both.

Terry Sweeny is right. Hacker attacks won’t hurt your company brand. And claims that they do hurt security’s brand.

[Update: I've responded to two classes of comments in "Requests for a proof of non-existence" and "A critique of Ponemon Institute methodology for “churn”." Russell has added an "in-depth critique of Ponemon’s method for estimating ‘cost of data breach’."]

I’m tempted to claim this as a nail in the coffin for the insider as the most important threat vector, but of late, I’ve decided that the insider is an near-unkillable boogeyman, and so ‘nails in the coffin’ is the wrong metaphor. Really, this just indicates that references to insiders are a best practice, and we can’t kill them. We can, however, treat those references as an indicator that the person speaking is probably not an empiricist, and discount appropriately.

]]> http://newschoolsecurity.com/2011/01/referencing-insiders-is-a-best-practice/feed/ 2 http://newschoolsecurity.com/2010/12/nate-silver-in-the-nyt-a-bayesian-look-at-assange/ http://newschoolsecurity.com/2010/12/nate-silver-in-the-nyt-a-bayesian-look-at-assange/#comments Wed, 15 Dec 2010 12:46:09 +0000 alex http://newschoolsecurity.com/?p=1948 From The Fine Article:

Under these circumstances, then, it becomes more likely that the charges are indeed weak (or false) ones made to seem as though they are strong. Conversely, if there were no political motivation, then the merits of the charges would be more closely related to authorities’ zealousness in pursing them, and we could take them more at face value.

The plan was to blow up the House of Lords during the State Opening of Parliament on 5 November 1605…

…Fawkes, who had 10 years of military experience fighting in the Spanish Netherlands in suppression of the Dutch Revolt, was given charge of the explosives.

The plot was revealed to the authorities in an anonymous letter sent to William Parker, 4th Baron Monteagle, on 26 October 1605. During a search of the House of Lords at about midnight on 4 November 1605, Fawkes was discovered guarding 36 barrels of gunpowder – enough to reduce the House of Lords to rubble – and arrested.

Guy Fawkes day is a celebratory event in the UK with fireworks and bonfires.  It’s also when some of my ex-pat friends stock up on fireworks to ensure they can be suitably obnoxious on the 4th of July, but that’s another story…

So why is it that in England, a failed terror plot has become an excuse to have a party, whereas in the U.S., a failed or thwarted terror plot is  an excuse to strip away Civil Liberties?

Richard Bejtlich has a post responding to an InformationWeek article written by Michael Healey, ostensibly about end user security.  Richard  upbraids Michael for writing the following:

Too many IT teams think of security as their trump card to stop any discussion of emerging tech deemed too risky…

Are we really less secure than we were 10 years ago? Probably not…

…security folks are so jumpy. But they’re missing the message that CIOs need to hear: Security is working. It’s been more than a decade (yes, 10 years) since any particular security flaw has had a truly widespread impact. The Melissa and the ILoveYou attacks were the last.

Now Richard dresses down Mike regarding his naivete’ about the threat landscape.  Using Melissa and ILoveYou as examples of aggregate risk to Internet participation is of course, silly.  But the lesson doesn’t stop there. Michael,even if your organization hasn’t had a recent, significant breach – there’s very little evidence to suggest that this is because “it’s working”.   It could very well be “good luck” based on a lack of frequency (in threat actions). Think of it this way, while I’m sure there are parts of Oklahoma that haven’t been hit by a Tornado in recorded history, that doesn’t mean that I’d move into a mobile home there.

Let me also pile on by mentioning that the Verizon DBIR data set shows a significant uptick in the use of custom malware by threat agents (you know, the kind designed to evade signature based defenses) in data breaches.

Speaking of which, let me share with you a few thoughts on impact and loss.  In the past four years, we can account for nearly a billion records (credit cards and other PII) known to be compromised.  And that’s *just* the Verizon/Secret Service data.  You could probably increase that number by 12 figures by including data at risk and lost from the DLDB.  Being a journalist, I’m sure you’ll recall that here in the US have had significant IP and military secret losses, as well.

Finally Mike, there’s the problem of trying to keep up with the threat landscape.  Take Gunnar’s excellent table around web security as an example:

Do we really need to say anymore?