Continuous Deployment and Security

From an operations and security perspective, continuous deployment is either the best idea since sliced bread or the worst idea since organic spray pancakes in a can. It’s all of matter of execution. Continuos deployment is the logical extension of the Agile development methodology. Adam recently linked to an study that showed that a 25% increase in features lead to a 200% increase in code complexity, so by making change sets smaller we dramatically  decrease the complexity in each release. This translates to a much lower chance of failure.  Smaller change sets also mean that rolling back in the case of a failure state is also much easier. Finally, smaller change sets make identifying what broke unit and integration tests easier and far easier to code review which increases the chances of catching serious issues prior to deployment. All of this points to building systems that are more stable, more reliable, have less downtime and are easier to secure. This assumes, of course, that you are doing continuos deployment well.

In order for continuous deployment (and DevOps in general) to be successful there needs to be consistent process and automation. There are lots of other factors as well, such as qualified developers, proper monitoring, the right deployment tools but those are for another discussion.

Consistent processes are essential if you are to guarantee that the deployment happens the same way every time. To put it bluntly, when it comes to operations and security, variation is evil. Look to Gene Kim’s research (Visual Ops, Visual Ops Security) or more traditional manufacturing methodologies like Six-Sigma for a deep dive into why variation is so very very bad. The short version though is that in manufacturing, variation  means products you can’t sell. In IT, variation means downtime, performance issues, and security issues. At the most basic level, if you are making changes and you are making changes to how you make the changes, you create a much harder situation from which to troubleshoot. This translates to longer incident response times and longer times to recovery which nobody wants. Especially in an online business.

The easiest way to keep deployment process consistent is to remove the human element as much as possible. In other words, automate as much it as possible. This has the added advantage of saving the humans for reviewing errors and identifying potential issues faster. It doesn’t matter which automation mechanism you use as long as it’s stable and supports your operating environment well. Ideally, it will either be the same system as currently being used the by the operations and applications teams (e.g. chef, puppet, cfengine) or be one that can integrated with those systems (e.g. hudson/jenkins).

With good check-in/build release messages, you even get automated logging for your change management systems and updates to your configuration management database (CMDB).

Please vote New School

We’re honored to be nominated in three categories for the Security Bloggers Awards:

  • Most Educational
  • Most Entertaining
  • Hall of Fame

On behalf of all of us who blog here, we’re honored by the nomination, and would like to ask for your vote.

We’d also like to urge you to vote for our friends at Securosis for “Best Representing the Security Industry.” We don’t think Securosis actually is the best representative of the industry today. But I think they represent what we all ought to aspire to be, a empirical, business-aware industry. So please consider them as a part of the broad “New School” sort of slate. We’d also like to put a word in for the ThreatPost podcast as a great mix of technical and non-technical content, and for Veracode for best corporate blog. We’re suggesting Veracode in large part for Chris Eng’s empirical and side-splittingly funny thought leadership videos, but also for a general avoidance of FUD in their blogging.

But whomever you like, please take a moment to vote.

The New School of Software Engineering?

This is a great video about how much of software engineering runs on folk knowledge about how software is built:

Greg Wilson – What We Actually Know About Software Development, and Why We Believe It’s True

There’s a very strong New School tie here. We need to study what’s being done and how well it works to figure out how to make better software more reliably.


Incidentally, at around 28 minutes in, Wilson mentions Nachi Nagappan‘s work on physical distance versus managerial distance, and then jumps to remote hires at a a startup. While I’m not sure of which paper Wilson is discussing, almost all of Nagappan’s work is done with Microsoft developers and products. As such, both have to be seen in the context of Microsoft’s deep and shared experience in shipping software. By definition, that shared experience doesn’t exist at a startup. And as to the managerial distance issue, it’s satirically discussed here. Assuming that his results generalize is a large jump, and one that I’m not sure I’d make.

New School Approaches to Passwords

Adam Montville left a comment on my post, “Paper: The Security of Password Expiration“, and I wanted to expand on his question:

Passwords suck when they’re not properly cared for. We know this. Any other known form of
authentication we have is difficult because of the infrastructure required to pull it off. That
sucks too. Does this leave us at a stalemate where we need to get people to care about their
passwords?

I think the answer is “almost.” We need to agree that passwords suck when they’re not properly cared for, and that caring for them is hard. So we need to assume that passwords will tend to be poor, reused, etc, and develop methods to deal with that. Most of our mechanisms today punish users. We tell them to memorize 100 or more unique passwords, and then “security experts” abuse them for re-use or using a password management tool.

Cormac Herley has claimed that the password has a set of properties including being subject to memorization that make it impossible to replace, and we should accept that and start engineering for it. (“A Research Agenda Acknowledging the Persistence of Passwords” and “Passwords: If We’re So Smart Why Are We Still Using Them?“)

Similarly, Nate Lawson posted “On the evolving security of password schemes” which closes “most admins focus too much on increasing entropy of user choices and not enough on decreasing the attacker’s guess rate and implementing responses to limit their access when they do get a hit.” Indeed.

We need to observe the world, and ask how we can work within the constraints it presents regardless of if those constraints are economic, sociological or evolutionary.

How to Send Adam into Hysterics

Via Nathan Yau’s awesome Flowing Data blog.

Paper: The Security of Password Expiration

The security of modern password expiration: an algorithmic framework and empirical analysis, by Yingian Zhang, Fabian Monrose and Michael Reiter. (ACM DOI link)

This paper presents the first large-scale study of the success of password expiration in meeting its intended purpose, namely revoking access to an account by an attacker who has captured the account’s password. Using a dataset of over 7700 accounts, we assess the extent to which passwords that users choose to replace expired ones pose an obstacle to the attacker’s continued access. We develop a framework by which an attacker can search for a user’s new password from an old one, and design an efficient algorithm to build an approximately optimal search strategy. We then use this strategy to measure the difficulty of breaking newly chosen passwords from old ones. We believe our study calls into question the merit of continuing the practice of password expiration.

This is the sort of work that we at the New School love. Take a best practice recommended by just about everyone for what seems like excellent reasons, and take notice of the fact that human beings are going to game your practice. Then get some actual data, and see how effective the practice is.

Unfortunately, we lack data on rates of compromise for organizations with different password change policies. So it’s hard to tell if password policies actually do any good, or which ones do good. However, we can guess that not making your default password “stratfor” is a good idea.

ACM gets a link because they allow you to post copies of your own papers, rather than inhibiting the progress of science by locking it all up.

Steve Bellovin’s “Lessons from Suppressing Research”

Steve Bellovin has a good deal of very useful analysis and context about “an experiment that showed that the avian flu strain A(H5N1) could be changed to permit direct ferret-to-ferret spread. While the problem the government is trying to solve is obvious, it’s far from clear that suppression is the right answer, especially in this particular case.”

Steve’s post contains excellent context, putting the issue in context of nuclear secrets, cryptography and software vulnerability disclosure. I want to follow up a bit on his closing:

The ultimate decision may rest on personal attitudes. To quote Fouchier one more time, “The only people who want to hold back are the biosecurity experts. They show zero tolerance to risk. The public health specialists do not have this zero tolerance. I have not spoken to a single public health specialist who was against publication.”

I think that personal preference is one way to think of this, and perhaps in fact, personal preference drives the choice of profession. But perhaps what’s really happening is that public health specialists are operating with a different set of drivers than “biosecurity experts.” In particular, given the very low incidence of ‘biosecurity incidents,’ perhaps ‘biosecurity experts’ are operating in a world where all threats exist only on paper (or in papers). In contrast, public health professionals have real epidemics and pandemics to deal with. They’re forced to deal with the propaganda of anti-vaccination nuts whose fear of autism is killing people with whooping cough and other diseases. They have to deal with contamination of the food supply. They can reasonably prioritize preventing salmonella or e.coli over theoretical terrorist threats.

However, this narrow focus on preventing all problems (in contrast to risk management, cost-benefit or other pragmatic approaches) is not unique to bio-security experts. The security professional, focused by definition on security, will naturally tend towards zero tolerance for risks.

An example, already reduced to absurdity, is visible in the TSA. Their goal is not balanced security, it’s a relentless and offensive pursuit of security at the expense of dignity, calm, and cupcakes. But we should not be surprised at their pursuit of the cupcake. It’s the natural result of having an agency focused entirely on security.

This is, by the way, relates to why CISOs should report into a functional area of the business, be it operations or IT, rather than reporting to the CEO. If the CISO is focused entirely on security, then those concerns need to be balanced with the overall operational picture by someone with accountability for delivering of a whole to the business, not treated as some special magic.

New podcast with Dave Birch

I really enjoyed a conversation with Dave Birch for Consult Hyperion’s “Tomorrow’s Transactions” podcast series. The episode is here. We covered the New School, lessons learned from Zero-Knowledge Systems, and games for security and privacy.

Discussing Norm Marks’ GRC Wishlist for 2012

Norm Marks of the famous Marks On Governance blog has posted his 2012 wishlist.  His blog limits the characters you can leave in a reply, so I thought I’d post mine here.

1.  Norm Wishes for “A globally-accepted organizational governance code, encompassing both risk management and internal control”

Norm, if you mean encompassing both so that they are tightly coupled, I respectfully disagree.  Ethically, philosophically, these should be separate entities and ne’r the twain should meet.  Plus, accountants & auditors make poor actuaries.  See the SoA condemnation of RCSA.

Second, the problem with a globally accepted something is that it limits innovation.  We already have enough of this “we can’t do things right because we’ll have to justify doing things differently than the global priesthood says we have to” problem to deal with now.  Such documentation will only exacerbate the issue.

2.  Norm wishes for: “The convergence of the COSO ERM Framework and the global ISO 31000:2009 risk management standard.”

See #1, part 2 above.

3.  Norm wishes for:  “An update of the COSO Internal Control Framework that recognizes that internal controls are the organization’s response to uncertainty (i.e., risk), and you need the controls to ensure the likelihood and effects of uncertainty are within organizational tolerances.”

First, risk only equals uncertainty if you’re one of those stuck in the early 20th century Knightians.  For those that aren’t, and esp. actuaries and Bayesians alike, uncertainty is a factor in risk analysis – not the existence of risk.

Second, this wish seems to be beholden to the fundamental flaw of the Accounting Consultancy Industrial Complex – that Residual Risk = Inherent risk – Controls.  Let me ask you, what controls do you personally have against an asteroid slamming into your house?  But is that “high” risk?  Do you operate daily as if it’s “high” risk?  Why not?  Certainly you have weak controls, and most people would argue that their house and familys are of high value…

The reason it’s not “high risk” is because of frequency.  Yes, frequency matters in risk – and your RCSA process doesn’t (usually, formally) account for that.

4.) Norm wants “guidance that explains how you can set guidance on risk-taking that works not only for (a) the board and top management (who want to set overall limits), but also for (b) the people on the front lines who are the ones actually making decisions, accepting risks, and taking actions to manage the risks. The guidance also has to explain the need to measure and report on whether the actions taken on the front lines aggregate to levels within organizational tolerances.”

Great idea, but for this one to work, you’d have to establish guidance around reward-taking, tolerance, etc., too.

5.) Norm wants “A change to the opinion provided by the external auditors, from one focusing on compliance with GAAP to one focusing on whether the financial reports filed with the regulators provide a true and fair view of the results of operations, the condition of the organization, and the outlook for the future.”

I’m going with “bad idea” on this one.  Accountants != entrepreneurs.  Despite all their longing for control, power, and self-importance.

6.)  For Norm, Regulators should receive ” An opinion by management on the effectiveness of the enterprise-wide risk management program. This could be based on the assessment of the internal audit function”

I’m confused, how is the internal audit function in any way at all related to the quality of decision making?  Assurance is *an* evidence, a confidence value for specific risk factors.  It seems that Norm is saying that assurance is *the* evidence in total.

Frankly, very few accountants have training or exposure to probability theory, decision theory, or complexity theory.  Until they *do*, my wish for 2012 is that CPAs  reserve judgement on people trying to use real methods to solve real problems.

7.) Norm wants:  A change in attitude of investor groups, focusing on longer-term value instead of short-term results.

AGREED and +1 to you Norm!

In 10.) a, Norm desires that “audit engagements should be prioritized based on the risk to the organization and the value provided by an internal audit project.”

ABSOLUTELY NOT.  Unless Audit engagements are to be prioritized by the faulty idea of “Inherent Risk”.

Example, as a risk manager – I may have relatively stable frequency and magnitude of operational losses.  They may fall into a “low” tolerance range established by an ERMC or something.  But even though I am doing a good job (or really lucky) I may really be concerned about the process enough to warrant a high frequency of audit.  There are just so many concerns about this sort of approach by an auditor (from a risk/actuary standpoint) that I can’t disagree more.

In point 11 Norm’s wish is “An improved understanding by the board and top management of the value of internal audit as a provider of assurance relative to governance, risk management”

Me too, but I don’t think Norm and I agree on that “value.”

Again, for a mature risk management group, the value of assurance is simply the establishment of confidence values for certain inputs.  And frankly, if the board and top management understood that, I’m not sure Norm would really want them too, because many times the assurance is really a reinforcement of confidence/certainty, and frankly is a job that can easily be done with a risk model that reduces SME bias.

Finally, Norm “would like to see the term “GRC” disappear”

AMEN.  To use the ISACA/Audit terminology, Compliance is just “a risk.”  To use risk terminology, Compliance is a factor that contributes to secondary or indirect losses.

So, I’m with you – I’d like to see GRC taken out behind the shed.  Where I differ is that’s not because it becomes coupled with risk management, but rather because for me compliance aligns better with the authoritarian world of audit rather than a discipline like risk whose goal is to reduce subjectivity, or a discipline like governance whose role is to optimize resource expenditures.

The New School of Security Predictions

Bill Brenner started it with “Stop them before they predict again!:”

My inbox has been getting hammered with 2012 vendor security predictions since Halloween. They all pretty much state the obvious:

  • Mobile malware is gonna be a big deal
  • Social networking will continue to be riddled with security holes
  • Technologies A, B and C will be dead
  • Microsoft will release a lot of security patches
  • Data security breaches will continue to get more expensive

Looking at the predictions I got this time last year for 2011, I found that any of them could be repackaged as 2012 predictions and nobody would know the difference. Here are some examples from the Zscaler Labs Research Team…

Jack Daniel followed up with “The Pandering Pentagram of Prognostication :”

The five points of the pentagram represent the key elements of “good” predictions, get them all and your prediction will land in the center of the pentagram, assuring a center brain shot to your victim. I mean reader. Whatever.

The five elements are outlined below, miss even one and your prediction may be off target and you will fail to hit your target.

  • Your prediction must be self-serving.
  • Your prediction must suck up to your customers, prospects, or others whose favor you are trying to win…

I’ll respond with a prediction that 90% of 2012 infosec predictions will contain no numbers and no dates. If someone selects a group of 10 or more predictors (say, bloggers in SBN, or 2011 BlackHat speakers with blogs) and proves me wrong, I’ll donate $100 to a charity of your

Both Bill and Jack are helping the community by pointing out the “best practices in predictions” so that people can recognize them for the self-serving (ad-serving) linkbait that most of them are.


To get something positive out of this, I encourage everyone to ask anyone who sends you predictions about the lack of underlying data.