<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>The New School of Information Security &#187; data</title>
	<atom:link href="http://newschoolsecurity.com/category/data/feed/" rel="self" type="application/rss+xml" />
	<link>http://newschoolsecurity.com</link>
	<description>The Blog Inspired By The Book</description>
	<lastBuildDate>Fri, 11 May 2012 16:20:19 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3.2</generator>
		<item>
		<title>Why Sharing Raw Data is Important</title>
		<link>http://newschoolsecurity.com/2012/05/why-sharing-raw-data-is-important/</link>
		<comments>http://newschoolsecurity.com/2012/05/why-sharing-raw-data-is-important/#comments</comments>
		<pubDate>Fri, 11 May 2012 16:20:18 +0000</pubDate>
		<dc:creator>adam</dc:creator>
				<category><![CDATA[data]]></category>
		<category><![CDATA[Data Analysis]]></category>
		<category><![CDATA[Reports and Data]]></category>

		<guid isPermaLink="false">http://newschoolsecurity.com/?p=2600</guid>
		<description><![CDATA[Bob Rudis has a nice post up &#8220;Off By One : The Importance Of Fact Checking Breach Reports,&#8221; in which he points out some apparent errors in the Massachusetts 2011 breach report, and also provides some graphs. Issues like this are why it&#8217;s important to release data. It enables independent error checking, but also allows [...]]]></description>
			<content:encoded><![CDATA[<p>Bob Rudis has a nice post up &#8220;<a href="https://rud.is/b/2012/05/10/off-by-one-the-importance-of-fact-checking-breach-reports/">Off By One : The Importance Of Fact Checking Breach Reports</a>,&#8221; in which he points out some apparent errors in the Massachusetts 2011 breach report, and also provides some graphs.</p>
<p>
Issues like this are why it&#8217;s important to release data.  It enables independent error checking, but also allows people to slice and dice the issues in ways that otherwise are only accessible to a privileged few with the raw numbers.</p>
<p>
]]></content:encoded>
			<wfw:commentRss>http://newschoolsecurity.com/2012/05/why-sharing-raw-data-is-important/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>Checklists and Information Security</title>
		<link>http://newschoolsecurity.com/2012/04/checklists-and-information-security/</link>
		<comments>http://newschoolsecurity.com/2012/04/checklists-and-information-security/#comments</comments>
		<pubDate>Tue, 10 Apr 2012 15:57:59 +0000</pubDate>
		<dc:creator>adam</dc:creator>
				<category><![CDATA[best practice]]></category>
		<category><![CDATA[compliance]]></category>
		<category><![CDATA[data]]></category>
		<category><![CDATA[disclosure]]></category>

		<guid isPermaLink="false">http://newschoolsecurity.com/?p=2589</guid>
		<description><![CDATA[I&#8217;ve never been a fan of checklists. Too often, checklists replace thinking and consideration. In the book, Andrew and I wrote: CardSystems had the required security certification, but its security was compromised, so where did things goo wrong? Frameworks such as PCI are built around checklists. Checklists compress complex issues into a list of simple [...]]]></description>
			<content:encoded><![CDATA[<p>I&#8217;ve never been a fan of checklists.   Too often, checklists replace thinking and consideration.  In the book, Andrew and I wrote:</p>
<blockquote><p>
CardSystems had the required security certification, but its security was compromised, so where did things goo wrong?  Frameworks such as PCI are built around checklists.  Checklists compress complex issues into a list of simple questions.  Someone using a checklist might therefore think he had done the right thing, when in fact he had not addressed the problems in depth&#8230;Conventional wisdom presented in short checklists makes security look easy.
</p></blockquote>
<p>So it took a while and a lot of recommendations for me to get around to reading &#8220;<a href="http://www.amazon.com/gp/product/0312430000/ref=as_li_ss_tl?ie=UTF8&#038;tag=httpwwwemer04-20&#038;linkCode=as2&#038;camp=1789&#038;creative=390957&#038;creativeASIN=0312430000">The Checklist Manifesto</a>&#8221; by Atul Gawande.  And I&#8217;ll admit, I enjoyed it.  It&#8217;s a very well-written, fast-paced little book that&#8217;s garnered a lot of fans for very good reasons.</p>
<p>
What&#8217;s more, much as it pains me to say it, I think that security can learn a lot from the Checklist Manifesto.  One objection that I&#8217;ve had is that security is simply too complex.  But so is the human body.  From the Manifesto:</p>
<blockquote><p>[It] is far from obvious that something as simple as a checklist could be of substantial help. We may admit that errors and oversights occur&#8211;even devastating ones. But we believe our jobs are too complicated to reduce to a checklist. Sick people, for instance, are phenomenally more various than airplanes. A study of forty-one thousand trauma patients in the state of Pennsylvania&#8211;just trauma patients&#8211;found that they had 1,224 different injury-related diagnoses in 32,261 unique combinations. That&#8217;s like having 32,261 kinds of airplane to land. Mapping out the proper steps for every case is not possible, and physicians have been skeptical that a piece of paper with a bunch of little boxes would improve matters.
</p></blockquote>
<p>The Manifesto also addresses the point we wrote above, that &#8220;someone using a checklist might think he&#8217;d done the right thing&#8221;:</p>
<blockquote><p>
Plus, people are individual in ways that rockets are not&#8211;they are complex. No two pneumonia patients are identical. Even with the same bacteria, the same cough and shortness of breath, the same low oxygen levels, the same antibiotic, one patient might get better and the other might not. A doctor must be prepared for unpredictable turns that checklists seem completely unsuited to address. Medicine contains the entire range of problems&#8211;the simple, the complicated, and the complex&#8211;and there are often times when a clinician has to just do what needs to be done. Forget the paperwork. Take care of the patient.
</p></blockquote>
<p>So it&#8217;s important to understand that checklists don&#8217;t replace professional judgement, they supplement it and help people remember complex steps under stress.</p>
<p>
So while I think security can learn a lot from The Checklist Manifesto, the lessons may not be what you expect.  Quoting the book that inspired this blog again:</p>
<blockquote><p>
A checklist implies that there is an authoritative list of the &#8220;right&#8221; things to do, even if no evidence of that simplicity exists.  This in turn contributes to the notion that information security is a more mature discipline than it really is.</p></blockquote>
<p>For example, turning back to the Manifesto:</p>
<blockquote><p>Surgery has, essentially, four big killers wherever it is done in the world: infection, bleeding, unsafe anesthesia, and what can only be called the unexpected. For the first three, science and experience have given us some straightforward and valuable preventive measures we think we consistently follow but don&#8217;t.
</p></blockquote>
<p>I think what we need, before we get to checklists, is more data to understand what the equivalents of infection, bleeding and unsafe anesthesia are.  Note that those categories didn&#8217;t spring out of someone&#8217;s mind, thinking things through from first principles.  They came from data.  And those data show that some risks are bigger than others:</p>
<blockquote><p>But compared with the big global killers in surgery, such as infection, bleeding, and unsafe anesthesia, fire is exceedingly rare. Of the tens of millions of operations per year in the United States, it appears only about a hundred involve a surgical fire and vanishingly few of those a fatality. By comparison, some 300,000 operations result in a surgical site infection, and more than eight thousand deaths are associated with these infections. We have done far better at preventing fires than infections. [So fire risks are generally excluded from surgical checklists.]
</p></blockquote>
<p>Security has no way to exclude <del>insiders</del> the fire risk.  We throw everything into lists like PCI.  The group who updates PCI is not provided in depth incident reports about the failures that occurred over the last year or over the life of the failure.  When  security fails, rather than asking, &#8216;did the checklist work&#8217;, the PCI council declares that they&#8217;ve violated the 11th commandment, and are thus not compliant.  And so we dan&#8217;t improve the checklists.  (Compare and contrast: don&#8217;t miss the long section of the Manifesto on how Boeing tests and re-tests their checklists.)</p>
<p>
One last quote before I close.  Gawande surveys many fields, including how large buildings are built and delivered.  He talks to a project manager putting up a huge new hospital building:</p>
<blockquote><p>
Joe Salvia had earlier told me that the major advance in the science of construction over the last few decades has been the perfection of tracking and communication.
</p></blockquote>
<p>Nothing for us security thought leaders to learn.  But before I tell you to move along, I&#8217;d like to offer up an alpha-quality DO-CHECK checklist for improving security after an incident:</p>
<ol>
<li>Have you addressed the breach and gotten the attackers out?
<li>Have you notified your customers, shareholders, regulators and other stakeholders?
<li>Did you prepare an after-incident report?
<li>Did you use Veris, the taxonomy in Microsoft&#8217;s SIR v11  or some other way to clarify ambiguous terms?
<li>Have you released the report so others can learn?
</ol>
<p>I believe that if we all start using such a checklist, we&#8217;ll set up a feedback loop, and empower our future selves to make better, and more useful checklists to help us make things more secure.</p>
<p>
]]></content:encoded>
			<wfw:commentRss>http://newschoolsecurity.com/2012/04/checklists-and-information-security/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
		<item>
		<title>Time for an Award for Best Data?</title>
		<link>http://newschoolsecurity.com/2012/02/time-for-an-award-for-best-data/</link>
		<comments>http://newschoolsecurity.com/2012/02/time-for-an-award-for-best-data/#comments</comments>
		<pubDate>Wed, 01 Feb 2012 17:15:54 +0000</pubDate>
		<dc:creator>adam</dc:creator>
				<category><![CDATA[data]]></category>
		<category><![CDATA[disclosure]]></category>
		<category><![CDATA[Reports and Data]]></category>
		<category><![CDATA[research papers]]></category>

		<guid isPermaLink="false">http://newschoolsecurity.com/?p=2489</guid>
		<description><![CDATA[Yesterday, DAn Kaminsky said &#8220;There should be a yearly award for Best Security Data, for the best collection and disbursement of hard data and cogent analysis in infosec.&#8221; I think it&#8217;s a fascinating idea, but think that a yearly award may be premature. However, what I think is sorta irrelevant, absent data. So I&#8217;m looking [...]]]></description>
			<content:encoded><![CDATA[<p>Yesterday, DAn Kaminsky said &#8220;<a href="https://twitter.com/#!/dakami/status/164424568088444928">There should be a yearly award for Best Security Data, for the best collection and disbursement of hard data and cogent analysis in infosec.</a>&#8221;   I think it&#8217;s a fascinating idea, but think that a yearly award may be premature.  However, what I think is sorta irrelevant, absent data.  So I&#8217;m looking for data on the question, do we have enough good data to issue an award yearly?</p>
<p>
Please nominate in the comments.</p>
<p>
Also, please discuss what the criteria should be.</p>
<p>
]]></content:encoded>
			<wfw:commentRss>http://newschoolsecurity.com/2012/02/time-for-an-award-for-best-data/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
		<item>
		<title>Paper: The Security of Password Expiration</title>
		<link>http://newschoolsecurity.com/2012/01/paper-the-security-of-password-expiration/</link>
		<comments>http://newschoolsecurity.com/2012/01/paper-the-security-of-password-expiration/#comments</comments>
		<pubDate>Thu, 05 Jan 2012 16:19:14 +0000</pubDate>
		<dc:creator>adam</dc:creator>
				<category><![CDATA[data]]></category>
		<category><![CDATA[research papers]]></category>

		<guid isPermaLink="false">http://newschoolsecurity.com/?p=2433</guid>
		<description><![CDATA[The security of modern password expiration: an algorithmic framework and empirical analysis, by Yingian Zhang, Fabian Monrose and Michael Reiter. (ACM DOI link) This paper presents the first large-scale study of the success of password expiration in meeting its intended purpose, namely revoking access to an account by an attacker who has captured the account&#8217;s [...]]]></description>
			<content:encoded><![CDATA[<p><a href="http://cs.unc.edu/~fabian/papers/PasswordExpire.pdf">The security of modern password expiration: an algorithmic framework and empirical analysis</a>, by Yingian Zhang, Fabian Monrose and Michael Reiter.  (ACM DOI <a href="http://dl.acm.org/citation.cfm?id=1866328">link</a>)</p>
<blockquote><p>
This paper presents the first large-scale study of the success of password expiration in meeting its intended purpose, namely revoking access to an account by an attacker who has captured the account&#8217;s password. Using a dataset of over 7700 accounts, we assess the extent to which passwords that users choose to replace expired ones pose an obstacle to the attacker&#8217;s continued access. We develop a framework by which an attacker can search for a user&#8217;s new password from an old one, and design an efficient algorithm to build an approximately optimal search strategy. We then use this strategy to measure the difficulty of breaking newly chosen passwords from old ones. We believe our study calls into question the merit of continuing the practice of password expiration.
</p></blockquote>
<p>This is the sort of work that we at the New School love.  Take a best practice recommended by just about everyone for what seems like excellent reasons, and take notice of the fact that human beings are going to game your practice.  Then get some actual data, and see how effective the practice is.</p>
<p>
Unfortunately, we lack data on rates of compromise for organizations with different password change policies.  So it&#8217;s hard to tell if password policies actually do any good, or which ones do good.  However, we can guess that not making your default password &#8220;stratfor&#8221; is a good idea.</p>
<p>ACM gets a <a href="http://dl.acm.org/citation.cfm?id=1866328">link</a> because they allow you to post copies of your own papers, rather than inhibiting the progress of science by locking it all up.</p>
]]></content:encoded>
			<wfw:commentRss>http://newschoolsecurity.com/2012/01/paper-the-security-of-password-expiration/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>Top 5 Security Influencers of 2011</title>
		<link>http://newschoolsecurity.com/2011/12/top-5-security-influencers-of-2011/</link>
		<comments>http://newschoolsecurity.com/2011/12/top-5-security-influencers-of-2011/#comments</comments>
		<pubDate>Mon, 12 Dec 2011 15:51:47 +0000</pubDate>
		<dc:creator>adam</dc:creator>
				<category><![CDATA[careers]]></category>
		<category><![CDATA[data]]></category>

		<guid isPermaLink="false">http://newschoolsecurity.com/?p=2361</guid>
		<description><![CDATA[I really like Gunnar Peterson&#8217;s post on &#8220;Top 5 Security Influencers:&#8221; Its December and so its the season for lists. Here is my list of Top 5 Security Influencers, this is the list with the people who have the biggest (good and/or bad) influence on your company and user&#8217;s security: My list is slightly different: [...]]]></description>
			<content:encoded><![CDATA[<p>I really like Gunnar Peterson&#8217;s post on &#8220;<a href="http://1raindrop.typepad.com/1_raindrop/2011/12/top-5-security-influencers.html">Top 5 Security Influencers</a>:&#8221;</p>
<blockquote><p>
Its December and so its the season for lists. Here is my list of Top 5 Security Influencers, this is the list with the people who have the biggest (good and/or bad) influence on your company and user&#8217;s security:
</p></blockquote>
<p>My list is slightly different:</p>
<ol>
<li>The Person Coding Your App
<li>Your DBA
<li>Your Testers
<li>Your Ops team
<li>The person with the data
<li>Uma Thurman
<li>You
</ol>
<p>That&#8217;s right, without data to argue an effective case for investing in security, you have less influence than Uma Thurman.  And even if you have more influence than her, if you want to be in the top 5, you better be the person bringing the data.</p>
<p>
As long as we&#8217;re hiding everything that might allow us to judge comparative effectiveness, we&#8217;re going to continue making no progress.<br />
<P><br />
<b>Ahh, but which Uma?</b><br />
<img src="http://newschoolsecurity.com/wp-content/uploads/2011/12/265446.1020.A1.jpg" alt="265446 1020 A" title="265446.1020.A.jpg" border="0" width="202" height="299" style="float:right;" /><br />
Update: Chris Hoff asks &#8220;<a href="https://twitter.com/#!/Beaker/status/147733651428347904">But WHICH Uma? Kill Bill Uma or Pulp Fiction Uma?</a>&#8221; and sadly, I have to answer: The Truth About Cats and Dogs Uma.  You remember. Silly romantic comedy where guy falls in love with radio veterinarian Janeane Garofalo, who&#8217;s embarrassed about her looks?  And Uma plays her gorgeous but vapid neighbor?  That&#8217;s the Uma with the more influence than you.  The one who spends time trying to not be bubbly when her audition for a newscaster job leads off with &#8220;hundreds of people feared dead in a nuclear accident?&#8221;  Yeah.  That Uma.  Because at least she&#8217;s nice to look at while going on about stuff no one cares about.  But you know? If you show up with some chops and some useful data to back your claims, you can do better than that.</p>
<p>On the downside, you&#8217;re unlikely to ever be as influential as Kill Bill Uma.  Because, you know, she has a sword, and a demonstrated willingness to slice the heads off of people who argue with her, and a don&#8217;t-care attitude about jail.  It&#8217;s hard to top that for short term influence.  Just ask the 3rd guy trying to code your app, and hoping it doesn&#8217;t crash.  He&#8217;s got eyes for no one not carrying that sword.</p>
<p>
]]></content:encoded>
			<wfw:commentRss>http://newschoolsecurity.com/2011/12/top-5-security-influencers-of-2011/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>Fixes to Wysopal’s Application Security Debt Metric</title>
		<link>http://newschoolsecurity.com/2011/03/fixes-to-wysophal%e2%80%99s-application-security-debt-metric/</link>
		<comments>http://newschoolsecurity.com/2011/03/fixes-to-wysophal%e2%80%99s-application-security-debt-metric/#comments</comments>
		<pubDate>Sat, 05 Mar 2011 09:47:27 +0000</pubDate>
		<dc:creator>Russell</dc:creator>
				<category><![CDATA[data]]></category>
		<category><![CDATA[Data Analysis]]></category>
		<category><![CDATA[metrics]]></category>
		<category><![CDATA[Science of Risk Management]]></category>

		<guid isPermaLink="false">http://newschoolsecurity.com/?p=2099</guid>
		<description><![CDATA[In two recent blog posts (here and here), Chris Wysopal (CTO of Veracode) proposed a metric called “Application Security Debt”.  I like the general idea, but I have found some problems in his method.  In this post, I suggest corrections that will be both more credible and more accurate, at least for half of the [...]]]></description>
			<content:encoded><![CDATA[<p>In two recent blog posts (<a href="http://www.veracode.com/blog/2011/02/application-security-debt-and-application-interest-rates/" target="_blank">here</a> and <a href="http://www.veracode.com/blog/2011/03/a-financial-model-for-application-security-debt/" target="_blank">here</a>), Chris Wysopal (CTO of Veracode) proposed a metric called “Application Security Debt”.  I like the general idea, but I have found some problems in his method.  In this post, I suggest corrections that will be both more credible and more accurate, at least for half of the formula.  The second half is harder to do right and needs more thinking.</p>
<p><span id="more-2099"></span><span style="font-weight: bold;">Overview</span></p>
<p>Application Security Debt is based on the concept of  “technical debt” proposed by Ward Cunningham (a programmer who developed the first wiki program): describes it like this:</p>
<blockquote><p>Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite… The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.</p></blockquote>
<p>Chris adds:</p>
<blockquote><p>The cost of technical debt is the time and money it will take to rewrite the poor code after you ship and bring it back to the quality required to maintain the software over the long haul.</p></blockquote>
<p>Here is Chris’ summary of <strong>Application Security Debt</strong>:</p>
<blockquote><p>Security debt is similar to technical debt. Both debts are design and implementation constructions that have negative aspects that aggregate over time and the code must be re-worked to get out of debt. Security debt is based on the latent vulnerabilities within an application. Application interest rates are the real world factors outside of the control of the software development team that lead to vulnerabilities having real cost. These factors include the cost of a security breach and attacker motivation to discover and exploit the latent vulnerabilities.</p></blockquote>
<p>Chris’ <a href="http://www.veracode.com/blog/2011/03/a-financial-model-for-application-security-debt/" target="_blank">second post</a> describes a financial model that estimates the cost of Application Security Debt.  Framing the metric in financial terms will presumably help managers compare the cost of the “debt” to the cost of developing more secure software or costs of fixing the vulnerabilities.  (Note: Veracode provides a range of <a href="http://www.veracode.com/solutions/application-security-testing.html" target="_blank">application security testing services</a>, so they have an interest in economically justifying their services.  This isn’t a criticism of Veracode, Chris, or his proposal.  Just a reality.)</p>
<p>Chris’ model is focused on the simplest case where the application developer and application user is the same organization, so that it bears the costs of development, maintenance, and also any security breaches that result.  Starting with the simplest case is a great idea when proposing a new method.  So far so good.</p>
<p>Chris defines his financial model this way:</p>
<blockquote><p>The basic financial model for security debt is monetary risk that can be expressed as <em>expected loss</em>. The formula for expected loss is <strong>event likelihood X impact in dollars</strong>. Event likelihood is based on the makeup of vulnerabilities in the application and the likelihood that the vulnerabilities will be discovered and exploited. The impact is the cost of a security breach based on an exploit of one of those vulnerabilities.  [Emphasis in original]</p></blockquote>
<p>This is, of course, a version of the bottom-up Annualized Loss Expectancy (ALE) formula for individual risk elements:</p>
<ul>
<li>ALE = Single Loss Expectancy X Annual Rate of Occurrence</li>
</ul>
<p>(Mike Rothman recently <a href="http://securosis.com/blog/firestarter-risk-metrics-are-crap" target="_blank">crapped on all “risk metrics”</a> by lumping them all into the ALE formula.  I’ll critique ALE and Mike’s post in a separate blog post.)</p>
<p>ALE issues aside, I think Chris is making mistakes in his definition of Application Security Debt that will lead to serious confusion.</p>
<h4>Debt = Expected Principal + Interest Costs</h4>
<p>Chris made a mistake when he defines monetary value of the Application Security Debt as expected loss due to security breaches.    Instead, the &#8216;Principal&#8217; part of the debt formula is the cost of fixing security problems beyond what is budgeted. Chris had it right in his summary in the first article:</p>
<blockquote><p>The cost of technical debt is the time and money it will take to rewrite the poor code after you ship and bring it back to the quality required to maintain the software over the long haul.</p></blockquote>
<p>Expected losses are in the category of “Interest Costs” as Chris said in his summary:</p>
<blockquote><p>Application interest rates are the real world factors outside of the control of the software development team that lead to vulnerabilities having real cost.</p></blockquote>
<p>Putting this together in simple language:</p>
<p><em>“Application Security Debt is a ‘loan’ with variable principal which could range from 0% to 100% of your original project costs. The &#8216;principal&#8217; is what you&#8217;ll eventually have to pay to fix security bugs or rewrite the code.  It also has varying and uncertain &#8216;interest costs&#8217;, which are the costs of security breaches due to these vulnerabilities. This includes the possibility of the mother-of-all balloon payments (i.e. a huge loss event).”</em></p>
<p>The good news is that Expected Principal is relatively easy to estimate with good accuracy and without a lot of outside data.  The not-so-good-news is that Interest Cost is a bear to estimate.</p>
<h4>Estimating ‘Expected Principal’</h4>
<p>For simplicity, let’s assume that cost of fixing code (above the budgeted costs) occurs in discrete increments, <em>F</em>:</p>
<ol>
<li>Zero  (i.e. your debt is ‘forgiven’)</li>
<li>Minor fixes and patches (&#8216;Principal&#8217; = 10% increase in project cost)</li>
<li>Major fixes and patches  (&#8216;Principal&#8217; = 25% increase in project cost)</li>
<li>Substantial rewrite (&#8216;Principal&#8217; = 50% increase in project cost)</li>
<li>Total rewrite   (&#8216;Principal&#8217; = 100% increase in project cost, or more)</li>
</ol>
<p>Thus, the best case is that you owe no principal and the worst case is that you owe principal equal to the entire cost of the project.  You could include other factors such as external costs of schedule delays, costs of rehiring your programmers after you fire them all <img src='http://newschoolsecurity.com/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' /> , or what ever.  My point is that these costs are not open-ended, but are a multiplier on your original development costs.</p>
<p>The Expected Principal (EP) is equal to each of these cost scenarios multiplied by their probability of management choosing that option:</p>
<p><a href="http://newschoolsecurity.com/wp-content/uploads/2011/03/EP-formula.png"><img class="aligncenter size-full wp-image-2100" src="http://newschoolsecurity.com/wp-content/uploads/2011/03/EP-formula.png" alt="" width="272" height="130" /></a></p>
<p>For example, if the original cost of the application development project is $1 million, and there is 5% chance of Zero costs, 80% of Minor code fix costs, and 15% chance of Substantial rewrite costs, then the Expected Principal would be $155,000, or 16% of the original cost.</p>
<p><strong>This is important: </strong>Expected Principal is ultimately determined by management decisions and ‘threshold of pain’.  This means that the value of <em>p(F)</em>, above, is a subjective probability.  It would be an ideal metric to estimate using prediction markets (PMs).   (PMs have been used successfully in software development to estimate shipment dates and defect rates, for example.)</p>
<p><strong>Another implication</strong>: you don’t need to accurately forecast future loss events or their economic impact to get a decent estimate of Expected Principal.  Instead, you only need to estimate the Interest Costs very roughly to determine which code fix scenario is most likely.    You could even estimate <em>p(F)</em> by setting thresholds for the number and severity of vulnerabilities discovered by certain levels of effort.  Better, you could combine these methods to ‘triangulate’ on estimates of <em>p(F).</em></p>
<p>To calibrate these subjective probability estimates, it would be <strong>very helpful to collect historical data on the % of applications that have some level of rewrite or schedule delay due to security problems</strong>.  (Hint hint!)</p>
<h4>Estimating ‘Interest Costs’ on the Debt will be Hard</h4>
<p>The second part of the Application Security Debt formula is ‘Interest Costs’.  This is where things get hairy.   All the members of the ALE family of risk calculations have a similar flaws: 1) prodigious data requirements and 2) propagation of uncertainty through the calculations.  Furthermore, some suffer by using only mean values and ignoring extreme values (i.e. the “tails” of the probability distribution curves).</p>
<p>Chris acknowledges these issues, at least the requirement for more and better data:</p>
<blockquote><p>Now you are probably thinking that this is getting a little tenuous and it is. We need better data on likelihood type and likelihood of an application breach by industry and other factors like company size.</p></blockquote>
<p>Data issues aside, I think there are flaws in his use of ALE and calculation methods.  Here’s one thought experiment to show how it could lead to the wrong conclusions, in my opinion.</p>
<p>Let’s use Chris’ ‘baseline expected loss’ table, where he calculates the expected loss for each type of vulnerability.  Imagine that we are comparing two similar applications, A and B.  Assume that each project is expected to have the same number of vulnerabilities, five each.  Let’s say the development cost of each project is $1 million.  Application A has five SQL injection vulnerabilities while application B has one SQL Injection vulnerability and four Remote File Inclusion vulnerabilities.  Doing the calculations:</p>
<ul>
<li>A’s expected losses = $19,220,000</li>
<li>B’s expected losses = $5,074,080</li>
</ul>
<p><em>Does project A really have four times more risk than project B?</em> Probably not.  From what I know, the number of vulnerabilities in an application is not proportional to the likelihood that the application will be breached.  Instead, I’d guess that the likelihood of being breached is a function of where the application is in the IT architecture, how accessible it is, how important it is to attackers, etc.</p>
<p>Also, there’s the ‘weakest link’ effect: “given enough random attackers or one persistent attacker, it only takes one vulnerability to lead to a breach”.  Assuming all SQL Injection vulnerabilities are equally discoverable and equally exploitable, then we should estimate that application B with one SQL Injection vulnerability is just as likely to get breached as application A with five, all other things being equal.</p>
<p>(I confess I’m not an expert in application security or vulnerability analysis, so these comments are my interpretation of what others have written or said.)</p>
<p>Even if my logic here is flawed somewhat, my main point is that the relation between number of vulnerabilities and likelihood of being breached is non-linear and it may even be indeterminate if contextual factors dominate.</p>
<p>This example also hints at another severe weakness in the ALE method – it ignores correlation and dependence between risk elements and factors.  We know from forensic analysis and the DBIR that severe security breaches involve a sequence of exploits and attacks.  This means that the likelihood of breach in one application is dependent on the likelihood of breach in other applications and systems.  An application might appear unimportant, but it might be a stepping-stone to other applications, databases, and networks.</p>
<p>It’s hard to account for all these factors and influences together without some sort of over-arching model for enterprise-level information security and risk.   Basically, you are looking for the ‘risk contribution’ of those specific application vulnerabilities to total costs, now and in the uncertain future.    Formally, the ‘Interest Cost’ for any given set of application vulnerabilities is the difference between the <a href="http://meritology.com/resources/Total%20Cost%20of%20Cyber%20(In)security.ppt" target="_blank">Total Cost of Security (TCoS)</a> in two possible worlds: World 1) application A has X vulnerabilities, vs. World 2) application A does not have X vulnerabilities (or if application A is not deployed at all).</p>
<p>What we really need are some short-cut approximations for this that doesn’t require a complete data set and risk estimates for the whole enterprise.  One approach I’m interested is in using modern AI methods (data mining, machine learning, inference methods).  This is on-going research.</p>
<h4>Summary</h4>
<p>I’m glad Chris proposed his Application Security Debt metric.  I hope my post has been helpful in correcting some of the errors, as I see them.  The good news is that the “Expected Principal” component of the metric looks like it can be estimated fairly easily and with good accuracy.  On the other hand, the “Interest Cost” component needs a lot of work.  I’m happy to collaborate with Chris or anyone else who wants to work on this.</p>
]]></content:encoded>
			<wfw:commentRss>http://newschoolsecurity.com/2011/03/fixes-to-wysophal%e2%80%99s-application-security-debt-metric/feed/</wfw:commentRss>
		<slash:comments>7</slash:comments>
		</item>
		<item>
		<title>Is Norton Cybercrime Index just &#8216;Security Metrics Theater&#8217;?</title>
		<link>http://newschoolsecurity.com/2011/02/is-norton-cybercrime-index-just-security-metrics-theater/</link>
		<comments>http://newschoolsecurity.com/2011/02/is-norton-cybercrime-index-just-security-metrics-theater/#comments</comments>
		<pubDate>Thu, 17 Feb 2011 12:27:17 +0000</pubDate>
		<dc:creator>Russell</dc:creator>
				<category><![CDATA[data]]></category>
		<category><![CDATA[disclosure]]></category>
		<category><![CDATA[fail]]></category>
		<category><![CDATA[metrics]]></category>
		<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[verification]]></category>

		<guid isPermaLink="false">http://newschoolsecurity.com/?p=2076</guid>
		<description><![CDATA[Symantec's new Norton Cybercrime Index looks like it is mostly a marketing tool.  They present it as though there is solid science, data, and methods behind it, but an initial analysis shows that this is probably not the case.  The only way to have confidence in this is if Symantec opens up about their algorthms and data.]]></description>
			<content:encoded><![CDATA[<p>I really hope that Symantec has invested serious money and resources to produce a good composite metric that meaningfully improves the ability of decision-makers to make better security decisions.  But an initial investigation leads me to believe that it is mostly a marketing ploy, at least in this initial version. Let me be the first to call it &#8216;Security Metrics Theater&#8217; (with nod to Bruce S.).</p>
<p>Here&#8217;s the website: <a href="http://www.nortoncybercrimeindex.com">www.nortoncybercrimeindex.com</a> (all in FLASH)</p>
<p style="text-align: center"><a href="http://newschoolsecurity.com/wp-content/uploads/2011/02/Norton-Cybercrime-Index.png"><img class="aligncenter size-full wp-image-2078" src="http://newschoolsecurity.com/wp-content/uploads/2011/02/Norton-Cybercrime-Index.png" alt="" width="479" height="232" /></a></p>
<p>Here&#8217;s a <a href="http://www.pcmag.com/article2/0,2817,2379856,00.asp">typical article</a>:</p>
<blockquote><p>Norton Cybercrime Index, unveiled today, rates the current state of cybercrime in a single, simple number and indicates whether the danger level is going up or down. Interested visitors can drill down for almost any level of detail. [...]</p>
<p>The index is open-ended, like the Dow Jones Industrial Average. Symantec&#8217;s proprietary algorithm draws on many sources to produce the index, among them the Symantec Global Intelligence Network, Norton Safe Web and the millions of customers using Norton 360 Version 4.0, Norton AntiVirus 2011, and Norton Internet Security 2011. To ensure the validity of the algorithm Symantec had it analyzed by experts at the University of Texas&#8217;s Institute for Cyber Security; the experts approved.</p></blockquote>
<p>What&#8217;s the goal?  From the FAQ (embedded in FLASH):</p>
<blockquote><p>Symantec created the Norton Cybercrime Index to show people that cybercrime is real, it can happen to anybody, and there is something you can do to protect yourself.</p></blockquote>
<p>How is it calculated?</p>
<blockquote><p>&#8230;using a statistical model and algorithm, which assigns values to the number of online threats observed each day.  Threats include malware, fraud, identity theft, spam, phishing, and social engineering trickery.  Once threats are quantified and processes through an algorithm, the Norton Cybercrime Index number is generated.  The algorithm has been endorsed by the University of Texas San Antonio as a valid measurement reflecting the risk of cybercrime.”</p></blockquote>
<h4>My initial judgement</h4>
<p>It looks like it is purely a product of Symantec’s marketing department.  There&#8217;s a massive PR effort underway via blogs, twitter, public places (e.g. London, Times Square), and probably at the RSA Conference, now underway in San Francisco.  The web advertising firm <a href="http://www.finedesigngroup.com/">Fine Design Group</a> created the FLASH UI, and tweeted about it first.</p>
<p>It will be interesting to probe their methods and data, assuming that Symantec will be transparent about the “proprietary algorithm” used to compute the index.  If they really want to establish credibility, it would be irrational to treat this as proprietary, confidential, and closed, for all the obvious reasons.  <a href="http://www.idanalytics.com/">ID Analytics</a> is listed as a data provider, but there&#8217;s no evidence that their &#8216;advanced analytics&#8217; are used by Symantec, only their summary data regarding personal identity theft in the US.</p>
<p>I’d be very surprised if any of Symantec’s metrics experts are behind it.  I don’t know of anyone in the security metrics community who has been contacted or involved as an outside expert.  They certainly haven’t presented it for peer review at last Monday&#8217;s <a href="http://www.securitymetrics.org">Mini-metricon</a> (why not?) or to the <a href="http://www.securitymetrics.org">securitymetrics.org</a> email list (why not?) or any academic conference or journal (why not?).  Searching the University of Texas at San Antonio, Institute of Cyber Security&#8217;s web site, I couldn&#8217;t find any mention of their work on this project, nor any presentation or report.  A search of Google Scholar for &#8220;cyber crime index&#8221; produced a few results, but not related to this and not from anyone at UT-SA.</p>
<p>Q: Who did have an early look at this?  A: <a href="http://www.gizmodo.com.au/2011/02/internet-security-trends-prove-we-are-a-sick-sick-bunch/">Angus Kidman</a>, a blogger from Gizmodo.  And what did he learn from his demo?  From <a href="http://www.gizmodo.com.au/2011/02/internet-security-trends-prove-we-are-a-sick-sick-bunch/">his blog post</a>:</p>
<blockquote><p>&#8220;On the day of the demo, these were the top search terms being targeted for poisoning:</p>
<ul>
<li>Invisible</li>
<li>Camel toe</li>
<li>Wifetube&#8221;</li>
</ul>
</blockquote>
<p>Right.  How very useful.  I&#8217;ll now modify my search patterns so I avoid those words today. <img src='http://newschoolsecurity.com/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' /> </p>
<h4>I don’t have  a good feeling about this</h4>
<p>It smells like FUD in a spiffy FLASH interface. Sure, there probably is <em>real data</em> behind it, but it’s aggregated into an index that is supposed to mean something.  A <em><strong>daily</strong></em> index!  The FUD label fits because this presentation gives the illusion of scientific validity, precision, reliable aggregation, and meaningful signals, when that none of these are present (it appears). Using fancy words like “statistical method” and “algorithm” gives it the air of scientific validity without really saying anything.  Worse, those words hide the assumptions, judgments, fudge factors, and who-knows-what that make the index work.</p>
<p>My intuition about this is that Symantec marketing manager wanted to create a &#8220;daily itch&#8221; to get average people to read what ever news blips were available that day about &#8216;cybercrime&#8217;, which would increase the chances that they would move from &#8216;awareness&#8217; to &#8216;action&#8217; (= buy more Symantec stuff).  By getting this out as a daily index, any up or down moves each day will trigger some people to click the buttons to find out &#8216;why?&#8217;.   But this will take them to news items, but not any credible justification of why they might be at greater risk on that day, compared to the day before.</p>
<p>As a thought experiment, imagine a similar &#8216;Risk Index&#8217; that is powered by <a href="http://www.astrologicalinvesting.com/">astrology readings</a>, <a href="http://www.newprophecy.net/">numerological interpretations of Nostradamus&#8217; texts</a>, or some other daily signal source.  With the appropriate shroud of credibility, some number of people are going to start following it, and when ever it changes, they will seek information as to &#8216;what does this mean for me?&#8217;  It would serve have exactly the same function as their current design.  This doesn&#8217;t prove anything, but establishes in my mind some plausibility.</p>
<h4>What&#8217;s the harm?</h4>
<p>Some might argue that this is harmless or even mildly beneficial if it prompts people to be more aware of security problems and to fix their security problems.  But I think it&#8217;s harmful because it promotes a false signal and a false method for doing information security metrics &#8212; for consumers or for anyone else.</p>
<p>Maybe I’m wrong and this may be an important advance, or at least a step forward.   At very the least, it shows that one  major security product/service vendor spent money to define a method, collect data, and make public the results.  Prior to this, no major vendor was even spending money on it.</p>
<h4>What to do now</h4>
<p>Is there any way this Index could be redirected to be a more valuable and extensible project?  I hope so.  But for that to happen, those of us how care about the New School approach to security need to apply the full-court press on Symantec to open up their method and data.</p>
<p><em><strong>Your action</strong></em> &#8212; contact Symantec, preferably in-person at RSA Conference, and demand they open up and also engage in the security metrics community in a serious way.  The burden of proof is on them, and if they can&#8217;t back it up then they should be shamed.</p>
]]></content:encoded>
			<wfw:commentRss>http://newschoolsecurity.com/2011/02/is-norton-cybercrime-index-just-security-metrics-theater/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>Java Security &amp; Criminals</title>
		<link>http://newschoolsecurity.com/2010/10/java-security-criminals/</link>
		<comments>http://newschoolsecurity.com/2010/10/java-security-criminals/#comments</comments>
		<pubDate>Tue, 12 Oct 2010 16:09:21 +0000</pubDate>
		<dc:creator>adam</dc:creator>
				<category><![CDATA[data]]></category>
		<category><![CDATA[disclosure]]></category>
		<category><![CDATA[Doing it Differently]]></category>

		<guid isPermaLink="false">http://newschoolsecurity.com/?p=1810</guid>
		<description><![CDATA[Brian Krebs has an interesting article on &#8220;Java: A Gift to Exploit Pack Makers.&#8221; What makes it interesting is that since information security professionals share data so well, Brian was able to go to the top IDS makers and get practical advice on what really works to secure a system. Sorry, dreaming there for a [...]]]></description>
			<content:encoded><![CDATA[<p>Brian Krebs has an interesting article on &#8220;<a href="http://krebsonsecurity.com/2010/10/java-a-gift-to-exploit-pack-makers/">Java: A Gift to Exploit Pack Makers</a>.&#8221;  What makes it interesting is that since information security professionals share data so well, Brian was able to go to the top IDS makers and get practical advice on what really works to secure a system.</p>
<p>
Sorry, dreaming there for a minute.  </p>
<p>
What Brian really did was go look at what attackers are doing in their commercial exploit kits, and discovered that Java exploits have surpassed Adobe exploits in &#8216;his&#8217; sample.</p>
<p>
I&#8217;m curious what you all think of the approach.  What can we learn from attacker toolkits and marketing pitches?  What are the limits of this?  </p>
<p>
]]></content:encoded>
			<wfw:commentRss>http://newschoolsecurity.com/2010/10/java-security-criminals/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>Fines or Reporting?</title>
		<link>http://newschoolsecurity.com/2010/10/fines-or-reporting/</link>
		<comments>http://newschoolsecurity.com/2010/10/fines-or-reporting/#comments</comments>
		<pubDate>Fri, 01 Oct 2010 15:40:36 +0000</pubDate>
		<dc:creator>adam</dc:creator>
				<category><![CDATA[data]]></category>
		<category><![CDATA[disclosure]]></category>
		<category><![CDATA[measurement]]></category>
		<category><![CDATA[metrics]]></category>

		<guid isPermaLink="false">http://newschoolsecurity.com/?p=1792</guid>
		<description><![CDATA[Over at the Office of Inadequate Security, Dissent does excellent work digging into several perspectives on Discover Card breaches: Discover&#8217;s reports, and the (apparent) silence of breached entities. I’m concerned that for many of the breaches they report, we have never seen breach reports filed by the entities themselves nor media reports on the incidents. [...]]]></description>
			<content:encoded><![CDATA[<p>Over at the Office of Inadequate Security, Dissent does excellent work digging into several perspectives on Discover Card breaches: Discover&#8217;s reports, and the (apparent) silence of breached entities.</p>
<blockquote><p>
I’m concerned that for many of the breaches they report, we have never seen breach reports filed by the entities themselves nor media reports on the incidents.  For now, though, let’s start with what I found when I received one batch of their reports to NYS. Keep in mind as you read the summaries that we are only talking about the number of  Discover card users affected by the incidents and for  only two states.  The numbers affected by each incident could be considerably higher, but since the entities themselves never filed breach reports with NYS or Maine, I have no additional information at this time.  (&#8220;<a href="http://www.databreaches.net/?p=14330">Staring into the abyss: how many breaches go unreported?</a>&#8220;)
</p></blockquote>
<p>As much as I&#8217;d like to encourage security and punish failures, I&#8217;d like to first see us know how much is wrong so we can estimate progress over time.</p>
<p>
]]></content:encoded>
			<wfw:commentRss>http://newschoolsecurity.com/2010/10/fines-or-reporting/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>ATTENTION: Securosis 2010 Data Security Survey results</title>
		<link>http://newschoolsecurity.com/2010/09/attention-securosis-2010-data-security-survey-results/</link>
		<comments>http://newschoolsecurity.com/2010/09/attention-securosis-2010-data-security-survey-results/#comments</comments>
		<pubDate>Wed, 15 Sep 2010 17:10:31 +0000</pubDate>
		<dc:creator>alex</dc:creator>
				<category><![CDATA[data]]></category>
		<category><![CDATA[Data Analysis]]></category>

		<guid isPermaLink="false">http://newschoolsecurity.com/?p=1776</guid>
		<description><![CDATA[The Securosis 2010 Data Security Survey results are out! http://bit.ly/aR4MuY Go, go and be NewSchool!  Seriously, don&#8217;t spend anymore time here, click the link!]]></description>
			<content:encoded><![CDATA[<p>The <a href="http://bit.ly/aR4MuY">Securosis 2010 Data Security Survey results are out! http://bit.ly/aR4MuY</a></p>
<p>Go, go and be NewSchool!  Seriously, don&#8217;t spend anymore time here, click the link!</p>
]]></content:encoded>
			<wfw:commentRss>http://newschoolsecurity.com/2010/09/attention-securosis-2010-data-security-survey-results/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>

