Oracle Database Server – 2 patches
Oracle Fusion Middleware – 11 patches
Oracle E-Business Suite – 3 patches
Oracle Supply Chain Products Suite – 1 patch
Oracle PeopleSoft – 6 patches
Oracle JD Edwards – 8 patches
Oracle Sun Products – 17 patches
Oracle Virtualization – 3 patches
Oracle MySQL – 27 patches

Fully 60% of the above patches are from OSS products. So which is more secure: open source or closed source. Or let’s compare Oracle DB vs MySQL: 2 versus 27 patches?

What do these numbers tell you? Absolutely nothing. Even with something like CVSS you still can’t tell which product is more secure. The whole thing is a load of malarkey. The product that is and will remain most secure is the one that you can manage and maintain the easiest for your organization.

Ries concieves as startups as businesses operating under conditions of high uncertainty, which includes things you might not think of as startups. In fact, he thinks that startups are everywhere, even inside of large businesses. You can agree or not, but suspend skepticism for a moment. He also says that startups are really about management and good decision making under conditions of high uncertainty.

He tells the story of IMVU, a startup he founded to make 3d avatars as a plugin instant messenger systems. He walked through a bunch of why they’d made the decisions they had, and then said every single thing he’d said was wrong. He said that the key was to learn the lessons faster to focus in on the right thing–that in that case, they could have saved 6 months by just putting up a download page and seeing if anyone wants to download the client. They wouldn’t have even needed a 404 page, because no one ever clicked the download button.

The key lesson he takes from that is to look for ways to learn faster, and to focus on pivoting towards good business choices. Ries defines a pivot as one turn through a cycle of “build, measure, learn:”

Ries jokes about how we talk about “learning a lot” when we fail. But we usually fail to structure our activities so that we’ll learn useful things. And so under conditions of high uncertainty, we should do things that we think will succeed, but if they don’t, we can learn from them. And we should do them as quickly as possible, so if we learn we’re not successful, we can try something else. We can pivot.

I want to focus on how that might apply to information security. In security, we have lots of ideas, and we’ve built lots of things. We start to hit a wall when we get to measurement. How much of what we built changed things (I’m jumping to the assumption that someone wanted what you built enough to deploy it. That’s a risky assumption and one Ries pushes against with good reason.) When we get to measuring, we want data on how much your widget changed things. And that’s hard. The threat environment changes over time. Maybe all the APTs were on vacation last week. Maybe all your protestors were off Occupying Wall Street. Maybe you deployed the technology in a week when someone dropped 34 0days on your SCADA system. There are a lot of external factors that can be hard to see, and so the data can be thin.

That thin data is something that can be addressed. When doctors study new drugs, there’s likely going to be variation in how people eat, how they exercise, how well they sleep, and all sorts of things. So they study lots of people, and can learn by comparing one group to another group. The bigger the study, the less likely that some strange property of the participants is changing the outcome.

But in information security, we keep our activities and our outcomes secret. We could tell you, but first we’d have to spout cliches. We can’t possibly tell you what brand of firewall we have, it might help attackers who don’t know how to use netcat. And we certainly can’t tell you how attackers got in, we have to wait for them to tell you on Pastebin.

And so we don’t learn. We don’t pivot. What can we do about that?

We can look at the many, many people who have announced breaches, and see that they didn’t really suffer. We can look at work like Sensepost has offered up at BlackHat, showing that our technology deployments can be discovered by participation on tech support forums.

We can look to measure our current activities, and see if we can test them or learn from them.

Or we can keep doing what we’re doing, and hope our best practices make themselves better.

[Edit -- fixed my mispelling of company name.  D'oh!]

Our assessment of the quality of cyber-crime surveys is harsh: they are so compromised and biased that no faith whatever can be placed in their findings. We are not alone in this judgement. Most research teams who have looked at the survey data on cyber-crime have reached similarly negative conclusions.

In the book, Andrew and I wrote “today’s security surveys have too many flaws to be useful as sources of evidence.” Dinei and Cormac were kind enough to cite that, saving me the trouble of looking it up.

I wanted to try here to carve out, perhaps, a small exception. I think of surveys as coming in two main types: surveys of things people know, and surveys of what they think. Both have the potential to be useful (although read the paper for a long list of ways in which they can be problematic.)

So there’s surveys of things people know. For example, what’s your budget, or how many people do you employ? There are people in an organization who know those things, and, starved as we are for knowledge, perhaps they would be useful to know. So maybe a survey makes sense.

But how many people Microsoft employs in security probably doesn’t matter to you. And the average of how many people Boeing, State Farm, Microsoft, Archer Daniels Midland, and Johnson & Johnson employ in security is even less useful. (Neighbors on the Fortune 500 list.) So even in the space that we might want to defend surveys, they’re not that useful.

So our desire for surveys is really evidence of how starved we are for data about outcomes and data about efficacy. We’re like the drunk looking for keys under the lamppost, not because we think the keys are there, but because there’s at least a little light.

So next time someone shows you a survey, don’t even bother to ask them what action they expect you to take, or what decision they expect you to alter, or ask them why you should accept what it says as acceptable arguments for that choice.

Rather, ask them to see the section titled “How we overcame the issues that Dinei and Cormac talked about.” It’ll save everyone a bunch of time.

]]> http://newschoolsecurity.com/2011/06/sex-lies-cybercrime-surveys/feed/ 4 http://newschoolsecurity.com/2011/03/fixes-to-wysophal%e2%80%99s-application-security-debt-metric/ http://newschoolsecurity.com/2011/03/fixes-to-wysophal%e2%80%99s-application-security-debt-metric/#comments Sat, 05 Mar 2011 09:47:27 +0000 Russell http://newschoolsecurity.com/?p=2099 In two recent blog posts (here and here), Chris Wysopal (CTO of Veracode) proposed a metric called “Application Security Debt”.  I like the general idea, but I have found some problems in his method.  In this post, I suggest corrections that will be both more credible and more accurate, at least for half of the formula.  The second half is harder to do right and needs more thinking.

Overview

Application Security Debt is based on the concept of  “technical debt” proposed by Ward Cunningham (a programmer who developed the first wiki program): describes it like this:

Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite… The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.

Chris adds:

The cost of technical debt is the time and money it will take to rewrite the poor code after you ship and bring it back to the quality required to maintain the software over the long haul.

Here is Chris’ summary of Application Security Debt:

Security debt is similar to technical debt. Both debts are design and implementation constructions that have negative aspects that aggregate over time and the code must be re-worked to get out of debt. Security debt is based on the latent vulnerabilities within an application. Application interest rates are the real world factors outside of the control of the software development team that lead to vulnerabilities having real cost. These factors include the cost of a security breach and attacker motivation to discover and exploit the latent vulnerabilities.

Chris’ second post describes a financial model that estimates the cost of Application Security Debt.  Framing the metric in financial terms will presumably help managers compare the cost of the “debt” to the cost of developing more secure software or costs of fixing the vulnerabilities.  (Note: Veracode provides a range of application security testing services, so they have an interest in economically justifying their services.  This isn’t a criticism of Veracode, Chris, or his proposal.  Just a reality.)

Chris’ model is focused on the simplest case where the application developer and application user is the same organization, so that it bears the costs of development, maintenance, and also any security breaches that result.  Starting with the simplest case is a great idea when proposing a new method.  So far so good.

Chris defines his financial model this way:

The basic financial model for security debt is monetary risk that can be expressed as expected loss. The formula for expected loss is event likelihood X impact in dollars. Event likelihood is based on the makeup of vulnerabilities in the application and the likelihood that the vulnerabilities will be discovered and exploited. The impact is the cost of a security breach based on an exploit of one of those vulnerabilities.  [Emphasis in original]

This is, of course, a version of the bottom-up Annualized Loss Expectancy (ALE) formula for individual risk elements:

• ALE = Single Loss Expectancy X Annual Rate of Occurrence

(Mike Rothman recently crapped on all “risk metrics” by lumping them all into the ALE formula.  I’ll critique ALE and Mike’s post in a separate blog post.)

ALE issues aside, I think Chris is making mistakes in his definition of Application Security Debt that will lead to serious confusion.

Debt = Expected Principal + Interest Costs

Chris made a mistake when he defines monetary value of the Application Security Debt as expected loss due to security breaches.    Instead, the ‘Principal’ part of the debt formula is the cost of fixing security problems beyond what is budgeted. Chris had it right in his summary in the first article:

The cost of technical debt is the time and money it will take to rewrite the poor code after you ship and bring it back to the quality required to maintain the software over the long haul.

Expected losses are in the category of “Interest Costs” as Chris said in his summary:

Application interest rates are the real world factors outside of the control of the software development team that lead to vulnerabilities having real cost.

Putting this together in simple language:

“Application Security Debt is a ‘loan’ with variable principal which could range from 0% to 100% of your original project costs. The ‘principal’ is what you’ll eventually have to pay to fix security bugs or rewrite the code.  It also has varying and uncertain ‘interest costs’, which are the costs of security breaches due to these vulnerabilities. This includes the possibility of the mother-of-all balloon payments (i.e. a huge loss event).”

The good news is that Expected Principal is relatively easy to estimate with good accuracy and without a lot of outside data.  The not-so-good-news is that Interest Cost is a bear to estimate.

Estimating ‘Expected Principal’

For simplicity, let’s assume that cost of fixing code (above the budgeted costs) occurs in discrete increments, F:

1. Zero  (i.e. your debt is ‘forgiven’)
2. Minor fixes and patches (‘Principal’ = 10% increase in project cost)
3. Major fixes and patches  (‘Principal’ = 25% increase in project cost)
4. Substantial rewrite (‘Principal’ = 50% increase in project cost)
5. Total rewrite   (‘Principal’ = 100% increase in project cost, or more)

Thus, the best case is that you owe no principal and the worst case is that you owe principal equal to the entire cost of the project.  You could include other factors such as external costs of schedule delays, costs of rehiring your programmers after you fire them all , or what ever.  My point is that these costs are not open-ended, but are a multiplier on your original development costs.

The Expected Principal (EP) is equal to each of these cost scenarios multiplied by their probability of management choosing that option:

For example, if the original cost of the application development project is $1 million, and there is 5% chance of Zero costs, 80% of Minor code fix costs, and 15% chance of Substantial rewrite costs, then the Expected Principal would be $155,000, or 16% of the original cost.

This is important: Expected Principal is ultimately determined by management decisions and ‘threshold of pain’.  This means that the value of p(F), above, is a subjective probability.  It would be an ideal metric to estimate using prediction markets (PMs).   (PMs have been used successfully in software development to estimate shipment dates and defect rates, for example.)

Another implication: you don’t need to accurately forecast future loss events or their economic impact to get a decent estimate of Expected Principal.  Instead, you only need to estimate the Interest Costs very roughly to determine which code fix scenario is most likely.    You could even estimate p(F) by setting thresholds for the number and severity of vulnerabilities discovered by certain levels of effort.  Better, you could combine these methods to ‘triangulate’ on estimates of p(F).

To calibrate these subjective probability estimates, it would be very helpful to collect historical data on the % of applications that have some level of rewrite or schedule delay due to security problems.  (Hint hint!)

Estimating ‘Interest Costs’ on the Debt will be Hard

The second part of the Application Security Debt formula is ‘Interest Costs’.  This is where things get hairy.   All the members of the ALE family of risk calculations have a similar flaws: 1) prodigious data requirements and 2) propagation of uncertainty through the calculations.  Furthermore, some suffer by using only mean values and ignoring extreme values (i.e. the “tails” of the probability distribution curves).

Chris acknowledges these issues, at least the requirement for more and better data:

Now you are probably thinking that this is getting a little tenuous and it is. We need better data on likelihood type and likelihood of an application breach by industry and other factors like company size.

Data issues aside, I think there are flaws in his use of ALE and calculation methods.  Here’s one thought experiment to show how it could lead to the wrong conclusions, in my opinion.

Let’s use Chris’ ‘baseline expected loss’ table, where he calculates the expected loss for each type of vulnerability.  Imagine that we are comparing two similar applications, A and B.  Assume that each project is expected to have the same number of vulnerabilities, five each.  Let’s say the development cost of each project is $1 million.  Application A has five SQL injection vulnerabilities while application B has one SQL Injection vulnerability and four Remote File Inclusion vulnerabilities.  Doing the calculations:

• A’s expected losses = $19,220,000
• B’s expected losses = $5,074,080

Does project A really have four times more risk than project B? Probably not.  From what I know, the number of vulnerabilities in an application is not proportional to the likelihood that the application will be breached.  Instead, I’d guess that the likelihood of being breached is a function of where the application is in the IT architecture, how accessible it is, how important it is to attackers, etc.

Also, there’s the ‘weakest link’ effect: “given enough random attackers or one persistent attacker, it only takes one vulnerability to lead to a breach”.  Assuming all SQL Injection vulnerabilities are equally discoverable and equally exploitable, then we should estimate that application B with one SQL Injection vulnerability is just as likely to get breached as application A with five, all other things being equal.

(I confess I’m not an expert in application security or vulnerability analysis, so these comments are my interpretation of what others have written or said.)

Even if my logic here is flawed somewhat, my main point is that the relation between number of vulnerabilities and likelihood of being breached is non-linear and it may even be indeterminate if contextual factors dominate.

This example also hints at another severe weakness in the ALE method – it ignores correlation and dependence between risk elements and factors.  We know from forensic analysis and the DBIR that severe security breaches involve a sequence of exploits and attacks.  This means that the likelihood of breach in one application is dependent on the likelihood of breach in other applications and systems.  An application might appear unimportant, but it might be a stepping-stone to other applications, databases, and networks.

It’s hard to account for all these factors and influences together without some sort of over-arching model for enterprise-level information security and risk.   Basically, you are looking for the ‘risk contribution’ of those specific application vulnerabilities to total costs, now and in the uncertain future.    Formally, the ‘Interest Cost’ for any given set of application vulnerabilities is the difference between the Total Cost of Security (TCoS) in two possible worlds: World 1) application A has X vulnerabilities, vs. World 2) application A does not have X vulnerabilities (or if application A is not deployed at all).

What we really need are some short-cut approximations for this that doesn’t require a complete data set and risk estimates for the whole enterprise.  One approach I’m interested is in using modern AI methods (data mining, machine learning, inference methods).  This is on-going research.

Summary

I’m glad Chris proposed his Application Security Debt metric.  I hope my post has been helpful in correcting some of the errors, as I see them.  The good news is that the “Expected Principal” component of the metric looks like it can be estimated fairly easily and with good accuracy.  On the other hand, the “Interest Cost” component needs a lot of work.  I’m happy to collaborate with Chris or anyone else who wants to work on this.

It’s very difficult to argue with a poorly constructed argument.  Especially when I have no idea what a “risk metric” is.  But best as I can tell, Mike’s position is that unless you are smart and/or have strong resources allocated to your InfoSec team, things like metrics, GRC, application security, SEIM, and a “host of other security processes or technologies.” are “science projects…”

Meaning, I suppose, that they provide no “pragmatic” use to security departments (whatever pragmatic means).

Problem is, for many folks, metrics, risk management, appsec and other “security processes or technologies” can and do have significant value.  In fact, in terms of managing a large, disparate enterprise, the data gathering process for risk analysis alone can be more valuable than the result (experience contrary  to what Rothman writes in comments: “the value of the (assessment) benefit is outweighed by the cost of gathering the data.”).

That said, it’s a shame that his argument is poorly constructed because, by in large, I have to agree that there’s plenty of poopy risk statements to pick on.  As I said in my CSO Magazine interview (shameless self promotion) and in my RSA Risk Management Smackdown panel – there have been times when I’ve counseled an organization to put off making risk statements until their visibility into their environment is much better.  In the public record you should be able to find past statements where I say it’s better for a small business to focus resources away from risk assessment when the required assessment was a bureaucratic quest rather than a quest for knowledge or wisdom.

Bottom Line, for risk and metrics, Rothman shouldn’t generalize for the sake of sensationalism and marketing.  And he probably shouldn’t be doing that for other security processes and technologies beyond risk analysis and security management, too.   But as a consulting firm, what Securosis could/should be doing is giving people help – helping them recognize their organizational maturity, and helping them understand what resource allocation is or isn’t appropriate at various levels of maturity.   Just a thought.

Here’s the website: www.nortoncybercrimeindex.com (all in FLASH)

Here’s a typical article:

Norton Cybercrime Index, unveiled today, rates the current state of cybercrime in a single, simple number and indicates whether the danger level is going up or down. Interested visitors can drill down for almost any level of detail. [...]

The index is open-ended, like the Dow Jones Industrial Average. Symantec’s proprietary algorithm draws on many sources to produce the index, among them the Symantec Global Intelligence Network, Norton Safe Web and the millions of customers using Norton 360 Version 4.0, Norton AntiVirus 2011, and Norton Internet Security 2011. To ensure the validity of the algorithm Symantec had it analyzed by experts at the University of Texas’s Institute for Cyber Security; the experts approved.

What’s the goal?  From the FAQ (embedded in FLASH):

Symantec created the Norton Cybercrime Index to show people that cybercrime is real, it can happen to anybody, and there is something you can do to protect yourself.

How is it calculated?

…using a statistical model and algorithm, which assigns values to the number of online threats observed each day.  Threats include malware, fraud, identity theft, spam, phishing, and social engineering trickery.  Once threats are quantified and processes through an algorithm, the Norton Cybercrime Index number is generated.  The algorithm has been endorsed by the University of Texas San Antonio as a valid measurement reflecting the risk of cybercrime.”

My initial judgement

It looks like it is purely a product of Symantec’s marketing department.  There’s a massive PR effort underway via blogs, twitter, public places (e.g. London, Times Square), and probably at the RSA Conference, now underway in San Francisco. The web advertising firm Fine Design Group created the FLASH UI, and tweeted about it first.

It will be interesting to probe their methods and data, assuming that Symantec will be transparent about the “proprietary algorithm” used to compute the index.  If they really want to establish credibility, it would be irrational to treat this as proprietary, confidential, and closed, for all the obvious reasons.  ID Analytics is listed as a data provider, but there’s no evidence that their ‘advanced analytics’ are used by Symantec, only their summary data regarding personal identity theft in the US.

I’d be very surprised if any of Symantec’s metrics experts are behind it.  I don’t know of anyone in the security metrics community who has been contacted or involved as an outside expert.  They certainly haven’t presented it for peer review at last Monday’s Mini-metricon (why not?) or to the securitymetrics.org email list (why not?) or any academic conference or journal (why not?).  Searching the University of Texas at San Antonio, Institute of Cyber Security’s web site, I couldn’t find any mention of their work on this project, nor any presentation or report.  A search of Google Scholar for “cyber crime index” produced a few results, but not related to this and not from anyone at UT-SA.

Q: Who did have an early look at this?  A: Angus Kidman, a blogger from Gizmodo.  And what did he learn from his demo?  From his blog post:

“On the day of the demo, these were the top search terms being targeted for poisoning:

• Invisible
• Camel toe
• Wifetube”

Right.  How very useful.  I’ll now modify my search patterns so I avoid those words today.

I don’t have  a good feeling about this

It smells like FUD in a spiffy FLASH interface. Sure, there probably is real data behind it, but it’s aggregated into an index that is supposed to mean something.  A daily index!  The FUD label fits because this presentation gives the illusion of scientific validity, precision, reliable aggregation, and meaningful signals, when that none of these are present (it appears). Using fancy words like “statistical method” and “algorithm” gives it the air of scientific validity without really saying anything.  Worse, those words hide the assumptions, judgments, fudge factors, and who-knows-what that make the index work.

My intuition about this is that Symantec marketing manager wanted to create a “daily itch” to get average people to read what ever news blips were available that day about ‘cybercrime’, which would increase the chances that they would move from ‘awareness’ to ‘action’ (= buy more Symantec stuff).  By getting this out as a daily index, any up or down moves each day will trigger some people to click the buttons to find out ‘why?’.   But this will take them to news items, but not any credible justification of why they might be at greater risk on that day, compared to the day before.

As a thought experiment, imagine a similar ‘Risk Index’ that is powered by astrology readings, numerological interpretations of Nostradamus’ texts, or some other daily signal source.  With the appropriate shroud of credibility, some number of people are going to start following it, and when ever it changes, they will seek information as to ‘what does this mean for me?’  It would serve have exactly the same function as their current design.  This doesn’t prove anything, but establishes in my mind some plausibility.

What’s the harm?

Some might argue that this is harmless or even mildly beneficial if it prompts people to be more aware of security problems and to fix their security problems.  But I think it’s harmful because it promotes a false signal and a false method for doing information security metrics — for consumers or for anyone else.

Maybe I’m wrong and this may be an important advance, or at least a step forward.   At very the least, it shows that one  major security product/service vendor spent money to define a method, collect data, and make public the results.  Prior to this, no major vendor was even spending money on it.

What to do now

Is there any way this Index could be redirected to be a more valuable and extensible project?  I hope so.  But for that to happen, those of us how care about the New School approach to security need to apply the full-court press on Symantec to open up their method and data.

Your action — contact Symantec, preferably in-person at RSA Conference, and demand they open up and also engage in the security metrics community in a serious way.  The burden of proof is on them, and if they can’t back it up then they should be shamed.

Especially interesting after comments on the CMO video.

Real dashboards

Dashboards are appealing because they are macho.  The metaphor is derived from the instrument panels of airplanes and cars, and especially fighter planes and racecars.  Managers like to think they are daring and brave fighter pilots and auto racers, so the dashboard as visual metaphor is superficially appealing.

To the right are three examples of instrument panels:  a vintage fighter plane, a modern sports car, and a modern helicopter.  The basic form is an array of individual meters, mostly analog meters on a continuous numeric scale.  Some have thresholds or warning levels (e.g. ‘red line’ on the RPM gauge or the ‘over temp’ on the temperature gauge).

Below are two examples of security dashboards that make obvious the visual metaphor: 1) a compliance dashboard offered by ISACA and 2) Seculert’s executive dashboard.  Of course, the ISACA dashboard is simplistic and stylized, but it does make the most obvious use of the visual metaphor.  (I’m not talking about ‘dashboards’ that are merely assemblies of charts, graphs, and tables.  They don’t really make use of the dashboard visual metaphor.)

ISACA compliance dashboard

Securlert dashboard

Dashboards work when controllers match the meters

Going back to the original context of instrument panels for airplanes and automobiles, we can describe their use case in simple terms as a feedback loop:

1. Look at the instruments
2. Adjust the associated controllers (throttle, joystick, etc.)
3. Repeat (1) to see the effect of (2), relative to goals or thresholds

The key to usability is the association between appropriate controllers and the individual meters.  In a car, the controllers are the steering wheel, the gas pedal, the brake pedal, the ignition switch, and the gearshift, primarily.   Generally, there are one or two controllers associated with each meter and the action of each controller is usually proportional to the metric that appears on the meter (e.g. Gas pedal and brake pedal control speed; gas pedal and gear shift control RPM, etc.). There are more controllers on a plane, but the same relationships hold between controllers and meters, at least for older planes.

Information security management is different

The dashboard metaphor doesn’t work for information security management because the ‘controllers’ – i.e. the actions and decisions of InfoSec management – are not in close correspondence to the output of the ‘meters’.  At a CISO level and above, managers take action through budgets, spending priorities, architectures, staffing levels and resource allocation, performance evaluations, security policy management, security awareness programs, compliance audits, vendor relations and contracts, security development lifecycles and methods, incident response policies, and (hopefully) collaboration with business executives on information security implications of business decisions.  In addition, there is a management function that is under-recognized in my opinion: InfoSec risk intelligence and organization learning/agility related to information security.

Even if your list of InfoSec management actions/decisions is different, I believe you’ll agree that almost none of these ‘controllers’ is in one-to-one or few-to-one correspondence with any of the ‘meters’ in the security dashboards.  This means that there is no simple way to see the effect of adjusting any ‘controller’ by monitoring a ‘meter’, which translates to usability problems.    Managers may still go through the motions of looking at dashboards, but their actual decisions will be guided by other information.

What visual metaphor would be better?

The right answer to this question might lead to a great business venture.  I don’t have the answer, but I can offer some suggestions on fruitful directions.

There’s a lot of research that shows that people think about risk in terms of stories.  Any visual metaphor that conveys a stylized story about risk and the ‘controllers’ associated with risk could be very useful.  One approach is ‘rich pictures’ associated with Soft Systems Methodology.  Here’s another example.  Of course, these rich pictures are static unless you watched them being created.  A brilliant designer could probably turn them into dynamic animations, maybe using video game techniques.

Speaking of video games, there are many possible ways to adapt the way they visualize their virtual worlds, which often entail threats, risks, and uncertainties.  One example is the campaign map from the Total War series for Windows PC, a personal favorite of mine.  It is somewhat analogous to the rich picture example given above. One interesting feature of this map is the areas of ‘light’ vs. ‘dark’.  The dark areas are where you, as a leader, have little or no intelligence.  Only when you send a unit or agent into the dark regions do you find out what is really there.   I really wish that security metric displays had a similar way of conveying uncertainty, ambiguity, and ignorance.

While not a solution in it self, another interesting visualization method for the social aspects of information security and risk is to use facial expressions to represent the risk perceptions or emotional disposition of various groups – users, adversaries, supply chain partners, regulators, etc.  I experimented with this with good results in the 1980s when I built a prototype of Michael Porter’s competitive analysis method in Hypercard (!).

Automated facial expression animations could be useful to express risk perceptions by stakeholders

Summary

Dashboards work when the user has proportional controllers or switches that correspond to each of the ‘meters’ and the user can observe the effect of using those controllers and switches in real time by observing the ‘meters’.

Dashboards don’t work when there is a loose or ambiguous connection between the information conveyed in the ‘meters’ and the actions that users might take.  Other visual metaphors should work better.

For more information on visual metaphors in design, check out these two seminal books: Donald Norman’s Design of Everyday Things and Edward Tufte’s The Visual Display of Quantitative Information.

Just because you can codify a standard or practice doesn’t mean that this practice is sane. There’s plenty of documentation around homeopathy, astrology, biorhythms, and other pseudosciences, but that doesn’t make them any more real.

In other words, just being able to reference a document for repeatability does not make the outcome of those acts real or valid. Almost everyone in that thread has focused on our industry’s ability to create documentation, not on the fundamental problems of creating a defensible method for risk expression.

This is why our standards blow.  And yes, I’m going to expand my focus beyond CRISC/Risk IT and include the 800 series from NIST (including the new releases), the ISO 27005/31000 document, and many others.  They are all very heavy on repeating the same idea that risk management is some OODA/PDCA type cycle and subsequent bureaucratic processes and very thin on the actual establishment of useful risk statements. Look, your P/D/C/A policy/procedures only need to be a few pages, and you certainly don’t need the time, expense, and hassle of certification.  Spending the time and effort to tailor a several hundred page document and get people all certifiable on the subject to fit your organizational culture is just a rabbit trail of waste.

I mean, as weird as OSSTMM is – at least Pete has done a really good job of trying to provide metrics and derivative values of meaning that are repeatable.