Adam Shostack at Ada’s Technical Books from Ada's Technical Books on Vimeo.

Thanks to Danielle for inviting me, and I’d be happy to do more readings in the future.

]]> http://newschoolsecurity.com/2011/10/new-school-of-information-security-book-reading-at-adas/feed/ 0 http://newschoolsecurity.com/2011/02/self-promotion-a-little-interview-about-alex-rsa/ http://newschoolsecurity.com/2011/02/self-promotion-a-little-interview-about-alex-rsa/#comments Tue, 01 Feb 2011 18:48:16 +0000 alex http://newschoolsecurity.com/?p=2055 Self Promotion time, sorry for the spam, but I think the stuff I’ll be participating in at RSA is pretty NewSchool.  Here’s an interview that talks about both of the things I’ll be doing and you can see if they’ll be interesting:

http://itacidentityblog.com/rsa-podcast-alex-hutton-principal-in-research-and-risk-intelligence-verizon-business

Interesting interactive data app from the Wall Street Journal about your privacy online and what various websites track/know about you.

http://blogs.wsj.com/wtk/

Full disclosure, our site uses Mint for traffic analytics.

What’s the least contrived situation in which a pie chart is better than a bar graph or table? (Pac man and pies are two obvious examples.)

Quick: which is larger, the grey slice on top, or the grey slice on the bottom? And ought grey be used for “sophisticated” or “moderate”?

I’m confident that both organizations are focused on accurate reporting. I am optimistic that this small example in the utlity of pie charts will inform report writers. The report writers and their graphics departments, loving their customers, will move to bar charts to help them compare numbers between sources.

I’m confident that not using pie charts is a best practice.

Elsewhere: “The only time it makes sense to use a pie chart.”

And elsewhere: “The Visual Display of Quantitative Information, 2nd edition“

Case in point:  I saw a documentary about “Abstinence only” sex education programs for teens in the public schools of New Mexico — one negative example in Albuquerque and one positive example in Socorro.   (This is federally funded.)  Skipping over the most aggregious errors and misstatements in these programs, I noticed one big blooper regarding risk estimation and risk communication.

The educators who developed and deliver this program emphasize the failure rate of condoms as argument against relying on them.  In contrast, abstinence-only is touted because it is 100% effective in preventing unplanned pregnancy and all the negative stuff that goes along with it.  Funny thing–they never mentioned the failure rate of abstinence-only when implemented by teenagers!     Sure, you can tell teenagers to be abstinent and they can even commit to it, but would you bet on it?   What odds would you demand for a large bet(say, $100,000 from your bank account) that a large group of teens would remain abstinent for five years?  There are plenty of studies (e.g. here and here) that demonstrate the limited capabilities of teens to avoid risky behavior, control impulses, rationally balance short-term gain against long-term pain, think beyond a short planning horizon, resist peer pressure, etc.    For most teens in the US, their “failure rate” (i.e. failing to avoid risky behaviors) is greater than 0%, and in cases of “multiple-risk adolescents ” the failure rate is far above 0%.

I would bet that condoms are much more reliable than the average teenager’s commitments to eschew immediate pleasures.   Of course, using both would be much more reliable than either alone.   This is “defense in depth”, of course.  Better still, take it to the max and advise that they add a “full-body condom”.  Then they would be “fer sher,  fer sher!”, as the Valley Girl might say.

Brett Miller just emailed me and asked (as part of a very nice email) “isn’t that an orrery, not an astrolabe?”

It appears that I’m going to have to update my commentary. Thanks, Brett!

[And thanks Scott--I misspelt orrery, now corrected.]

]]> http://newschoolsecurity.com/2009/10/ooops/feed/ 1 http://newschoolsecurity.com/2009/10/speaking-in-michigan-on-tuesday/ http://newschoolsecurity.com/2009/10/speaking-in-michigan-on-tuesday/#comments Fri, 16 Oct 2009 15:14:40 +0000 adam http://newschoolsecurity.com/?p=756 Andrew Stewart and I will be speaking at the University of Michigan SUMIT_09 on Tuesday. We’re on 10:30-11:25. If you’re in the area, please come by.

]]> http://newschoolsecurity.com/2009/10/speaking-in-michigan-on-tuesday/feed/ 0 http://newschoolsecurity.com/2009/09/visualization-friday-beautiful-functional-and-effective/ http://newschoolsecurity.com/2009/09/visualization-friday-beautiful-functional-and-effective/#comments Fri, 25 Sep 2009 07:07:18 +0000 Russell http://newschoolsecurity.com/?p=682 We can all learn from this great role model, aimed at personal nutrition awareness and education: Nutritiondata.com .

I encourage you to click on the images below to visit the site and explore interactive features. 

If only security awareness web sites aimed at end-users and consumers were this good.

I applaud the effort and goals of the study and it may have some useful conclusions. We should have more of this type of study, especially at a large scale.

Unfortunately, the report has some major problems, listed roughly in order of severity:  (for details, read on…)

1. The most basic charts are missing.  What was the total number of attacks in the study period?  The total number of vulnerabilities?  What percentage of identified vulnerabilities were actually attacked?  Were the most prevalent vulnerabilities attacked most frequently?  What how prevalent are zero-day vulnerabilities as a percentage of total vulnerabilities?
2. Some of the most important statements in the report have no backing data or detailed analysis.  Here are examples from the Executive Summary:

   “Waves of targeted email attacks, often called spear phishing, are exploiting client-side vulnerabilities in commonly used programs such as Adobe PDF Reader, QuickTime, Adobe Flash and Microsoft Office.”

   There is no analysis or data in the report regarding targeted email attacks (a.k.a. spear phishing).  The rise of spear phishing has been documented elsewhere, but if there was data coming from this study to support that assertion, it didn’t appear in the detailed section of the SANS report. 

   “Attacks against web applications constitute more than 60% of the total attack attempts observed on the Internet.” 

   Looking at the detailed sections of the report, I could not find any table or chart that listed the total number of attacks in the study and the percentage of attacks on each on category (i.e. web applications vs. app platform vs. OS).  (Figure 1 doesn’t do it.  Here is my critique of this bad graphic.)

   “Rising numbers of zero-day vulnerabilities”

   I couldn’t find any support in the report for the assertion that the number of zero-day vulnerabilities were rising.  There was no data analysis and no charts in the “Zero-day Vulnerability Trends” section.

   “On average, major organizations take at least twice as long to patch client-side vulnerabilities as they take to patch operating system vulnerabilities. In other words the highest priority risk is getting less attention than the lower priority risk.”

   There isn’t any chart that shows the average time to patch client-side vulnerabilities vs. OS vulnerabilities.  They don’t mention or analyze server-side application patching, which seems weird.  How about a chart with two bars, one for web applications and other for OS, showing the average number of days to reduce vulnerabilities by half?  That would be simple and clear.  Regarding the charts they do show…

3. The charts in the “Vulnerabilities” section are poorly executed and poorly explained.  The wiggly “percentage vs. days” graphs leave me scratching my head.  The data is probably in there, but it’s very hard to see in this format. Here’s my interpretation of these charts (but I might be wrong).  These are supposed to “aging charts” that graph the % of vulnerabilities that remain unresolved or unpatched after X days.  If none are resolved, then the line should be flat at 100%.  If all are resolved on day one, then it would be vertically down from 100% to 0%.  If they are gradually resolved, then the line will slope downward.  If some of the vulnerabilities are never resolved, then the line should plateau at some residual level. But why don’t the charts all start with a value of 100% on Day zero?  I don’t know and it’s not explained.  I’m guessing that it takes some number of days before all the vulnerabilities appear in the scans. More disturbing is the periodic oscillations in data series.  These jump out visually.  But what does this pattern mean?   You might be inclined to think that vulnerabilities were resolved every 5 to 7 days, only to reappear the next week, and so on.  That would be a huge process problem.  But it turns out that these oscillations are a spurious result of the scanning process and the data analysis method.  Consider this phrase from the report:

   “Periodic drops in detection rates occur during the weekends when scanning focuses on servers rather than desktop machines and the detection rates of vulnerabilities related to desktop software fall accordingly.”
   
   If this is really true (and it could probably be verified), then it would make sense to drop weekend data points and just include weekday data points for client-side vulnerabilities.  The data analysis could be further improved by using a smoothing or curve fitting method, because what counts in this analysis is not the day-by-day squiggles but the overall slope of the curve and whether it plateaus.  Finally there should be some marker for “half-life” or something, to allow easy visual identification of how long it takes to resolve half of the vulnerabilities in each class.

4. One finding was not highlighted but deserves more attention: the intentional use of security-violating methods by legitimate web applications.  This blurs the distinction between “good guys” and “bad guys” and makes security management much more difficult.  Examples from the report:

   “…an advertiser’s banner might be embedded in a web page which is set up to reflect some JavaScript off of the advertiser’s HTTP server for tracking purposes. However, in this case, there is little risk because the site in question (usually) has full control over his/her page, so this request to the advertiser is not generally malicious. It is the “reflection” attacks, along with attacks that leverage flaws in form data handling, that make up the vast majority of XSS attacks that we have seen in the last six months.” [emphasis added]

   ”A very large spike in SQL Injection attacks in July was caused mostly by an online advertiser who distributed code to many affiliates using SQL injection as functionality. The application was quickly pulled, resulting in a large drop in events for the month of August.”

   Related to this trend is the intentional use of security- and privacy-violating technologies to support marketing purposes, e.g. FTC’s action against Sears .

5. The analysis of Cross-site Scripting is confusing at best.  In the attack section, they say: “Cross Site Scripting (XSS) is one of the most prevalent bugs in today’s web applications.”  But looking at the graph immediately below (Fig. 13), it doesn’t look like XSS attacks are frequent – numbering in the thousands while other attacks number in the millions.  Looking down to the vulnerabilities section, there is no data or analysis of XSS vulnerabilities as a percentage of the total.
6. There is no mention of uncertainty or confidence levels in the analysis or conclusions.   For example, was the TippingPoint attack data adjusted for false positives and false negatives?  It’s well known that all Intrusion Detection/Prevention Systems (IDPS) are prone to false positives and false negatives, no matter how well designed or tuned.  This doesn’t discredit the data or analysis, but it does add uncertainty and reduce confidence levels. 
7. There was no mention of the limitations of the attack or vulnerability data.  Of course, all data sources will have limitations.  It’s essential that the authors factor this in to the analysis and make clear that they are not analyzing the full spectrum of cyber attacks or vulnerabilities.  For example, there is no coverage of insider threats, social engineering, combined physical and cyber attacks, or data leakage.  There is also no coverage in the data sets for complex targeted attacks.

   Attack data limitations. As a network appliance, TippingPoint’s Intrusion Detection/Prevention Systems (IDPS) do not detect all types of attacks.  Specifically, they cannot detect attacks within encrypted network traffic, attacks that come through wireless protocols, or attacks that utilize application payload, e.g. code injection.  Also, the IDPS are susceptible to various types of attacks, most involving large volumes of traffic, which can “blind” the IDPS for a period of time.  IDPS appliances also don’t detect authorization or authentication attacks.  (I’m no expert in IDPS or intrusion detection in general, so I’m drawing from other sources, including NIST 800-94: Guide to Intrusion Detection and Prevention Systems.) 

   Vulnerability data limitations.  Qualys vulnerability scanning service identifies many vulnerabilities in hardware configuration, operating systems, networking, libraries, platforms (e.g. Adobe Acrobat and Flash), etc., including: Misconfigured or unpatched servers, laptops and desktops; Out-of-date or misaligned security policy; Unauthorized hardware, software or applications; Easily-guessed passwords; and Inadequate controls on traffic from trusted third-party networks.But there are limitations. 

   Most obvious but unstated, Qualys identifies known vulnerabilities only, not vulnerabilities that have not yet been discovered.  While CERT has been quoted as saying “99% of attacks exploit known vulnerabilities”, this does not mean that there is no risk associated with undiscovered vulnerabilities. 

   More important is that automated vulnerability assessment scanning doesn’t provide a macro view on the relative riskiness of vulnerabilities. According to a Gartner report, vulnerability assessment scanning systems “provide data on devices within the network, but it remains difficult to understand how the overall network is vulnerable, or how vulnerabilities within a device affect their neighbors. What may appear as a benign or low-priority vulnerability on a host may be used as a launching point for an attacker to penetrate other devices on the network.”   Thus, the prioritization of vulnerabilities in the report seems to be based on their prevalence in the total population, not based on the relative riskiness of each vulnerability in it’s IT and network context.  (That would require other analysis, including attack graphs, penetration testing, etc.)

   Finally, Qualys only added Web Application Scanning (WAS) to their QualysGuard Security and Compliance Suite on May 26, 2009. This was half-way through the study period (March – August 2009), and it’s not clear from the report whether WAS produced any data used in this study or how widely it is being used in the customer base.  (Qualys suite is software as a service (SaaS), so new functionality should be immediately available to all subscribers.  Who knows what percentage of customers started using that functionality.)

8. There was very little explanation of the methodology.  I’m guessing that they drew conclusions based on total number of attacks of various types, compared to the total number of vulnerabilities of various types.  I didn’t see any analysis of the severity of each attack type, or any data regarding how many of these attacks succeeded in exploiting vulnerabilities.
9. There is insufficient background on the data sets.  For example, what are the demographics of the TippingPoint data (attacks) and the Qualys data (vulnerabilities)?  How many TippingPoint customers are also Qualys customers?  Some of the context information that was listed seemed irrelevant, e.g. “TippingPoint intrusion prevention systems protect 6,000 organizations” and “vulnerability data from 9,000,000 systems compiled by Qualys”.  Yes, that’s a lot of organizations and systems, but compared to what?
10. What was the point of the charts and analysis on the geographic origins and destinations of attacks?  I presume that this could be very important for understanding the attacking parties (“bad guys”) and the resources they use. Maybe it could also to help guide IT managers on the likelihood of certain attacks in their geography.  But the analysis falls short of either purpose.  As it is, these charts simply make me say “hmmmmm….”.

As a general comment, the report is poorly organized.  I found it hard to read, and it looked like a “cut and paste” job with no overall editing.  It was especially hard to trace the line of reasoning from the executive summary to the detailed sections.  Also, “…Four Key Attacks” section is confusing.  They actually list five key attacks, not four, but then say there are really only two categories: Server-side HTTP attacks and Client-side HTTP attacks.  Mis-numbering like this is a basic editing mistake.

Lastly, I wonder if there is any way that this study could be augmented and extended with data from other sources.  I bet that other web application scanning services (White Hat, et al), penetration testing companies, managed security services companies, and forensic analysis services could add a lot to get a more comprehensive picture.  As a professional community, we need to find ways to do this sort of study in a way that can be extended and combined with other data sets and studies.