I applaud the effort and goals of the study and it may have some useful conclusions. We should have more of this type of study, especially at a large scale.

Unfortunately, the report has some major problems, listed roughly in order of severity:  (for details, read on…)

1. The most basic charts are missing.  What was the total number of attacks in the study period?  The total number of vulnerabilities?  What percentage of identified vulnerabilities were actually attacked?  Were the most prevalent vulnerabilities attacked most frequently?  What how prevalent are zero-day vulnerabilities as a percentage of total vulnerabilities?
2. Some of the most important statements in the report have no backing data or detailed analysis.  Here are examples from the Executive Summary:

   “Waves of targeted email attacks, often called spear phishing, are exploiting client-side vulnerabilities in commonly used programs such as Adobe PDF Reader, QuickTime, Adobe Flash and Microsoft Office.”

   There is no analysis or data in the report regarding targeted email attacks (a.k.a. spear phishing).  The rise of spear phishing has been documented elsewhere, but if there was data coming from this study to support that assertion, it didn’t appear in the detailed section of the SANS report. 

   “Attacks against web applications constitute more than 60% of the total attack attempts observed on the Internet.” 

   Looking at the detailed sections of the report, I could not find any table or chart that listed the total number of attacks in the study and the percentage of attacks on each on category (i.e. web applications vs. app platform vs. OS).  (Figure 1 doesn’t do it.  Here is my critique of this bad graphic.)

   “Rising numbers of zero-day vulnerabilities”

   I couldn’t find any support in the report for the assertion that the number of zero-day vulnerabilities were rising.  There was no data analysis and no charts in the “Zero-day Vulnerability Trends” section.

   “On average, major organizations take at least twice as long to patch client-side vulnerabilities as they take to patch operating system vulnerabilities. In other words the highest priority risk is getting less attention than the lower priority risk.”

   There isn’t any chart that shows the average time to patch client-side vulnerabilities vs. OS vulnerabilities.  They don’t mention or analyze server-side application patching, which seems weird.  How about a chart with two bars, one for web applications and other for OS, showing the average number of days to reduce vulnerabilities by half?  That would be simple and clear.  Regarding the charts they do show…

3. The charts in the “Vulnerabilities” section are poorly executed and poorly explained.  The wiggly “percentage vs. days” graphs leave me scratching my head.  The data is probably in there, but it’s very hard to see in this format. Here’s my interpretation of these charts (but I might be wrong).  These are supposed to “aging charts” that graph the % of vulnerabilities that remain unresolved or unpatched after X days.  If none are resolved, then the line should be flat at 100%.  If all are resolved on day one, then it would be vertically down from 100% to 0%.  If they are gradually resolved, then the line will slope downward.  If some of the vulnerabilities are never resolved, then the line should plateau at some residual level. But why don’t the charts all start with a value of 100% on Day zero?  I don’t know and it’s not explained.  I’m guessing that it takes some number of days before all the vulnerabilities appear in the scans. More disturbing is the periodic oscillations in data series.  These jump out visually.  But what does this pattern mean?   You might be inclined to think that vulnerabilities were resolved every 5 to 7 days, only to reappear the next week, and so on.  That would be a huge process problem.  But it turns out that these oscillations are a spurious result of the scanning process and the data analysis method.  Consider this phrase from the report:

   “Periodic drops in detection rates occur during the weekends when scanning focuses on servers rather than desktop machines and the detection rates of vulnerabilities related to desktop software fall accordingly.”
   
   If this is really true (and it could probably be verified), then it would make sense to drop weekend data points and just include weekday data points for client-side vulnerabilities.  The data analysis could be further improved by using a smoothing or curve fitting method, because what counts in this analysis is not the day-by-day squiggles but the overall slope of the curve and whether it plateaus.  Finally there should be some marker for “half-life” or something, to allow easy visual identification of how long it takes to resolve half of the vulnerabilities in each class.

4. One finding was not highlighted but deserves more attention: the intentional use of security-violating methods by legitimate web applications.  This blurs the distinction between “good guys” and “bad guys” and makes security management much more difficult.  Examples from the report:

   “…an advertiser’s banner might be embedded in a web page which is set up to reflect some JavaScript off of the advertiser’s HTTP server for tracking purposes. However, in this case, there is little risk because the site in question (usually) has full control over his/her page, so this request to the advertiser is not generally malicious. It is the “reflection” attacks, along with attacks that leverage flaws in form data handling, that make up the vast majority of XSS attacks that we have seen in the last six months.” [emphasis added]

   ”A very large spike in SQL Injection attacks in July was caused mostly by an online advertiser who distributed code to many affiliates using SQL injection as functionality. The application was quickly pulled, resulting in a large drop in events for the month of August.”

   Related to this trend is the intentional use of security- and privacy-violating technologies to support marketing purposes, e.g. FTC’s action against Sears .

5. The analysis of Cross-site Scripting is confusing at best.  In the attack section, they say: “Cross Site Scripting (XSS) is one of the most prevalent bugs in today’s web applications.”  But looking at the graph immediately below (Fig. 13), it doesn’t look like XSS attacks are frequent – numbering in the thousands while other attacks number in the millions.  Looking down to the vulnerabilities section, there is no data or analysis of XSS vulnerabilities as a percentage of the total.
6. There is no mention of uncertainty or confidence levels in the analysis or conclusions.   For example, was the TippingPoint attack data adjusted for false positives and false negatives?  It’s well known that all Intrusion Detection/Prevention Systems (IDPS) are prone to false positives and false negatives, no matter how well designed or tuned.  This doesn’t discredit the data or analysis, but it does add uncertainty and reduce confidence levels. 
7. There was no mention of the limitations of the attack or vulnerability data.  Of course, all data sources will have limitations.  It’s essential that the authors factor this in to the analysis and make clear that they are not analyzing the full spectrum of cyber attacks or vulnerabilities.  For example, there is no coverage of insider threats, social engineering, combined physical and cyber attacks, or data leakage.  There is also no coverage in the data sets for complex targeted attacks.

   Attack data limitations. As a network appliance, TippingPoint’s Intrusion Detection/Prevention Systems (IDPS) do not detect all types of attacks.  Specifically, they cannot detect attacks within encrypted network traffic, attacks that come through wireless protocols, or attacks that utilize application payload, e.g. code injection.  Also, the IDPS are susceptible to various types of attacks, most involving large volumes of traffic, which can “blind” the IDPS for a period of time.  IDPS appliances also don’t detect authorization or authentication attacks.  (I’m no expert in IDPS or intrusion detection in general, so I’m drawing from other sources, including NIST 800-94: Guide to Intrusion Detection and Prevention Systems.) 

   Vulnerability data limitations.  Qualys vulnerability scanning service identifies many vulnerabilities in hardware configuration, operating systems, networking, libraries, platforms (e.g. Adobe Acrobat and Flash), etc., including: Misconfigured or unpatched servers, laptops and desktops; Out-of-date or misaligned security policy; Unauthorized hardware, software or applications; Easily-guessed passwords; and Inadequate controls on traffic from trusted third-party networks.But there are limitations. 

   Most obvious but unstated, Qualys identifies known vulnerabilities only, not vulnerabilities that have not yet been discovered.  While CERT has been quoted as saying “99% of attacks exploit known vulnerabilities”, this does not mean that there is no risk associated with undiscovered vulnerabilities. 

   More important is that automated vulnerability assessment scanning doesn’t provide a macro view on the relative riskiness of vulnerabilities. According to a Gartner report, vulnerability assessment scanning systems “provide data on devices within the network, but it remains difficult to understand how the overall network is vulnerable, or how vulnerabilities within a device affect their neighbors. What may appear as a benign or low-priority vulnerability on a host may be used as a launching point for an attacker to penetrate other devices on the network.”   Thus, the prioritization of vulnerabilities in the report seems to be based on their prevalence in the total population, not based on the relative riskiness of each vulnerability in it’s IT and network context.  (That would require other analysis, including attack graphs, penetration testing, etc.)

   Finally, Qualys only added Web Application Scanning (WAS) to their QualysGuard Security and Compliance Suite on May 26, 2009. This was half-way through the study period (March – August 2009), and it’s not clear from the report whether WAS produced any data used in this study or how widely it is being used in the customer base.  (Qualys suite is software as a service (SaaS), so new functionality should be immediately available to all subscribers.  Who knows what percentage of customers started using that functionality.)

8. There was very little explanation of the methodology.  I’m guessing that they drew conclusions based on total number of attacks of various types, compared to the total number of vulnerabilities of various types.  I didn’t see any analysis of the severity of each attack type, or any data regarding how many of these attacks succeeded in exploiting vulnerabilities.
9. There is insufficient background on the data sets.  For example, what are the demographics of the TippingPoint data (attacks) and the Qualys data (vulnerabilities)?  How many TippingPoint customers are also Qualys customers?  Some of the context information that was listed seemed irrelevant, e.g. “TippingPoint intrusion prevention systems protect 6,000 organizations” and “vulnerability data from 9,000,000 systems compiled by Qualys”.  Yes, that’s a lot of organizations and systems, but compared to what?
10. What was the point of the charts and analysis on the geographic origins and destinations of attacks?  I presume that this could be very important for understanding the attacking parties (“bad guys”) and the resources they use. Maybe it could also to help guide IT managers on the likelihood of certain attacks in their geography.  But the analysis falls short of either purpose.  As it is, these charts simply make me say “hmmmmm….”.

As a general comment, the report is poorly organized.  I found it hard to read, and it looked like a “cut and paste” job with no overall editing.  It was especially hard to trace the line of reasoning from the executive summary to the detailed sections.  Also, “…Four Key Attacks” section is confusing.  They actually list five key attacks, not four, but then say there are really only two categories: Server-side HTTP attacks and Client-side HTTP attacks.  Mis-numbering like this is a basic editing mistake.

Lastly, I wonder if there is any way that this study could be augmented and extended with data from other sources.  I bet that other web application scanning services (White Hat, et al), penetration testing companies, managed security services companies, and forensic analysis services could add a lot to get a more comprehensive picture.  As a professional community, we need to find ways to do this sort of study in a way that can be extended and combined with other data sets and studies.