Archive for the 'measurement' Category

Data void: False Positives

There’s a good post at Gartner pointing out the lack of data reported by vendors or customers regarding the false positive rates for anti-spam solutions.  

Although Gartner customers almost never complain about false positive rates, I wonder if false positives are under estimated. End users rarely complain about false positives, but they are very vocal reporting Spam in their inbox. Box Sentry (www.boxsentry.com) recently did a tests in a number of organizations and found the false positive rate in some organizations using popular anti-spam tools was as high as 13% of legitimate emails. The largest proportion of false positives in their study was legitimate person-to-person traffic.  While it could be that these organizations have over-tuned their systems to block more Spam at the expense of quarantining more legit email, the reality was the email administrators had no idea they had such a high false positive rate because they never checked.  Have you? 

Going further, it would be very valuable to estimate the cost of false positives.

As I’ve discussed in a previous post, this is just another instance of a general problem in the security industry.  You can’t do rational analysis of effectiveness, cost-effectiveness, risk, and the rest without some estimate of false positive rates and their costs.

The Visual Display of Quantitative Information

In Verizon’s post, “A Comparison of [Verizon's] DBIR with UK breach report,” we see:

pie-charts-suck.jpg

Quick: which is larger, the grey slice on top, or the grey slice on the bottom? And ought grey be used for “sophisticated” or “moderate”?


I’m confident that both organizations are focused on accurate reporting. I am optimistic that this small example in the utlity of pie charts will inform report writers. The report writers and their graphics departments, loving their customers, will move to bar charts to help them compare numbers between sources.

I’m confident that not using pie charts is a best practice.

Elsewhere: “The only time it makes sense to use a pie chart.”

And elsewhere: “The Visual Display of Quantitative Information, 2nd edition

Help EFF Measure Browser Uniqueness

The EFF is doing some measurement of browser uniqueness and privacy. It takes ten seconds.

Before you go, why not estimate what fraction of users have the same
transmitted/discoverable browser settings as you, and then check your
accuracy at https://panopticlick.eff.org. Or start at http://www.eff.org/deeplinks/2010/01/help-eff-research-web-browser-tracking for a bit more detail.