Please nominate in the comments.

Also, please discuss what the criteria should be.

]]> http://newschoolsecurity.com/2012/02/time-for-an-award-for-best-data/feed/ 3 http://newschoolsecurity.com/2012/01/sharing-research-data/ http://newschoolsecurity.com/2012/01/sharing-research-data/#comments Mon, 30 Jan 2012 15:45:38 +0000 adam http://newschoolsecurity.com/?p=2484 I wanted to share an article from the November issue of the Public Library of Science, both because it’s interesting reading and because of what it tells us about the state of security research. The paper is “Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results.” I’ll quote the full abstract, and encourage you to read the entire 6 page paper.

Background
The widespread reluctance to share published research data is often hypothesized to be due to the authors’ fear that reanalysis may expose errors in their work or may produce conclusions that contradict their own. However, these hypotheses have not previously been studied systematically.

Methods and Findings
We related the reluctance to share research data for reanalysis to 1148 statistically significant results reported in 49 papers published in two major psychology journals. We found the reluctance to share data to be associated with weaker evidence (against the null hypothesis of no effect) and a higher prevalence of apparent errors in the reporting of statistical results. The unwillingness to share data was particularly clear when reporting errors had a bearing on statistical significance.

Conclusions
Our findings on the basis of psychological papers suggest that statistical results are particularly hard to verify when reanalysis is more likely to lead to contrasting conclusions. This highlights the importance of establishing mandatory data archiving policies.

Despite the fact that the research was done on papers published in psychology journals, it can teach us a great deal about the state of security research.

First, the full paper is available for free online. Compare and contrast with too many venues in information security.

Second, the paper considers and tests alternative hypotheses:

Although our results are consistent with the notion that the reluctance to share data is generated by the author’s fear that reanalysis will expose errors and lead to opposing views on the results, our results are correlational in nature and so they are open to alternative interpretations. Although the two groups of papers are similar in terms of research fields and designs, it is possible that they differ in other regards. Notably, statistically rigorous researchers may archive their data better and may be more attentive towards statistical power than less statistically rigorous researchers. If so, more statistically rigorous researchers will more promptly share their data, conduct more powerful tests, and so report lower p-values. However, a check of the cell sizes in both categories of papers (see Text S2) did not suggest that statistical power was systematically higher in studies from which data were shared. [Ed: "Text S2" is supplemental data considering the discarded hypothesis.]

But most important, what does it say about the quality of the data we so avariciously hoard in information security? Could it have something to do with higher prevalence of apparent errors?

Probably not. It might surprise you to hear me saying that, but hear me out. We almost never have hypotheses to test, and so our ability to perform statistical re-analysis is almost irrelevant. We’re much for fond of saying things like “It calls the same DLLs as Stuxnet, so it’s clearly also by the Israelis.” Actually, there are several implied hypotheses in there:

1. No code by different authors calls the same DLL
2. No code calls any undocumented APIs
3. Stuxnet DLLs are not documented

Stuxnet being written by the Israelis is clearly not a hypothesis, but a fact, as documented by Nostradamus.

More seriously, read the paper, see how good science is done, and ask if anyone is holding us back but ourselves.

Thanks to Cormac Herley for the pointer.

]]> http://newschoolsecurity.com/2012/01/sharing-research-data/feed/ 1 http://newschoolsecurity.com/2012/01/paper-the-security-of-password-expiration/ http://newschoolsecurity.com/2012/01/paper-the-security-of-password-expiration/#comments Thu, 05 Jan 2012 16:19:14 +0000 adam http://newschoolsecurity.com/?p=2433 The security of modern password expiration: an algorithmic framework and empirical analysis, by Yingian Zhang, Fabian Monrose and Michael Reiter. (ACM DOI link)

This paper presents the first large-scale study of the success of password expiration in meeting its intended purpose, namely revoking access to an account by an attacker who has captured the account’s password. Using a dataset of over 7700 accounts, we assess the extent to which passwords that users choose to replace expired ones pose an obstacle to the attacker’s continued access. We develop a framework by which an attacker can search for a user’s new password from an old one, and design an efficient algorithm to build an approximately optimal search strategy. We then use this strategy to measure the difficulty of breaking newly chosen passwords from old ones. We believe our study calls into question the merit of continuing the practice of password expiration.

This is the sort of work that we at the New School love. Take a best practice recommended by just about everyone for what seems like excellent reasons, and take notice of the fact that human beings are going to game your practice. Then get some actual data, and see how effective the practice is.

Unfortunately, we lack data on rates of compromise for organizations with different password change policies. So it’s hard to tell if password policies actually do any good, or which ones do good. However, we can guess that not making your default password “stratfor” is a good idea.

ACM gets a link because they allow you to post copies of your own papers, rather than inhibiting the progress of science by locking it all up.

If you or your child came down with influenza during the H1N1, or swine flu, outbreak in 2009, it may not have happened the way you thought it did.

A new study of a 2009 epidemic at a school in Pennsylvania has found that children most likely did not catch it by sitting near an infected classmate, and that adults who got sick were probably not infected by their own children.

Closing the school after the epidemic was under way did little to slow the rate of transmission, the study found, and the most common way the disease spread was a through child’s network of friends.

The work he discusses is “Role of social networks in shaping disease transmission during a community outbreak of 2009
H1N1 pandemic influenza” by Simon Cauchemeza, Achuyt Bhattaraib, Tiffany L. Marchbanksc, Ryan P. Faganb, Stephen Ostroffc, Neil M. Fergusona, David Swerdlowb, and the Pennsylvania H1N1 working group.

The first thing that comes to mind is that closing schools is a best practice. It’s something that makes so much sense that it’s hard to argue against, even if it does no good. The next thing is look at what happens when they have data available to them. They can study their prescriptions and test to see if they did any good. But note how detailed the data is: social graphs, seating charts. This isn’t something we would obviously get from more detailed breach notices. It’s going to require in-depth investigations, and investigators who talk about their methods. VERIS is a step in this direction, and I’m looking forward to seeing critiques or even competitors that can help us move forward and learn.

But the data we have is the data we have, and while we work to get more, there’s a good deal that we can probably learn from what’s out there. We just have to be willing to ask if our practices really work.

]]> http://newschoolsecurity.com/2011/02/infosecs-flu/feed/ 1 http://newschoolsecurity.com/2010/03/more-bad-news-for-ssl/ http://newschoolsecurity.com/2010/03/more-bad-news-for-ssl/#comments Fri, 26 Mar 2010 18:51:41 +0000 Chandler http://newschoolsecurity.com/?p=1520 I haven’t read the paper yet, but Schneier has a post up which points to a paper “Side-Channel Leaks in Web Applications: a Reality Today, a Challenge Tomorrow,” by Shuo Chen, Rui Wang, XiaoFeng Wang, and Kehuan Zhang.about a new side-channel attack which allows an eavesdropper to infer information about the contents of an SSL connection in certain contexts, some of them fairly common.  For example (from Schneir’s link to Ed Felten’s commentary on the paper):

The new paper shows that this inference-from-size problem gets much, much worse when pages are using the now-standard AJAX programming methods, in which a web “page” is really a computer program that makes frequent requests to the server for information. With more requests to the server, there are many more opportunities for an eavesdropper to make inferences about what you’re doing — to the point that common applications leak a great deal of private information.

Consider a search engine that autocompletes search queries: when you start to type a query, the search engine gives you a list of suggested queries that start with whatever characters you have typed so far. When you type the first letter of your search query, the search engine page will send that character to the server, and the server will send back a list of suggested completions. Unfortunately, the size of that suggested completion list will depend on which character you typed, so an eavesdropper can use the size of the encrypted response to deduce which letter you typed. When you type the second letter of your query, another request will go to the server, and another encrypted reply will come back, which will again have a distinctive size, allowing the eavesdropper (who already knows the first character you typed) to deduce the second character; and so on. In the end the eavesdropper will know exactly which search query you typed. This attack worked against the Google, Yahoo, and Microsoft Bing search engines.

SSL has been touted as a Web security panacea for years, but the harsh reality is that its weaknesses are growing rapidly, made worse by the changing ways that HTTP is used–when the expected SSL-protected transaction was a page request followed by the return of a full page of content, it was extremely difficult to infer the contents of the connection.  Now that the requests and responses are relatively atomic, even down to the characte, this is no longer the case.

And as old assumptions fail, so does security built on top of those assumptions.

• A Roadmap for Cybersecurity Research, by DHS
• National Cyber Security Research and Development Challenges , by the I3P
• Toward a Safer and More Secure Cyberspace, National Academies
• Report to the President on Cyber Security: A Crisis of Prioritization , by PITAC
• Ensuring (and Insuring?) Critical Information Infrastructure Protection, 2005 Rueschlikon Conference on Information Policy
• Four Grand Challenges in Trustworthy Computing , Computing Research Association Conference, 2003
• Others

But no one seems to be able to mobilize any signficant research into solutions.   It’s been very frustrating to see so much talk and so little action.   

This reminds me of the quote by Mark Twain, shown at left, which is the inspiration for the title of this post.

The latest iteration of this was a panel at RSA: “The role of research in industry and government“.  SC Magazine summarized the discussion this way:

A disconnect between the primary research sectors and a lack of appropriate funding in each is leading to decreased technological progress, exposing a huge gap in security that is happily being exploited by cybercriminals.

(read on for a diagnosis and two proposed solutions…)

Part of the problem is the the incentives to focus research on problems and not solutions.  I run into this a lot at academic and other “thought leadership” conferences.  Here’s how it was explained to me: It’s much easier to do a modest-sized research project that shows yet another failure in the economics of security than it is to do the complex, large-scale research that would be necessary to develop both theory and empirical support for solutions. 

The bias toward complaining and against doing research work is even stronger at industry conferences.  I don’t blame any individuals.  Simply put, everyone has a day job that pays them to solve near-term problems and deliver immediate payoffs.   High-risk, fundamental research does not fit that template.

There was one recent attempt to mobilize breakthrough research — the “National Cyber Leap Year Summit” last August, sponsored by NITRD.  As I’ve previously written, that effort was largely a waste of time and money because you can’t brainstorm your way through hard problems like this.

Gene Spafford (a.k.a. “Spaf”) is one person who has thought long and hard about how to effectively mobilize and support interdisciplinary information security research.  In the second half of this blog post, he mentions a white paper that he has been circulating in DC for feedback.   The white paper advocates “changing the way we fund some of the research and education in the US in cybersecurity” and makes specific recommendations.  It’s a good read and very thoughtful suggestions.  The second of his two suggestions can be summarized:

I suggest a program similar in nature to the MacArthur “Genius Grants” program: the ISPEG, or Information Security and Privacy Extended Grant. Some agency or agencies would provide ISPEG funding to a small number of researchers in multi-year fashion, to “do good things” in cybersecurity and privacy. The intent would be to fund these individuals without requiring specific proposals or highly structured budgets, and with minimal requirements for deliverables and constraints. The researchers would be encouraged to exercise vision and leadership to the betterment of the country and the field of cybersecurity. If they are carefully selected, this will naturally follow.

A small set of ISPEG awardees [should be] chosen annually. These individuals will be senior academic, tenured faculty, chosen on the basis of past accomplishments specifically in the fields of information security and privacy, and because of a commitment to service and education. [emphasis added]

I think this is a keen idea overall.  Several formal studies of scientific performance have shown that the most productive method for acheiving major research innovations is through senior, experienced researchers who have both freedom and adequate support over an extended period of time.  However, Spaf’s model is aimed at supporting only academic researchers and only those researchers who have been blessed by the academic system (“tenured”).  Yes, they merit this sort of support, but they aren’t the only people who can or should play in the advanced research arena. Therefore I want to propose another idea that could work in parallel with ISPEG.

Proposal: Information Security Pioneers Fellowship Program (ISPFP)

Here’s how it might work. A non-profit organization would administer the program and would be the “home” for a number of individuals (the “Pioneer Fellows”) who would have financial and institutional support for a period of time. In return for this support, they would serve as catalysts, leaders, orchestrators, and even program managers for innovative interdisciplinary research projects, esp. those that involve industry, academic, and government partners. They could also work on projects and activities that enable advanced research or help bring advanced research results to the masses: in education, industry, or government policy. For example, here are some specific project ideas that would be well suited for Pioneer Fellows:

• Organizing and leading multi-organization proposal teams for advanced interdisciplinary InfoSec research projects (“Broad Agency Announcments” from DARPA, DHS, NSF, NIST, others).
• Leading the specification and field testing of security metrics, e.g. Center for Internet Security’s consensus metrics , and also pilot implementations.
• Leading the design and implementation of a statistically robust survey of information security practices, metric results, and costs, to displace the current “Computer Crime and Security Survey” (CSI/FBI).  (“Statistically robust” would include random sampling of organization populations, for example.)
• Design and help implement a “Cyber CDC” for advanced vulnerability and threat research and intelligence.
• Organize, lead, and/or collaborate in international research projects. 
• Help integrate economics, organization science, and behavioral science into education, training, and certification programs for security managers and executives.

Being a non-profit (preferably 501c3), they could accept and administer donations from many sources — corporations, foundations, and government. This would open the door to funding from many sources, including organizations that don’t usually provide funding, including VCs, industry associations, privacy advocates, IT vendors and consultants of all stripes, etc.

The fellowship period and applicant qualifications are open to consideration.  Ideally, this program should be “idea capitalists”, knowing some people and ideas won’t payoff but others will be huge winners.  One thing for sure — we shouldn’t focus this program only on people who have been “officially” annointed by some hierarchy, some certification program, or by credentials alone. 

OK… now for all of you who might be frustrated with lack of action, this message is for you:  THIS IDEA COULD BE IMPLEMENTED IMMEDIATELY!

Sorry to shout, but I want that message to hit you between the eyes.

First, there are several candidates for host institution:

• Center for Internet Security
• Security Innovation Network (SINET)
• European Network and information Security Agency (ENISA)

Second, there are a good list of possible projects, not only the list above but also ideas from any of the reports listed at the top of this post. 

Third, there are plenty of good candidates for Pioneer Fellows.  Just look for the people who are already doing pioneer work on their own dime or in their “spare time”.

Fourth, the funding would probably start flowing if the right executives were in the same room at the same time, and someone with sufficient “gravitas” asked for the order.  $35K to $50K per major sponsor is reasonable and comparable to other sponsorship arrangements.  Ten major sponsors would fund 8 to 10 Fellows, assuming they paid full salaries. Once this is all in place, we could probably solicit a “foundational grant” from a major government agency to ramp up recruitment and other administrative parts of the process.

That’s a sketch of the idea.  What do you think?

Human error has been implicated in 70 to 80% of all civil and military aviation accidents. Yet, most accident reporting systems are not designed around any theoretical framework of human error. As a result, most accident databases are not conducive to a traditional human error analysis, making the identification of intervention strategies onerous. What is required is a general human error framework around which new investigative methods can be designed and existing accident databases restructured. Indeed, a comprehensive human factors analysis and classification system (HFACS) has recently been developed to meet those needs.

Consider that pilots, whether private, commercial, or military, are one of the more stringently trained and regulated groups of people on the planet.  This is due, at least in part, to the history of aviation.  As the report notes,

In the early years of aviation, it could reasonably be said that, more often than not, the aircraft killed the pilot. That is, the aircraft were intrinsically unforgiving and, relative to their modern counterparts, mechanically unsafe. However, the modern era of aviation has witnessed an ironic reversal of sorts. It now appears to some that the aircrew themselves are more deadly than the aircraft they fly (Mason, 1993; cited in Murray, 1997). In fact, estimates in the literature indicate that between 70 and 80 percent of aviation accidents can be attributed, at least in part, to human error (Shappell & Wiegmann, 1996).

One upon a time, operating an airplane was so dangerous that only highly-skilled experts could do it, and even then the equipment would get out of their control and crash.  Later (yet still almost twenty years ago), the equipment improved to the point that equipment failure no longer overshadowed operator error, but planes still get out of control and crash.

Other than the fact that pilots are almost universally still highly-skilled and/or trained operators, this doesn’t sound all that different from the evolution of computing.

Flight has obviously never really had the adoption rate explode like PC’s in the Age of the Web, but there is still a strong parallel between aircraft accidents and Information Security failures.  This assertion becomes even more true once the paper gets into James Reason’s “Swiss Cheese” model of understanding root causes of aircraft accidents.

Reason identifies four factors that interact with each other increase accident rates, which I’ll paraphrase as:

1. Unsafe Acts — This is the cause of the active failure (i.e. crash), such as a poor decision or a failure to watch the instruments or otherwise recognize the unsafe situation was forming or occurring
2. Preconditions for Unsafe Acts– Situations that increase risk of an accident, such as miscommunication between aircrew members or with others outside the aircraft, such as air traffic control
3. Unsafe Supervision– failures of management or leadership to recognize when they are, for example, pairing inexperienced pilots together in less-than-optimal conditions
4. Organizational Influences — Usually business-level decisions, such as reducing training hours to reduce costs

How familiar does this sound?  If you’ve ever read an IT Audit report, this should seem painfully familiar, even if only analogously.  The paper provides a strong taxonomy within each area, and I could easily drill down at least one more level into each one.  Read the paper to learn more and become a better professional problem solver, security-related or otherwise.

For example, using a real-world case I dealt with recently.  This is an easy example which ties the four levels together more neatly than many, so consider it an “Example-Size Problem” and extend as you see appropriate.

The incident was the loss of sensitive business information, which I personally believe hurt the company in a negotiation:

1. Unsafe Act:  The VP left his unencrypted laptop unattended while at a meeting — this was the Active Failure/Unsafe Act that led to the Mishap
2. Preconditions:  The VP assumed that others were watching his laptop, but did not explicitly confirm this fact
3. Unsafe Supervision:  Despite knowing that Executives are high-risk users with regards to sensitive information on their laptops, the IT Executive Support Team had recommended against deploying Full-Disk Encryption on executives’ laptops because they feared being held accountable if an executive lost information due to an encryption system failure
4. Organizational Influences:  While a Laptop Encryption Policy existed and specified that the VP should have been encrypted for multiple reasons, the policy was widely ignored, there was no cultural pressure to ensure that mobile information was protected, and thus compliance was unacceptably low.  No pressure to comply was generated by Executive management because the cost associated with doing so was considered to be prohibitive.

In this case, the damage (opportunity cost) of lost revenue due to that single lost laptop was many multiples of the complete cost of deploying a Full-Disk Encryption system.  Unfortunately, in the absence of a comprehensive analysis of the series of failures leading up to the unsafe act, the real root cause of an incident may be ignored or mis-assigned, leading to either an incomplete or unsustainable remediation course.

When incidents occur, it’s rare to see a true and honest assessment not just what went wrong, but why.  Too often, in fact, the culture seems to be to put it down to, “nobody could have predicted it.”  Reject these assessments.  To improve an organization, we must refuse to accept these explanations.  Instead, find the root cause–all the way up to the Organizational Influences–and then Fix It.

I think the paper’s key hypothesis “securtity can be correctly represented with quantitative information” is overly broad. Can you replace the term security with something more precice? For example, I would take issue with the claim “health can be correctly represented…” but there are lots of usefully measurable aspects of health. Also, I would argue that there are lots of useful things which are not correct. (In this I take the view that we can disprove hypotheses and thus come closer to correct, but the best we do is either “wrong” or “well tested and not easily shown false”) There’s testable/faslifiable, and there’s operational improvement, and neither requires correctness. That would lead to something like “information confidentiality can be made less bad through quantification,” which I think is nearly semantically equivalent (or can be made into a set of equivallent statements) which are stronger Popperian hypotheses. Going a little further afield, I’d like to offer up two alternatives:

“Information security is no different than other disciplines which
must be measured to be improved.”

“Information security is different from other operational/engineering
disciplines in ways which make quantification irrelevant.”

Anyway, it’s a thought provoking paper, and worth a look.

]]> http://newschoolsecurity.com/2010/01/is-quantified-security-a-weak-hypothesis/feed/ 3