(how probability underlies knowledge outside understood conditions)
(a swipe at short-termism in accepting excessive risk)
(common statistical illiteracy affecting real-life (medicine, courts))
Marcus has written The Anatomy of Security Disasters and a version 2.1 in PDF
This is an interesting article I largely agree with but I thought some aspects of it have room for another view. I'm going to make a series of short quotes with my comments between them.
I'm very skeptical of the notion that "Risk Management" has any value beyond the butt-covering obviousness of having made an attempt."Due diligence" is the expression that captures this.
The inspiration for this paper came from a discussion I had with Alan Paller, the founder of SANS and the CIO Forum. He was quoted in an article as saying that CIOs were regularly lied to regarding security by their technical staff. Bear in mind that this is the viewpoint as expressed from the position of the CIO: corporate executives felt that they had done their job when they told technical staff to "make it secure." Technical staff had cheated by doing naughty things like leaving unauthorized connections between critical networks, leaving systems unpatched and so forth.
Marcus does a great job of explaining how those apparent lies may in fact be misunderstandings; but it doesn't completely exclude the possibilty the CIOs were lied to.
Things I would ask (and I think I can guess the answers) are:
It's easy to grumble that people in an organisation don't follow policy; but it's hard to be surprised about it if those at the top don't enforce their policies. That presupposes that the policy expresses something possible and is enforceable (which should be considerations when writing policy).
Has anyone fired project managers for putting things live with unacceptable security failings? What about publishing (internally) a list of all the people who've been punished recently and for what? I can read in the local paper who's been convicted of drink-driving but there's no organisational or social pressure like it at work. So is anyone surprised that staff (including technical staff) find it easier to ignore the policy?
I have seen this play itself out dozens of times, in critical decisions made regarding IT security during the course of my career. The reality gap comes into play when the executive decided to shop the idea around: he asked for this thing to be done securely, and got a resounding "no" from the security group, but a "yes" (it can be done, but not securely) from the web group. All he hears is the "yes" and when the whole thing blows up a year later, his memory will be that he asked if it could be done safely, but someone lied to him.
And if there's been a reshuffle meanwhile it may be a different manager.
It used to be difficult for a security practitioner to argue against the idea of risk management. It sounds so pure and mathematical. Unfortunately for us all, the Wall St crash of Dec 2008 serves as a complete debunking of the value of risk management. All the big firms that lost billions or went out of business had risk management departments and practices and felt they were taking acceptable risks.
This is criticism of bad risk management; rather than of risk management as such. Schneier's Beyond Fear is what I recommend as an introduction to risk assessment.
The difference between a risk-taking skydiver and a risk-taking corporate executive is that the skydiver's risk is direct and personal whereas the executive's typical worst-case scenario is that they have to use their golden parachute and then find a new company to lead off a different cliff.
That's a matter involving incentives .. about which more later.
... required reading for anyone who claims to believe risk management is practical. To summarize it: you can only play Las Vegas odds-maker when you're working on small numbers of variables and extremely well-understood conditions.
Knowledge, evidence, risk and probability are wider than that. I'll mention some books (with ridiculously-brief one-line reviews).
A fad that is related to risk management is the use of economic models to talk about security. I'm not, I confess, up to date on the latest literature in this area, but the gist of the idea seems to be to take a risk management approach and then try to rationalize what actions make the most sense from a standpoint of cost-effectiveness. I have already asserted my assumption that risk management consists largely of compounded wild-ass guesses, so an economic model built to optimize wild-ass guesses is not going to be worth any more than the paper it's printed on. I've talked to fans of economic models that who explained security to me in terms of it being a "market failure" - a market failure being what happens when a market does not adequately self-regulate because customers don't have anything rational on which to base their decisions. Outside of Las Vegas, there must be very few markets that are not "failures." Security is, certainly, one. When economists talk about a "market failure" I hear the sound of hands waving; I do not think we can expect any help or illumination from that quarter.
Here's a quote from the book
Computer Security Basics
... assess the impact of security costs in relation to expected security benefits. ... Dennis Steinauer of the National Institute of Standards and Technology put it this way, "Controls that are more expensive than the value of the information they protect are not cost-effective. Absolute security is achieved only at unlimited cost."This points out that security decisions (and not just in IT) are fundamentally economic - about what's worth doing. It's not worth hiring guards to protect my old car. I would describe this by saying security involves economic considerations rather than economic models.
It's poor economic models (and their elevation along with stories of highly-paid "brilliance") that Taleb writes about. And Michael Lewis writes about; e.g. here on AIG where the model seems to have been the simple one of not noticing that one thing differs from another.
My suspicion is that the "reality gap" between management's expectations and what they actually have out there on their networks is larger than they realize. I think it is vastly larger.
I agree; it's very easy to get a reality gap. It happens when bosses don't have the time, inclination or knowledge to follow the detail of what's in their network - i.e. it applies practically everywhere. It can be fed by reluctance to pass on bad news and by wishful thinking that says if you've got a security policy that's a good reason to be confident everyone follows it.
If the attitude of "this risk is acceptable because it has not resulted in a failure yet" sounds familiar to you, it should. That's the history of the computer security disaster in a nutshell.
That's the history of nearly everything. Concorde had a good history of 30 years before an unfortunate event that got it discontinued. And the RBMK had a similar good run - developed in the 1950s before Chernobyl became a household word in 1986. I don't think IT has any major differences that make it exempt from the problems described elsewhere.
E.g.: "By the way, since you asked us to keep the management costs on that system down, we have followed your directive and are now using an Internet-based remote control interface. Since the old system had zero chance of compromise over the Internet, and the new one has something more than zero chance, we can realistically say that it is infinitely more dangerous to proceed in this fashion." I believe that is how a nay-sayer would put it.
There Marcus is using a biased risk figure.
Infinity (something/zero) is the relative risk increase
(rather than absolute)
and it is also the risk arising from Internet-based control as opposed
to the risk from all causes.
Ben Goldacre's Bad Science (chapter 14)
Bad Science
calls it
the single most melodramatic and misleading way of
describing any statistical increase in risk
In 1997, at The Black Hat Briefings, I suggested that we should scrap the Internet and our installed base of Internet apps, and start over (blaming the whole thing on Y2K). Everyone in the room laughed. But I wasn't joking.
That was 1998 rather than 1997 and in the powerpoint archived at blackhat.com Marcus points out:
Those points raise economic ideas - some of the same ones you'll find
at these links (due to Ross Anderson):
Economics and Security Resource Page
video
Incentives are definitely a factor in how decisions get made.
Apparently "ship it on Tuesday and get it right by version three" was
the common saying inside Microsoft - which makes sense because it's not
the software vendor who suffers when you get hacked.
The conclusion of
Reactor Accidents: Institutional Failure in the Nuclear Industry
In a situation like this robbery from The Real Hustle TV show the trick is to convince people that a policy ("shopping must be paid for before departing with it") doesn't apply this time. Relaxing the adherence to security policy needs to be recognised as risky even when it doesn't feel like it.
And two books I haven't read (yet)
Human Error
The PDF version of the article has a footnote saying
The only way the outsourcing customer can be sure the outsourcer is doing what they claim to is to duplicate the effort, which is something that they are obviously unwilling to attempt in the first place. You don't need to be an expert in game theory to realise that an outsourcer is always more highly rewarded the more extremely they lie.Now I'm sure Marcus realises that's not always true. If the outsourced work consists of many units of something uniform then it can be sampled. And there are kinds of computation (e.g. sorting, factorisation) where it is easier to check a result than to obtain it in the first place.
I hope that encourages people to look at the economic aspects of security.
Blog Links
chuvakin.blogspot.com blog
blog.liquidinfo.net
blog.spaf.us
2009-07-21