Dominic Sayers

Root cause, part 2

I read an excellent essay by Joel Spolsky today about Problem Management (except that he cleverly doesn't mention Problem Management by name at all). Some things I took away: 1. Joel's approach of talking about a real-life incident is exactly the strategy I'm using to implement Problem Management at my organisation. Nobody wants to sit through Powerpoint slides with process flow charts on them, but get them talking about a real problem and they are hooked. Thanks for the confirmation that my approach is not totally mad. 2. He talks about root cause as the process by which you decide where to target your long-term solution. This is exactly the pragmatic approach to root cause analysis I was trying to describe in my last post. I wasn't aware of Sakichi Toyoda's Five Whys maxim, but it makes some sort of sense. The first couple of questions you ask are going to be about the immediate symptoms of a problem. Beyond 5 questions and you are in danger of looking too deeply at the mysteries of the universe. You could well end up trying to implement too general a solution, at great expense, which fixes problems you don't actually have. So my amended principle is this: a root cause is that which we can alter to effect a permanent fix to a problem such that the solution addresses more than just the immediate symptoms of the problem. The important points again are that the solution domain is limited to the space we can directly affect ourselves and that the solution should be a permanent or long-term fix, not a palliative for the symptoms.

Definition of root cause

I was asked for a definition of root cause today. Here is what I came up with:

ITIL doesn’t define “root cause” in any useful way. The official ITIL definition is “the underlying or original cause of an Incident or Problem” (ITIL v3 glossary).

CMMI is more helpful: “A root cause is a source of a defect such that if it is removed, the defect is decreased or removed.” (CMMI v1.1, Causal Analysis and Resolution process area). There are two effective parts to a useful definition: a root cause is that which we can alter to effect a permanent fix to a problem. So the important points are (i) addressing the root cause fixes the problem and (ii) it is something we can directly affect. Examples: 1.       Database fails because it is full. The root cause is that we failed to monitor its usage effectively to take preventative action in time. So the action we take to fix the Incident (increase the database size) is different to the action we take to fix the Problem (start monitoring capacity usage effectively). 2.       Database fails because a virus infects the database software. Here we are not interested in the coding error that allowed the virus to strike, we are only interested in addressing our technology choice or patching regime. In other words, the root cause for the vendor (software bug) is different to the root cause for us (technology management). Wikipedia is very helpful on this and has good articles on root cause and root cause analysis. The temptation is to get too philosophical about causation when a useful definition is circumscribed by your own sphere of influence.

A degree of risk (part 2)

I was talking with a colleague a couple of weeks ago about risk in software development projects and I said, "so, the aim of a mature development process is to reduce the number of errors that happen after a code release as far as possible". And he said, "No, that's not right - the aim is to optimise the number of errors. You can have too few errors". I think I understand what he meant and on reflection I agree with him. This is my attempt to explain why. Unless you're a TV presenter, it's quite easy to understand that risk is nearly always present but is mostly manageable. Often this is inescapable - no amount of effort will reduce the risk to zero. The question is, should we ever try? Should zero risk be the goal of any undertaking? It depends, of course. You will struggle to make a business profit without taking risks. That's accepted. But what if you are developing the control system for a passenger aircraft? What is the acceptable risk in that undertaking? One fault in your output could have fatal consequences. Should you pursue zero risk in that sort of project? Well, aircraft control systems are complex and there is a limit to how thoroughly they can be tested (although that limit is quite high). You can take action to ensure that many sources of error are eliminated systematically, but I don't believe you can ever get to 100% certainty that an aircraft control system is safe. At some point you have to install your system on a plane and watch it take off. The things you can do to make that experience less nerve-racking are ordinary bits of good engineering practice. Lessons learned from centuries of building stuff from wood, stone, iron, steel, concrete and glass can be translated into software projects fairly seamlessly. Transparent project management, repeatable processes, eliminating unnecessary manual tasks, identifying valuable metrics, keep it as simple as possible, etc. The important point is to understand where the risk arises and mitigate it whenever it is sensible to do so. And that's the point I've been trying to get to. There are many sources of risk and not all are equal. If a risk can be mitigated cost effectively then you should do it. The key word is "cost-effective" and the trick is to get the cost-benefit sums right. This can lead you to some surprising results. The point my colleague was making was that if you have too few errors after a code release, you may be spending too much on mitigating your project risks. You could have delivered the release earlier or more cheaply. How do you compare the value of expediting the release with the downside of delivering buggy software? The savings are relatively clear: if you deliver your software earlier or with fewer resources then there are direct and measurable cost savings. The other factor is opportunity cost - the sooner you deliver your software, the sooner it can start earning money for you or your customer. That loss of revenue (or loss of downstream cost savings) can be very significant and is certainly a factor in your calculations. How do you measure the downside? This is less clear because the errors are not predictable. One disastrous error could wipe out all the supposed benefits of the entire project. You could analyse the maximum loss from a disastrous error then try to estimate how likely it is to occur. For a trading system this should be doable - what is the effect of mispricing a trade by an order of magnitude? The maximum loss could be terrifying, but you can take steps to reduce the likelihood of it happening: (i) release code frequently so each release is only an incremental change from the previous one, (ii) pilot your new version with a restricted set of expert users, (iii) build sanity checks into the software to raise an alert when an unlikely event happens. I guess you are comparing the direct savings of your aggressive delivery schedule with the premium you would pay to insure yourself against the loss due to delivering bad software. The insurance premium will be lower if you can make the event less likely. You should take the risk if the savings exceed the premium. That's about as clear as I can make it. So anything you can do to mitigate risk with only a small cost, you should do. Definitely. Errors that arise from easily avoidable risk are just plain stupid and unprofessional. Doing things manually that you could easily automate, writing code in a bad style, using new technology just because it's new - these are commonplace sources of error which can be eliminated by adopting simple hygiene measures. On the other hand, taking expensive traders off a desk to spend time testing your new software in a test environment may not be such a good investment. It prevents them earning money (so it is not popular with them) and you are unlikely to get their full attention. So long as they understand and accept the risk of using software they have not tested personally then that is a business decision for them to make. This is an acceptable, managed risk.

A degree of risk (part 1)

We seem to believe that degrees of risk are too complicated for the average person to understand. You'll see this next time there's a health scare; the TV presenter will ask the expert "is it safe?", the expert will try to say something like "the risk is very small" and the TV presenter (probably a Humanities graduate) will say "but is it safe?" and so on. The presenter can't be bothered to understand that there is a continuum between absolutely safe and unacceptably unsafe, and assumes that the viewer cannot do so either. This annoys me for a number of reasons, but mainly because I feel like an oppressed minority. Just because I am numerate I am not part of the target audience. In trying to make the message comprehensible to the dumbest viewer (and Humanities students) the opportunity to convey useful information is lost. So we get used to this picture of risk being all or nothing, black or white, safe or unsafe. Look at this report celebrating 30 years of the UK's Health & Safety Executive (HSE):

"HSC’s annual report for 1977/78 states: ‘Our overriding concern is… to stimulate awareness of the risks and encourage the joint participation of workers and management in efforts to eliminate them.’ In 2004, the mission for HSC and HSE is to work with LAs ‘to protect people’s health and safety by ensuring that risks in the changing workplace are properly controlled’. The style may be different and the message broader but the core objective is essentially the same."

Really? Is eliminating risk really the same as controlling it? I would say absolutely not, and the 2004 mission statement is a huge improvement on the 1977 version. The fact that the HSE's own self-congratulatory pamphlet fails to point out the difference is an assumption by them that their target audience can't understand the difference. So we are led to a situation where an acceptance of risk is regarded as a Bad Thing. Gamblers and traders and other disreputable characters embrace risk. If the man in the street contemplates taking a financial risk, the regulator insists on apocalyptic warnings being read to him. If a popular investment turns out not to have been such a good idea after all, the newspapers blame the financial services industry or the government or the company selling the instrument, never giving the actual small investors credit for knowing what risk they were accepting when they bought it. I think it's a damaging mindset and it seems to be rooted in 1970s thinking: that's when the HSE was invented, that's when comprehensive schools started eliminating competition from sports periods, that's when licenses became necessary for bingo and gaming machines. Hopefully that sort of thinking has had its day and we can start taking a more mature attitude to everyday risk and the way we communicate it. I'm reminded of a scientist I once heard on the radio, who perceptively complained that although arts programmes are allowed to assume some prior knowledge on the part of the viewer or listener, science programmes have to explain everything from scratch every time. In other words, you can talk about Baudrillard as if everybody's read him but woe betide you if you mention a probability distribution without explaining in laborious detail what it is. Go ahead and quote Descartes on philosophy but don't dare mention cartesian geometry. Anyway, I was trying to get to a point where I could start writing about risk management of projects but I seem to have come to a natural break anyway. More later.

Books I should have read, update

Latest status: 1. The Tipping Point: How Little Things Can Make a Big Difference, Malcolm Gladwell (job done) 2. Blink: The Power of Thinking Without Thinking, Malcolm Gladwell 3. The Wisdom of Crowds, James Surowiecki 4. The Cathedral & the Bazaar, Eric S. Raymond 5. The Mythical Man Month, Frederick P. Brooks (purchased but not opened yet) 6. Liar’s Poker, Michael Lewis (now finished. Yes, good stories but not sure what it signifies today. Enjoyable nonetheless) 7. The Long Tail: How Endless Choice Is Creating Unlimited Demand, Chris Anderson 8. The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail, Clayton Christensen 9. The Elegant Solution: Toyota’s Formula for Mastering Innovation, Matthew May Oh dear, the list is getting longer faster than I can read. To add to my woes Mike Masnik has now published the bibliography for his collection of essays on the economics of ideas and content. I suspect that I should have read more of his list than I actually have.

Decommissioning stuff

I've been thinking about decommissioning IT systems, mainly because I've been asked to turn a few systems off at the place where I work. I've done this a few times before and it's never as easy as you might think. Removing 80% of the functional value of a system and 80% of the users is the easy bit. The last 20% of both is usually difficult both technically and politically. So a way of making the process of decommissioning an IT system more deterministic would be a good thing. Before I start trying to turn a system off I'd like to know what my chances of succeeding are.

Stakeholders

One thing I think you need to establish at the start is who the stakeholders are. For systems that have been around for a long time this can be surprisingly hard. These systems are often part of the furniture to the extent that users don't even realise they are using the system, it's just how they naturally do their jobs. An old system may have no formal governance because it was introduced at a time when the control environment was not as far-reaching as it is today. All this can make it hard to find out who is affected by the decision to decommission the system. What you would have ideally is a clear ownership of the system by the business, with a named owner and some sort of representation from each part of the business that uses the system. From the technical side you would have a named IT owner who understands the dependencies that other systems have on your target system. If these things don't exist for your target system then you are probably going to have to invent them. Because the first thing you need is a statement of the system's functional value from its users - if you can't get that then you should probably just turn it off anyway and see who screams. Armed with a statement of value and an analysis of functional dependencies from your IT owner, you at least know what work needs to take place to make conditions acceptable to turn off the system.

The needs of the many outweigh the needs of the few

A timeless story of innovation is that a functional gap has been plugged by a tactical system. The system is highly specific to the functional niche it inhabits and over time it comes to do everything the users want and they have learned to operate it efficiently. Then along comes Strategy, lumbering through the IT undergrowth like a tyrannosaurus. Whole swathes of tactical applications are replaced by one strategic system because it consolidates costs and organisation into something that senior management can understand. The strategic system is specified on the basis of functionality that is required by the user. Inevitably you are proposing to replace a tactical system that does everything the user wants with a strategic system that only does what they need. The cost distribution of the new system will have been worked out "fairly" by accountants. Which often means that your longest-standing users end up paying more for a system that does less. It can be a hard sell. Fortunately you have senior management on your side and if the users don't like it they can do their complaining from the deserted outpost they are transferred to.

Data retention

Your users are going to ask for all the data in the system to be retained indefinitely, and made available on line for ad hoc queries. Ignore them, they always say this. In my experience of having acceded to user requests like this, the data is never looked at after the end of the first month of the new system. Suggest a rational data retention policy and a straightforward migration of any applicable to data to the replacement system (if any). Be aware of legal and regulatory requirements.

Execution

I think the steps towards making a decision to decommission a system are these:

1. Establish a competent governance framework for the system.
2. Get a statement of the functional value of the system from its business owners
3. Get an analysis of the dependencies on the system from its IT owner
4. Get a signed agreement that certain named projects will replace any valuable functionality
5. Agree a rational data retention and migration policy
6. Get a signed agreement that the system can be turned off subject to the aforementioned project deliverables and data policy.
7. Plan the decommissioning like any other change using your best practice Change Management procedures.

Turning IT systems off is a good thing. I think it signifies that an IT organisation is mature but not senile. If understand your system landscape well enough to be able to say with confidence that this system is no longer required then you are doing something right. If you can actually execute that decision then it also reflects well on your relationship with your customers.

More on open source and security

Now Wired and TechDirt have chimed in on the same theme, that voting machines can only be acknowledged to be secure if the source code for their software is publically available. This isn't quite the same as open source, of course. They are not suggesting that the code should be publically changeable. Wired makes the valid point that scrutiny of the source code ought to be (and quite possibly is) a constitutional right. Also I didn't know that Australia had already done this two years ago.

The Economist on open source and security

I haven't yet attempted to support my belief that open source code is the only secure code, it remains a postulate. External support comes not just from Bruce Schneier et alii but this week also from The Economist. Unfortunately the online version of the article is hidden behind a paywall, but if you happen to be a subscriber then check it out here.

It's only a sentence at the end of the article, and I'm not sure the writer has any particular authority on the subject, but hey, it's The Economist :-)

Vendor lock-in and open document formats

Microsoft are investing in a converter that will allow their customers to read and write documents in the OpenOffice format. This is a good thing. It means if I choose to save my documents in OpenOffice format then I have a choice of two office suites to open them in (three if you count StarOffice).

NB My advice would therefore be to adopt the OpenOffice format to save your documents, whatever software you use to edit them.

Why are they doing this? Well, you will note they are not investing in a Microsoft Office converter for OpenOffice. That would simply encourage people to switch away from their product. I'm sure Novell are already working on such a converter for exactly that reason.

An OpenOffice converter allows them to retain their market share by (i) enabling their customers to open all documents, (ii) pacifying organisations like the Commonwealth of Massachusetts who don't want their documents tied up in a proprietary format and (iii) looking like good citizens.

They could have been really cynical and only funded the part of the project that converts from OpenOffice to Microsoft Office, and not the other way round, but that would have made them look bad. This is the next best option for them.

They may be doing this for slightly the wrong reasons but no matter, it brings their product back into consideration for me. Now the only decision is whether the usability advantage of Office 2007 is worth the big bucks you have to pay for it.

Climate change

Some simple logic that is causing me to furrow my brow. Please, if you can refute one of my premises or dispute any of my conclusions then let me know. Thinking about Microsoft Windows, Microsoft Internet Explorer and Microsoft Office: (1) Security vulnerabilities continue to be found in these products today (2) Forthcoming releases of these products are subject to the same patches as the current releases THEREFORE (1+2) (A) Unpatched vulnerabilities still exist in current and future versions of these products. Now, (3) Security patches, virus checkers and firewalls protect against known vulnerabilities THEREFORE (A+3) (B) We have no protection against an exploit of an unpatched vulnerability. Furthermore, (4) Microsoft Windows and Microsoft Office have around 100% market share on the corporate desktop (5) Many people and organizations have a financial or political interest in attacking or threatening to attack wealthy corporations. (6) Some of these organizations have substantial resources to mount a sophisticated attack THEREFORE (B+4+5+6) (C) There are organizations planning to attack the corporate desktop using an unpatched vulnerability for financial or political gain. (D) There is considerable risk of a severely damaging attack What is perplexing me is the unwillingness of any corporation I know to do something about this. I am talking about an attack that could threaten the viability of a major global corporation. Top headline on the evening news. Pundits talking domesday scenarios. Some organizations have publicly announced moves to an open source desktop platform (Telstra, City of Munich etc.) but this has always been for cost reasons. Why isn't this the most talked about issue in IT? The only reason I can think of is the analogy with climate change: (a) the threat increases slowly, and there is no natural trigger for action (b) the cost of mitigating the threat is substantial (c) the mitigation is not guaranteed to work (d) the cost is incurred today and the benefit is gained tomorrow or the day after (e) it is boring I want to discuss this some more. Any contributions gratefully received.