Great paper on failure of complex systems

People wanting to read more on complex systems and their failures will find the book "Normal Accidents: Living with High-Risk Technologies" by Charles Perrow of interest. Written in layman terms, it is very interesting. ISBN 0-691-00412-9.

Often when we have a major incident, there is single-point-of-failure elimination campaign. Which is kind of humorous. During the almost two dozens of years I have been involved in such things, very rarely is there such a thing - the elusive SPOF; the one thing which we can fix that we prevent *all* future failures. The silver bullet. With very few exceptions, major incidents involve at least five failures - in my experience. Trivial failures only cause trivial incidents.
I think these things, in the context of complex systems are a bit like Black Swans (See the book by Nassim Nicholas Taleb). A Black Swan is by definition unpredictable. So, in theory. most service disruptions in complex systems are NOT black swans. They would be predictable if you we all of the data. But large corporate IT systems are too dynamic to have all of it in order to predict reliably.
Arguably specific human behaviour is not predictable; will a given operator react in a predictable manner in the timeframe required? We may know what the documentation says, but we cannot account for all possible human errors.

Way back in the late 70s, when I was a young technician and thought I knew it all, I read a book that changed my thinking forever: SYSTEMANTICS: How Systems Really Work and How They Fail, by John Gall. (now titled The Systems Bible) http://en.wikipedia.org/wiki/Systemantics

This paper reinforces and expands on the lessons in that book. What a wonderful reminder that complexity itself creates unique problems.

Russell L. Ackoff has passed away at 90 years young. 4 p.m. Oct. 29, 2009.

He was one of the greatest organizational thinkers of the last 100 years. Condolences to the systems thinking community, his students, readers and colleagues.