Time and time again the global software industry failed to deliver designs that worked in good order, delivered on time and within budget. Many projects failed either because they performed poorly, did not meet user expectations or they could not catch up with technological momentum.
There are historical reasons why software engineering projects failed more often than other types of engineering projects. In the past thirty years immense market demand on software as a result of exponential growth of cheaper synthetic memory availability (Moore's Law) created an avalanche effect known as the PC revolution. The tidal wave effects of the PC Revolution was immense and widespread. It did not stop at households. The collapse of the Mainframe Computer and consequent emergence of vast IP networks in the corporate landscape quickly followed. On top of that the Internet Revolution that emerged in the last two decades pushed competition to unprecedented levels.
Historical developments aside software engineering is also very different by its nature. Most notably software engineering blueprints are thousands sometimes millions of lines of code that can easily be erroneously constructed or ill-configured, inducing defects with much less chance of being detected due to difficulties of human brain comprehending code as opposed to say a suspension bridge blueprint. In addition there are vast number of mathematical possibilities in which you can construct a software program. It is like infinitely many and sometimes irregular ways of cutting a jigsaw puzzle, then recycling the pieces to make more puzzles.
So software engineering is a difficult discipline. Due to the factors we described above standard development and management practices of other engineering disciplines did not work out well when applied to software development. This led to emergence of the Open Source movement, and more dynamic development methodologies such as XP (eXtreme Programming) and Agile in the past decade or so.
What is common in these novel practices is that products are collaboratively, adaptively and continuously designed, developed and deployed in iterative cycles. Design evolves over time similar to biological evolution of species by Natural Selection.
In Nature no species is perfect at any given time. In successive and many generations species simply adapt to changing conditions or fail to adapt and become extinct. There is no perfection that warrants indefinite survival but a continuum of struggle we call survival by adaptation. There is no concept of an upfront design or designer yet alone a perfect one, in fact there is no beginning or end in the design process, design evolves indefinitely so long as it survives Nature's scrutiny.
Similarly in many Open Source projects there is no centralized vision of design (akin to Blind Watchmaker analogy). The design evolves by itself without a designer. Everyday thousands of coders worldwide collaborate on different parts of hundreds of software products similar to gene coalitions established within our cells. There is no seek for perfection, akin to the fact that perfection does not exist in Mother Nature. Resources can be deployed and utilized more efficiently by allowing mistakes rather than chasing an impossible Platonic vision of perfect design. In return product versions are released at a much faster rate with greater chance of seizing survival opportunities when they are exposed to users' scrutiny. This process is called Evolutionary Design.
To show my point lets look at Mother Nature's design process and see how imperfect it can be. The following is one of countless examples of faulty designs we inherit from our vertebrate lineage.
In this extract from The Blind Watchmaker, Richard Dawkins talks about how imperfect our eyes are from design perspective. Somewhere along the evolution of vertebrates a group of genes might have been copied in error but in a way not enough to jeopardize emergence of our species millions of years later.
"My second example of an evolutionary progression ... concerns the retina of our eyes (and all other vertebrates). Like any nerve, the optic nerve is a trunk cable, a bundle of separate 'insulated' wires, in this case about three million of them. Each of the three million wires leads from one cell in the retina to the brain. You can think of them as the wires leading from a bank of three million photocells (actually three million relay stations gathering information from an even larger number of photocells) to the computer that is to process the information in the brain. They are gathered together from all over the retina into a single bundle, which is the optic nerve for that eye.Evolutionary Design will continue to change the software engineering practice in a profound way. We will see fitter and better products emerging, satisfying complex and varying needs of demanding consumers.
Any engineer would naturally assume that the photocells would point towards the light, with their wires leading backwards towards the brain. He would laugh at any suggestion that the photocells might point away from the light, with their wires departing on the side nearest the light. Yet this is exactly what happens in all vertebrate retinas. Each photocell is, in effect, wired in backwards, with its wire sticking out on the side nearest the light. The wire has to travel over the surface of the retina, to a point where it dives through a hole in the retina (the so-called 'blind spot') to join the optic nerve. This means that the light, instead of being granted an unrestricted passage to the photocells, has to pass through a forest of connecting wires, presumably suffering at least some attenuation and distortion (actually probably not much but, still, it is the principle of the thing that would offend any tidy-minded engineer!)."
On the other hand today software engineers must come into terms with the concept of imperfect design or design without a designer. This is a completely novel mindset that developers need to understand and adapt. In the Evolutionary Design era catching up adaptation cycles and focusing on delivery is far more important than chasing up an ideal design that warrants failure.
