Jun 22, 2021
Waterfall was a mistake on most software projects
In case that’s not yet obvious
A while ago I was reading the “Waltzing with Bears: Managing Risk on Software Projects” book by Tom DeMarco and Timothy Lister, and there was a paragraph in there that jumped at me:
There is probably no job on earth for which an ability to believe six impossible things before breakfast is more of a requirement than software project management. We are routinely expected to work ourselves into a state of believing in a deadline, a budget, or a performance factor that time subsequently may prove to be impossible.
One of the lies we sadly still believe is that Waterfall can be used to manage large software projects, when in fact, the model was just a mistake.
I remember this every time I see Project Manager Job Descriptions that demand “working knowledge of the Waterfall process”, and inevitably combine that with a requirement for “proven track record of delivering projects on time, within budget, and with the agreed scope”. Sometimes the job posting doesn’t simply mention the “Waterfall” process, but the “Standard Waterfall” process.
This image is familiar, right? Maybe you’ve seen a more cheerful, colored version of this figure, but you’ve seen it. Right? It’s the idea that you can build any kind of software using a linear-sequential development model in which work flows through several phases. You can’t go to the next phase until the current phase is complete. Work progresses in only one direction, like water flowing downstream in a Waterfall.
The figure above is from the document to which the definition of Waterfall is most often attributed to, “Managing the development of large software systems” by Dr. Winston W. Royce, 1970. The word “Waterfall” doesn’t show up anywhere in that paper. That happened later, in a document from 1976 named “Software requirements: are they really a problem?” by T.E. Bell and T.A. Thayer. In this latter document, the figure above is presented under the “Development Phases of the System Development Cycle” caption, is named “Waterfall” and is attributed to Royce. The document goes on to say that “in this approach software is developed in the disciplined sequence of activities shown […]”.
Then things become fuzzy and history isn’t totally clear on what happened next. Story goes that some guy from the military needed a uniform software development process for building systems for the U.S. Department of Defense, found Royce’s paper, saw the figure above at the beginning of the document, and said “Yup! That will do”. Then the Waterfall software development cycle gets described in the DOD-STD-2167A standard which then spread also in the private field as a model to use — as the title of the 1970 paper said — in managing the development of large software systems.
But not only did Royce not name the model, but he didn’t invent it either. The model was known before that, from the manufacturing and construction industries, and adapted to software development (which was a relatively new type of industry at that time). What Royce did was to present the approach and — in fact — point out some of its flaws when applied to the development of large software systems. He didn’t recommend it for building large software systems.
If you read the 1970 paper, you will see that the author starts from the model presented in the figure above, which he describes as a “fundamentally sound” concept, but then goes on to say that it is “risky and invites failure” and additions must be made “to this basic approach to eliminate most of the development risks”. The author recognized that you can’t have a linear model. You are inevitably forced to iterate. “Hopefully, the iterative interaction between the various phases is confined to successive steps”, but “unfortunately, for the process illustrated, the design iterations are never confined to the successive steps”. These iterations can sometimes necessitate changes that can be “so disruptive” that “one can expect up to a lO0-percent overrun in schedule and/or costs”. He also mentions the risk associated with the testing phase, which happens at the end of the project, when you most likely have no alternatives left to correct the problems created from previous phases and discovered too late in the process.
So it was not about sequential phases where activity only flowed from one phase to the other in a linear-sequential fashion.
To fix the limitations of the model, Royce proposes — among other additions — things like “a preliminary program design” to be performed before analysis begins, like a prototype or a simulation to test assumptions and hypotheses and find “trouble spots”. He also advised keeping the client close because what “a software design is going to do is subject to wide interpretation even after previous agreement”. Then mentions that the customer involvement “should be formal, in-depth and continuing”.
Do these things sound anything like what you heard about Waterfall? Do these things describe the figure above?
Anyone who has used a predictive approach to build software knows that the standard model — sequential and non-iterative — is pure fantasy on any project that requires considerable effort to be built. I never saw “Standard Waterfall” in my 15+ years of working in the software development industry. And I never encountered anyone else that saw it either. Simply because it was a mistake. In reality, you start from there and end up with some sort of predictive + adaptive hybrid that doesn’t resemble “the standard”.
Don’t get me wrong, Waterfall sometimes has its place. But to avoid making a mistake using it, you first have to evaluate the characteristics of your project. Do you work with a stable and familiar business domain? Can you gather upfront requirements that can precisely define what you need to build? Are expectations clear for everyone? Have you performed this kind of project before and can build a good level confidence estimate? Are there no expected changes, or can they be avoided, covered by risk buffers or a solid contract? The client can’t stick around for the entire project duration and you need a simple approach to manage the relationship with them? Has the client asked for a fixed-price contract because they want to know the delivery date so they can use it to integrate the project into other things they are doing? Is the project short-lived?
That last point is critical. The longer the duration of the project, the fewer chances all of these things can remain true. Using a predictive approach (especially the “Standard Waterfall”) on large projects where you can’t answer those questions with “yes”, is simply a mistake. On most software development projects you cannot define all of the requirements from the start. You can’t build a good estimate. You can’t create the perfect plan. You can’t outline an exact schedule. You can’t create the precise design. Developers are not perfect or may not know the domain well enough. The implementation phase is not based on facts only but also on various assumptions. Testing is not just a formality that validates the implementation. You cannot think of everything as statically defined from the beginning and then treat changes as an exception or something unwanted that must be avoided at all costs. And you can’t go through all of these stages in a straight line without going back over some of them to make adjustments.
This model had issues long before it had a name or a standard.
We might not have known better when this was all taking shape, and nobody knew that this model became a model of reference by mistake, but now, 50 years later, we still insist on using it and requiring people to know it even before inviting them for a job interview. There are almost no job descriptions for a Project Manager in software development that don’t ask for Waterfall in complex and large software development projects (as if we’re doing construction or manufacturing and we can build software like we build a house or fabricate some screw nuts).
Companies don’t usually tolerate mistakes from their employees, yet we demand professionals to use a broken model. We like to dump everything on the shoulders of Project Managers and say “You are responsible for the successful implementation of the required solution”, but whatever you do, “use Waterfall” because we like the sound of “all projects delivered on time within budget and within the agreed scope”. And if you fail with it, it’s on you… and you’re fired… or you don’t get a promotion… or you get to do all of the crappy projects from now on.
So unless your project is suitable for a predictive approach — after carefully looking at its characteristics — you should stop using Waterfall. The model is a mistake and a lie. And if you do need that job in software development that asks for “experience with Waterfall”, hope that’s just HR speak for “Managed at least a few software development projects using a predictive approach and things didn’t go to hell in a handbasket”. Try to figure out if they are making a mistake or actually believe the lie, to at least know what you are getting yourself into.
