The Waterfall

Lost in Translation

We recently published a post on the similarities between FTTM and Agile (see here). This was written in response to confusion which sometimes arises where clients are under the impression that one must choose between the FTTM methodology or the Agile methodology, and somehow they are seen as competing methodologies. As we explained in the post, the FTTM and Agile methodologies have more in common than they do differ (high customer satisfaction through continuous improvement, flexible requirements, self-organizing teams, etc.). Where we differ, is in the implementation.

Another excellent article is The history-and legacy-of Agile development. This article explains (as you can deduce from the title) the history of Agile and how it came about in response to long software development times caused by “the waterfall methodology”…and this is part of the perception problem. FTTM is perceived to be a waterfall methodology (old) whereas Agile is perceived to be more dynamic methodology (modern). This post is intended to put to bed any confusion.

Why Waterfalls Can’t Flow Up

To start, let’s summarize the history of Agile. During the early 1990s (coincidentally when we did our best practice study), enterprise PC computing was exploding yet there was an “application delivery lag”. The time from when a set of business needs were identified to the time those needs were met with a software application was typically three years, by which time the original business needs usually (inevtiabley) changed. The typical response was to either cancel the project or release the application anyway, but this often failed because the business needs had changed, even though the application may have met the original needs from three years earlier.

The problem was the software development process teams were using at the time: requirements had to be frozen before the functional design could start, functional design had to be complete before detailed design could start, and so on and so forth. The process was rigid, so if requirements changed, there was no method by which they could not be incorporated back into the product definition - there was not feedback loop. The requirements were either tabled for the next product or junked. This “phased” development approach is what Agile aficionados call the waterfall methodology - you can only go forward and not back (a waterfall can’t flow up). (It is interesting to note that when the original waterfall approach was originally conceived, you could go back and iterate, but it never happened in practice due to needing to stick to the schedule and budget.)

After many years of frustration, a number of like-minded professionals got together around 2000-2001 to figure out a better approach to software development. The term Agile had not been coined yet, but it was the start of a new and different way to develop software that would eventually evolve into Agile some years later. The idea was to be more process lightweight than process heavyweight, and to deliver software incrementally and get end-user feedback (vs. the big bang approach of releasing software at the end of three years). If features were prioritized, then the most important features were delivered first, and the less important features would either be included later or potentially omitted if it was deemed more important to get to market faster.

What came out of this work as a set of guiding principles for a more lightweight approach. By having a system of continuous delivery, it meant development would be kept “fresh” as end-user business needs evolved and their feedback was constantly brought back into the development team and product definition. There is also less rigidity/formality; for example, this lightweight approach takes the view that if requirements are unknown or sketchy at the start, implement what you think, get it in the hands of the end-user, then the user will tell you what the requirements are (or are not!). It’s all about speed and fast iterations.

In many respects, the original waterfall steps (phases) do apply, they just happen at the software feature (micro) level and repeated multiple times throughout the development rather than at the product (macro) level where they are performed just once.

Fast-Time-To-Market (FTTM)

As we said earlier, during the early 90s - around the time project development teams were figuring out how to get to market faster - we conducted a best practice study into what made teams fast, which for us meant bringing the Right Product to the market at the Right Time (there is usually a trade-off between product functionality and time). At a high-level, these practices covered three areas: the customer, the internal (company) environment and how the project is executed. All three are tightly related. Not surprisingly, many of the best practices (30 in total) are very similar to Agile, involving high customer satisfaction through Voice of the Customer (VOC), continuous improvement, flexible requirements, frequent releases, self-organizing teams and ownership, etc. FTTM actually pre-dates Agile, so in the end, two independent teams concluded the same approach.

Over the past ~30 years as part of our consulting business implementing these practices in companies, we developed a software product called fastProject, that implements many of the best practices related to project execution. For practical reasons, fastProject is integrated into Microsoft Project. When we develop a schedule of the project, we do so by identifying tasks and linking them in order to get an idea of the timeframe of the project. Linking activities creates a “waterfall”. The screenshot below depicts what we would call a waterfall schedule in the sense that dependencies flow “down” from one task to another

A “Waterfall” schedule

The Waterfalls

So this is where the confusion over the term “waterfall” comes in. In Agile terms, waterfall refers to a rigid development process that goes through a number of phases, where you can’t go to the next phase until the current one is completed. This is a traditional phase-gate approach, where at the completion of each phase, there would be a review and a sign-off. At that point, the next phase could start. This approach was originally developed to control budgets and is well-known (now) for being slow.

In FTTM scheduling terms, a waterfall refers to a set of activities with dependencies (predecessors and successors) which, when linked together, flow like a waterfall to create a timeline of the project. This can sometimes be misinterpreted as all tasks must be sequential - not so. As can be seen from the screenshot above, tasks can be parallel and the start of one task (Task 5) can be offset from its predecessor task (Task 3).

So here we have the same word representing two different concepts, and this is where confusion arises.

The problem with Agile (that we have seen) is not with the concept but in the implementation. There is a list of items (called a product backlog) to be completed for the product release. Teams are allocated a subset of these to be completed within a defined timeframe (typically 2 weeks). This timeframe is called a Sprint and the items to be completed are called a Sprint Backlog. For those items not completed, they carry forward to the next Sprint in a edition to new Product Backlog items. The problem with this approach is there is no schedule, i.e. there is no order of the work. It’s a list of items that need to be completed and typically categorized into one of four buckets: Forecast, To-Do, In-Progress and Done. By not putting them into some kind of order or timeline, you are unable to predict the completion dates of key milestones. Ask any team following a typical Agile implementation and ask them when they will hit a key milestone or will be done with the project, and they can’t tell you. All they know is what’s in the backlog.

The reason for building a schedule is so the teams can identify and organize the work in the order they intend (in advance of doing he work) which then creates a timeline and an estimate of key milestone dates. In FTTM, we take one step further and identify target dates which gives us a gap, and we can see which activities are driving the gap - the critical path. This allows us to make high-level strategic decisions right from the start in order to accelerate. Rarely do we find a team using Agile that has this capability.

A Blend of FTTM & Agile - A Case Study

Of course, it’s not black and white. We have been brought in to “fix” a number of projects that were late that were using the Agile methodology, and we ended up with a hybrid solution. In one engagement, a customer was developing a large software application, and had repeated missed delivery dates that was affecting customer confidence). We created Sprints in the schedule and allocated Backlog items to the Sprints. We then put the Sprints in some common sense order. Admittedly, it was somewhat of an artificial order because there wasn’t any true technical dependency between Sprints; however, it did show that with the current thinking, we were going to be late by 6+ months. This drove us to optimize both the Sprint Backlogs (which features went into which Sprint) whilst also tackling the higher level problem of too many features for the next release. To solve this problem, we prioritized the Product Backlog using decisionAccelerator, and moved some features to the next release.

All of this was iterative, but by combining both the Agile/Sprint methodology with an FTTM critical path to predict customer delivery dates, the team got their next release in customer’s hands on-time, as they did with future releases. hey now had a system for continuous product releases. The key point here is because they had a critical path schedule, they were able to see the gap, make adjustments and derive an end date which they continually kept the customer informed about.

In Summary

So in summary, FTTM and Agile are not competing methodologies. They have the same goals, but from an implementation perspective, FTTM adds the ability to put the work in a timeline, identify the gap, make high-level strategic decisions to close the gap, and implement a process of continuous acceleration in order to target release dates. The post here discusses this in more detail. The word waterfall doesn’t even need to be mentioned.