Is Agile Really Better than Waterfall?

The simple and short answer to this question is "No".

There is no single methodology which can claim to be a panacea for the problems faced by software projects.

Evolution of Agile

Agile methods like XP, Scrum, FDD etc. have evolved based on the principles enshrined in the Agile manifesto which supposedly came into being as a result of frustrations experienced with the Waterfall method.

Agile has become quite fashionable since the last few years and as the die-hard Agile fans would love to hear "Agile takes care of all that is wrong with the monster named Waterfall".

However, reality is far removed from this statement.

One thought that often comes up is related to Agile and Fr(agile) and does accept the fact that Agile has its own set of challenges and shortcomings just like Waterfall or any other methodology.

In fact, in many ways Waterfall is much better than Agile and as some might say Waterfall Model, Original and Forever.

The "Waterfall"  Concept

Waterfall is based on the concept of staggering as it advocates starting an activity only when its precedent activities have been completed.

Agile is based on the concept of parallelism as it advocates performing all activities in parallel without bothering about their precedence relationships.

Agile approach is better when the product or technology is a new or innovative one and changes to try out several permuations and combinations is budgeted in its cost.

However, no organization can provide infinite budget for any project and hence projects are forced to do upfront planning to the extent they  have visibility.

On this front waterfall wins hands-down.

Agile says changes are welcome.

Waterfall also says changes are welcome but adds that changes don't come free of cost.

This is a good incentive for customer and developers to think through and ensure the high-level requirements and overall architecture is laid down before coding and testing.

The Agile Concept

In Agile method estimation, planning, requirements and design documentation, test planning, meetings, status reports are viewed as overheads.

And the focus is on delivering the working software to the customer day in and day out (or rather sprint in and sprint out).

It follows from above that Agile is focused on coding, testing and bug-fixing.

Code is continuously spewed out by the project, tested against the requirements and changed either due to design and implementation issues or due to changes in requirements.

Code changes are effected through re-factoring which essentially amounts to re-design and re-coding.

In this process the overall architecture is not focused upon and evolves along with the requirements.

This may lead to weak architecture and impact non-functional requirements like performance, scalability etc.

Waterfall versus Agile

It can be concluded that Agile seems to focus on the short-term at the cost of long-term.

For complex projects done by big, distributed teams Agile requires much higher rigor and discipline that what is advocated by the Agile purists.

In such situations Waterfall is a better choice.

It is also a fact that no one uses pure Waterfall.

Intermediate deliveries have been a common practice in case of Waterfall projects as well.

In the end, Agile seems to be more of a marketing ploy than just a well-grounded method.

It attempts to make a virtue out of lack of discipline by suggesting that trying to get upfront clarity in the requirements and approach in the beginning, to the extent possible but limited by visibility and constraints, is futile as they will change.

Thinking hard and having a good plan while remaining open for changes to the plan is a better stratgey than the above. 

Since, as is commonly accepted, well begun is half done.

In conclusion, it can also be said that "well planned is half execcution done"!

Role of Sponsors in Improvement Initiatives

Sponsors play a crucial role in the improvement initiatives in any organization. The sponsors are generally part of the executive management in the organization and charter improvement initiatives based on burning or strategic needs of the business. They not only lay down the vision for improvement initiatives but also allocate the budget and resources. At times, sponsors may do more harm than good for the initiatives to be completed successfully in a real sense. This may happen due to following reasons:

Setting unrealistic time line for the improvement initiatives - this arises due to the tendency to equate more resources being available to reduction in the time line by the same proportion. This is akin to assuming that if 1 person can run a mile in 10 minutes then 10 persons can together run a mile in 1 minute.

Focusing on short-term, superficial benefits over long-term, profound benefits - this arises due to the tendency to show quick results. This is often reflected in statements like "I would like to see us getting certified for a standard or model by so and so date". The end result may be that the certificate adorns the walls of the organization but the "improvement" part remains missing from the initiative.

Allocating inadequate resources for the initiatives - this arises due to the fact that sponsors choose to ignore the effort requirement by viewing it as a cost than an investment. This is reflected in creating virtual and part-time teams without budgeting for a part of their effort for the improvement initiative. This in the end may result in virtual improvement since virtual teams can only achieve virtual improvements. In such a situation virtual team members have a full-time job for which they are accountable while holding additional responsibility for improvement initiative but without having the authority to invest part of their effort in a formal way.

The role of sponsors should then be defined keeping in mind the above points. This way the improvement initiatives will have higher effectiveness. Sponsors should not ignore the long-term, profound benefits and must try to balance it with the short-term, superficial benefits. This would ensure an organization will get the certificate as well as the improvement!