Mixing Agile and Waterfall at Scale: A Technical Perspective
The Scaled Agile Framework is being adopted extensively throughout the industry. In most large enterprises however, there is a transition period wherein many existing programs will continue to apply the waterfall model, while other programs are moving quickly to agile delivery via the Agile Release Train approach. Moreover, these programs are not independent and some degree of organizational, and code level dependencies exist. This can lead to conflicting methods and expectations and can even put common delivery objectives into serious jeopardy. Since this situation will exist for some time, it makes sense to provide guidance as to effective practices than can be applied to enable productive interactions between these two approaches.
An earlier guidance article, Mixing Agile and Waterfall Development in the Scaled Agile Framework discusses some of the different organizational aspects of this problem and how the programs can interact to mitigate the differences and assure alignment and delivery. In this article, we will consider the technical perspective of “interoperability” between Agile and waterfall groups, in order to provide better risk management strategies.
To establish effective collaboration between such diverse groups, our primary goal is to validate all assumptions as early as we can—fail-and-fix early and often—rather than to discover significant problems near the end, which could dramatically affect expected outcome. To achieve this, we provide four suggestions for mechanisms that teams can use to address this mixed mode of operation.
Joint Requirements and Design Workshops
Even though waterfall is largely based on Big Upfront Requirements and Design , there is no reason why these activities cannot be conducted in a highly collaborative manner. It is much easier to identify and resolve dependencies when representatives of both Agile and waterfall groups meet together in front of the whiteboard. Here’s few tips for running such workshops effectively:
- Do the homework before the meeting. Don’t necessarily pursue all dependencies, but first, try to build a holistic view of the entire initiative. The dependencies will start popping up. Each group should conduct this review internally before meeting with the other group. This is necessary to acquire the right understanding and context as preparation for effective joint meetings.
- Use whiteboards, walls, and flipcharts extensively 
- Use Specification by Example  to derive concrete system behaviors and eliminate ambiguities whenever possible. “Give me an example” should be the key phrase at such workshops. A “Demo-driven” approach is also very useful here: imagine you are at the finish line and going to demonstrate the working software. Describe what you “see”… Those examples should be captured and then used by both groups over the course of the project.
- Then identify dependencies and plan for integration points and mechanisms.
Mockups and Designing to Interfaces
Mockups (mocks) allow Agile teams to test interface assumptions before the system/code is available. Nothing helps as much as a physical integration and when one of the two groups will only deliver working software in the end, interfaces become “the most real thing” possible. Here’s a few tips to remember:
- Mockups are good when they bridge both the system integration and personal communication gaps between the teams. Mocks simulate interfaces and thus should be discussed first at the design workshop (see above). The best case scenario occurs when the waterfall teams themselves create mockups for the Agile teams. If this is not possible, the Agile teams can do it, but it has to be carefully reviewed by their waterfall counterparts. Otherwise mocks will only test a set of invalid assumptions.
- Design to interfaces when using mocks. It is highly recommended to couple mockup development with the native means of “capturing” the interface structure that the programming languages provide (e.g.: Interfaces in Java and C#, abstract classes in C++ and so on). This will allow the developers to simply replace each mockup with real implementations as soon as it is available. If inconsistencies are found – capture them immediately.
Frequent integration tests the code and the assumptions that went into it. As Figure 1 illustrates, by “frequent,” we mean far more frequently than once near the end of the project. Mockups beg for implementation of the logic associated with the mockup interface, which on its own fosters integration.
Integration can be full or partial, but in either case what’s important is that teams integrate without fear. If there are integration failures, (and, yeah, there will be many failures in the beginning) teams should be driven by innate desire to fix the integration errors first, and only then go back to their own agenda. While realistically, this is nothing like automated Continuous Integration at this level, it still has immense value even though its on-demand and manual, not automated.
Use Design Patterns and Refactoring
Most system architects who operate in non-Agile environments quite fairly claim to use design patterns in their systems. However Agile has a slightly different interpretation of pattern usage, which nevertheless has very important consequences. Patterns are important because of the inherent complexity associated with software development and our inability to predict either requirements or system design long term. Certain system behaviors (both internal and external) will inevitably change over the time of implementation. Patterns provide design approaches that are more immune to change, and lend themselves better to refactoring. For example, the Open-Closed Principle  suggests that variability be encapsulated and can be changed in relative isolation from the rest of the functionality.
Design patterns such as Bridge, Adapter, Decorator, and Chain of Responsibility  are also helpful here—they foster separation of concerns. Factories  foster the use of mockups, testability, and early integration as instantiation of the entities can be separated from their use. Facade pattern fosters early validation of integration with legacy systems, etc. So, consider patterns not so much as “building blocks” for your system, but rather as ways to isolate variability, which will likely be higher if it is dependent on later-breaking developments from the waterfall teams.
In this article we’ve described a few technical approaches to better integrate Agile and waterfall development paradigms. It is important to keep in mind that while these techniques are intended to provide increased probably of success in mixed mode development, that also also help waterfall teams to see certain Agile practices in action. They can then decide based on the facts what to do next to improve their development process.