The epiphany of integration points is that they control product development. They are the leverage points to improve the system. When timing of integration points slip, the project is in trouble.
—Dantar Oosterwal, The Lean Machine
Continuous Integration (CI) is the process of taking features from the Program Backlog and developing, testing, integrating, and validating them in a staging environment where they are ready for deployment and release.
CI is the second element in the four-part Continuous Delivery Pipeline, preceded by Continuous Exploration(CE), and followed by Continuous Deployment(CD), and Release on Demand, as illustrated in Figure 1.
Continuous integration is a critical technical practice for each Agile Release Train (ART). It improves quality, reduces risk and establishes a fast, reliable, and sustainable development pace.
With continuous integration, the “system always runs,” meaning it’s potentially deployable, even during development. CI is most easily applied to software solutions where small, tested vertical threads can deliver value independently. In larger, multi-platform software systems the challenge is harder. Each platform has its own technical constructs and the platforms themselves must be continuously integrated to prove new functionality.
In complex systems—those with mechanical subsystems, software, cyber-physical systems, suppliers, subassemblies, and others—CI is harder still. But the fact remains that integrating and testing collaborating components together frequently is the only practical way to fully validate a solution.
As a result, teams need a balanced approach, one that allows them to build quality in and receive fast feedback from the integrated increments. For the software elements, truly, continuous integration can be more easily applied. For larger and complex systems, in particular, a ‘continuish integration’ process is required (see Business Solutions and Lean Systems Engineering article) to maintain the economic trade-offs between frequency, the scope of integration, and testing.
The Four Sub-dimensions of Continuous Integration
As illustrated in Figure 2, SAFe describes four sub-dimensions of Continuous Integration:
- Develop – covers the skills necessary to implement stories and commit the code and components to the trunk
- Build – covers the skills needed to create deployable binaries and merge the development branches into the trunk
- Test end-to-end – covers the skills necessary to validate the solution in the staging environment
- Stage – covers the skills to host and validate the solution in a staging environment, prior to production
Each sub-dimension is further described in the sections below.
Develop the Solution
Developing the solution refers to the implementation of stories by coding, testing, and committing the work product into the source control system. This is the work done on individual workstations before the code is built by an automated build process. Testing in this sub-dimension is necessarily limited to unit and story level testing and most times requires test doubles (see Test-Driven Development) to replicate other components or subsystems.
There are seven skills necessary:
- Break features into stories – Splitting features into stories enables continuous delivery via small batches and smooth integration.
- Behavior-Driven Development (BDD) –BDD (see article) is a process for Product owners and teams to understand the requirements better by creating acceptance criteria and acceptance tests, and often automating them before the code is even written.
- Test-Driven Development (TDD) – TDD involves writing the unit test first, then building the minimal code needed to pass the test. This leads to better design, higher quality, and increased productivity.
- Version control – Effective version control allows teams to recover quickly from problems and to improve quality by makings sure the right components are integrated together. Putting all assets under version control is a leading indicator of continuous integration maturity.
- Built-in quality – Built-In Quality prescribes practices around flow, architecture & design quality, code quality, system quality, and release quality. The most relevant to this sub-dimension is the code quality practices.
- Application telemetry – Application telemetry is the primary mechanism by which usage data could be tracked and measured against the hypothesis.
- Threat modeling – In addition to the threat modeling done in the Architect sub-dimension of continuous exploration, attention should be given in the design of the system to possible vulnerabilities which may be introduced into the system.
Build the Solution
During the build phase, teams continuously integrate new code. This can be accomplished by automating the build and test tools to run upon code commit. Passing versus not-yet-passing and broken automated tests are the real indicators of progress. Automating the building of code enables teams to fix problems quickly before they affect larger parts of the system. Addressing a broken build should be the highest priority. A ‘gated commit’ ensures software has passed the gate (e.g. unit tested, performance tested, and free of known defects, etc.) before being checked into the main codebase or trunk. Code that passes the gate is automatically integrated into the trunk; which removes the complications of managing multiple branches. This trunk-based development helps to ensure the code can be reliably released on demand without the need for costly code freezes or hardening iterations.
There are five skills which can help build the solution:
- Continuous code integration – Code commit should automatically trigger compilation and testing of the changes. Ideally, this happens on each commit but should happen at least several times a day.
- Build and test automation – The compilation process should be automated and include unit- and story-level tests to verify the change. These tests often use test doubles to replicate other parts of the systems and enable fast builds.
- Trunk-based development – Long-lived branches must be avoided. Teams should merge back as quickly as they can, at least once per day, and all teams should work off a single trunk.
- Gated commit – Committing to a single trunk is risky as broken changes can impact many teams. This is why only changes validated through the build and test process are merged into the trunk.
- Application security – Code analysis tools inspect the code and third-party packages for known vulnerabilities.
Test the solution end-to-end
While critical, automated local story and component testing aren’t enough. In order to fully test the features, system-level integration and testing are required. With the support of the System Team, the work of all teams on the ART must be frequently integrated to assure that the solution is evolving as anticipated, as Figure 3 illustrates.
System-level testing happens as frequently as possible during the iteration, ideally after every commit. However, whatever the circumstances, such full-system integration must be accomplished at least once per iteration. Otherwise, the late discovery of defects and issues reflects all the way back to earlier iterations, causing substantial rework and delays.
End-to-end testing requires the following skills:
- Test and production environment congruity – Environment congruity assures that testing exercises the solution as it would behave in front of live users and decreases the probability that defects will escape into production.
- Test automation – Many types of tests need to be run: functional testing, integration testing, regression testing, etc. The Agile Testing article details a testing matrix of what can and should be automated.
- Test data management – To create consistency in testing, tests must be consistent and realistic, as production-like as much as possible, and under source control.
- Service virtualization – Different kinds of testing require different environments. Service virtualizations allow teams to simulate a production environment without the costs and effort associated with creating and managing real environments.
- Testing nonfunctional requirements (NFRs) – system attributes such as security, reliability, performance, maintainability, scalability, and usability must also be thoroughly tested.
- Continuous integration with suppliers – Suppliers bring unique contributions that can have a large impact on lead-time and value delivery. Their work must be continuously integrated as well. It helps to adopt a common integration cadence and establish objective evaluation milestones.
Validate on Staging
Finally, the entire solution must be validated on a staging environment., which requires the following skills:
- Maintain a staging environment –A staging environment, which matches production provides the place for such validation.
- Blue/Green deployment – “Blue/green deployment provides two environments, one live and one idle. In order to deploy a switch is flipped (or a load balancer) and the idle environment becomes the staging environment, while the other becomes the new idle environment. This enables quick transition and recovery where necessary.
- System demo – This is the event where stakeholders validate the solution.
Enabling a Culture of Continuous Integration
Continuously integrating large and complex systems is a journey that takes time. The following section provides some suggestions for building a successful CI culture and practice.
- Integrate often – The more frequently teams integrate, the quicker they find problems. The harder it is to do, the more often they need to do it—eliminating impediments and adding automation along the way. This results in faster learning cycles and less rework.
- Make integration results visible – When the integration process breaks, everybody should know how and why it broke. When it’s fixed, new tests should be added to detect the problem earlier and prevent it from happening again.
- Fixing failed integrations is a top priority – If teams just keep working during an integration failure, it doesn’t create the right sense of urgency and importance to fix the problem. To highlight the problem, teams often use flashing lights to draw attention to a broken build, and visible indicators displaying the percentage of the time the system remains broken.
- Establish common cadence – Integration points are easier when all the teams are moving at the same consistent rhythm. If full CI can’t be accomplished during the course of an iteration, teams can make near-term trade-offs on what can be integrated, while continuously improving their techniques and infrastructure toward this goal.
- Develop and maintain proper infrastructure – Effective continuous integration depends on the availability of test and staging environments. Infrastructure is, of course, an investment. But Lean-Agile Leaders take the long view and make the investments necessary today to increase velocity for the marathon ahead.
- Apply supportive software engineering practices – Continuous integration is easier when the system is designed with those concerns in mind. Test-first development and designing for testability call for better solution modularity and separation of concerns, as well as the use of primary interfaces and physical test points.
It’s also important to note that continuous system integration and frequent solution integration represent a special challenge for groups early in their transition to SAFe. They just haven’t done it previously, nor have they built the necessary infrastructure. But the current state cannot be an excuse to simply reduce the scope of integration. Most of these challenges will disappear in a better, future state, but only if the teams start now.
Learn More Oosterwal, Dantar P. The Lean Machine: How Harley-Davidson Drove Top-Line Growth and Profitability with Revolutionary Lean Product Development. Kindle Edition.  Leffingwell, Dean. Scaling Software Agility: Best Practices for Large Enterprises (Agile Software Development Series). Pearson Education. Kindle Edition.  Kim, Gene and Jez Humble, Patrick Debois, John Willis. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press. Kindle Edition.
Last update: 12 September 2018