The epiphany of integration points is that they control product development. They are the leverage points to improve the system. When timing of integration points slip, the project is in trouble.

—Dantar Oosterwal, The Lean Machine

Continuous Integration

Definition: Continuous Integration (CI) is an aspect of the Continuous Delivery Pipeline in which new functionality is developed, tested, integrated, and validated in preparation for deployment and release.

CI is the second aspect in the four-part Continuous Delivery Pipeline of Continuous Exploration (CE), Continuous Integration (CI), Continuous Deployment (CD), and Release on Demand (Figure 1).

Details

Continuous integration is a critical technical practice for each Agile Release Train (ART). It improves quality, reduces risk, and establishes a fast, reliable, and sustainable development pace.

With continuous integration, the system always runs, meaning it’s potentially deployable, even during development. CI is most easily applied to software solutions where small, tested vertical threads can deliver value independently. In larger, multi-platform software systems, the challenge is harder. Each platform has technical constructs which need continuous integration to validate new functionality. CI is even more complicated when systems comprise software, hardware, components, and services provided by suppliers. But the fact remains that frequently integrating and testing features together is the only practical way to validate a solution fully.

As a result, teams need a balanced approach that allows them to build-in quality and gets fast feedback on their integrated work. For purely software-based solutions, continuous integration is relatively easy to achieve with modern tools. For more complex systems with hardware and software, a continuous integration approach is required (see the Enterprise Solution Delivery article) to balance the economic trade-offs between frequency, the scope of integration, and testing.

The Four Activities of Continuous Integration

As illustrated in Figure 2, SAFe describes four activities associated with continuous integration:

1. Develop describes the practices necessary to implement stories and commit the code and components to version control
2. Build describes the techniques needed to create deployable binaries and merge development branches into the trunk
3. Test end-to-end describes the practices necessary to validate the solution
4. Stage describes the steps required to host and validate solutions in a staging environment before production

Develop

Developing the solution refers to implementing stories by refining features from the ART Backlog as may be needed and then coding, testing, and committing the work product into the source control system. Testing in this activity tends to focus on unit and story-level testing and often requires test doubles (see Test-Driven Development) to replicate other components or subsystems that are not readily available or easily tested.

Several practices are associated with developing the solution:

• Break features into stories – This enables continuous delivery via small batches and smooth integration, including creating user story maps to ensure that workflows meet customer needs.
• Behavior-Driven Development (BDD) –BDD is a process Product Owners and teams use to understand requirements better and improve quality by creating acceptance criteria and automating tests, often before the code is written. BDD works with TDD and is described here.
• Test-Driven Development (TDD) – TDD involves writing the unit test first, then building the minimal code needed to pass the test. This testing technique leads to better design, higher quality, and increased productivity. TDD works with BDD and is described further here.
• Version control – Effective version control allows teams to recover quickly from problems and improve quality, ensuring the integration of the right components. Aggregating assets under version control is a leading indicator of continuous integration maturity.
• Built-in quality – Built-In Quality prescribes practices around flow, architecture & design quality, code quality, system quality, and release quality.
• Application telemetry – Application telemetry is the primary mechanism that acquires and then uses application data to help determine the results of relevant hypotheses.
• Threat modeling – In addition to threat modeling done in continuous exploration (architect step), the system design should identify possible vulnerabilities that teams may introduce with new functionality.

Build

During the build phase, teams continuously integrate new code. Automating the build and test tools to run upon code commit is one of the best ways to integrate. Passing versus not-yet-passing and broken automated tests are objective indicators of progress. Automating code building enables teams to fix problems quickly before they affect more significant parts of the system. Addressing a broken build should be the highest priority. A ‘gated commit’ ensures software has passed the gate (e.g., unit tested, performance-tested, free of known defects, and so on) before being checked into the main codebase or trunk. Code that passes the tests is automatically integrated, which removes the complications of managing multiple source code branches. Trunk-based development helps ensure code can be released on demand reliably without costly code freezes.

Five practices can help build a high-quality solution:

1. Continuous code integration – Code commit should automatically trigger the compilation and testing of changes. Ideally, this happens on each commit and should happen several times daily.
2. Build and test automation – The automated compilation process includes unit- and story-level tests to verify changes, often requiring test doubles to enable fast builds and replicate other systems.
3. Trunk-based development – Teams should integrate code quickly, at least once daily, or ideally upon commit, and all teams should work off a single trunk, avoiding long-lived branches.
4. Gated commit – Committing to the main trunk is risky, as broken changes can impact many teams. Therefore, only the modifications validated through the build and test process merge into this branch.
5. Application security – Code analysis tools inspect the code and third-party packages for known vulnerabilities.

Test the solution end-to-end

While critical, automated local story and component testing aren’t enough. System-level integration and testing are required to test features thoroughly. Figure 3 illustrates how the System Team helps integrate the work of all teams on the ART frequently, providing some objective evidence of progress.

System-level testing frequently happens during the iteration, ideally after every commit. However, whatever the circumstances, such full-system integration must be accomplished at least once per iteration. Otherwise, the late discovery of defects and issues from earlier iterations causes substantial rework and delays.

Six practices can help improve end-to-end system testing:

1. Test and production environment congruity – Environment congruity assures that testing exercises the solution as it would behave in front of live users and decreases the probability that defects will escape into production.
2. Test automation – Automated testing should include a variety of tests, such as functional, integration, regression, and more. The Agile Testing article details a testing matrix of what can and should be automated.
3. Test data management – To create stability, tests must be consistent and realistic, replicating production as much as possible and under source control.
4. Service virtualization – Different kinds of testing require different environments. Service virtualizations allow teams to simulate a production environment without the costs and effort associated with creating and managing physical infrastructure.
5. Testing nonfunctional requirements (NFRs) – System attributes, such as security, reliability, performance, scalability, and usability, are critical and require testing.
6. Continuous integration with suppliers – Suppliers bring unique contributions that reduce lead time and improve value delivery, requiring continuous integration. It helps to adopt a shared integration cadence and establish objective evaluation milestones.

Stage

Finally, the ART must validate the entire solution in staging based on the following practices:

• Maintain a staging environment –A staging environment closely resembling production provides the place for such validation.
• Blue/Green deployment – The blue/green deployment pattern involves two environments–live (production) and idle (staging). Changes flow continuously to idle, stage, and ready them for production deployment. When ready, a configuration change switches the two environments. Idle becomes live, while the old live becomes the new idle. This approach enables continuous delivery, zero-downtime deployment, and fast failure recovery.
• System demo – This is where stakeholders evaluate a solution’s readiness for production deployment.

Enabling a Culture of Continuous Integration

Continuously integrating large and complex systems is a time-consuming journey. The following section provides some suggestions for building a thriving CI culture and practice.

• Integrate frequently – The more often teams integrate, the quicker they find problems. And the harder it is to do, the more often they need to do it. This practice eliminates impediments and adds automation along the way, resulting in faster learning cycles and less rework.
• Make integration results visible – When the integration process breaks, everybody should know why it failed. When it’s fixed, creating new tests should detect the problem earlier and prevent it from happening again.
• Fixing failed integrations is a top priority – Teams that continue working through integration failures fall short of the values and culture associated with building production-ready systems. Teams often use flashing lights or other notifications, drawing attention to a broken build and establishing visible indicators displaying the percentage of the time the system remains broken.
• Establish a shared cadence – Integration points are more accessible when all teams follow the same consistent rhythm. Suppose teams cannot do a complete integration within an iteration. In that case, they can make near-term trade-offs on what’s possible while continuously improving their techniques and infrastructure toward this goal.
• Develop and maintain proper infrastructure – Effective continuous integration depends on the availability of test and staging environments. Infrastructure is, of course, an investment. But Lean-Agile Leaders take the long view and make the investments necessary today to increase velocity for the marathon ahead.
• Apply supportive software engineering practices – Continuous integration is more accessible when designing systems with those concerns in mind. Test-first development and planning for testability call for more modular solutions and separation of concerns, as well as using primary interfaces and physical test points.

Another important aspect of CI culture is ensuring a fast flow of value through the pipeline. See the ART Flow article for more information on making value flow without interruption (Principle #6).

Enabling Continuous Integration with DevOps

Continuous integration involves crucial ‘development’ activities that originally inspired the ‘Dev’ in DevOps. These activities focus on solution development and pipeline flow through pre-production environments. Applying DevOps thinking, practices, and tooling in this segment of the value stream enables rapid development, frequent code integration, and built-in quality and compliance.

As illustrated in Figure 4, SAFe’s CALMR approach to DevOps (center) enables continuous integration and several practice domains (inner rings). Each of the four activities (in green) is a collaborative effort that draws upon DevOps expertise from multiple disciplines to maximize delivery speed and quality.

For example, building solutions in the continuous delivery pipeline crosses multiple DevOps domains. Checking code into version control triggers the deployment pipeline to invoke automated merge, quality, and security checks, then apply configurations stored as code to build shippable, full-stack binaries. Using DevOps, this process typically turns source code into tested, deployable solutions quickly.

All four continuous integration activities are enabled by DevOps, though with different combinations of technical practices and tooling. See the DevOps article series for more detailed guidance on DevOps and how it allows the continuous delivery pipeline.

---

Learn More

[1] Oosterwal, Dantar P. The Lean Machine: How Harley-Davidson Drove Top-Line Growth and Profitability with Revolutionary Lean Product Development. AMACOM, 2010.

[2] Leffingwell, Dean. Scaling Software Agility: Best Practices for Large Enterprises (Agile Software Development Series). Pearson Education, 2007.

[3] Kim, Gene, Jez Humble, Patrick Debois, and John Willis. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press, 2016.

Last update: 06 January 2023