If you don’t care about quality, you can meet any other requirement.
—Gerald M. Weinberg
Agile Testing
Note: This article is part of Extended SAFe Guidance and represents official SAFe content that cannot be accessed directly from the Big Picture.
Agile Testing is an approach to systems and software testing that develops and tests systems in small increments, often developing tests before writing the code, Story, or Feature. Although traditional development often uses a deferred, big-batch testing approach, tests help elaborate and better define the intended system behavior, even before the system is implemented. Quality is built-in from the beginning. Like Agile development, Agile testing is a collaborative, team-oriented process. All team members share responsibility for testing the system.
This test-first approach applies to all types of Agile work. That includes capabilities, features, stories, NFRs, and code. It can even be used for the hardware components of a system. In the same way, tests are written during coding, and acceptance tests for capabilities, features, and stories are written during elaboration. The just-in-time practice of elaborating the proposed system behavior also mitigates the need for overly detailed requirement specifications and sign-offs. Even better, unlike conventionally-written requirements, these tests are automated wherever possible. But even when they are not, they still provide a definitive statement of what the system does rather than a statement of early thoughts about what it was supposed to do.
Details
Agile testing is a continuous process integral to Lean and Built-In Quality. Agile Teams and Agile Release Trains (ARTs) cannot go fast without high quality. Without constant, primarily automated testing, it’s challenging to implement a fast, Continuous Delivery Pipeline and achieve Release on Demand. Wherever possible, that means testing—and even test automation—comes first.
In Agile testing, everyone on the team is a tester. Using Behavior-Driven Development (BDD), Product Managers and Product Owners collaborate with their teams to create tests for features and stories. Developers create tests for code changes using Test-Driven Development (TDD). With these general tests created by their team members, testers can focus on exercising edge cases and the less intuitive interactions with the system.
Agile testing strives to be test-first. This approach has many benefits:
- Multiple perspectives provide a broad view of the required system behavior and the best approach to testing it
- Collaboration creates alignment across the team and a shared understanding of how to implement the behavior
- It forces developers to think broadly about a change before implementing it
Other tests—such as performance, security, and user acceptance—are typically less collaborative and more challenging to write first. Nonetheless, developers should still perform these tests early and often to minimize surprises later in development.
The Agile Testing Matrix
At scale, many tests are necessary to ensure quality: tests for code, interfaces, security, stories, larger workflows, and so on. Brian Marick (Extreme Programming (XP) advocate and Agile Manifesto [1] co-author) helped pioneer Agile testing by describing an Agile testing matrix that guides the reasoning behind various tests and provides a taxonomy for organizing tests. This approach was further developed in Agile Testing [2] and extended to scaling the Agile paradigm in Agile Software Requirements [2, 3].
Figure 1 describes the latest matrix [4] with guidance on what to test and when.
The horizontal axis of the matrix contains business (top) or technology (bottom) facing tests. Written in business terminology, business-facing tests are designed to be understandable by the user. Written in the developer’s language, technology-facing tests are used to evaluate whether the system delivers the behaviors the developer intended.
The vertical axis on the left side guides development by having the development team think about how they will test a story or section of code before they write it. To find defects or missing features, the right side contains tests critiquing the solution by evaluating the system against the user requirements.
Creating tests across the four quadrants (Q1 – Q4) fosters a comprehensive strategy that helps ensure quality. (Note: The quadrant number is arbitrary and implies no priority to creating tests.)
- Q1 (Unit and component tests) –tests are written to run before and after code changes to confirm that the system works as intended.
- Q2 (Functional tests) – tests for stories, features, and capabilities validate functionality works the way the Product Owner (or user) intended.
- Q3 (System-level acceptance tests) – validate that the whole system’s behavior meets usability and functionality requirements. These may include exploratory tests, user acceptance tests, scenario-based tests, and final usability tests. Because they involve users and testers engaged in actual or simulated deployment scenarios, these tests are often manual. They are frequently the final system validation before delivery of the system to the end-user.
- Q4 System qualities test – verify the system meets its Nonfunctional Requirements (NFRs). Typically, they’re supported by a suite of automated testing tools (such as load and performance) designed for this purpose. Since any system changes can violate NFR, they must be run continuously, or at least whenever it’s practical.
SAFe’s TDD and BDD articles detail practices for quadrants one and two, respectively. Continuous Integration provides more information on quadrant three. NFRs, Continuous Integration, and Release on Demand offer more details on quadrant four.
Learn More
[1] Manifesto for Agile Software Development. http://AgileManifesto.org/ [2] Crispin, Lisa, and Janet Gregory. Agile Testing: A Practical Guide for Testers and Agile Teams. Addison-Wesley, 2009. [3] Leffingwell, Dean. Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise. Addison-Wesley, 2011. [4] Gregory, Janet, and Lisa Crispin. More Agile Testing: Learning Journeys for the Whole Team. Addison-Wesley, 2015.
Last update: 27 February 2023