Luck is what happens when preparation meets opportunity.

–- Seneca

Architectural Epic Abstract

Architecture epics are large technology initiatives that are necessary to evolve portfolio solutions to support current and future business needs. Sources of architectural epics can include:

Mergers and acquisitions, which require technological integration
Technological change and infrastructure obsolescence
Performance and scalability challenges of existing solutions
Cost and economic drivers, such as avoiding duplication of effort

Typically, these are initiatives of “epic proportions”, as they typically cut across three dimensions:

Time – requiring multiple PSI/Releases to implement, perhaps taking up to a year or two to complete
Scope – affecting multiple products, applications and solutions
Organizations – affecting multiple teams, programs, business units, and even external entities

Architectural epics are captured in the portfolio backlog, which is part of the architectural epic kanban system. There, they are processed through various states of maturity until they are either flushed from the system or are moved to implementation. This process assumes reasoned economic analysis of the business drivers behind the epic, its technical impact, and strategies for incremental implementation. Work-in-progress (WIP) limits are limits on resources imposed at any stage.

Details

Epic Flow Through the Kanban System

Most architectural epics are portfolio-level concerns and are therefore initiated and tracked at the portfolio level. The Architectural Epic Kanban exists for that purpose, as illustrated in Figure 1.

Figure 1. A high-level outline of the architectural epic kanban system.

Funnel

The identification of architecture epics takes place at the funnel stage, where ideas are captured. This backlog represents a “low cost holding pattern” where ideas can be captured and discussed at little or no cost or impact. All ideas are welcome here, so there are no work-in-progress (WIP) limits imposed at this stage. Epics can be stated in any format here, typically just a short keyword or phrase such as ”migrate all products to JBOSS”.

Backlog

Epics that reach the Backlog queue warrant a further time investment. One format for stating the purpose and value of the epic at this stage is the Epic Value Statement Format. In the backlog queue, WIP is limited to match resources and an Epic Owner, typically a business analyst or architect, is assigned to shepherd the epic forward through the system. In this stage, epics are also roughly sized for effort and potential value.

Analysis

Epics (both architecture and business) are the primary economic drivers for the program portfolio, so they must be carefully analyzed before being committed to implementation. The most worthy epics from the backlog are passed to analysis when space becomes available in that queue. There, effort and economic impact are better defined, Weighted Shortest Job First WSJF Prioritization is applied and a lightweight business case ([1],Chapter 20) is developed. Analysis efforts can include:

Workshops with business stakeholders to understand and describe business benefits of the technological epic
Workshops with system architects and Agile team leads to understand implementation effort and impact on current systems
Developing concrete examples to resolve ambiguities (Specification by Example)
Defining the Success Criteria for an epic

A primary purpose of this analysis is to identify risk, which is especially important for architectural epics that span multiple systems. There, risk analysis may influence the implementation plan associated with the epic, based in part on its cost of delay, as shown in Figure 2.

Figure 2. Two scenarios for implementing the large epic spanning multiple products: a) sequential and b) concurrent.

The result of the analysis phase is a lightweight business case that will highlight stakeholder sponsors, impact on sales and distribution, development effort, WSJF rating, and more. The business case is then used by the appropriate authorities to make a go/no-go decision for the epic. (See [1], Ch 21 for a full business case format).

Incremental Implementation

Historically, implementing wholesale architectural changes to a system was conceptually simple: make a new branch, start a new development, and merge it to the baseline down the road somewhere. In Agile, however, it isn’t done that way.The Agile alternative is to commit to a continuous redesign of the system, branching rarely, if at all, and even then only for very short periods. This is accomplished by multiple incremental implementation paths.

Strategies for Incremental Implementation

As we have described in System Architect and [1], we consider three options:

Case A: The epic is big, but there is an incremental approach to implementation. The system always runs.
Case B: The epic is big, but it can’t be implemented entirely incrementally. The system will need to take an occasional break.
Case C: The epic is really big, and it cannot be implemented incrementally. The system runs when needed; do no harm.

Examples of incremental patterns are also described in ([2] Chapter 2), whereby the legacy subsystems are gradually “strangled” over time using proven patterns like asset capture or event interception.

Splitting Epics

Implementing incrementally means that architectural epics must be split into smaller sub-epics (see below) and Architectural Features. In [1], we describe eight approaches to splitting architectural epics. Here are a few examples:

Partition by subsystem or product. Epic: “All applications must run in native 64-bit mode”. Split via “… do one application each PSI/Release” and “… do client side to support labeling claims to the consumer first, and defer server side for performance until later”.
Complex/simple. Epic (complex): “Productize the APIs”. The first step might be to “Build an adapter to expose only the meter API”.
System qualities incrementally. Epic: “Support 10,000 homes per server deployment”. The result of splitting could be “support 5,000 in PSI/Release 1” and then “support 10,000 in PSI /Release 2”.

Sub-epics

The Big Picture provides three levels of abstraction for expressing system behaviors—epics, features, and stories. However, these are arbitrary distinctions, labels to describe how to think about the system at various levels of abstraction. But there is no perfect, one-size-fits-all, hierarchy. For example, work implied by epics must be allocated to the appropriate release trains, where they will eventually be decomposed into features. In such cases an epic such as “move CRM to the cloud” will spawn a number of smaller epics, one for each affected train. To manage this, some have found it convenient to call these derived epics sub-epics, which create an additional level in the hierarchy but also provide an additional container for all the program specific features. The added information at the sub-epic level helps everyone understand where the epics came from.

Lean Economic Perspective

Architectural epics drive better economics by enabling the technology platforms that deliver business functionality. But innovative product development cannot occur without risk taking. Therefore, initial technology-related decisions cannot always be correct. Hence, the Agile enterprise must be prepared to reverse course on occasion. The Ignore Sunk Costs principle of product development flow ([3], principle E17) provides essential guidance: do not consider money already spent. Incremental implementation helps, as corrective action can be taken before the investment grows too large. Using Spikes to drive exploration is another approach to course correction. And finally, another principle, the Principle of Optimum Decision Timing ([3], Principle E15), suggests that every decision has its optimum economic timing, therefore architectural decision-making must be continuous and incremental as well.

Learn More

[1] Leffingwell, Dean. Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise.Addison-Wesley, 2011.
[2] Fowler, Martin. Strangler Application. http://martinfowler.com/bliki/StranglerApplication.html
[3] Reinertsen, Donald. The Principles of Product Development Flow: Second Generation Lean Product Development. Celeritas Publishing, 2009.