To improve entrepreneurial outcomes and hold innovators accountable, we need to focus on the boring stuff: how to measure progress, how to set up milestones, and how to prioritize work. This requires a new kind of accounting designed for startups—and the people who hold them accountable.

—Eric Ries, The Lean Startup [1]

Applied Innovation Accounting in SAFe

Note: This article is part of Extended SAFe Guidance and represents official SAFe content that cannot be accessed directly from the Big Picture.

Introduction

Developing innovative world-class solutions is an inherently risky and uncertain process. But this level of uncertainty causes some enterprises to avoid taking prudent risks. When they do, it increases the likelihood they will spend too much time and money building the wrong thing based on flawed data or invalid assumptions. As Eric Ries observed, “What if we found ourselves building something nobody wanted? In that case, what mattered if we did it on time and within budget?” [2]

Unfortunately, traditional financial and accounting metrics have not evolved to address the need to support investments in innovation and business agility. As a result, organizations often use lagging economic indicators such as profit and loss (P&L) and return on investment (ROI) to measure the progress of their technology investments.

While these are helpful rear-view mirror business measures, these results occur far too late in the solution lifecycle to inform the actual solution development. Even NPV (Net Present Value) and IRR (Internal Rate of Return), though more forward-looking, are based on estimating the unknowable future cash returns and speculative assumptions of investment costs and discount rates. Moreover, even with a sensitivity analysis (such as “what if” calculations), these financial metrics neither reflect nor inform the learning from incremental product development. Consequently, these traditional metrics aren’t helpful in our move to iterative, incremental delivery and Business Agility.

We need a better plan that quickly validates product assumptions and increases learning. In SAFe, this is accomplished, in part, by applying the Lean Startup Cycle (see Figure 1), a highly iterative process that provides the opportunity to quickly evaluate large initiatives and measure viability using a different financial measure: innovation accounting.

Figure 1. Lean Startup cycle with SAFe

What is Innovation Accounting?

Innovation accounting is a term coined by Eric Ries’ book, The Lean Startup [2]. This process cycle consists of 3 learning milestones:

Minimum Viable Product (MVP) – establish a baseline to test assumptions and gather objective data.
Tune the Engine – quickly adjust and move towards the goal based on the data gathered.
Pivot or persevere – Decide to deliver additional value or move on to something more valuable based on the validated learning. (Lean thinking defines value as providing benefit to the customer; anything else is waste.)[2]

A fast feedback loop is essential to validate learning and reduce waste, also called ‘build-measure-learn,’ as illustrated in Figure 2 [2]. Applying the knowledge obtained from customer feedback results in increased predictability, decreased waste, and improved shareholder value.

Figure 2. Build, Measure, and Learn feedback loop from Lean Startup [2]

Leading Indicators versus Vanity Metrics

Innovation accounting asks us to consider two questions:

Are we progressing toward our outcome hypothesis?
How do we know?

In The Lean Startup, this is known as a “leap of faith assumption,” which requires that we understand and validate our value and growth hypothesis before we move forward with execution. This becomes essential to the economic framework that drives effective solution development.

To answer these questions and make better economic decisions, we support innovation accounting using leading indicators and actionable metrics focused on measuring specific early outcomes using objective data. Leading indicators are designed to harvest the results of developing and releasing a Minimum Viable Product (MVP). These indicators may include non-standard financial metrics such as active users, hours on a website, revenue per user, net promoter score, and more.

It’s essential to be aware of vanity metrics, which are indicators that do not truly measure the potential success or failure of the actual value of an initiative. While they may be easy to collect and manipulate, they do not necessarily provide insights into how the customer will use the product or service. Measures such as the number of registered users, raw page views, and downloads may provide helpful information or make us feel good about our development efforts. Still, they may be insufficient to provide the evidence to decide if we should pivot or persevere with the MVP.

There are some practical ways we can avoid being deceived by vanity metrics and instead work to evaluate our hypothesis. A/B or split testing enables us to validate our outcome hypothesis using actionable data.

For example, group A may get the new feature, and group B does not. By establishing a control group, we can evaluate the results against our hypothesis and make decisions as part of our feedback loop. We can also avoid vanity metrics by focusing on customer-driven data.

We can also use cohort analysis to examine the use of a new product, service, feature, and so on over time as it pertains to a cohort (group). For example, suppose we wanted to see how a new feature on our website improved the conversion rate to paying customers. We could look at new weekly registrations of the cohort and report on the percentage conversion to paying customers. We can analyze this information weekly and see if each cohort’s conversion rate remains constant. If it does, we have a clear indication of how the feature is affecting the conversion rate. If it does not stay constant, then we have a chance to tune the engine or pivot.

Applying Innovation Accounting in SAFe

Measuring Epics

Implementing large, future-looking initiatives is an opportunity for organizations to reduce waste and improve economic outcomes. In SAFe, large initiatives are represented as Epics and are captured using an epic hypothesis statement. This tool defines the initiative, expected benefit outcomes, and the leading indicators to validate progress toward its hypothesis.

Example of Airline Website Epic

For example, consider an airline that wants to develop a website for purchasing tickets. This is a significant endeavor that will consume considerable time and money. Before attempting to design and build the entire initiative, the epic hypothesis statement template should be used to develop a business outcomes hypothesis, define leading indicators and gain knowledge regarding the expected outcome (see Figure 3).

Figure 3. Epic Hypothesis Statement for an Airline Website

We might hypothesize that the website will help reduce call center volume and ultimately reduce costs to the airline, resulting in better profit margins per ticket sold. Thus, we are assuming that the website will be faster and easier than a phone call to the customer service department.

To test that hypothesis, we could release an incremental feature or set of features, such as an MVP, that allows customers to research flight schedules. We could analyze the call volume and the types of questions the help desk received to validate and measure the features’ effectiveness. Then we could quickly compare the trends of inquiries on the website vs. those at the call center. Additionally, we could build telemetry for collecting data into the feature using DevOps practices and analyzing customer interaction. The information captured in Figure 4 shows that call center activity has decreased, and website use has increased. These leading indicators demonstrate that the MVP appears to validate our hypothesis.

Figure 4. Call center telemetry

Leading indicators can provide immediate feedback on usage patterns. By analyzing the objective data, we can test our hypothesis and decide to continue releasing additional features, tune the engine, or pivot to something else. Thus, the Epic’s MVP and the leading indicators enable us to make faster economic decisions based on facts and findings. Interestingly, in figure 4, the visits to the site metric by itself might be considered a vanity metric. This metric doesn’t tell us much about our MVP’s success or Epic’s viability. However, placed in context with the other metrics, it indicates where customers spend their time visiting the website.

Example of Autonomous Vehicle Epic

With their direct connection to large numbers of users and their interactions, as well as the ability to quickly evolve the UI, websites are a convenient place to consider how to apply innovation accounting. However, it has far broader applicability than that. Let’s consider a different example, an epic, which describes the sensor system of a new autonomous vehicle.

Our epic hypothesis (see Figure 5) is that the sensor system will quickly detect and help us to avoid collisions with objects. We assume the information will be provided fast enough for the vehicle to stop or take evasive action (as that is the entire point!).

Figure 5. Example Epic for an Autonomous Vehicle

To test the hypothesis, we would like to find out if the sensor system can detect objects and if it’s fast enough for our purposes before building the entire system. We could make a single sensor and essential data capture software to validate the distance between the object and the speed of the vehicle control system interface. As an MVP, suppose we mounted the sensor on the front of a car and connected it to a laptop within the vehicle.

Next, we place several objects on a test track and drive the car toward them. We could use the software to record information from the sensor as our leading indicator. We could also use software to measure how long it takes for the message to be generated and sent to the vehicle control system interface by using a mock-up instead of waiting for the vehicle control system to be built. This would give us an early indication of compliance with our NFR. Figure 6 describes the leading indicators for this cyber-physical system.

Figure 6. Leading Indicators for autonomous vehicle sensor

Based on the data generated during the MVP testing, we have more questions to answer before moving forward with additional features. Perhaps we can tune the engine and see if we can decrease the message-sending time. Why didn’t the sensor detect the road sign? Would going slower have helped for the initial tests? What happens if we place the sensor on the roof of the car? We may need to pivot to a different technology based on the answer to these and other questions.

‘Failing fast’ is an Agile value. The implication is that failure in small batch sizes is acceptable if we learn from it. Thus, validated learning becomes the primary objective of testing the hypothesis. As previously mentioned, SAFe uses the Lean Startup Cycle to evaluate the MVP of Epics iteratively and to pivot (change direction, not strategy) or persevere (keep moving forward) decisions against the outcomes hypothesis. This is done incrementally by developing and evaluating features from Epic. We use the empirical metric to prioritize features by applying Weighted Shortest Job First (WSJF). With WSJF, we can rapidly assess the feature’s economic value and epic’s overall progress toward its hypothesis. This allows us to quickly and iteratively make a pivot-or-persevere decision based on objective data.

The decision to persevere indicates there is still additional economic benefit. Leading indicators validate our hypothesis and MVP, resulting in the development and release of other features. The decision to pivot may occur when sufficient value has been delivered or upon learning that our hypothesis was incorrect. Deciding to pivot is so difficult that many companies fail to do it.[2] Often companies will continue to invest in an initiative despite (or lack of) data to the contrary. We can reduce the waste of time and money by using fast feedback loops and leading indicators to avoid working on features that customers don’t want or need.

It’s important to note that innovation accounting, as applied to SAFe, does not consider the sunk cost (such as money already invested). To pivot without mercy or guilt, we must ignore sunk cost, as discussed in SAFe Principle #1 – Take an economic view. Although Lean Budgeting may allocate funding to a value stream upfront, we continually use the Lean Startup Cycle to evaluate the benefit hypothesis. Consequently, the initial allocation of funds does not equate to actual spending, so deciding to pivot or preserve is crucial.

Summary

Variability and risk are part of every significant technology initiative. However, traditional financial metrics used to measure the value of those initiatives during development have not evolved to address the need for innovation and business agility. Innovation accounting was developed as part of the Lean Startup Cycle to provide validated learning and reduce waste. Creating and measuring an MVP is used to validate the hypothesis, obtain results, and minimize the risk before investing in the entire system. This fast feedback loop, based on objective data and actionable metrics, enables incremental learning. Focusing on the right leading indicators ensures we make the vital pivot or preserve decision. SAFe’s use of innovation accounting and the Lean Startup Cycle enables the enterprise to reduce waste and accelerate learning while enhancing business outcomes.

Learn More

[1] http://knowledge.wharton.upenn.edu/article/eric-ries-on-the-lean-startup/

[2] Ries, Eric. The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. The Crown Publishing Group, 2011.

Last update: 24 February 2023