Product development metrics for investment and deployment decision making

1. Market Readiness Levels: Product Development Metrics for Investment
and Deployment Decision Making
Michael Mealling, Starbridge Venture Capital, Washington, DC.
Abstract
Investment professionals routinely analyze and evaluate, often in great detail, how a new product or
service progresses from initial idea to full production-ready entry into the market. While each firm has its
own strategies for analyzing this process, it is often useful to summarize that process into a simple metric
in order to quickly communicate the project’s status, compare it to a larger portfolio, or determine where
to allocate resources.
The process of developing a new technology is similarly just as complex, diverse, and context dependent.
Yet NASA has developed just such a summary metric in the form of Technology Readiness Levels
(TRLs) that has proven to be very useful in evaluating where a technology development effort is in its life
cycle. In this paper we will attempt to synthesize a Market Readiness Level (MRL) based on several
existing models (Sustainability, Cloudwatch Consortium, Demand Readiness Level, The Lean Startup).
The overarching goal is a metric that, from an investors view, helps indicate whether a viable business
model exists and what stage the new offering is at.
1. Existing Business Models
Each of the existing models for evaluating a new offering along its development path falls into two rough
categories: offering-centric and market-centric. The focus of this paper is on evaluating a specific effort
within a firm to develop a new offering so it will focus on the ‘offering-centric’ view but will pull various
concepts from ‘market-centric’ models. The overarching goal is a metric that meets the following criteria
that, at least from the investment world’s view, will drive a new offering toward responding to an actual
market need based on demonstrated facts.
Sustainability
In ​(Solberg Hjorth and Brem)​, the authors examine some of the same frameworks in order to evaluate the
market readiness level of various sustainability innovations. A key difference is that it represents one of
the few frameworks that includes the development of the organization responsible for the development in
the metric. While interesting, it is attempting to include organizational behavior metrics within the
framework that would be best included elsewhere and developed more fully. ​(Solberg Hjorth and
Brem)​’s basic framework is listed here:
1. An acceptance that viable improvements can be made
2. Ability to highlight where the improvement can be made
3. Being able to identify what the system should do
4. Putting numbers on what is expected in terms of a solution, financially and technically
5. Ability to define how the system should operate and integrate
1

2. 6. Identify on a component level what the system should be comprised of
7. An understanding of who should be planning, designing, and implementing the solution
8. Having contact with the people, internally or externally, who will design and create the solution
9. Solution is being created to solve a defined problem
Cloudwatch Consortium
The Cloudwatch Consortium is made up of European cloud service providers and focuses on government,
business, and research initiatives. In many cases, they deal with determining market readiness for both
cloud services and the initiatives they support.
This consortium is a combination of sections from the Business Model Canvas, validation steps from the
Lean Startup Methodology, and the Four Fits Model ​(​Four Fits For $100M+ Growth — Brian Balfour​)​.
The Business Model Canvas develops and validates the basic value proposition through direct
communication with potential customers, while the Lean Startup Methodology focuses on developing and
testing against a company's Minimum Viable Product.
An essential feature of the Cloudwatch Consortium model is how each section is focused on a higher level
conceptual step that has specific ‘gating functions’ before the next stage can begin. This is a feature
2

3. NASA incorporated into Technology Readiness Levels. The requirements for exactly what outputs are
expected in order to progress to the next level are crystal clear.
Demand Readiness Levels
Demand Readiness Levels ​(F. Paun)​DRL) ​(Florin Paun)​ were developed to have a direct, even
mathematical, relationship with TRLs. The general intent is that the sum of the DRL and TRL indicates
stages of investment and or market launch; e.g. a DRL of 1 but a TRL of 8 suggests a purely R&D
application whereas a DRL of 9 but a TRL of 3 suggests that increasing the technology development
budget was justified.
While correlating demand directly to the technology is an interesting approach, it imposes an unnecessary
structure on the steps of evaluating and documenting demand that often does not correlate to current best
practices. The mathematical relationship also runs the risk of bureaucracies adopting rigid guidelines on
the relationship that conceals important learnings from both metrics.
The Lean Startup - Steve Blank
Steve Blank, the author of The Startup Owner's Manual ​(Blank and Dorf)​, is credited with developing the
customer development methodology that forms the core of the “lean startup” movement standardized in
Eric Ries’, “The Lean Startup” ​(Ries)​. By 2014, many startup incubators adopted this new methodology
and investors were eager to invest in new companies that had proven market demand and tested business
models. In response to discussions with investors, Blank developed ‘Investment Readiness Level’ metrics
as a way for incubators and investors to quickly allocate effort and investment dollars.
3

4. (​Steve Blank​)
2. Level Advancement Metrics
One of the key features of NASA’s TRL model is that it is very clear what steps need to be completed
before the technology can advance to the next level and what opportunities open up at each level. This
feature needs to exist within a Market Readiness Level (MRL) as well. With TRLs there are universally
applied metrics such as ‘Peer Reviewed Publication of Research’ or ‘Experimental Results Data that
Validates Predictions’. The exact details of the metrics used to prove moving from one step to the next are
situation specific. A Market Readiness Level is also situational and depends not only on the product or
service but on the market and the business model. In the case of a TRL, these metrics are usually obvious
and easily testable: did it work; did it work well enough; did it work long enough; etc.
The same can’t be said about MRLs. There is a long history of companies using ‘vanity metrics’ to
evaluate progress in a way that makes the company look exciting and popular but does not provide useful
information about how well the business model is working. This problem is significant enough in younger
companies that specifying the key metrics is considered a core piece of information to discover and refine
over time. It is also important that all aspects of the business model are measured, not just the financial
ones. The intent is to be able to evaluate the entire business model and improve it, not just measure the
outputs.
In the MRL model below, documentation of specific types of metrics are required deliverables in order to
advance to the next level.
3. Minimum Viable Products
Eric Ries, defined an MVP as “that version of a new product which allows a team to collect the maximum
amount of validated learning about customers with the least effort” ​(Ries)​. This goal is often lost in the
4

5. product development process as an organization feels budget or time pressure to launch the product and
the low fidelity MVP ​becomes ​the product. This is where the discipline required by an external entity
measuring the process by the Market Readiness Level can keep the organization from fooling itself into
skipping steps.
The two key terms are “maximum” learning and “least” effort. The learning that is maximized is whether
the proposed solution matches the customer’s needs. It is not a prototype to validate whether the
engineering works, to demonstrate features, or to produce an actual production ready product.
Low Fidelity MVP
The role of a low fidelity MVP is to test the basic value proposition and determine whether the firm's
understanding of the problem matches the customer’s. Features that explore aspects of the business model
such as pricing, contractual terms, etc should be avoided unless they are a critical component of the
solution ​from the customer’s viewpoint​. Firms will often discover that seeing the solution in some
concrete form for the first time triggers a customer’s own understanding of their problem and can often be
used to discover entirely new business models.
An important consideration for an MVP at this level is that the underlying technologies or processes may
not need to be proven. Many low fidelity MVPs substitute manual human labor for more difficult or
expensive features. This technique is colloquially known as ​flintstoning​ from the fact that Fred
Flintstone’s car looked like a car but had no engine, just Fred’s feet. For non-software MVPs, especially
deep-tech ones, this can be difficult but should not be discounted. It is important to be clear with the
customer that this is occurring so there is no deception.
High Fidelity MVP
The role of the high fidelity MVP is to refine and revalidate what was learned in Step 6 but also to begin
verifying that the complete business model matches the customer’s assumptions, especially with respect
to pricing, contract terms, partners, packaging, etc. This MVP can become the first iteration of the product
but that is not the goal. It is still valid to ​flintstone​ some administrative processes such as billing,
on-boarding, and support but the core product offering should be complete enough for the customer to
actually use the solution in real world situations.
If the high fidelity MVP is used beyond Step 7 it should be capable of capturing any of the key usage
metrics needed for subsequent Levels and post-launch iterations. The firm should avoid any pressure to
launch the high fidelity MVP as a final product or to skip Levels 6 through 9. This is the point where
pressure to launch can have considerable costs and consequences.
4. Proposed Market Readiness Level Model
The two main objectives of the MRL listed here are to prevent a product development team from fooling
itself into thinking it has a business model and to allow an external entity to evaluate how far a firm is
along the product development path. The MRL coupled with a TRL can tell quite a bit about how mature
5

6. a technological product is. For example, a high TRL with a MRL that has gone through multiple
iterations, between Level 3 and Level 5, would suggest a solution in search of a problem. An MRL that
passes Level 6 but with a very low TRL suggests that additional R&D investment in the basic technology
is justified.
The framework outlined below is a synthesis of the above referenced source materials. As with TRLs, no
level should be skipped and each level’s required outputs should be fully documented. Returning to a
previous level to further validate assumptions and test hypotheses is not considered an error or failure.
The process is the goal and a ‘No-Go’ decision is just as valid and important as a ‘Go’ decision;
especially when it prevents wasteful or even distorting expenditure of resources.
Level 0 - Perceived Need
The first stage of any product development process is understanding a given industry or market
segment well enough to be able to have an informed intuition that a problem exists and that a
solution might be possible. At this stage, the idea is only roughly formed but market research and
information gathering should be done. Subject matter experts can be consulted but none of the
information gathered should be considered as trustable validation that the intuition is valid. Any
notional solutions should be documented but they are not considered part of the output of this
Level.
Output:​ Documentation of conversations, existing market research, and rough outlines of the
problem
Level 1 - Notional Value Proposition
Prerequisite:​ Documentation related to the general market and the problem from subject matter
experts
At this level, it is permissible to develop notional solutions to the problem in order to write the
notional value propositions. Developing a value proposition statement is not a trivial process. It
must be stated from the point of view of the customer and their perception of the problem.
Output: ​At least one draft value proposition statement
Level 2 - Notional Customer Characterisation
Prerequisite: ​One or more value propositions and market documentation
Using the value propositions from Level 1 and information from subject matter experts, including
potential customers, fill in the right side of the Business Model Canvas with testable views of the
customer relationships, segments, channels, and revenue streams. The closer to the bottom of the
chart the less specific it should be because you do not know enough at this point to describe
revenue streams or channels in any detail.
6

7. Output: ​Business Model Canvas with the right side completed and potential customers identified
to validate the value proposition and the canvas
Level 3 - Customer Discovery
Prerequisite: ​One or more Business Model Canvases and a statistically significant number of
customers to interview
Customer discovery is an art that has much in common with psychiatry. The key task is for the
potential customer to communicate their problems and thoughts in their own voice and not in
response to any prompting by the interviewer. At this stage, there should be no mockups or
detailed discussions of solutions. The goal is to document whether the identified customers are
the right ones, whether they do indeed have the problem you have identified, and finally, how
painful is that problem.
The most important aspect of this Level is that there must be NO attempt to convince the
customer that the problem exists or that the interviewer has any solution to the problem. The
interviewer must not interpret or filter what the customers have to say.
7

8. This is the first step in the process where the goal is to prevent the firm from fooling itself. It is
also expected that the firm will iterate at this Level several times.
Output: ​A Business Model Canvas with the entire right side validated from customer feedback
plus documented interviews of potential customers with the goal of understanding their view of
the problem, solutions they may have tried previously, underlying assumptions surrounding the
problem, the potential value of a solution without identifying your specific solution, and the
financial value of having the problem solved. Wherever possible, this documentation should be a
recording to prevent the firm from misinterpreting the customer’s statements.
Level 4 - Low-Fi MVP Design
Prerequisite: ​A customer-validated Business Model Canvas
Using the feedback from the previous Level, identify the expected functionality of a solution and
build a low fidelity minimum viable product (MVP). The low fidelity MVP should only provide
enough of the solution for the customer to evaluate whether it actually solves their problem. The
MVP can include tests of the business model in order to gauge the customer's view of the value.
Output:​ A low fidelity MVP, metrics to quantify how well the Canvas and previous hypotheses
match actual customer usage, and a set of diverse customers large enough to be statistically
significant
Level 5 - Low-Fi MVP Campaign
Prerequisite: ​Low Fidelity MVP, a statistically significant number of candidate customers (at a
minimum), and a set of hypotheses to be tested
A testing campaign is run against the customers identified in Level 4. Wherever possible, the
customer must be allowed to interact with the MVP as unattended and uncoached as possible. As
with previous Levels, the goal is to understand the MVP and the problem/solution it embodies
from the customer’s point of view, not to sell them on the solution.
It is expected that the firm will iterate considerably between Levels 3, 4, and 5 as these form the
core of understanding the problem well enough to make the customer happy.
Output: ​Usage metrics and all feedback from customers and stakeholders
Level 6 - Revalidate Solution and Model
Prerequisite: ​Metrics and feedback from the Low-Fi MVP
Based on data from MVP, re-validate the entire Canvas against the expected Metrics. If they do
not match then return to Level 4 and create tests sufficient to explain the discrepancies. If the
predictions do match then solidify Partner agreements, access to Key Resources, and any details
8

9. sufficient to fully test the Business Model. The result should be a complete and validated Canvas
with sufficient data to make a Go/No-Go decision to move a High Fidelity MVP.
Output: ​A fully validated Canvas and all resources necessary to build and test a Hi-Fi MVP.
Level 7 - High-Fi MVP Campaign
Prerequisite: ​All information necessary to create a Hi-Fi MVP able to test all aspects of the
customer problem, the solution, and the business model.
Build and deploy an early adopter campaign using updated HIGH FIDELITY MVP , including
instrumentation around Metrics, integration with Partners, and exposure of the business model(s).
A High Fidelity MVP capable of testing and iterating on the basic business model, including all
customers and stakeholders. It is OK for this MVP to generate revenue but it is not required. All
Metrics necessary for a "go to market" decision must be included.
Output: ​Data, customer feedback, and metrics
Level 8 - Validate Against Model
Prerequisite: ​All data related to the Hi-Fi MVP sufficient to prove traction between statistically
significant number of paying customers and MVP's business model. All Metrics fit or exceed the
model and any discrepancies are understood.
The results from the high fidelity MVP are analyzed and the final Canvas is updated. The final
Canvas and all Metrics and assumptions are communicated to all stakeholders
Output: ​A validated business model
Level 9 - Go to Market
Prerequisite: ​A validated business model
Metrics and all gained knowledge allow sufficient confidence in cost, revenue and market
projections to financially justify building the full customer solution. A decision is made to either
develop the go to market solution and launch, to abandon the effort, or to iterate on previous
steps.
Output: ​A yes or no decision to launch
5. Conclusion
The Market Readiness Levels outlined here should prevent a firm from fooling itself into believing there
is a market and also help the firm or outside investors gauge whether further investment in the product
9

10. development effort is warranted. Firms and investors are cautioned to not view multiple iterations at
various Levels as a mistake or an error as long as the iterations are converging. All data produced at each
level should be preserved since market conditions and technologies can change and what may have
looked like a failed business model may become relevant again.
10

11. Citations
Blank, Steve, and Bob Dorf. ​The Startup Owner’s Manual: The Step-by-Step Guide for Building a Great
Company. Studyguide for the Startup Owners Manual : The Step-by-Step Guide for Building a Great
Company​. 2013.
Four Fits For $100M+ Growth — Brian Balfour​. ​https://brianbalfour.com/four-fits-growth-framework​.
Accessed 14 Dec. 2020.
Paun, F. “Demand Readiness Level as Equilibrium Tool for the Hybridization between Technology Push
and Market Pull Approaches.” ​ANR-ERANET Workshop. 8th February​, 2011.
Paun, Florin. “The Demand Readiness Level Scale as New Proposed Tool to Hybridise Market Pull with
Technology Push Approaches in Technology Transfer Practices.” ​Technology Transfer in a Global
Economy​, 2012, pp. 353–66, doi:​10.1007/978-1-4614-6102-9_18​.
Ries, Eric. ​The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create
Radically Successful Businesses​. Currency, 2011.
Solberg Hjorth, Sune, and Alexander Michael Brem. “How to Assess Market Readiness for an Innovative
Solution: The Case of Heat Recovery Technologies for SMEs.” ​Sustainability: Science Practice and
Policy​, vol. 8, no. 11, Multidisciplinary Digital Publishing Institute, Nov. 2016, p. 1152.
Steve Blank​. 1 July 2014,
https://steveblank.com/2014/07/01/how-investors-make-better-decisions-the-investment-readiness-le
vel/​.
11