Calculating the ROI of Digital Prototyping

Product teams frequently confront the challenges of aligning leading practices in the market with their organization’s established processes. In particular, rapid prototyping is an increasingly popular practice that enables product teams to generate feedback from users about product concepts before they commit investment to build them. But because user research does not have a predictable outcome, it doesn’t fit into any of a large organization’s neatly defined budget categories. Prototyping seemingly adds costs to the product development lifecycle without guaranteeing any financial return.

This conundrum has considerable ramifications for product managers who often feel that it’s easier to get a multi-million dollar budget and ‘green light’ for an unsubstantiated product concept than it is to actually generate user feedback on that product concept beforehand. Being able to make a financial case for prototyping and other user research activities should be a weapon in every product manager’s arsenal.

Think like an analyst

Product teams are accustomed to practicing empathy, so there are few better places to start than by talking to a number of corporate finance teams to better understand the budgeting process from their perspective. Unsurprisingly, it turns out that they take a very pragmatic (as in, non-buzzwordy) approach to budgeting, where things like ‘user-centric’ and ‘lean’ have no bearing.

Finance teams recognize that most products fail at an astonishing rate, and make projections accordingly. Making a financial case for digital prototyping must rest on the premise that any initiative that has an unpredictable outcome (i.e. there is a chance it can fail) is by definition an experiment.

Because developing a product is, essentially, one big experiment, finance teams conduct an analysis based on scenarios ranging from highly optimistic to severely conservative. Each scenario will be assigned some perceived likelihood of occurring. The overall expected value or ROI of an initiative is the cumulative value of every individual scenario multiplied by its respective likelihood of happening. For simplicity, let’s say that some initiative is assigned two potential scenarios.

Developing a product without user research is essentially one expensive experiment which, according to aforementioned industry benchmarks, has a 90% chance of not paying off (at least without modifications post-launch).

To get an idea for what this means in dollars and cents, a survey of product managers found that more than half of respondents spend $50,000 or more per month, while more than 70% spend at least seven months to build a product. For large organizations, this roughly translates into somewhere between $350,000 to $1M to build a product. Without user research, this is one expensive experiment. It’s quite costly to fail, not to mention the toll on morale, the time and resources that could’ve been spent elsewhere, or the potential brand ramifications.

Place more bets

At most large organizations, the status quo is that once a product becomes a budget line item, it’s going to be built. No amount of user research or data is likely to prevent that. Digital prototyping must therefore be positioned to finance teams as an inexpensive way to assess the market’s response to a product at any given time. Prototyping, in other words, enables product teams to break the massive product development bet into multiple smaller bets.

Despite the overwhelming odds against each individual bet, your finance team will be able to quickly pick up on the mathematical phenomena taking place. In our example, if we were to add two phases of user research to the development lifecycle, the probability of assessing that we’ve discovered what it takes to reach the optimistic case can be expressed as:

[(10%) + (90% * 10%) + (90% * 90% * 10%)] = 27.1%

Rapid prototyping shifts the projected outcome of any initiative, thereby increasing its expected value. Tom Chi, the head of Google X, calls this “Rate-Based Goals.” In a 2012 talk at Mind The Product, he elaborated:

No matter what we start working on, there’s maybe a 5% chance that it’s the right thing. But, here’s a little bit of math. By the time you try 20 things, even if each individual thing only has a 5% chance of success, your chance of success skyrockets up to 64%. And by the time you try 50 things, your chance of success goes up to 92%. It’s almost like you can’t fail! Then someone said, ‘well if you go over 100, does it go over 100%?’ It doesn’t! It asymptotically approaches 100%!

This is obviously an oversimplified explanation, as it is not nearly as easy as it sounds to determine you’ve struck gold from user feedback. But in financial terms, it is a legitimate investment.

Take heed of the diminishing return on betting

It’s now clear how, from a financial perspective, digital prototyping can increase the expected value of a project. Therefore, prototyping is a sound investment so long as the value (or learning) from a user insights outweighs the cost associated with generating that insight.

Not all user insights are created equal. Vlad Lokshin of web and mobile app development firm, DarwinApps, explains an important concept to each of his clients at the start of every project. As products move through the development lifecycle, they become more complex, and the cost of changing them exponentially increases.

In other words, the value of an insight is far greater before rather than after a product team invests resources into building a robust product. As in, the relative value of an insight over time can be modeled like this:

The ROI of rapid prototyping is greatest early on, making it a wise financial investment perhaps even before a product concept becomes a line item. As the old saying goes, a stitch in time saves nine.

The same principles apply elsewhere which is why pharmaceutical companies, as one example, work tirelessly to test assumptions as early as possible in the clinical trial lifecycle.

Do what’s tried and true early and often

It’s difficult to imagine a future during which most products succeed. Nevertheless, for product managers and finance teams that want to optimize the expected value of investments, there is no better way than to budget for rapid prototyping.

By breaking up product development into multiple smaller bets rather than one large bet, prototyping offers the organization multiple opportunities to determine the most promising routes to success and allocate resources accordingly. Compared to the cost for developing an entire digital product in one go, rapid prototyping is an order of magnitude cheaper.

Sure, sometimes product teams get lucky rolling the dice: the big bet pays off and they get the job, the car, and money all on the first try. Just remember - developing products is risky business, and there’s a more sensible game plan that going ‘all in’ blind.