Stop Using the Wrong Product Metrics: Valuable Metrics Framework [Part 1]

— July 24, 2019

This article is Part 1 in a two-part series that will help your product team find the right metrics to demonstrate and improve business impact. Check out Part Two here, when you’re finished with this article.

As a product manager, your job is to drive value for your customers and for the business.

The thing is, you can’t tell whether you’re driving value unless you have a way to measure the value that you’re creating! After all, you need an objective way of determining your impact outside of qualitative feedback from customers and stakeholders.

That’s why it’s so valuable for product managers to be data-informed, and why it’s such a key skillset for any product manager.

Yet, even when product managers are data-informed, one challenge that I regularly see across the industry is that it’s difficult to align current product metrics to the targeted business value.

Let me first explain why product metrics might not align with business value. Then, from there, I’ll dive into a framework for how to leverage metrics to truly drive business impact.

When Product Metrics Don’t Line Up

Many times, product managers inherit an existing product, and therefore they also inherit its current set of metrics and instrumentation.

As an example, at one point I inherited a mobile app into my product ownership. Rather than starting from scratch, I already had the following metrics instrumented:

• Number of installations on iOS vs. Android
• Last login date of user
• Logins per week per user

But, just because you have existing metrics doesn’t mean that they’re the right ones.

The Dangers of Vanity Metrics

For example, it’s dangerous to use the vanity metric of logins per week per user, because that doesn’t explicitly point towards tangible customer value or tangible business value.

When a user logs in, it doesn’t mean that they successfully completed a valuable task. For example, a user might log in just to check for notifications—and if they didn’t have any notifications, then they didn’t get any value from that login.

Therefore, logins aren’t a valid indicator of value, even though it’s an easy metric to measure.

Another vanity metric that I regularly see is “number of times a new feature is used.”

By definition, a new feature will naturally be used over time, because human beings are curious and will try out new things. New feature usage doesn’t necessarily mean value.

Measure What’s Meaningful, Not What’s Easy

The key reason why product metrics might not align to business value is because most instrumented metrics are the ones that are easy to instrument, and not the ones that actually matter to the business.

For example, it’s usually pretty easy to measure how long someone is active in your application, which doesn’t speak to business value. It’s usually harder to instrument the metric of “dollars saved by the customer by using your product,” because it’s inherently difficult (if not impossible) to get live cost data from your competitors.

It’s natural that once you’ve implemented some set of metrics—even if they’re not the right ones!—you won’t look for additional metrics to implement.

After all, implementing new metrics takes development time. On top of that, once you’ve implemented them, you then have to train your stakeholders on what the new metrics mean.

That’s why it’s so crucial to set the right metrics from the very beginning, even if they’re hard to implement.

Rather than falling for the trap of “measuring what’s easy to measure”, we should start from the top and begin with business value. When we begin with business value, we can then determine what metrics to focus on.

So, let me walk you through a framework for how to decide on which metrics to instrument.

A Framework for Valuable Metrics

We can think of metrics as streams of value that flow into larger rivers of value, which eventually flow into the ocean of business value.

Imagine a stream that flows into a lake, which doesn’t connect to the ocean. That stream is never going to contribute to the ocean because it’s flowing into a lake that is disconnected.

It doesn’t matter how fast or how wide the stream is—it’s simply not contributing to the ocean. Focusing additional efforts on that stream is wasteful because it never winds up flowing into the ocean.

Therefore, it’s critical to first define what business value is. What are the key drivers of the business? Every business will have different key drivers, and even the same business will wind up having different key drivers over time. Again, in our metaphor above, the ocean represents business value.

Then, once you know what the various drivers of the business are, you’ll need to select one of them to focus on. Remember, product management is all about tradeoffs—you can’t do everything at once.

Once you know the business value you want to target, then define the north star objective. What’s the core qualitative objective of your team?

Once you have an objective, you can then define a north star metric that moves you towards the objective. Remember, an objective is aspirational, whereas a metric is tangible. In our metaphor, the river represents the north star metric.

Once you have the north star metric, you can then decompose the metric into distinct qualitative problem areas.

As you define a problem area, you can then ideate with business stakeholders and development teams to flesh out a constellation of metrics that you can track and improve. In our metaphor, each stream represents a valuable product metric.

Once you’ve knocked out the highest priority problem area, you can then focus your attention on the next highest priority.

As you strengthen each of your streams of product value, you build additional velocity and momentum into a river of value, which then flows into the ocean of business value.

To summarize:

1. Determine the key drivers of the business
2. Select the single driver you want to focus on
3. Define a qualitative north star objective
4. Define a north star metric
5. Decompose the north star metric into qualitative problem areas
6. Define product metrics that capture each problem area
7. Solve a problem area and measure progress
8. Iteratively knock out problem areas

Note how the framework cycles between qualitative and quantitative steps. That’s intentional! Human beings are naturally driven by stories, and the most robust stories are naturally driven by data.

Summary and Additional Thoughts

Product managers usually use the metrics that are the most convenient to get—those that they’ve inherited through previous implementation, and those that are easy to implement next.

But, the convenience of getting a metric isn’t correlated with how well-aligned that metric is to the business value.

So, to ensure that you drive business value as a product manager, use the framework above.

Begin with the business value that you want to drive, then define a north star objective and metric, then break down into sub-metrics, and iteratively solve problems that will ladder back into the business value.

Of course, frameworks aren’t helpful without live examples.

Business & Finance Articles on Business 2 Community

(33)