Product Metrics: How & What

When you wok as a product manager, you ask yourself questions all the time. Most common amongst those are — Is my product working? Or if the product is already established, is my feature working? How many users are using it? When do they use it the most?
That list is never ending

Software products today are much more complex (in a nice way). Some of this complexity comes with a treasure trove of data. We then integrate with other software & SDK’s which help us understand users behavior and experience.
This data is important and can help you understand product/feature performance. What is working well and what is not. If something needs a little nudge or a complete makeover.

Unfortunately this data is in its raw format. It is up to us to give it shape and understand it. The numbers are known by different names. The one that I use the most — KPI, metrics, user adoption.

How many metrics?

There is no single set of metric that works for everyone. Businesses and products (features pithing a product) are all unique and have different goals based on the state and ambition. For example, a startup will want to acquire new customers quickly, where as a 50 year old business would be more focused on retention and upset to an already established customer base.
Most organizations have a handful of metrics that sum up their product’s (or service) overall performance. From these handful of metrics there is always one (or two) metrics that is slightly more important in comparison. These typically are termed as focus metrics.
These focus metrics are supported by a more granular metrics that teams and individuals spend their time on to drive the momentum of the focus metric. Once you identify and set them up; it forms a hierarchy with an upward flow of impact. Any individual granular or lower level metric that performs well, will advance the focus metric.

What should be my Focus Metric?

Let’s talk about not one but more than one metrics. A “North Start” metric does not exists. Experts have mulled over this concept and have concluded that a business should have a constellation of metrics, rather than one single metric that matters. A North Star metric can be very limiting. For example, Facebook no longer has a single metric in Monthly Active Users. They follow a group of focus metrics that allows them to innovate and continue to grow their product.

Trying to maximizing your focus metric can hurt your product too. If YouTube cares only about videos watched, they might autoplay vides when you visit their website. Take it from me, that is an extremely frustrating user experience. You are basically trying to force your users to do something at random which has an impact on retention – which should be one of your focus metric.

Focus Metrics should be your top priority, not your only priority, and improving focus metric should be organic rather than accomplishing them at the expensive of harming other KPIs.

Focus Metrics Recommendations

Choose a focus metric tied to active usage such as Weekly or Monthly Active Users. They do a good job of summing up trending in other metrics – typically acquisition and retention. This number will show you if your customers are using the provide over time.

Once you have established your focus metric, its time to to start adding metrics that complete your focus metric. An easy way to validate that is to ask yourself a question “if we improve this number will the product’s long-term performance improve?”

At this point you start layering your metrics (like a layered cake) and you can go as many layers deep depending on the hierarchy and complexity of your product.
The first layer of metrics typically have checks to make sure that the product is growing in a healthy direction. For example, if your focus metric is WAU, a good layer 1 metric is a 7 day retention to ensure you aren’t spending previous funds to acquire new users who leave after a day or two.

The second layer onwards these metrics are more customized to your specific needs. Continuing the above example a layer 2 metric would be to check retention on your various platform like Desktop, Mobile, Web, etc. You can go one more layer below to break it down by region.

You can keep going as many layer deep you want but keep clarity in mind cause the more layers you add there is a tendency to add confusion. A good thumb rule is to focus on what matters. Too many goals can be just as ineffective as having none.

Key Metric Types

There is no secrete sauce here. The industry has a standard set of metric types/categories today.


Reach
This is the total # of people who have used the product in a recent time period. For consumer products, it could be the # of paid accounts, or users who have made a purchase in the last 3 months. For B2B, this key metric is often product install base or number of paid licenses within the past quarter or year.
Reach is important because it represents the maximum number of users who could reasonably become active, whether organically or thorough re-engagement campaigns.

Activation
This is the foundational step that primes a new user to become an active user. This was made famous by Facebook who identified adding 7 friends in the first 10 days as their activation metric when they were a startup. This metric drove their Active user key metric and they they focused adding friends as the central part of their onboarding experience.

It is recommended viewing the metrics as a % of new users rather than count in order to isolate it form natural user growth. That way you can know if you’re more successfully at activating users over time.

Active Users
Active users are users who take a key action and received value from your product within a recent time period. Here the value can be defined as one action – like uploading a picture or set of actions, such as uploading 3 pictures and tagging a picture.

A lot of consumer-focused products promote habitual usage (Twitter, Instagram) look at Daily Active Users. B2B is better viewed through the lens of Weekly Active users since it’s not always used every day (weekends are off).

Monthly Active users also is a good fit, if your product has a monthly billing cycle since bills are usually due monthly.


This is also one of the most common focus metric.

Engagement
This is all about how committed your customers are. It accounts for both the frequency and cadence of completing key actions.
This metric can be defined as the number of key actions taken, minute of video watched, or number of transactions completed. It’s important to divide this by your active user count to measure the depth of engagement per users with your product. Otherwise, user grown might mislead you into thinking your product is more sticky than it actually is.

Retention
This is all about checking if your product has staying power. Retention can be driven two ways:

  1. Are you bringing in the kind of people who stick around?
  2. Are you giving those who have already come through the door enough reason to come back?
    When deciding on the time frame for retention goals, pick a range that is long enough to capture the reasonable repeat visit cycle of your customers, yet short enough that teams can get feedback to iterate quickly.

Business Specific
It is possible that your product has specific needs or have a unique ability where the above metrics won’t work to capture the right number for you. In this case it is absolutely fine to define your metric.


Further Reading:

Product School – These are the metrics great product managers track

One more gadget: The Ember Mug 2

Coffee is probably one of the most ubiquitous and social products in the world. I’m sure tea is also somewhere right up there. So it doesn’t hurt to say a Hot Beverage.

I have seen messy desks (I have one) and extremely clean ones and a mug for a hot beverage is ubiquitous. One of the things that drives me nuts is the rate at which my beverage goes from pipping hot to stone cold. It’s not that I am lazy to get up and heat it – its the fact that every time I wake up i have the urge to make a fresh cup of hot beverage to replace my cold one. I hate wasting food.

I’m sure there are studies that would say Coffee/Tea is good or bad for your health. For me apart from Coffee being a fuel it helps me be alert (especially when you are in those long and boring meetings), it has certainly helped me improve productivity (also proved by this MIT study).

Earlier this year I was diagnosed with Cervical Radiculitis – a pathological condition of the spine and one of the reason for this was my sitting posture. It’s hard to maintain a good posture when you work in front of a screen for more than 12 hours a day. It’s even harder to correct a habit that is ~30+ years old. A hot beverage allows me to stretch back, straighten my poster and take a nice hot sip. Doing this every 20-30 minutes helps tremendously.

So when Ember came out with their temperature control mug, it was a no brainer to get one of these.

Ember has a travel mug and a regular mug. Personally I prefer the mug.

They come in 2 sizes, 10 oz and 14 oz. To put it in perspective, the 10 oz although small looks like a normal mug that’s available in the market. The 14 oz mug – although they do have that extra 4 oz; stand out cause the shape and size is not like any other normal cup available in the market.

Also remember these mugs come with a batter – which is typically heavy, so the more bigger you cup, the much heavier it gets. I went with the Black Ember Mug2 10 oz size.

I generally prefer the black color, but when it comes to utilities, I do prefer white. There was one reason I chose the Black Mug – Coffee Stains. These mugs are made of durable stainless-steel with a ceramic reinforced coating (presumably food grade) which gives these mugs the nice matt finish. And this finish will retain coffee stains. I’m sure the black mugs also retains coffee stains, but they are barely visible.

The App

The Ember Mug comes with an app. Following the instructions and I was able to pair this mug with my phone in less than 30 seconds. That was awesome.

It also walked me through to create a profile that allows me to choose a coffee brand, brew style, and my desired temperate. They suggest 135 F to being with; after setting this temperate for a week, i reduced mine to 132. Personalization is a great thing. The app also comes with multiple presets for different kinds of coffee brews.

The overall app is OK. Not the best usable app. The main screen is basically a screen with a single color and a number that shows your current coffee temperature in the middle and at the very bottom an animation showing how far off it is from your desired temperature and your profile if it’s selected one.

The ^ right below the animation suggests more options, and every time instead of tapping it, i swipe up and close the app. When you get there, you can select a coffee preset, or add/remove presets; a Tea timer if you need one, and some coffee recipes. I have never used this page after the first setup.

Settings
The settings screen allows you to:
1. change the color of your LED – I set mine to green
2. Adjust brightness (of the LED maybe?)
3. shows the Battery level
4. Change Temperature – I have mine setup at Fahrenheit
5. Notifications – I turned mine on to notify me when my coffee reaches the right temperature.

Menu
The menu has a lot more options. I enabled Ember X Health and every once in a while when I’m thinking what exactly this app does, I venture into insights.

The LED Indicator
One of the things that has always confused me is the LED indicator on the mug. It flashes Red when the battery is running low or out of charge. Most of the time it is white; and I’m have seen it go Green only when its on my charging plate. Basically IO have absolutely no idea what these indicators mean. For now I think or it as the mug has battery and its working.

Conclusion

So far its the best $99 that I have spent this year.

It keeps my coffee at my desired temperature. The battery on this lasts long enough for me to take my mug and walk around or go to a hour long meeting. Because of the ceramic coating, they do feel fragile, but I’m assuming the stainless-steel makes it durable. The confusing LED indicator – try and tune out of this and think of the white light as the mug’s working.

iPad Pro gets a trackpad

iPad Pro was announced this week with a load of Hardware goodness.
But what got the most attention was that iPad Pro now has a trackpad. Well, the Magic keyboard did (not the iPad) and the iPad OS was updated to support the trackpad.

You can always get the regular Magic Keyboard too.

Steve Sinofsky on the iPad Pro getting a trackpad:

2/ Hardware evolves just like software but we don’t often see it the same way. We’re used to talking about the cycle software bundling and unbundling, but hardware does the same thing. Every new generation of hardware begins this cycle anew.

Users change, they way the interact and interface with a piece of hardware also changes over time. What apple is doing great here is that they are keeping the medium intact for users who want it to work the way it should as well as introducing changes for new users and their behaviors.

And why is a trackpad so important to some users – give a listen to this episode on The Talk Show with John Gruber and Federico Viticci.

And after all of that, just head to apple.com and watch the video and look at the beautiful piece of hardware(s).

Dropbox: Rewriting the heart of our sync engine

Sujay Jaykar for Dropbox:

Shipping any change to sync behavior required an arduous rollout, and we’d still find complex inconsistencies in production. The team would have to drop everything, diagnose the issue, fix it, and then spend time getting their apps back into a good state. Even though we had a strong team of experts, onboarding new engineers to the system took years. Finally, we poured time into incremental performance wins but failed to appreciably scale the total number of files the sync engine could manage.

Once you have a successful product at your hand; things start to get complex. It’s not that the world has got complex problems; adding features and at times making this simple add complexity. Also, in today world, technology is so ingrained that anything you build should have the ability to scale almost instantly. But predicting how much to scale is hard. At a given point in time you do have to make a decision to build something again. The approach that you take makes or breaks your new shiny product.

I loved Dropbox’s approach here. They created a “rewrite” checklist.
I understand that one needs to refactor, optimize constantly. But at a certain time and due to resource constraints, this stops happening.

Can you deliver incremental value?

A PM is typically delivering value to their customers every release. I feel this is a very important checklist item; rewrites are slow and the teams needs to come together to pull this off.

They followed this up with a “can you pull off a rewrite” checklist

It’s much easier to write new code than fully understand existing code. So, before embarking on a rewrite, you must deeply understand and respect the “Classic” system.

This comes back to bite you hard. This I feel is the single most important item for a developer and a PM to understand how your existing product works.

Do you have the domain experts who understand the current system?

This is important. If you don’t have some; hire a contractor.

I have been involved in a couple of rewrites myself and things did not go as planned. This checklist made me realized how easy it would have been for me to apply this to every single aspect of the product and then move forward.

Apple’s Worldwide Developers Conference 2020 kicks off in June with an all-new online format

Apple Newsroom:

“We are delivering WWDC 2020 this June in an innovative way to millions of developers around the world, bringing the entire developer community together with a new experience,” said Phil Schiller, Apple’s senior vice president of Worldwide Marketing. “The current health situation has required that we create a new WWDC 2020 format that delivers a full program with an online keynote and sessions, offering a great learning experience for our entire developer community, all around the world. We will be sharing all of the details in the weeks ahead.”

With Microsoft, Google and Facebook cancelling their keynote (main) events this year; everyone were speculating if Apple will follow suit.

Classic Apple way to announce this; not a cancellation of the conference but an all-new online format accessible to ALL developers – the show much go on.

Apple prepares for the keynote events for months and WWDC is a beat amongst all; they may not have the hassle of managing crowds but making this accessible online when WWDC kicks off requires a whole set of challenges. Monitoring the performance of the even before and during the event will be key.

What I liked about this announcement was the fact that there was no mention of “coronavirus” or “COVID-19”.

perf top for debugging and checking if your app is hogging your CPU

A lot of products today live in the cloud and when it comes to its performance, developers typically have the knowledge or tools that they use to ensure the performance is acceptable.
However, as a PM I like looking at this too cause it tells me if my performance is degrading based on a feature that was added.

You can look at performance of an application by multiple means. You can use synthetic tools like Google Puppeteer and Lighthouse to see the performance of a web application. But what about that server-side code that sits in the background to process and serve this data to your application.

There are some handy local Linux tools that you can use to get a quick idea on how your application is impacting your CPU.

Your applications/programs spend a lot of time on the CPU – like billions of cycles (a billion cycle is normal). and you want to know what is your application really doing and which process is using or impacting your CPU the most.

perf is a well-known linux utility that is helpful to get some of your answers quickly. Keep in mind perf is a weird tool, that gives you useful information in different ways so give it some time to use it an understand the output.

Try running $ sudo perf record python
After running it for sometime quit by hitting Ctrl + C
The results are saved in a file called perf.data

To view the results run $ sudo perf report

This will show you the C functions from the CPython interpreter (not the Python functions) and its % usage on the CPU.

perf works on any linux machine. The exact features will vary depending on your kernel version.

When you are noticing CPU spikes on your server in very short durations there are two things you can do.

First by running $ top will show you the list of all programs and its % usage of the CPU.
You can then run $ perf top
This is just like top but for functions instead of programs. This will help you determine what function in the program is causing the CPU to spike so much.

perf top doesn’t always help but its easy to try and sometimes you are surprised by the results.

Also check out Flamegraphs by Brendan Gregg if you ant to visualize your CPU performance. Follow the instruction on his GitHub to generate report.
The graph is built from collections (usually thousands) of stack traces sampled form a single program.

Cache

Your CPU has a small cache on it called the L1 cache that it can access in about ~0.5 nanoseconds. Its 200x faster than accessing the RAM. If you are trying to do operation where ever 0.001 seconds matter, CPU cache usage matters. But you don’t want anyone to be abusing it.

Run $ perf stat
Let it run for a few seconds and then quit by hitting Ctrl + C
This will show you if any program is using those cache and how much.

You can also run $perf stat ls which simply runs the ls command and prints a report at the end.
You can also pass -e to get specific stats.

Your CPU tracks all kinds of counters and what its doing. $ perf stat asks it to count things (like L1 cache misses) and reports the results.

It’s Roadmap season

The (fiscal) year is coming to an end and most of us are recharged coming back from the holiday season. It’s the time of the year when product managers are the busiest, emotions run high and frustration sets in. It’s time to make the roadmap for the next year.

Within the organization Roadmaps are a fearcly debated topic. It’s not the principle of a product roadmap; it’s the misunderstandings they bring.

Google “What is a product roadmap” and you will be treated with a variation of definition (classic):

1. A high-level visually summary that maps out vision and direction: What is a Product Roadmap?

2. A high-level, visual representation of the direction your product offering will take over time: What is a Product Roadmap? | Customer Success Software | Gainsight

3. Plan for how your product is going to meet a set of business objectives: Product Roadmap Examples and Definition | Aha!

I like to call product roadmaps as a strategic document or in Gibson Biddle words “The roadmap is an artifact — an expression — of your product strategy”.

Reading Gibson Biddle’s artilce on The Product Roadmap

When I share a roadmap, I express confidence about the next quarter’s projects but highlight that the content and timing of the subsequent quarters are highly speculative. There’s lots of near-term learning that will cause plans to change.

This is extremely important. The classic mistakes we all make is try to set the roadmap in stone at the beginning of the year or if you are flexible enough we try to put in delivery dates for each item.

Having such a roadmap is important. Keep in mind, its the one thing that you can share with your customers to let them know if the direction your product is headed works for them and how by how much.

Remember, a product roadmap is NOT a project plan, its not permanent and most importantly its not a list of features or requirement.

For your product roadmap to express your product strategy focus on outcomes, understand your audience (your stake holders), keep it simple and stupid (KISS), state your product vision and most importantly keep it updated.

I personally follow Gibson Biddle’s quarterly format for my roadmap. Focus on themes its always helped me in aligning teams behind a common goal. It allows for healthy discussion and alignment and it helps to keep the teams focused.

Yours truly speaking to SBS Radio – Australia

Yesterday (Today) I had a short conversation with SBS (Australia) Radio’s host Mosiqi Acharya on how to click amazing photos on your smartphone!

Learn how to click amazing photos on your smartphone!Mosiqi Acharya & Vandan Desai Download

Spoiler Alert: The interview was recorded in Hindi. Head over to read the conversation in English.