User Experience (UX) Metrics for Product Managers

As Product Managers we are obsessed with what we build. Well, we all want to build the best darn product ever. We immerse ourselves in understanding

  1. how our users are finding the product or if the campaigns are working (Reach, Activation),
  2. How many users and if they are engaging well with the product (Active Users, Engagement), and
  3. Last but not least, if your users come back (Retention).
    The priority of these metrics changes depending on the nature of your app.

Give your product managers complete autonomy / authority to drive campaigns, onboard users. They are the best and most aware of the product they are building.

Automation is on the rise. With CI/CD and other similar processes; we are able to ship code at a faster rate then ever before. Delivering value to customers at this rate is great; but, it is important you focus on quality over quantity.

When we talk about quality, Performance is right up top. It is a common practice to check for performance before your application goes live. Most organizations do this.
Google’s Lighthouse is widely used for this purpose for web based consumer application. Google’s Lighthouse-CI is integrates with your CI/CD tool which passes or fails a builds based on performance rules.

Note: You can ignore the SEO numbers/suggestions for your SaaS application.

These tools ensure quality. This helps with SEO, accessibility and best practices and measures performance metrics. Performance metrics are important to understand how your page loads as this impacts user experience.

Important Metrics to track User Experience

Start Render or First Paint & Largest Contentful Paint (LCP)

The reason I have suggested an option to choose from 2 metrics is because one of them is easier to capture then the other. You can be the best judge when it comes to accuracy of the metric and if this is something that will work for you.

Start Render is measured by capturing a video of the page load and looking at each frame for the first time the browser displays something other than a blank page. This is typically measured in lab or using a synthetic monitoring tool like Catchpoint and is the most accurate measurement for it.

First Paint (FP) is a measurement reported by the browser itself; that is, when it thinks it painted the first content. It is fairly accurate but sometimes it reports the time when the browser painted nothing but a blank screen.

Largest Contentful Paint (LCP) is yet another metric by Google as they continue to foray into web performance metrics and is part of their Web Vitals offering.

FP should typically happen under 2 seconds. Imagine the application you are trying to use sows up a blank screen for a few seconds before it starts rendering the content. This is NOT a good user experience. Show a blank screen for more than 2 seconds after entering the URL can cause page abandonment. You want to tell the user as soon as possible that some activity is happening. This could be as simple as chaining the background color which alerts the user that the application is loading.

According to LCP‘s definition; it is the time it takes for the largest above the fold content to load. For example, breaking story on a news website. This is an important metric, because users typically expect to see something relevant quickly.

Together with FP (or start render) and LCP measures the Loading Experience for a user.

Time To Interactive (TTI)

According to web.vitals:

TTI metric measures the time from when the page starts loading to when its main sub-resources have loaded and it is capable of reliably responding to user input quickly.

TTI follows FP. Having a big gap between these two would mean your users are waiting until the entire page completes rendering. This means if you have an extremely fast loading web application but a horrible TTI; the performance is worst compared to a slower application.

We tend to build muscle memory often for things that we use constantly. This is true for SaaS applications. Most of the tools that I use at work; I have built in muscle memory, which makes me position my mouse on the browser at a location the link/button will render and the moment it does; I tend to click it.

Speed Index

Speed Index is one of the metrics that is tracked by Google’s Lighthouse (or Web.Vitals) as part of performance report:

Speed Index measures how quickly content is visually displayed during page load. Lighthouse first captures a video of the page loading in the browser and computes the visual progression between frames. Lighthouse then uses the Speedline Node.js module to generate the Speed Index score.

In simple words, Speed Index metric tells you at what rate does the visible content (above the page fold) is loading. The lower the score, the better is the user experience.

Typically all of these metrics should be tracked by your engineering or performance teams; however, it is good practice to keep an eye on these as they would be benchmarked based on historical data or competitive data. Breaching the benchmark or any changes in these benchmark can have a direct impact on the user experience of your application.

If you are curing to now more, here is a great article on Understanding Speed Index | Performance Metrics.


How to track metrics?

You can use a Synthetic Monitoring tool like Catchpoint. If you are the adventurous kind you can use Google Puppeteer to run a lighthouse test to capture the above metrics and Grafana to show a historical time series of these performance metrics.

As a product manger I track a lot more metrics and have built my entire dashboard on Grafana (more on this in a later post). I have a set up using Google Puppeteer and Lighthouse libraries that I use to push these metrics and other performance metrics provided by Google Lighthouse in my Dashboard every 24 hours. This allows me to see my performance numbers along with other KPI’s.

Apple’s Worldwide Developers Conference 2020 kicks off in June with an all-new online format

Apple Newsroom:

“We are delivering WWDC 2020 this June in an innovative way to millions of developers around the world, bringing the entire developer community together with a new experience,” said Phil Schiller, Apple’s senior vice president of Worldwide Marketing. “The current health situation has required that we create a new WWDC 2020 format that delivers a full program with an online keynote and sessions, offering a great learning experience for our entire developer community, all around the world. We will be sharing all of the details in the weeks ahead.”

With Microsoft, Google and Facebook cancelling their keynote (main) events this year; everyone were speculating if Apple will follow suit.

Classic Apple way to announce this; not a cancellation of the conference but an all-new online format accessible to ALL developers – the show much go on.

Apple prepares for the keynote events for months and WWDC is a beat amongst all; they may not have the hassle of managing crowds but making this accessible online when WWDC kicks off requires a whole set of challenges. Monitoring the performance of the even before and during the event will be key.

What I liked about this announcement was the fact that there was no mention of “coronavirus” or “COVID-19”.

Banner Blindness

An update to my previous post on Google Ads, here is an interesting article from Nielsen Norman Group. 

The most prominent result from the new eyetracking studies is not actually new. We simply confirmed for the umpteenth time that banner blindness is real. Users almost never look at anything that looks like an advertisement, whether or not it’s actually an ad.

The trouble with eye tracking is that it only tracks where your direct line of vision is. Human eyes are equipped with peripheral vision as well and that is where the trouble lies. Although the user is focused on reading what is important, he/she is completely aware of their surroundings. No wonder you would see banner ads that are “gif-fy” with saturated colors or text on them.

Technology is changing us (for the good I think)

In the last ~ten years I have changed my mobile phone a number of times, from Nokia 5110 to Palm Treo to Blackberry Curve to Android Droid X and finally settling down with an iPhone. Its amazing how technology has evolved from large mobile phones to small tiny Nokia phones and back to massivesmartphones phones.

Designing for mobile is different and not just with regards to the shape and size. Because mobile devices are lighter and more portable (in some cases), we find it more convenient to use them. And because we use them so often, we feel a unique, emotional connection to them.

One more thing I realized in the past few week is my transition from being a true Microsoft user to an everything Apple user in the past five or so years. I realize I have every single Apple product and everything is connected to the cloud. I am not talking about the wonderful (pun intended) ecosystem that Apple provides, but my motivation to move everything to Apple. And this does not end with just hardware; of course the transition to Mac OS X was smooth, I missed MS Office (IE was replaced with Chrome a while back). But in the past year, I have seen myself transitioned to iWork and when I think about this transition, I questioned myself why? Why did I move away from a software that I used for almost a decade.

Microsoft Word

Microsoft Word

“Clunky!”

It struck me when I heard this from my customers complaining about a system that they have used for years. It baffled me that the same users who have used this system for so many years are now finding this system clunky. Why?

Apple Pages

Tracing back to my own transition, I realized how simple Gtalk, Gmail and Chrome browser were to start with. After getting my hands on my first tablet in 2010, I slowly transitioned to Apple mail and it didn’t take me a while to move to iWork when I used their 30 day trial version. The fact that I was not looking at too many things in my face made me feel better. Just like my iPhone (in my case) I could concentrate on a task that I was currently doing rather than find something in that massive ribbon on the top with icons that I found it difficult to understand at times.

We have different attitude, behavior and priorities while using our gadgets. They come with us on the bus, walking down the street or watching TV. Users expect more contextual based actions then an al-a-carte menu system where they are expected to find an action they want to take. Because mobile devices have changed user expectations, it is extremely important to consider them is user studies since their influence is much larger then we can think of.

Wireframing is not Prototyping

With the rise of UX role in the industry, wireframes have played increasingly leading role in the modern world. Its a simple way to validating user interface and layout and are cheaper and faster to produce than a final visual comp.

Wait a minute did I say cheaper and faster? Really?

Over the last couple of years, the number of wire framing tools that have come up in the market is alarming. From a world of Visio, Omnigraffle and to a certain extent Rational Rose in the past, today you have options for any platform. You could do a google search today and you would be surprised to see the number of tools available today and with links to tell you the top 5, 10 & 20 tools for wireframing. With more options today, we run into more complexity.

Paper & Pencil have been probably the cheaper and faster way of evaluating the interface and layout, however I see very little of those today.

Today, there are applications that allow you to create wireframes that look hand drawn.

The last few years the wireframes have looked more and more different and detailed. Images, actual look and feel, every single mouse movement or tap or gesture and what happens next, etc. That definitely throws cheaper and faster out of the window.

Lets look at a classic definition of wireframe

wire·frame
ˈwīrˌfrām/

noun: wireframe; plural noun: wireframes; noun: wire-frame; plural noun: wire-frames
1. 
a skeletal three-dimensional model in which only lines and vertices are represented.

Exactly, a wireframe is a three-dimensional illustration of a page’s interface that specifically focuses on space allocation, prioritization of content, functionalities available, and intended behaviors. For these reasons, wireframes typically DO NOT include any styling, color, or graphics.

However, with the next generation of devices already here, and with users employing more touch-based system, a few changes should be made in the way we do classic wireframes. In spite of these progressions, though, creating wireframes should remain cheaper & faster.

Today we have taken to wire-framing to actually deliver a full blown spec. Do you think a developer is going to make sense of a step-by-step wireframe that captures every single gesture? (most of the developers/QA know what happens when you click a drop down or on tapping the button it turns blue from grey). These are not High-Fidelity wireframes (if that is your arguing point) but a spoon feeding document which runs into 100’s of pages.

A High-fidelity wireframe has increased level of details which should simply point out details like dimensions, behavior and/or actions related to any interactive element.

There have been some rules that designers need to consider while creating wireframe. Lets see how or what we can change so that wireframes are better suited for today’s interaction. It’s important to keep in mind to use wireframes as guides and not deliver a spec.

Some old school rules:

Do not use colors — If you want feedback on your functionality, do not use colors, however always use a different color to show highlights or selection. Stakeholders always get confused when they don’t see a selection color; in other cases avoid colors.

Do not use images — Images distract from the task at hand. Do not use an image unless it is part of your navigation and you cannot show any interaction with that image. Else just show a placeholder.

Typography — Use one font, but vary their sizes to differentiate.

Keep them short and to the point — Don’t spoon feed. Use annotations and try to capture your use cases.

One of the things old school wireframes always said was to show what things go where irrespective of shape and size since it was important to give the users an idea as to what content goes where. Later on it became important to do wireframes that would show details of content before the web page fold.

Today, users use more touch devices compared to desktops/laptops. Screen sizes have become more important and users don’t have to worry about the web page fold anymore. Every single pixel has become important. Thus making it even more important to deliver pixel perfect wirefames.

One of the biggest down side of delivering wireframes as a spec is that at times you do not know from where to start. Should I do the dashboard now? Oh but making an appointment is more important then looking at summary. At the end, you are scampering to get things done, with 100’s of wireframe slides and still no starting point.

A good technique that I have relied in the past is to write down high-level scenarios for your personas that you create. Once you have these written down; prioritize them — there you have your starting point.

Now take a single scenario and try to come up with uses cases, or business rules whatever you are comfortable with. Start with a single positive use case, document all the business rules. Do your wireframe, as minimal as possible and annotate them if required.

Create a summary page at the end of this, and capture all the negative use cases and error cases. Typically you wouldn’t need wireframes to show these unless you need a different interaction all together.

Keep your wireframes short, simple and annotated and remember to keep them cheaper and faster.