User Experience (UX) Metrics for Product Managers

As Product Managers we are obsessed with what we build. Well, we all want to build the best darn product ever. We immerse ourselves in understanding

  1. how our users are finding the product or if the campaigns are working (Reach, Activation),
  2. How many users and if they are engaging well with the product (Active Users, Engagement), and
  3. Last but not least, if your users come back (Retention).
    The priority of these metrics changes depending on the nature of your app.

Give your product managers complete autonomy / authority to drive campaigns, onboard users. They are the best and most aware of the product they are building.

Automation is on the rise. With CI/CD and other similar processes; we are able to ship code at a faster rate then ever before. Delivering value to customers at this rate is great; but, it is important you focus on quality over quantity.

When we talk about quality, Performance is right up top. It is a common practice to check for performance before your application goes live. Most organizations do this.
Google’s Lighthouse is widely used for this purpose for web based consumer application. Google’s Lighthouse-CI is integrates with your CI/CD tool which passes or fails a builds based on performance rules.

Note: You can ignore the SEO numbers/suggestions for your SaaS application.

These tools ensure quality. This helps with SEO, accessibility and best practices and measures performance metrics. Performance metrics are important to understand how your page loads as this impacts user experience.

Important Metrics to track User Experience

Start Render or First Paint & Largest Contentful Paint (LCP)

The reason I have suggested an option to choose from 2 metrics is because one of them is easier to capture then the other. You can be the best judge when it comes to accuracy of the metric and if this is something that will work for you.

Start Render is measured by capturing a video of the page load and looking at each frame for the first time the browser displays something other than a blank page. This is typically measured in lab or using a synthetic monitoring tool like Catchpoint and is the most accurate measurement for it.

First Paint (FP) is a measurement reported by the browser itself; that is, when it thinks it painted the first content. It is fairly accurate but sometimes it reports the time when the browser painted nothing but a blank screen.

Largest Contentful Paint (LCP) is yet another metric by Google as they continue to foray into web performance metrics and is part of their Web Vitals offering.

FP should typically happen under 2 seconds. Imagine the application you are trying to use sows up a blank screen for a few seconds before it starts rendering the content. This is NOT a good user experience. Show a blank screen for more than 2 seconds after entering the URL can cause page abandonment. You want to tell the user as soon as possible that some activity is happening. This could be as simple as chaining the background color which alerts the user that the application is loading.

According to LCP‘s definition; it is the time it takes for the largest above the fold content to load. For example, breaking story on a news website. This is an important metric, because users typically expect to see something relevant quickly.

Together with FP (or start render) and LCP measures the Loading Experience for a user.

Time To Interactive (TTI)

According to web.vitals:

TTI metric measures the time from when the page starts loading to when its main sub-resources have loaded and it is capable of reliably responding to user input quickly.

TTI follows FP. Having a big gap between these two would mean your users are waiting until the entire page completes rendering. This means if you have an extremely fast loading web application but a horrible TTI; the performance is worst compared to a slower application.

We tend to build muscle memory often for things that we use constantly. This is true for SaaS applications. Most of the tools that I use at work; I have built in muscle memory, which makes me position my mouse on the browser at a location the link/button will render and the moment it does; I tend to click it.

Speed Index

Speed Index is one of the metrics that is tracked by Google’s Lighthouse (or Web.Vitals) as part of performance report:

Speed Index measures how quickly content is visually displayed during page load. Lighthouse first captures a video of the page loading in the browser and computes the visual progression between frames. Lighthouse then uses the Speedline Node.js module to generate the Speed Index score.

In simple words, Speed Index metric tells you at what rate does the visible content (above the page fold) is loading. The lower the score, the better is the user experience.

Typically all of these metrics should be tracked by your engineering or performance teams; however, it is good practice to keep an eye on these as they would be benchmarked based on historical data or competitive data. Breaching the benchmark or any changes in these benchmark can have a direct impact on the user experience of your application.

If you are curing to now more, here is a great article on Understanding Speed Index | Performance Metrics.


How to track metrics?

You can use a Synthetic Monitoring tool like Catchpoint. If you are the adventurous kind you can use Google Puppeteer to run a lighthouse test to capture the above metrics and Grafana to show a historical time series of these performance metrics.

As a product manger I track a lot more metrics and have built my entire dashboard on Grafana (more on this in a later post). I have a set up using Google Puppeteer and Lighthouse libraries that I use to push these metrics and other performance metrics provided by Google Lighthouse in my Dashboard every 24 hours. This allows me to see my performance numbers along with other KPI’s.

One more gadget: The Ember Mug 2

Coffee is probably one of the most ubiquitous and social products in the world. I’m sure tea is also somewhere right up there. So it doesn’t hurt to say a Hot Beverage.

I have seen messy desks (I have one) and extremely clean ones and a mug for a hot beverage is ubiquitous. One of the things that drives me nuts is the rate at which my beverage goes from pipping hot to stone cold. It’s not that I am lazy to get up and heat it – its the fact that every time I wake up i have the urge to make a fresh cup of hot beverage to replace my cold one. I hate wasting food.

I’m sure there are studies that would say Coffee/Tea is good or bad for your health. For me apart from Coffee being a fuel it helps me be alert (especially when you are in those long and boring meetings), it has certainly helped me improve productivity (also proved by this MIT study).

Earlier this year I was diagnosed with Cervical Radiculitis – a pathological condition of the spine and one of the reason for this was my sitting posture. It’s hard to maintain a good posture when you work in front of a screen for more than 12 hours a day. It’s even harder to correct a habit that is ~30+ years old. A hot beverage allows me to stretch back, straighten my poster and take a nice hot sip. Doing this every 20-30 minutes helps tremendously.

So when Ember came out with their temperature control mug, it was a no brainer to get one of these.

Ember has a travel mug and a regular mug. Personally I prefer the mug.

They come in 2 sizes, 10 oz and 14 oz. To put it in perspective, the 10 oz although small looks like a normal mug that’s available in the market. The 14 oz mug – although they do have that extra 4 oz; stand out cause the shape and size is not like any other normal cup available in the market.

Also remember these mugs come with a batter – which is typically heavy, so the more bigger you cup, the much heavier it gets. I went with the Black Ember Mug2 10 oz size.

I generally prefer the black color, but when it comes to utilities, I do prefer white. There was one reason I chose the Black Mug – Coffee Stains. These mugs are made of durable stainless-steel with a ceramic reinforced coating (presumably food grade) which gives these mugs the nice matt finish. And this finish will retain coffee stains. I’m sure the black mugs also retains coffee stains, but they are barely visible.

The App

The Ember Mug comes with an app. Following the instructions and I was able to pair this mug with my phone in less than 30 seconds. That was awesome.

It also walked me through to create a profile that allows me to choose a coffee brand, brew style, and my desired temperate. They suggest 135 F to being with; after setting this temperate for a week, i reduced mine to 132. Personalization is a great thing. The app also comes with multiple presets for different kinds of coffee brews.

The overall app is OK. Not the best usable app. The main screen is basically a screen with a single color and a number that shows your current coffee temperature in the middle and at the very bottom an animation showing how far off it is from your desired temperature and your profile if it’s selected one.

The ^ right below the animation suggests more options, and every time instead of tapping it, i swipe up and close the app. When you get there, you can select a coffee preset, or add/remove presets; a Tea timer if you need one, and some coffee recipes. I have never used this page after the first setup.

Settings
The settings screen allows you to:
1. change the color of your LED – I set mine to green
2. Adjust brightness (of the LED maybe?)
3. shows the Battery level
4. Change Temperature – I have mine setup at Fahrenheit
5. Notifications – I turned mine on to notify me when my coffee reaches the right temperature.

Menu
The menu has a lot more options. I enabled Ember X Health and every once in a while when I’m thinking what exactly this app does, I venture into insights.

The LED Indicator
One of the things that has always confused me is the LED indicator on the mug. It flashes Red when the battery is running low or out of charge. Most of the time it is white; and I’m have seen it go Green only when its on my charging plate. Basically IO have absolutely no idea what these indicators mean. For now I think or it as the mug has battery and its working.

Conclusion

So far its the best $99 that I have spent this year.

It keeps my coffee at my desired temperature. The battery on this lasts long enough for me to take my mug and walk around or go to a hour long meeting. Because of the ceramic coating, they do feel fragile, but I’m assuming the stainless-steel makes it durable. The confusing LED indicator – try and tune out of this and think of the white light as the mug’s working.

Apple’s Worldwide Developers Conference 2020 kicks off in June with an all-new online format

Apple Newsroom:

“We are delivering WWDC 2020 this June in an innovative way to millions of developers around the world, bringing the entire developer community together with a new experience,” said Phil Schiller, Apple’s senior vice president of Worldwide Marketing. “The current health situation has required that we create a new WWDC 2020 format that delivers a full program with an online keynote and sessions, offering a great learning experience for our entire developer community, all around the world. We will be sharing all of the details in the weeks ahead.”

With Microsoft, Google and Facebook cancelling their keynote (main) events this year; everyone were speculating if Apple will follow suit.

Classic Apple way to announce this; not a cancellation of the conference but an all-new online format accessible to ALL developers – the show much go on.

Apple prepares for the keynote events for months and WWDC is a beat amongst all; they may not have the hassle of managing crowds but making this accessible online when WWDC kicks off requires a whole set of challenges. Monitoring the performance of the even before and during the event will be key.

What I liked about this announcement was the fact that there was no mention of “coronavirus” or “COVID-19”.

Documenting UX

A lot of companies today implement an agile process and the remaining ones like to call themselves agile. A few years back any designer (UX/IX) would pull out their hair because turning around design details wasn’t easy given the amount of time a sprint would end. Years go by and processes have evolved. Processes like LeanUX and staggered sprints or design and R&D sprints in parallel have more or less solved this issue. However these processes have reduced the amount of documentation.

We have user research data, but never used it to create personas. We have analytics data but never exported them out of their software/databases. There are a list of scenarios or stories that are important or have a higher priority, but they never made it to the official todo list or backlog. All, I see is wireframes, trolls of em. Some of them make sense like login screens and forgot passwords; the others I have no idea. Today wireframes have become one of the most important means of documentation. I even see scanned sketches, photographed whiteboards and clickable wireframes. One change in the use case or workflow, and you are back to square one.

It does capture details; however, chances are when I see someone else’s wires, I may not like a solution. How do you support your solution? 

And I have made them all; mistakes. The fact that the requirements or priorities are in my head and not anywhere else does not do good for me to convince someone why these things need to be done the way I intend to. 

Over the last couple of months I started maintaining a combination of few documents that has helped me transition stuff and propose design easily. Here is a short run down of these documents and processes: 

We all do our bit by venturing out and talking to users. We capture notes and lot of em. When you do so, try to capture them as quotes. There are two important advantages in doing so. One, Its faster to write them while you are interviewing/talking to them and Two, it preserves the context. 

“I have to apply the same filter every fucking time I visit this page” 

“See I like this pages concept in OneNote. This way I can just have multiple notebooks and go and write down in pages within them…”

One of the important thing is to digitize these notes. It could be in a notepad or Evernote; I found excel helpful (yeah!). I was able to create buckets/categories and then write the quotes in these specific buckets. Also, when I digitized these notes; I was able to add attributes to them; like screen resolution, position, firm, interview date, system usage, type of user, etc. These become important when I really wanted to narrow down to specific kind of users.  Apart from that I color codeed these quotes – for example. Red for negative, Blue for positive and Greens for opportunities; this gave me an idea on the state of the existing software.  

I haven’t created any personas (and I might not for the time being) for my work. But when I was working on a specific problem, I was able to filter down to a handful of target users in my excel sheet then parsing through notebooks of research data. Once I have narrowed it down to a bunch of users, it is easier for me to identify patterns in their quotes (colors added value). Those attributes that you add now allows you to see how much is in common between these users.

After looking at my excel, I note down tasks/use cases/stories. I maintain a list in a todo list. Apple reminders does the job for me. Nothing fancy. This allows me to scope out, prioritize stuff. Anything that is high priority gets a date and bubbles up, rest of them just remain at the bottom. Simple.

Analytics – If you have it, great. This comes in use when you really want to see what users are doing with existing systems. How are they using it. Export them and keep them handy. Don’t rely on your system that you did log in and get data. Historic data is fine. Patterns are not going to change overnight. Export them because you can quickly pivot data. That is important to figure out supporting numbers for your uses cases. 

This accompanying my wireframes/sketches is a good combination. I try to ensure my wireframes are also not too elaborate. Stick to the specific workflow or use case and thats good enough. 

Trying to Keeping iSimple and Stupid. So far it has worked for me. Maybe over the next few months or a year I will know if it was effective enough.

Please leave your comments and suggestions or any of the processes you personally follow or send an email to get in touch.

Banner Blindness

An update to my previous post on Google Ads, here is an interesting article from Nielsen Norman Group. 

The most prominent result from the new eyetracking studies is not actually new. We simply confirmed for the umpteenth time that banner blindness is real. Users almost never look at anything that looks like an advertisement, whether or not it’s actually an ad.

The trouble with eye tracking is that it only tracks where your direct line of vision is. Human eyes are equipped with peripheral vision as well and that is where the trouble lies. Although the user is focused on reading what is important, he/she is completely aware of their surroundings. No wonder you would see banner ads that are “gif-fy” with saturated colors or text on them.

Technology is changing us (for the good I think)

In the last ~ten years I have changed my mobile phone a number of times, from Nokia 5110 to Palm Treo to Blackberry Curve to Android Droid X and finally settling down with an iPhone. Its amazing how technology has evolved from large mobile phones to small tiny Nokia phones and back to massivesmartphones phones.

Designing for mobile is different and not just with regards to the shape and size. Because mobile devices are lighter and more portable (in some cases), we find it more convenient to use them. And because we use them so often, we feel a unique, emotional connection to them.

One more thing I realized in the past few week is my transition from being a true Microsoft user to an everything Apple user in the past five or so years. I realize I have every single Apple product and everything is connected to the cloud. I am not talking about the wonderful (pun intended) ecosystem that Apple provides, but my motivation to move everything to Apple. And this does not end with just hardware; of course the transition to Mac OS X was smooth, I missed MS Office (IE was replaced with Chrome a while back). But in the past year, I have seen myself transitioned to iWork and when I think about this transition, I questioned myself why? Why did I move away from a software that I used for almost a decade.

Microsoft Word

Microsoft Word

“Clunky!”

It struck me when I heard this from my customers complaining about a system that they have used for years. It baffled me that the same users who have used this system for so many years are now finding this system clunky. Why?

Apple Pages

Tracing back to my own transition, I realized how simple Gtalk, Gmail and Chrome browser were to start with. After getting my hands on my first tablet in 2010, I slowly transitioned to Apple mail and it didn’t take me a while to move to iWork when I used their 30 day trial version. The fact that I was not looking at too many things in my face made me feel better. Just like my iPhone (in my case) I could concentrate on a task that I was currently doing rather than find something in that massive ribbon on the top with icons that I found it difficult to understand at times.

We have different attitude, behavior and priorities while using our gadgets. They come with us on the bus, walking down the street or watching TV. Users expect more contextual based actions then an al-a-carte menu system where they are expected to find an action they want to take. Because mobile devices have changed user expectations, it is extremely important to consider them is user studies since their influence is much larger then we can think of.

Wireframing is not Prototyping

With the rise of UX role in the industry, wireframes have played increasingly leading role in the modern world. Its a simple way to validating user interface and layout and are cheaper and faster to produce than a final visual comp.

Wait a minute did I say cheaper and faster? Really?

Over the last couple of years, the number of wire framing tools that have come up in the market is alarming. From a world of Visio, Omnigraffle and to a certain extent Rational Rose in the past, today you have options for any platform. You could do a google search today and you would be surprised to see the number of tools available today and with links to tell you the top 5, 10 & 20 tools for wireframing. With more options today, we run into more complexity.

Paper & Pencil have been probably the cheaper and faster way of evaluating the interface and layout, however I see very little of those today.

Today, there are applications that allow you to create wireframes that look hand drawn.

The last few years the wireframes have looked more and more different and detailed. Images, actual look and feel, every single mouse movement or tap or gesture and what happens next, etc. That definitely throws cheaper and faster out of the window.

Lets look at a classic definition of wireframe

wire·frame
ˈwīrˌfrām/

noun: wireframe; plural noun: wireframes; noun: wire-frame; plural noun: wire-frames
1. 
a skeletal three-dimensional model in which only lines and vertices are represented.

Exactly, a wireframe is a three-dimensional illustration of a page’s interface that specifically focuses on space allocation, prioritization of content, functionalities available, and intended behaviors. For these reasons, wireframes typically DO NOT include any styling, color, or graphics.

However, with the next generation of devices already here, and with users employing more touch-based system, a few changes should be made in the way we do classic wireframes. In spite of these progressions, though, creating wireframes should remain cheaper & faster.

Today we have taken to wire-framing to actually deliver a full blown spec. Do you think a developer is going to make sense of a step-by-step wireframe that captures every single gesture? (most of the developers/QA know what happens when you click a drop down or on tapping the button it turns blue from grey). These are not High-Fidelity wireframes (if that is your arguing point) but a spoon feeding document which runs into 100’s of pages.

A High-fidelity wireframe has increased level of details which should simply point out details like dimensions, behavior and/or actions related to any interactive element.

There have been some rules that designers need to consider while creating wireframe. Lets see how or what we can change so that wireframes are better suited for today’s interaction. It’s important to keep in mind to use wireframes as guides and not deliver a spec.

Some old school rules:

Do not use colors — If you want feedback on your functionality, do not use colors, however always use a different color to show highlights or selection. Stakeholders always get confused when they don’t see a selection color; in other cases avoid colors.

Do not use images — Images distract from the task at hand. Do not use an image unless it is part of your navigation and you cannot show any interaction with that image. Else just show a placeholder.

Typography — Use one font, but vary their sizes to differentiate.

Keep them short and to the point — Don’t spoon feed. Use annotations and try to capture your use cases.

One of the things old school wireframes always said was to show what things go where irrespective of shape and size since it was important to give the users an idea as to what content goes where. Later on it became important to do wireframes that would show details of content before the web page fold.

Today, users use more touch devices compared to desktops/laptops. Screen sizes have become more important and users don’t have to worry about the web page fold anymore. Every single pixel has become important. Thus making it even more important to deliver pixel perfect wirefames.

One of the biggest down side of delivering wireframes as a spec is that at times you do not know from where to start. Should I do the dashboard now? Oh but making an appointment is more important then looking at summary. At the end, you are scampering to get things done, with 100’s of wireframe slides and still no starting point.

A good technique that I have relied in the past is to write down high-level scenarios for your personas that you create. Once you have these written down; prioritize them — there you have your starting point.

Now take a single scenario and try to come up with uses cases, or business rules whatever you are comfortable with. Start with a single positive use case, document all the business rules. Do your wireframe, as minimal as possible and annotate them if required.

Create a summary page at the end of this, and capture all the negative use cases and error cases. Typically you wouldn’t need wireframes to show these unless you need a different interaction all together.

Keep your wireframes short, simple and annotated and remember to keep them cheaper and faster.