Getting More Out of Usability Testing: Understanding Product Usage Over Time

By Paula Barraza

Published: September 23, 2013

“By including a long-term testing approach as part of your UX research plan, you will develop a more comprehensive model of user engagement and a better understanding of how and why engagement changes over time.”

Understanding the relationship between a product and its users over a period of time is a key aspect of user research that is often overlooked. By including a long-term testing approach as part of your UX research plan, you will develop a more comprehensive model of user engagement and a better understanding of how and why engagement changes over time.

Just as the real world quickly changes around us, so does our virtual, online world. We are perpetually being introduced to new ways of fitting the Internet into our lives, and we must often learn how to use new technologies and devices. These, in turn, affect and change our online behaviors over time, as well as our perceptions of our online experiences. Who actually engages with a Web site in exactly the same way as they did when they first used it? During our first visit to a Web site, most of us probably try to get oriented and become familiar with the site, with the goal of figuring out how and whether we want to engage with it. Typically, the things that we looked at and interacted with when using the site for the first time are not the same things that we look at after several months of use—or at least we do not need to look at them in detail or with the same level of attention.

Gaining Insights Through Long-Term Research

“Understanding long-term use leads to product innovations and improvements that enable us to retain users.”

As user researchers, we certainly need to understand first-time experiences, but understanding long-term use leads to product innovations and improvements that enable us to retain users. Usability testing in a lab has largely dominated the field of user research. One of the areas in which it has proven to be valuable in is identifying users’ painpoints around key tasks during a first-time experience.  I am suggesting that user researchers expand this knowledge by carrying out studies that tap into users’ perceptions of and their engagement with a product over several weeks—even several months. This long-term research can provide insights that traditional lab usability testing cannot.

Long-term research studies can

• identify inefficiencies in routine interactions
• surface unmet needs
• uncover how new features change usage behavior
• reveal changes in product perception over time
• provide reasons for changes in engagement, whether increased, decreased, or discontinued use
• help map usage patterns to business goals and design objectives
• pinpoint usability issues that surface later in time
• more accurately tell the user’s story

“Think of long-term usability testing as a series of smaller studies that you conduct over several weeks or several months, with the goal of understanding key states of product usage.”

Planning a Long-Term Usability Study

You can think of long-term usability testing as a series of smaller studies that you conduct over several weeks or several months, with the goal of understanding key states of product usage. When you put the results of these studies together, they paint a more complete picture of the user experience. Here are some things to consider when planning a long-term usability study:

• duration—The number of weeks or months over which a long-term usability study should continue varies based on a team’s resources and the amount of time that is appropriate for answering the research questions. The more long-term testing you do, the better you’ll become at determining what is the right amount of time for testing your product. Keep in mind that, after a certain period of time, user behavior when using your product may remain the same, so you’ll have to assess the tradeoffs in considering whether to continue your research past that point. For tablet testing, I have found that two months is often sufficient.
• participants—Another consideration in long-term usability testing is whether to use the same or different participants for the series of studies. The true value of long-term research comes from using the same set of participants for the entire duration of the study. This provides the opportunity to develop comprehensive models of engagement patterns, as well as an understanding of how and why these patterns may change over time.
• method—Approach long-term usability testing as a series of touchpoints with users whose goal is to gather users’ feedback at strategic points in their product usage cycle. Wherever there is an opportunity to do so, be creative in coming up with methods that let you leverage these user touchpoints. However, keep in mind that you don’t want to overburden participants by making them take part in research every week. Make sure that there are some weeks during which they don’t have any research activities in which they must participate. Table 1 shows an example of the user touchpoints for a two-month study that combines field visits, online journals, video calls, and surveys.

Table 1—An example of five touchpoints in a two-month study

Weeks	Activities
1	Home visits
2	Online journals and field-visit analysis
3	Online journals analysis and participant’s week off
4	Online journals
5	Online journal analysis and participant’s week off
6	Survey
7	Survey analysis
8	Video call and start working on the final report

Challenges of Long-Term Research

“Your research would be of little value if there weren’t enough time to analyze all of the data.”

One of the biggest challenges for UX researchers who are conducting long-term usability testing can be the need to devote several weeks or months to the study. This is especially difficult if you are concurrently conducting other research for your company. This is why it is important to carefully plan your long-term research: the dates of the study, the number of touchpoints with users, and the amount of data that you’ll capture. Thinking through these aspects of conducting long-term research will help ensure that you have adequate time to analyze and share your data as you go. Your research would be of little value if there weren’t enough time to analyze all of the data. Plus, it is good for stakeholders to see updates to your data as the study is taking place. 

Long-term research, like other types of UX research, does present some challenges, but if you have the time and the resources to follow the same participants over the course of several weeks or months, you’ll have an opportunity to gather valuable user feedback over time that can significantly increase users’ productivity and the usability of your product.

- See more at: http://www.uxmatters.com/mt/archives/2013/09/getting-more-out-of-usability-testing-understanding-product-usage-over-time.php#sthash.zQnZvr3G.dpuf

via UXmatters http://www.uxmatters.com/mt/archives/2013/09/getting-more-out-of-usability-testing-understanding-product-usage-over-time.php