Go to article

Published: May 6, 2013

“By comparing the user research methods that never really caught on to those that have become popular, we can determine what it is that makes user research techniques valuable to UX professionals.”

Remember GOMS analyses? Pluralistic walkthroughs? Have you written any scenarios lately? When was the last time you performed a cognitive walkthrough? Maybe in grad school? Never?

Of all the user research methods that have emerged over last few decades, why did some catch on and become renowned, while others are still waiting for their big break or have declined from their previous glory to has-been status? By comparing the user research methods that never really caught on to those that have become popular, we can determine what it is that makes user research techniques valuable to UX professionals.

Hey, I Still Use That!

In this column, for each of the user research methods that I’ll describe asstill waiting for their big break or as has-beens, I’m sure there are some fans who will protest, “Hey, I still use that!” While I’m sure there arepeople who still do use these methods, I’d bet that most would admit that they aren’t in wide use.

Still Waiting for Their Big Break

Several research methods from the early days of usability—in the 1980s and 1990s—never really caught on with UX professionals:

GOMS analysis
cognitive walkthrough
pluralistic walkthrough

Perhaps their rigidity, impracticality, or the lack of user involvement doomed these methods to a mostly academic existence in textbooks and courses on human-computer interaction.

GOMS Analysis

“GOMS analysis is a method for determining how long it would take a user to perform a task with a user interface by breaking the task down into each user interaction and the time it would take to perform it.”

GOMS (Goals, Operators, Methods, and Selection) analysis and similar variants such as KLM-GOMS (Keystroke-Level Model GOMS) could be the poster child for techniques that, while they have been widely discussed in academia, never caught on with UX professionals. Like many others who have a degree in human-computer interaction, I first learned about and last used GOMS in graduate school.

GOMS analysis is a method for determining how long it would take a user to perform a task with a user interface by breaking the task down into each user interaction and the time it would take to perform it. The UXPA’s Usability Body of Knowledge [1] describes GOMS as, “A family of predictive models of human performance that can be used to improve the efficiency of human-machine interaction by identifying and eliminating unnecessary user actions…. Empirically derived values for basic operators like keystrokes, button presses, double-clicks, and pointer movement time are used to estimate task times…. To obtain the predicted time for a task, you add the times for individual operators.” This complicated description alone might tell us why UX professionals never adopted this method as part of their practice.

Why Didn’t GOMS Catch On?

There are several reasons why GOMS never really caught on:

GOMS’s narrow focus is on efficiency and the time it takes to perform actions, which is a very limited view of user experience. Most UX and usability professionals are interested in the entire user experience rather than individual, minute interactions that allow them to predict efficiency.
It’s not very practical to use GOMS to measure every one of the many interactions in a typical user interface.
GOMS doesn’t involve users. In the early days of usability, when it wasn’t as easy to get access to users, some saw techniques like GOMS as good alternatives. As usability testing and user research became more widely accepted and practiced, UX professionals came to perceive activities that didn’t involve users as less valuable—and even frowned upon them.
GOMS never gained much exposure outside human factors and human-computer interaction programs. Few students who learned GOMS found it practical to use this approach in the real world after graduating.

Cognitive Walkthrough

“One of several types of usability inspection methods from the early years of usability, the cognitive walkthrough provides a way to evaluate user interfaces without involving users.”

Over my 12-year career in user research, the only time I performed a cognitive walkthrough was for a human-computer interaction class in graduate school. One of several types of usability inspection methods from the early years of usability, the cognitive walkthrough provides a way to evaluate user interfaces without involving users.

In a cognitive walkthrough, one or more usability professionals walk through a user interface, attempting to perform selected tasks, while asking four questions from the user’s perspective:

Will the user try to achieve the right effect?
Will the user notice that the correct action is available?
Will the user associate the correct action with the effect he wants to achieve?
If the user performs the correct action, will he see that he is making progress toward a task’s solution?

Why Didn’t the Cognitive Walkthrough Catch On?

Doing cognitive walkthroughs failed to catch on for a couple of reasons:

They don’t involve users.
This is a rather complex, inflexible method in comparison to simpler inspection methods such as heuristic evaluation and expert review. If you’re going to use a technique that doesn’t involve users, why not use something that’s simpler and more flexible?

Pluralistic Walkthrough

“The pluralistic walkthrough is an even more obscure usability inspection method than the cognitive walkthrough.”

The pluralistic walkthrough is an even more obscure usability inspection method than the cognitive walkthrough. You’ve probably never even heard of it unless you have a human-computer interaction background. I first read about this technique in grad school in Nielsen and Mack’s book,Usability Inspection Methods. I’ve never used this method; nor have I ever come across anyone else who has used it.

The UXPA’s Usability Body of Knowledge [2] describes the pluralistic walkthrough as, “A systematic group evaluation of a design in which usability practitioners serving as walkthrough administrators guide users through tasks simulated on hard-copy panels and facilitate feedback about those tasks while developers and other members of the product team address concerns or questions about the interface.”

Why Didn’t the Pluralistic Walkthrough Catch On?

UX professionals didn’t adopt this technique because:

Other than hard-core usability professionals, few have even heard of this technique.
Although this technique does involve users, the product team does notobserve them in a realistic situation. Walking users through paper-based designs with a group of developers and other product team members in a conference room hardly presents a realistic approximation of how people would normally use a user interface.
A pluralistic walkthrough involves much more work than a cognitive walkthrough or heuristic evaluation. If you’re going to go through the trouble of recruiting and scheduling all participants, why not just conduct usability testing or field studies?

Has-beens

These user research methods achieved some fame and were in wide use in the early days of user research:

focus groups and surveys
heuristic evaluation

However, other more useful and valid techniques have since overshadowed them. Yes, UX professionals still employ these methods from time to time, but their reputation and popularity have greatly diminished.

Focus Groups and Surveys

“Although UX professionals still occasionally conduct focus groups and surveys, they widely regard them as less useful and valid techniques for user research.”

Although UX professionals still occasionally conduct focus groups and surveys, they widely regard them as less useful and valid techniques for user research. Some clients still insist on conducting focus groups, and there are times when surveys are useful for gathering supplemental information, but very few people depend on either approach as their primary research method.

How Did Focus Groups and Surveys Become Has-beens?

The popularity of focus groups and surveys has diminished because:

Focus groups and surveys focus on what people say, not what they do. User research methods that observe user behavior in context are much more valuable. Focus groups and surveys take users out of their natural context and, at best, ask them to describe their behavior, which isn’t very accurate. These methods are best for gathering opinions and perceptions, which are helpful, but not enough to understand the user experience.
Because of these limitations, UX professionals now frown upon focus groups and surveys as user research techniques—or at most, use them only to supplement other methods.

Heuristic Evaluation

“Very few use the traditional method of heuristic evaluation that Jakob Nielsen originally described….”

“Hey, wait a minute,” you might be saying, “A lot of people still use heuristic evaluation!” Well, yes, they do something similar to heuristic evaluation, but very few use the traditional method of heuristic evaluation that Jakob Nielsen originally described in his book Usability Inspection Methods. These days, most people instead employ an evolution of the heuristic evaluation that is actually an expert review. (I’ll discuss the expert review later, as a technique that did catch on.)

Jakob Nielsen described his method of heuristic evaluation as: “Having a small set of evaluators examine the interface and judge its compliance with recognized usability principles (the ‘heuristics’).” [3] The key to this method is that it requires several usability professionals to do their own individual evaluations of a user interface, using the list of heuristics that Nielsen described in his “10 Usability Heuristics for User Interface Design,” then get together to compare and combine their findings.

How Did Heuristic Evaluation Become a Has-Been?

Why did the traditional method of heuristic evaluation fade away, getting replaced by what is essentially an expert review?

The traditional heuristic evaluation requires from three to five usability professionals. In the real world, it’s an extremely rare luxury for a UX team to have so many usability professionals who can work on the same project. In reality, a single usability professional incorrectlyconducts most so-called heuristic evaluations—making them, in actuality, expert reviews.
Although Nielsen intended his heuristics to be general usability principles that could apply to many types of problems, some UX professionals began to feel that they were too abstract and didn’t cover everything that it was important to evaluate. So they either developed new heuristics, guidelines, or checklists—or simply began to rely on their own expertise.
As a method that doesn’t involve users, some UX professionals frown upon heuristic evaluation and question its value.
The effectiveness of heuristic evaluation has become somewhat discredited. Studies have shown that heuristic evaluations often miss the majority of usability problems—especially when they don’t follow Nielsen’s traditional method.

Need a Better Agent

User research methods in this category still find some work, but are in danger of becoming has-beens, so need stronger advocates for their use. In other words, they need a better agent. One example of such a user research method is the use of scenarios.

Scenarios

“Scenarios are stories about how people would perform tasks in a planned system. … They act as a bridge between research and design, describing what a design needs to provide.”

Few researchers or UX designers today take the time to create scenarios. It would be very controversial to call scenarios a has-been because they still have some strong proponents. But I think most would agree that scenarios need a better agent to encourage their wider use.

Scenarios are stories about how people would perform tasks in a planned system. Since they are based on user-research findings, they act as a bridge between research and design, describing what a design needs to provide.

Scenarios came into use as a user-centered replacement for use cases, which provide stark descriptions of user actions and system reactions. Use cases do not consider user characteristics or needs at all. Scenarios provide a way to use the information that we gain about users through research to explore possible design solutions.

However, over time, personas and other more visual depictions of user experiences such as workflow diagrams, storyboards, and customer journey maps have supplanted scenarios. Some people still write scenarios for their personas or have incorporated scenario-like elements into their personas, so scenarios haven’t completely disappeared. But they aren’t in as wide use as they once were.

Why Haven’t Scenarios Become More Ubiquitous?

There are several reasons that scenarios are no longer in wide use:

Scenarios sit at an awkward juncture between user research and UX design. When different people are doing user research and design, scenarios can seem like the responsibility of the other guy, so tend to get neglected in favor of other techniques.
For many UX designers, writing scenarios in a Word document doesn’t feel like design. Designers are often more comfortable beginning with something more visual—such as workflow diagrams, storyboards, screen-flow diagrams, sitemaps, or just sketches of design ideas.
For UX designers who don’t have strong writing skills, it can be very time consuming to write scenarios for the many different tasks that a system involves. The time it takes to explore scenarios might not always feel productive, so it can be difficult to justify the effort it takes to clients and project team members.
Prose-based descriptions of user experiences can be difficult for some UX designers and project team members to digest and leverage. Often, visual depictions of user experiences such as storyboards, flow diagrams, and customer journey maps are easier for clients and project team members to quickly understand.

The A-List

“In an expert review, a UX professional examines a user interface to identify usability, design, accessibility, and other user experience problems.”

The research methods that have became famous and are still in wide use are the A-list stars of the user-research world.

expert reviews
usability testing
remote, moderated usability testing
eyetracking
field studies
card sorting
personas

Examining what has made these methods so popular and successful can help us to determine what elements make a user research method useful.

Expert Reviews

In an expert review, a UX professional examines a user interface to identify usability, design, accessibility, and other user experience problems. Unlike the other more traditional usability inspection methods—pluralistic walkthrough, cognitive walkthrough, and heuristic evaluation—that I discussed earlier, an expert review doesn’t follow a strict process. It relies on the reviewer’s own expert knowledge of usability and design best practices.

Why Did Expert Reviews Become Famous?

Doing expert reviews has become popular because:

They are easier, quicker, and less expensive than traditional usability inspection methods.
A single UX professional can perform an expert review.
While we acknowledge that expert reviews are not as comprehensive or accurate as research techniques that involve users, the intention is usually that an expert review will be a preliminary activity that occurs before using a research method that does involve users.

Usability Testing

“There is no better way to get realistic feedback on a design than to observe representative users trying to perform tasks using it.”

Usability testing is one of the oldest user research methods. Although it has evolved over the years, it is still one of the most common and frequently used research techniques.

Why Did Usability Testing Become Famous?

Usability testing has long been famous because:

There is no better way to get realistic feedback on a design than to observe representative users trying to perform tasks using it.
This is a very practical technique for both identifying specific problems and revealing their solutions. Unlike the findings from up-front, generative user research, the findings and solutions from usability testing are very specific and actionable.
Research findings that derive from actual user behavior and comments are more difficult to argue against than the opinion of a researcher.
Unlike other research methods, it’s easy for clients and project team members to observe usability test sessions. Usability testing clearly and powerfully demonstrates its value to observers, often converting them into enthusiastic supporters.

Remote, Moderated Usability Testing

“Until tools for remote, moderated usability testing became available, cost and time constraints limited the numberof people who could participate in testing, as well as who could participate.”

A major limitation of usability testing has been the need either to bring participants to you or to travel to them. Until tools for remote, moderated usability testing became available, cost and time constraints limited thenumber of people who could participate in testing, as well as who could participate. By making a phone call and using Web conferencing software, a usability professional can conduct usability tests with participants anywhere in the world.

Why Did Remote, Moderated Usability Testing Become Famous?

Remote, moderated usability testing has become popular because:

It opens up usability testing to a wider group of participants.
More people are willing to participate because doing so requires a smaller time commitment and no travel.
Participants are often more comfortable because they remain in their natural environment—avoiding the potentially intimidating environment of the usability lab, where observers and a facilitator may be in the same room.
This approach to testing eliminates the need for a usability lab, avoiding scheduling problems and saving organizations money.
Remote testing allows people to observe test sessions from anywhere in the world.

Eyetracking

“UX professionals and their clients have discovered the value in seeing where participants look when performing tasks.”

Eyetracking technology was in use in academic and medical research for many years before it caught on as a supplement to usability testing in industry. As the technology has improved and become more accessible, UX professionals and their clients have discovered the value in seeing where participants look when performing tasks.

Why Did Eyetracking Become Famous?

Eyetracking is now popular because:

Modern eyetrackers have become less invasive and almost imperceptible to participants, allowing us to use eyetrackers in usability testing without negatively affecting the results.
The prices of eyetrackers have come down to a more reasonable level.
The improved usability and reliability of eyetracking hardware and software have made it easier for researchers to run eyetracking studies.
Eyetracking is very impressive to clients, so helps sell usability testing.

Field Studies

“Field studies … involve going into the field to talk with and observe users performing their tasks in their natural context.”

Field studies is the general term for research techniques that involve going into the field to talk with and observe users performing their tasks in their natural context. There are various field-study techniques, including contextual inquiry, ethnography, observation, and interviews. Each technique involves different approaches to interviewing and observing behavior.

Why Have Field Studies Become Famous?

User researchers have enthusiastically embraced field studies as a key part of their practice because:

Field studies are the holy grail for user researchers, who deem nothing else to be more effective than going out into the field to interview and observe people performing tasks in their natural environment. For this reason, researchers are highly motivated when it comes to convincing clients to include field studies in their projects.
With a little explanation of the concept of field research, going to visit users to observe and talk with them about what they currently do makes obvious sense to most clients and project team members.
When researchers do field studies well, they provide extremely valuable findings that clearly demonstrate their value.

Card Sorting

“Researchers adopted card sorting from the field of psychology as a way to gather user input into the organization and labeling of site content.”

In the early days of Web design, user researchers adopted card sorting from the field of psychology as a way to gather user input into the organization and labeling of site content. Having users sort index cards into related piles, then label each pile provides insights into both categorization and labeling. Researchers soon developed variations on the original open-sorting method, including sorting by groups of participants, closed card sorting, reverse card sorting, and modified-Delphi card sorting. More recently, online card-sorting tools have allowed researchers to reach many more participants, anywhere in the world.

Why Did Card Sorting Become Famous?

Card sorting has become popular because:

It’s an easy way to get user input and validation very early on in a project without requiring a lot of initial preparation.
This is a simple technique that’s easy for participants and clients to understand.
It’s relatively inexpensive and easily demonstrates its value.
Online card sorting lets you reach many participants, in diverse locations, in a very short period of time.

Personas

“Personas are descriptions of fictional, archetypal users that include their goals, attitudes, characteristics, and expected use of a system.”

Personas are descriptions of fictional, archetypal users that include their goals, attitudes, characteristics, and expected use of a system. Because personas are based on user research findings, they should seem like descriptions of actual people, allowing the project team to keep a vivid picture of each user group in mind throughout a project. Creating a design for Beth and Joe, the most important personas, helps ensure that the design will both meet the needs of the primary user groups and suffice for other user groups.

Why Did Personas Become Famous?

UX professionals created user profiles—general descriptions of user groups rather than specific people—long before personas. So, why did personas catch on and almost replace user profiles?

Personas provide more specific, vivid portraits of users than the lists of general characteristics in user profiles.
Personas are easier for people to relate to, and they tend to stick around longer because team members refer to them throughout a project.
Personas are fun to create.
Personas are more interesting and quicker and easier to read than most user research deliverables.

Rising Stars

A new breed of up-and-coming user research techniques is gaining popularity. These are the rising stars:

remote, unmoderated usability testing
tree testing
diary studies, or longitudinal studies

Some of these may become the A-listers of the future, while others will likely remain less well known, but widely respected niche techniques for certain types of projects.

Remote, Unmoderated Usability Testing

“Remote, unmoderated usability testing tools … allow researchers to set up a series of tasks that participants from anywhere in the world can perform online….”

Over the last few years, there’s been an explosion in the number of remote, unmoderated usability testing tools. These tools allow researchers to set up a series of tasks that participants from anywhere in the world can perform online, during unmoderated usability testing sessions.

Why Is Remote, Unmoderated Usability Testing Becoming Popular?

Unmoderated studies save time and can include many participants, from anywhere in the world, allowing researchers to conduct more studies and include more participants than traditional, in-person usability testing allows.
The latest tools are better designed and easier to use for both researchers and participants.
While early remote, unmoderated usability testing tools typically required expensive, yearly licensing agreements, the prices of more recent tools have come down to reasonable, per-test rates.

Tree Testing

“Tree testing is a method of evaluating the organization of a Web site’s content before creating a site design.”

Similar to reverse card sorting, tree testing is a method of evaluating the organization of a Web site’s content before creating a site design. Participants use a simple, clickable site hierarchy to perform tasks that involve finding items in a hierarchy. They click links until they find the section where they think they would find a particular item.

Why Is Tree Testing Becoming Popular?

Tree testing is gaining in popularity because:

Like card sorting, you can do tree testing to evaluate a site’s organization and labeling very early during a project—before investing much effort in a design and when it’s easy to make changes.
With online tree testing, it’s possible to reach many participants anywhere in the world, which increases the validity of the results, but adds little extra expense.
It’s quick and inexpensive.
It provides specific and actionable findings, making it easy for clients to see the value of the technique.

Diary Studies, or Longitudinal Studies

“As user research has moved away from focusing exclusively on usability, techniques that can gather a longer-term and more realistic picture of an entire user experience have become more valuable.”

A limitation of most user research methods is that they capture only the specific moment in time when the researcher is with a participant. However, in reality, a user experience usually takes place over a longer period of time. Diary studies overcome this limitation by gathering information about a user experience over an extended period of time. Participants write about their experiences with a particular product or service in a diary. They may also take photos or perform other activities to record their experiences. Once the study period is over, the researcher analyzes the findings.

Why Are Diary Studies Becoming Popular?

The popularity of diary studies is increasing for the following reasons:

As user research has moved away from focusing exclusively on usability, techniques that can gather a longer-term and more realistic picture of an entire user experience have become more valuable.
Without the artificial environment of the lab or the presence of a researcher, diary studies can capture a more realistic picture of a user experience.

What Is It That Makes a User Research Method Useful?

“Through this examination of both user research methods that have achieved widespread adoption and others that have failed to catch on, we can assess what characteristics make any method of research useful and, therefore, popular.”

Through this examination of both user research methods that have achieved widespread adoption and others that have failed to catch on, we can assess what characteristics make any method of research useful and, therefore, popular. I have concluded that a user research method becomes widely used if it meets the following criteria:

It’s practical. A method of research is practical if it fits into a realistic project timeframe, is not too expensive, and doesn’t require too many researchers or UX designers to conduct it.
It’s flexible. A method is flexible if it doesn’t require strict adherence to a set of rules, a researcher can modify it somewhat to fit the needs of a specific project, and a researcher can perform the research quickly, if necessary.
It’s easy to understand. A research method is easy to understand if researchers can readily understand how to perform the technique.
It’s easy to remember. A method of research is easy to remember if a researcher knows what to do each time he uses the method without having to continually refer back to a book to recall what steps to perform.
It’s easy to perform. A research method that is easy to perform may require some learning—and it may take time to become an expert—but it isn’t too difficult to get started using the technique.
It’s easy to explain to others. A research method that you can easily explain to clients and project team members is one that they can readily understand.
It easily demonstrates its value. A research method that inherently conveys its value doesn’t require too much effort to convince clients and project team members that it is useful.
It involves users. This sounds kind of obvious, but research that involves users, performing realistic tasks, is more valuable than methods that rely only on expert opinions.
It produces valuable results. A research method that produces useful, actionable, and valuable results provides results in a format that clients and project team members can easily understand.

Conclusion

“The future will undoubtedly bring many completely new methods, as well as some variations on existing techniques.”

Developing and trying out new techniques is what keeps user research interesting. The future will undoubtedly bring many completely new methods, as well as some variations on existing techniques. What will remain the same are the foundations for understanding user experience: visiting people in their natural contexts, observing people performing their usual tasks, and talking with people about what they are doing. Nothing will ever replace user research methods that deliver such great value.

References

[1] User Experience Professionals’ Association. “GOMS.”

Usability Body of Knowledge, 2012. Retrieved on August 22, 2012.

[2] User Experience Professionals’ Association. “Pluralistic Usability Walkthrough.”