Why You Shouldn’t Trust Positive Online Reviews—Or Negative Ones, For That Matter

SMART SPENDING
Go to article
By BRAD TUTTLE | @bradrtuttle | August 28, 2012 |

Researchers and online review sites alike are trying to root out efforts to manipulate the system. By outing dubious, planted, solicited, and otherwise inauthentic reviews, consumers will have more reason to trust the reviews that remain, the thinking goes. But judging on how prevalent the manipulation seems to be, it’s arguable that online user ratings and reviews are less trustworthy than ever.

Over the weekend, a New York Times story about online reviews focused on the experiences of Todd Rutherford, who has this to say about the world of online reviews:

“When there are 20 positive and one negative, I’m going to go with the negative,” he said. “I’m jaded.”
Rutherford should know: He spent years writing and commissioning others to write thousands of what he calls ““artificially embellished reviews” of books. Often, the book “reviews” required little more than a 10- or 15-minute literal “review” of the book in order to produce 300 words of glowing, somewhat relevant and customized praise. Before Google suspended his ad account due to his pay-for-positive-reviews business, Rutherford was pulling in as much as $28,000 per month. One author admitted to paying $20,000 over the years to various services so that they would review his books.

The expose on fake book reviews is the latest example of why you shouldn’t trust online reviews. Or at the very least, you should review the reviews with an especially skeptical eye.

Another example comes from a recent post at Automotive News, which relates that Google is suddenly and without warning deleting dozens, sometimes hundreds of reviews of car dealerships at Google+ Local. Google isn’t explaining why, exactly, reviews are disappearing. But it did release a boilerplate statement noting that “these measures help everyone by ensuring that the reviews appearing on Google+ Local are authentic, relevant, and useful,” giving the indication that there was reason to believe the reviews were fake or somehow inauthentic.

The dealerships maintained that the deleted ones “are legitimate, obtained by requests over several months to sales and service customers,” according to Automotive News. But some would question the authenticity of reviews that come as the result of the pleas of the businesses being reviewed. Review giant Yelp actively discourages review requests by businesses because it taints the objectivity of reviews. What business, after all, would ask a displeased customer to offer his honest thoughts, feedback, and gripes in an online review? “Self-selected reviews create intrinsic bias in the business listing,” Yelp explains, “a bias that savvy consumers (read: yelpers) can smell from a mile away.”

Yelp tries to filter such reviews out of its ratings, and apparently so does Google+ Local. As a result, one auto dealership saw its number of reviews drop from 300 to just 11, and the reviews that were left were mostly negative. Before the mass deletion, the dealership had a rating of 29 out of 30 points; afterward, the rating was in the single digits.

Which rating is more accurate—the one before or after Google got rid of hundreds of reviews? It’s hard to say. Google and Yelp admit that their systems sometimes wind up flagging perfectly legitimate reviews. But that’s better than leaving up reviews that are very likely bogus, they claim.

Bear in mind that it’s not just positive reviews that are sometimes fake. After examining half a million online hotel reviews, a new study highlighted by the Talking Travel Tech blog comes to the conclusion that small, indie hotels are likely faking reviews. They’re not only generating fake positive reviews for their businesses, but they’re also producing fake negative reviews for nearby chain hotel competitors.

How do the researchers reach these conclusions? They compared reviews at TripAdvisor with those from Expedia. The big difference with reviews from these sites is that at Expedia, it’s necessary to have stayed in a hotel—and booked it through Expedia for proof—to write a review of it. TripAdvisor reviews are under no such obligation, and as a result, anyone can pen a review of a hotel, even someone who has never been a guest of the property.

Previously, other researchers have concluded that it’s sites like TripAdvisor, where no proof of customerhood is required for reviews, have the highest prevalence of fake reviews. The latest study backs up this theory by focusing on small mom-and-pop hotels—properties that stand the most to benefit from the reviews system because, unlike chain hotels, they have no national reputation to lean on.

The study shows that small hotels are “about 10% more likely to receive five-star reviews on TripAdvisor than they are on Expedia, relative to hotels owned by large corporations.” What’s more, researchers say that when a better-known hotel brand is located near an indie hotel, the chain hotel is more likely than one of its isolated sister hotels to get one- and two-star ratings at TripAdvisor. Overall, it’s estimated that a chain hotel down the block from an indie hotel will get hit with five more fake negative reviews than the same brand hotel that isn’t directly competing with a small, independently run hotel.

The researchers found that there was “relatively more positive manipulation than negative manipulation, even though the order of magnitude of the two is similar.” The big takeaway is that the system is being manipulated with fake positive and fake negative reviews—and that’s all negative for consumers who are using them to try and make smart choices.

Brad Tuttle is a reporter at TIME. Find him on Twitter at @bradrtuttle. You can also continue the discussion on TIME’s Facebook page and on Twitter at @TIME.

Making Sense of the Cross Channel Experience

Go to article

19th July 2012 by Jon Fisher | User Experience | View 4 Comments

A few weeks ago I had the pleasure of attending the Architectures of Meaning workshop at Pervasive 2012. The workshop discussed how information architecture is adapting and changing in light of the explosion of multiple channels (e.g. desktop, laptop, mobile, retail store, kiosk etc) that now deliver information and that information transcends.

Attendees at the workshop were invited to submit papers discussing the topic of the formation of “Architectures of Meaning”. That is systems, services or eco-systems that operate across both digital and physical spaces and are designed to support a user in the formation of a conceptual model of how the overall experience works (NB: This is my definition of the term “Architectures of Meaning” derived from our research perspective on cross channel information architecture. The discussion around this term is still ongoing).

Having just returned from discussing our work at Pervasive 2012, I felt it was an opportune moment to present some introductory thoughts about our framework for ”sense making in cross channel design”. In particular, I would like to demonstrate a potential method for visualising the information space from which understanding can be supported in a system. I should like to caveat that it is recognised that some of the concepts covered in this blog post are very deep, much deeper than could be covered in 1500 words. My intention is to spark the reader’s interest and highlight some of the directions that we have been exploring.

Cross-Channel services

As I’m sure many readers are familiar with, more and more in recent years we are being asked to consider wider design problems than simply websites. The long predicted age of ubiquitous computing is fast approaching (or is already here). The layers of information are continuously building up and a method of meaningfully representing them is now more important than ever.

We are typically being asked to work within expanded problem spaces that deliver a service across several channels. For example, it’s no longer enough to discuss concepts such as “retail vs. e-commerce” rather we are looking at how retail and e-commerce support one another in the greater good of the goal of “sell more stuff”.

I think the following diagram from Peter Morville’s 2011 UserFocus keynote sums up the concept of cross channel delivery nicely.

Figure 1: Diagram showing delivery of a service through individual channels vs. delivery of a service across several channels

Information architecture as the “Glue”

In their excellent book on designing Pervasive Information Architectures, Andrea Resmini and Luca Rosata explored the concept that Information Architecture can act as the “sense making glue” holding together the user’s conceptual model of a particular service, system or ecosystem.

I will not go into details over this concept here but suffice to say it has been proposed that a pervasive informational layer (“ethereal” to quote my colleagues in the AoM workshop) exists over a system that contributes to the formation of a conceptual model of understanding about the wider system user experience (interested readers should also read about Information Foraging Theory by Pirolli).

Transitions are critical

From an extensive research review and practical pilot study of our hypothesis, we explored the way in which this pervasive informational layer can be formed by users and most importantly how it can degrade across an eco-system of products and places. We argued that there are a number of ways that a user’s meaningful understanding of a cross channel service can be “chipped away” through poor information architecture. Of particular pertinence to the success of a system was the various ways we transition between channels i.e. when we move from a digital channel and into a physical retail store.

One of the major problems early on when designing cross channel services is it is very difficult to model and predict the different ways with which a user can transition between various channels. I acknowledge that we already utilise a number of user experience methods such as customer journey mapping and service blueprints. My problem with tools such as Customer Journey Maps and Service Blueprints in this instance (I think they are great tools most of the time) is that they are typically conveying a user journey or interaction over time i.e. in a linear format.

When evaluating cross channel services and the construction of a pervasive information architecture the user is often free to drift back and forth between channels as many times as they wish and (depending on the service) over an undefined time period. Dan Willis has explored some excellent and interesting ways to display user journeys in a cross channel context by drawing Intent Paths.

Figure 2: Dan Willis (@uxcrank) work on showing user intent paths

A new visualisation

Having identified the critical nature of designing for channel transitions (and how they can degrade meaning to a user), we began to discuss various ways that we could visualise the informational needs that users require in a cross channel context. It is argued that there will be a core set of informational needs or requirements that a user must carry between channels that help them form a conceptual understanding of the wider service. We need a way, very early in a design process, to identify, visualise and map these informational needs so that we can begin to construct an idea of how information will flow across our wider product eco-systems. The diagram below demonstrates a first iteration of one of these visualisations. For simplicity, let’s call it a Cross Channel IA Diagram (in our office we have affectionately termed it a “meaning map”).

Figure 3: Cross Channel IA Diagram

Let me describe a number of key points about the diagram:

• Channel Corners: Each corner of the hexagon contains a dedicated channel that is part of the wider service being designed. The nice thing about this approach is you can use any number sided shape to represent your cross channel information architecture. I have had discussions with people that some services or systems will have a huge number of channels but being realistic how often are we asked to design systems with a number greater than say 8?
• Channel Membranes: Around each channel is what I have called the channel membrane. What this essentially shows is the entrance and exit points to that channel. Again I appreciate that there are potentially hundreds of entrance and exit points from a channel but what I am really talking about is the common entrance and exit points based around our business objectives and user research.
• The Between Channel Information Space: This is the workhorse section of the diagram! This is the area where we can map the informational needs / requirements that are carried between channels by users and that effectively contribute to the formation of a common understanding of the system. What the centre of the diagram represents is the space between channels, the transitional space where (at the moment) many cross channel services typically break down. It is the content of the centre of the diagram that I propose can be used to start to create the pervasive informational layer that contributes to the development of “architecture of meaning”.

So I hope you can see is that what we potentially have is a very practical visualisation tool that can be used early in a cross channel design / service delivery project to help us construct a strong informational layer that pervades all of our design work. I propose that these Cross-Channel IA diagrams (in conjunction with a solid understanding of how meaning can degrade with channel transitions) can add value by:

• Helping us understand better our users cross channel goals without worrying about the order and manor with which a user interacts with a channel;
• Acting in a channel agnostic manor whereby we can see the overarching needs that need to be supported irrespective of the channel design silo that we may find ourselves working in;
• Forming the beginnings of our UX / Product / Service Strategy by helping us bound the problem space for ourselves and our stakeholders;
• Can help us formalise and document the “ethereal” information space that will inform our users’ conceptual model of the systems that we are designing, ensuring that this is supported and retained across the lifecycle of a design project.

Next Steps

I acknowledge that I touch upon many large topics of discussion in this blog post and some have not been covered at all (for example responsive web design or UX strategy). These are bigger topics for elsewhere.

The intention of this article has been to highlight some of our thoughts on creating pervasive information architectures. Our goal has always been to try to develop a practical framework that can be used early on in a design process to help us visualise the information space that we are so commonly being asked to design for nowadays.

Our work is not complete and we are continuing to refine and explore our work further. For example, a major element to the next stage of our research is to further explore and identify the psychological aspects that can influence a user’s transition between channels and what can lead to channel abandonment.

A more detailed explanation is due to follow from the proceedings of the AoM workshop and most probably in the Journal of Information Architecture (or a larger e-publication). I shall be presenting further on this subject at UX Bristol on the 20th July 2012.

I hope you will join me in the discussion.

Does Paid Search Cannibalize Organic Search Clicks?

Go to article

By David Felfoldi on August 21st, 2012

If your paid search campaign includes keywords also found well-ranking organically, you might be cannibalizing your organic search traffic. Cannibalization occurs when a user clicks on your sponsored ad instead of your organic one, causing you to pay for a click rather than receive a “free” one from organic search.

Google states that their research confirms that cannibalization doesn’t occur.

In what we call “Search Ads Pause Studies,” our group of researchers observed organic click volume in the absence of search ads. Then they built a statistical model to predict click volume for given levels of ad spend. This model generates estimates for the incremental clicks attributable to search ads or, in other words, the percentage of paid clicks that are not made up for by organic clicks when search ads are paused. On average, the incremental ad clicks (IAC) percentage across verticals is 89%. This means that a full 89% of the traffic generated by search ads is not replaced by organic clicks when ads are paused. This number was consistently high across verticals.

But our research without own clients indicates differently. And we can present it in a simple graph.

Cannibalization of Organic Search Clicks

Take a closer look at the graph above. In it, we reviewed the past 7 months search performance of a single, specific phrase to find the level of keyword cannibalization. The graph is pretty clear — when paid search was turned on, organic search traffic from that specific keyword decreased.

The most telling example is February 2012 — while it was the highest paid search month for the specific phrase, it was the lowest for organic. For the months before and after, when there is a decreased paid search spend, the organic performs better. Another telling example is July 2012 – paid search was at a 6 month low, but organic began an uptick.

What extent did paid search cannibalize organic?

The next question was — to what extent did paid search cannibalize organic? For this, we looked at organic traffic as a percentage of traffic by each month.

Here we see the inverse correlation in closer detail — when the percentage of traffic from paid search for the specific keyword is up — the months 5, 6, 8, 11 and 12 — the percentage of traffic from organic search for the specific keyphrase is down.

In other words, when you turn on the paid search ads, traffic from organic search drops. When you turn it off, traffic from organic search goes back up.

Should I Avoid Cannibalizing Organic Search with Paid Search Ads?

Our conclusion, however, was that while paid search may attract some of the clicks that may have originally gone towards organic, the months with the most keyword-sourced traffic are when paid and organic worked in conjunction.

Historically, the organic search keyphrase drove about 4-5% of total traffic. During periods of paid ads, that keyphrase now drove 10-11% of total traffic. This is what Google meant by “a full 89% of the traffic generated by search ads is not replaced by organic clicks when ads are paused” — while paid search does cannibalize some organic traffic, it also attracts and drives clicks that might never had occurred without it. It’s more of a symbiotic relationship, than predatory one.

Your results, however, may differ. If you have an ecommerce site, check out this great sources on paid search cannibalization. By looking at your average sales price, you can calculate and decide if your level of organic search cannibalization from paid search is worth it.

Millennials: Confident. Connected. Open to Change

Go to article

This is part of a Pew Research Center series of reports exploring the behaviors, values and opinions of the teens and twenty-somethings that make up the Millennial Generation

Executive Summary

Generations, like people, have personalities, and Millennials — the American teens and twenty-somethings who are making the passage into adulthood at the start of a new millennium — have begun to forge theirs: confident, self-expressive, liberal, upbeat and open to change.

They are more ethnically and racially diverse than older adults. They’re less religious, less likely to have served in the military, and are on track to become the most educated generation in American history.

Their entry into careers and first jobs has been badly set back by the Great Recession, but they are more upbeat than their elders about their own economic futures as well as about the overall state of the nation.(See chapter 4 in the full report)

They embrace multiple modes of self-expression. Three-quarters have created a profile on a social networking site. One-in-five have posted a video of themselves online. Nearly four-in-ten have a tattoo (and for most who do, one is not enough: about half of those with tattoos have two to five and 18% have six or more). Nearly one-in-four have a piercing in some place other than an earlobe — about six times the share of older adults who’ve done this. But their look-at-me tendencies are not without limits. Most Millennials have placed privacy boundaries on their social media profiles. And 70% say their tattoos are hidden beneath clothing. (See chapters 4 and 7 in the full report)

Despite struggling (and often failing) to find jobs in the teeth of a recession, about nine-in-ten either say that they currently have enough money or that they will eventually meet their long-term financial goals. But at the moment, fully 37% of 18- to 29-year-olds are unemployed or out of the workforce, the highest share among this age group in more than three decades. Research shows that young people who graduate from college in a bad economy typically suffer long-term consequences — with effects on their careers and earnings that linger as long as 15 years.¹(See chapter 5 in the full report)

Whether as a by-product of protective parents, the age of terrorism or a media culture that focuses on dangers, they cast a wary eye on human nature. Two-thirds say “you can’t be too careful” when dealing with people. Yet they are less skeptical than their elders of government. More so than other generations, they believe government should do more to solve problems. (See chapter 8 in the full report).

They are the least overtly religious American generation in modern times. One-in-four are unaffiliated with any religion, far more than the share of older adults when they were ages 18 to 29. Yet not belonging does not necessarily mean not believing. Millennials pray about as often as their elders did in their own youth. (See chapter 9 in the full report)

Only about six-in-ten were raised by both parents — a smaller share than was the case with older generations. In weighing their own life priorities, Millennials (like older adults) place parenthood and marriage far above career and financial success. But they aren’t rushing to the altar. Just one-in-five Millennials (21%) are married now, half the share of their parents’ generation at the same stage of life. About a third (34%) are parents, according to the Pew Research survey. We estimate that, in 2006, more than a third of 18 to 29 year old women who gave birth were unmarried. This is a far higher share than was the case in earlier generations.² (See chapters 2 and 3 in the full report)

Millennials are on course to become the most educated generation in American history, a trend driven largely by the demands of a modern knowledge-based economy, but most likely accelerated in recent years by the millions of 20-somethings enrolling in graduate schools, colleges or community colleges in part because they can’t find a job. Among 18 to 24 year olds a record share — 39.6% — was enrolled in college as of 2008, according to census data. (See chapter 5 in the full report)

They get along well with their parents. Looking back at their teenage years, Millennials report having had fewer spats with mom or dad than older adults say they had with their own parents when they were growing up. And now, hard times have kept a significant share of adult Millennials and their parents under the same roof. About one-in-eight older Millennials (ages 22 and older) say they’ve “boomeranged” back to a parent’s home because of the recession. (See chapters 3 and 5 in the full report)

They respect their elders. A majority say that the older generation is superior to the younger generation when it comes to moral values and work ethic. Also, more than six-in-ten say that families have a responsibility to have an elderly parent come live with them if that parent wants to. By contrast, fewer than four-in-ten adults ages 60 and older agree that this is a family responsibility.

Despite coming of age at a time when the United States has been waging two wars, relatively few Millennials-just 2% of males-are military veterans. At a comparable stage of their life cycle, 6% of Gen Xer men, 13% of Baby Boomer men and 24% of Silent men were veterans. (See chapter 2 in the full report)

Politically, Millennials were among Barack Obama’s strongest supporters in 2008, backing him for president by more than a two-to-one ratio (66% to 32%) while older adults were giving just 50% of their votes to the Democratic nominee. This was the largest disparity between younger and older voters recorded in four decades of modern election day exit polling. Moreover, after decades of low voter participation by the young, the turnout gap in 2008 between voters under and over the age of 30 was the smallest it had been since 18- to 20-year-olds were given the right to vote in 1972. (See chapter 8 in thefull report)

But the political enthusiasms of Millennials have since cooled -for Obama and his message of change, for the Democratic Party and, quite possibly, for politics itself. About half of Millennials say the president has failed to change the way Washington works, which had been the central promise of his candidacy. Of those who say this, three-in-ten blame Obama himself, while more than half blame his political opponents and special interests.

To be sure, Millennials remain the most likely of any generation to self-identify as liberals; they are less supportive than their elders of an assertive national security policy and more supportive of a progressive domestic social agenda. They are still more likely than any other age group to identify as Democrats. Yet by early 2010, their support for Obama and the Democrats had reced
ed, as evidenced both by survey data and by their low level of participation in recent off-year and special elections. (See chapter 8 in the full report)

Our Research Methods

This Pew Research Center report profiles the roughly 50 million Millennials who currently span the ages of 18 to 29. It’s likely that when future analysts are in a position to take a fuller measure of this new generation, they will conclude that millions of additional younger teens (and perhaps even pre-teens) should be grouped together with their older brothers and sisters. But for the purposes of this report, unless we indicate otherwise, we focus on Millennials who are at least 18 years old.

We examine their demographics; their political and social values; their lifestyles and life priorities; their digital technology and social media habits; and their economic and educational aspirations. We also compare and contrast Millennials with the nation’s three other living generations-Gen Xers (ages 30 to 45), Baby Boomers (ages 46 to 64) and Silents (ages 65 and older). Whenever the trend data permit, we compare the four generations as they all are now-and also as older generations were at the ages that adult Millennials are now.³

Most of the findings in this report are based on a new survey of a national cross-section of 2,020 adults (including an oversample of Millennials), conducted by landline and cellular telephone from Jan. 14 to 27, 2010; this survey has a margin of error of plus or minus 3.0 percentage points for the full sample and larger percentages for various subgroups (for more details, see page 110 in the full report). The report also draws on more than two decades of Pew Research Center surveys, supplemented by our analysis of Census Bureau data and other relevant studies.

What’s in a Name?

Generational names are the handiwork of popular culture. Some are drawn from a historic event; others from rapid social or demographic change; others from a big turn in the calendar.

The Millennial generation falls into the third category. The label refers those born after 1980 – the first generation to come of age in the new millennium.

Generation X covers people born from 1965 through 1980. The label long ago overtook the first name affixed to this generation: the Baby Bust. Xers are often depicted as savvy, entrepreneurial loners.

The Baby Boomer label is drawn from the great spike in fertility that began in 1946, right after the end of World War II, and ended almost as abruptly in 1964, around the time the birth control pill went on the market. It’s a classic example of a demography-driven name.

The Silent generation describes adults born from 1928 through 1945. Children of the Great Depression and World War II, their “Silent” label refers to their conformist and civic instincts. It also makes for a nice contrast with the noisy ways of the anti-establishment Boomers.

The Greatest Generation (those born before 1928) “saved the world” when it was young, in the memorable phrase of Ronald Reagan. It’s the generation that fought and won World War II.

Generational names are works in progress. The zeitgeist changes, and labels that once seemed spot- on fall out of fashion. It’s not clear if the Millennial tag will endure, although a calendar change that comes along only once in a thousand years seems like a pretty secure anchor.

Some Caveats

A few notes of caution are in order. Generational analysis has a long and distinguished place in social science, and we cast our lot with those scholars who believe it is not only possible, but often highly illuminating, to search for the unique and distinctive characteristics of any given age group of Americans. But we also know this is not an exact science.

We acknowledge, for example, that there is an element of false precision in setting hard chronological boundaries between the generations. Can we say with certainty that a typical 30-year-old adult is a Gen Xer while a typical 29-year-old adult is a Millennial? Of course not.
Nevertheless, we must draw lines in order to carry out the statistical analyses that form the core of our research methodology. And our boundaries-while admittedly too crisp-are not arbitrary. They are based on our own research findings and those of other scholars.

We are mindful that there are as many differences in attitudes, values, behaviors and lifestyles within a generation as there are between generations. But we believe this reality does not diminish the value of generational analysis; it merely adds to its richness and complexity. Throughout this report, we will not only explore how Millennials differ from other generations, we will also look at how they differ among themselves.

The Millennial Identity

Most Millennials (61%) in our January, 2010 survey say their generation has a unique and distinctive identity. That doesn’t make them unusual, however. Roughly two-thirds of Silents, nearly six-in-ten Boomers and about half of Xers feel the same way about their generation.

But Millennials have a distinctive reason for feeling distinctive. In response to an open-ended follow-up question, 24% say it’s because of their use of technology. Gen Xers also cite technology as their generation’s biggest source of distinctiveness, but far fewer-just 12%-say this. Boomers’ feelings of distinctiveness coalesce mainly around work ethic, which 17% cite as their most prominent identity badge. For Silents, it’s the shared experience of the Depression and World War II, which 14% cite as the biggest reason their generation stands apart. (See chapter 3 in the full report)

Millennials’ technological exceptionalism is chronicled throughout the survey. It’s not just their gadgets — it’s the way they’ve fused their social lives into them. For example, three-quarters of Millennials have created a profile on a social networking site, compared with half of Xers, 30% of Boomers and 6% of Silents. There are big generation gaps, as well, in using wireless technology, playing video games and posting self-created videos online. Millennials are also more likely than older adults to say technology makes life easier and brings family and friends closer together (though the generation gaps on these questions are relatively narrow). (See chapter 4 in the full report)

Work Ethic, Moral Values, Race Relations

Of the four generations, Millennials are the only one that doesn’t cite “work ethic” as one of their principal claims to distinctiveness. A nationwide Pew Research Center survey taken in 2009 may help explain why. This one focused on differences between young and old rather than between specific age groups. Nonetheless, its findings are instructive.

Nearly six-in-ten respondents cited work ethic as one of the big sources of differences between young and old. Asked who has the better work ethic, about three-fourths of respondents said that older people do. By similar margins, survey respondents also found older adults have the upper hand when it comes to moral values and their respect for others.

It might be tempting to dismiss these findings as a typical older adult gripe about “kids today.” But when it comes to each of these traits — work ethic, moral values, respect for others — young adults agree that older adults have the better of it. In short, Millennials may be a self-confident generation, but they display little appetite for claims of moral superiority.

That 2009 survey also found that the public — young and old alike — thinks the younger generation is more racially tolerant than their elders. More than two decades of Pew Research surveys confirm that assessment. In their views about interracial dating, for example, Millennials are the most open to change of any generation, followed closely by Gen Xers, then Boomers, then Silents.

Likewise, Millennials are more receptive to immigrants than are their elders. Nearly six-in-ten (58%) say immigrants strengthen the country, according to a 2009 Pew Research survey; just 43% of adults ages 30 and older agree.

The same pattern holds on a range of attitudes about nontraditional family arrangements, from mothers of young children working outside the home, to adults living together without being married, to more people of different races marrying each other. Millennials are more accepting than older generations of these more modern family arrangements, followed closely by Gen Xers. To be sure, acceptance does not in all cases translate into outright approval. But it does mean Millennials disapprove less. (See chapter 6 in the full report)

A Gentler Generation Gap

A 1969 Gallup survey, taken near the height of the social and
political upheavals of that turbulent decade, found that 74% of the public believed there was a “generation gap” in American society. Surprisingly, when that same question was asked in a Pew Research Center survey last year — in an era marked by hard economic times but little if any overt age-based social tension — the share of the public saying there was a generation gap had risen slightly to 79%.

But as the 2009 results also make clear, this modern generation gap is a much more benign affair than the one that cast a shadow over the 1960s. The public says this one is mostly about the different ways that old and young use technology — and relatively few people see that gap as a source of conflict. Indeed, only about a quarter of the respondents in the 2009 survey said they see big conflicts between young and old in America. Many more see conflicts between immigrants and the native born, between rich and poor, and between black and whites.

There is one generation gap that has widened notably in recent years. It has to do with satisfaction over the state of the nation. In recent decades the young have always tended to be a bit more upbeat than their elders on this key measure, but the gap is wider now than it has been in at least twenty years. Some 41% of Millennials say they are satisfied with the way things are going in the country, compared with just 26% of those ages 30 and older. Whatever toll a recession, a housing crisis, a financial meltdown and a pair of wars may have taken on the national psyche in the past few years, it appears to have hit the old harder than the young. (See chapter 3 in the full report)

But this speaks to a difference in outlook and attitude; it’s not a source of conflict or tension. As they make their way into adulthood, Millennials have already distinguished themselves as a generation that gets along well with others, especially their elders. For a nation whose population is rapidly going gray, that could prove to be a most welcome character trait.

Read the full report for more details.

1. Lisa B. Kahn. “The Long-Term Labor Market Consequences of Graduating from College in a Bad Economy,” Yale School of Management, Aug. 13, 2009 (forthcoming in Labour Economics). ↩
2. This Pew Research estimate is drawn from our analysis of government data for women ages 18 to 29 who gave birth in 2006, the most recent year for which such data is available. Martin, Joyce A., Brady E. Hamilton, Paul D. Sutton, Stephanie J. Ventura, Fay Menacker, Sharon Kirmeyer, and TJ Mathews. Births: Final Data for 2006. National Vital Statistics Reports; vol 57 no 7. Hyattsville, Maryland: National Center for Health Statistics. 2009. ↩
3. We do not have enough respondents ages 83 and older in our 2010 survey to permit an analysis of the Greatest Generation, which is usually defined as encompassing adults born before 1928. Throughout much of this report, we have grouped these older respondents in with the Silent generation. However, Chapter 8 on politics and Chapter 9 on religion each draw on long-term trend data from other sources, permitting us in some instances in those chapters to present findings about the Greatest Generation. ↩

Developing Effective Scenarios

Go to article

By Sean Carmichael

August 5th, 2011

Podcast: Play in new window | Download (Duration: 34:53 — 18.5MB)

[ Transcript Available ]

Combining storytelling with research data can help you craft realistic scenarios to guide your design process. Getting to know the specific needs of your users allows you to address any potential problems they may have. As a consultant, Kim Goodwin uses her experience and expertise in working with teams to develop effective scenarios. In this podcast, Kim discusses the role that scenarios play in the design process with Jared Spool.

Kim will be part of the User Interface 16 Conference November 7-9 in Boston, MA. She will present a full-day workshop on using scenarios, helping to focus and prioritize. Find out more details about the conference at UIConf.com.

Here’s an excerpt from the podcast.

“…The thing is, a story has a character, somebody with skills, and goals, and feelings, and other real human characteristics. User stories employ roles which are real abstractions of users. A story has a plot, it has a beginning and an end. It has something that starts off the action and some logical conclusion that’s a satisfying ending to the story.

Whereas, user stories are just like… Use cases or scenarios, they’re sequential thinking, which is good. That’s a helpful way to approach interaction because it always happens over time. But they’re often fragments of complete stories…”

Tune in to the podcast to hear Kim cover these points:

• How do you help people start to use personas and scenarios?
• Is it good to use abstract scenarios in conjunction with more specific ones?
• Where do the scenarios come from?
• How important of a skill is storytelling when developing scenarios?
• Have you experimented with breaking teams in to smaller groups in the scenario creation process?
• What are some techniques you’ve used with teams to collect data?
• How do scenarios compare to user stories?

Do you design with scenarios? Share your thoughts in our comments section.

Recorded: July, 2011
[ Subscribe to our podcast via  ←This link will launch the iTunes application.]
[ Subscribe with other podcast applications.]

Full Transcript.

Jared Spool: Hello, everyone. Welcome to another episode of the SpoolCast. I am very happy today because I have a chance to talk to my good old friend, Kim Goodwin, who has presented at more of our events, I think, than anybody else on the planet. And she wrote a fabulous book called “Designing for the Digital Age.” She’s going to be presenting at our upcoming User Interface conference, User Interface 16. Kim, can you believe it’s the 16th year?

Kim Goodwin: I know, that’s crazy. I’ve lost count of how many of these things I’ve done.

Jared: I know, I know. It’s really insane. So at User Interface 16, you’re going to be talking about really pulling the value out of using scenarios. Once you have your personas and your scenarios, how do you really get the value out of them? I’m really excited about this workshop because I think scenarios are the sort of untapped treasure that we as designers don’t really use enough.

I try and use them all the time, and I still don’t think I use them enough, because they really do have an effect on every aspect of the design, from the first moment you start to talk about what the hell you’re going to build, all the way through the deployment process. Do you find that to be true, that there’s use everywhere?

Kim: Pretty much. I mean, certainly, if you have a hammer, everything looks like a nail. But I think that scenarios are my go-to design tool. I think if I had to give up every other tool in my kit, I could still get an awful lot done with scenarios.

Jared: Yeah. Now, you and I were talking, and you referenced this great quote of Plato’s, which was, “Those who tell the stories rule society,” which is just brilliant. And of course, the society that many of us live in is our work society. I think that the personas that we create and the scenarios that we use to describe those persona situations are really a great way to get control of a product that possibly has gone astray in other ways. How do you help people start to use personas to bring their design conversations around to what they need to build really great stuff?

Kim: I think there are two ways, fundamentally, that personas and the scenarios they star in really help to drive that conversation. One is that scenarios help you develop and articulate your vision. I think that Plato says, “The people who tell the stories rule society” because… Think about all the great leaders that you’ve heard of in our time. Right?

Martin Luther King didn’t stand up there and say, “Hey, you guys aren’t including us, and civil rights are not what they should be.” He said, instead, “I have a dream.” And he painted a vision, and I think that’s what made him an inspiring leader.

And so, we want to do the same thing in design. We want to inspire people and say, not, “Here’s how all the usability is bad, and you should include design more, whine, whine, whine,” but instead, say, “Look, guys, here’s where we want to go. Wouldn’t that be great if we could build that and really get everyone behind it?”

So I think that’s one part of the story. The other is that, when people start to talk about doing this or that with the product or this or that feature, you can challenge other people on the team to use scenarios so that everybody’s using a shared reference point and speaking the same language. And it gives everybody a shared framework for making good decisions.

Jared: There’s a lot in common with both the points, this idea of a shared reference point and this idea of a vision. Right? Because what I’ve been helping teams with is giving them this idea that one way to construct your vision is really to just understand what the stories are for today’s experiences. What is it like to be a user today? And then, if you got rid of all the nasty, ugly, frustrating bits, what would that experience turn into? How would it change?

And so, there you’ve got, basically, one scenario, one in the dark, dismal, frustrating world of today’s experience, and one in the golden, blue-sky, flower-and-field version of the future where everything just goes smoothly and perfectly.

And so, at one level you have an abstract scenario that would cover both things fairly accurately. And then, you can use those scenarios to dive deep and say, “OK, what changes when it’s frustrating and people have to compensate for that frustration? And what is different when we’ve gotten rid of that frustration, and now they’re getting delighted by everything, and how does that change what they do?”

That distinction, being able to shift from that higher-level, abstract notion down to more specifics, depending on whether you’re in today’s world or a future world, is that something you find yourself doing?

Kim: Actually, not at all.

Jared: Oh, OK.

[laughter]

Kim: I find that part of what works about scenarios is that they’re not abstract.

Jared: OK.

Kim: They’re fairly concrete. They give people handles to grab onto. And I think there’s a lot of value in what you’re talking about, with painting a picture of today, right, and helping everybody understand where processes are broken and that sort of thing.

I tend to do that with research findings and personas and say, “OK, everybody, can we get agreement on the state of the world today? Here’s what we saw in research. Here’s what the goals and pain points and characteristics of our personas are. Do we agree that this is an accurate depiction of reality?”

Because, if I can get agreement on the current state of the world, that gives me a more solid foundation for building the future. Right? Because the last thing I want to have happen when I’m describing a future scenario or showing design is for people to start arguing about what the data is.

So we try to get commitment at that initial level without using scenarios, per se, but say, “Look, here are the pain points we saw in the process. Here are some of the really fabulous quotes that we got out of the research where people are banging their heads against walls. Maybe here are some snippets of video or some photographs that demonstrate those points,” but really using the data to make that visceral, and then in the scenarios, which are focused on the future, saying, “OK, now let’s imagine the better world.”

Jared: Right. I guess I’m curious, where do the scenarios then get pulled out? Because what I was thinking was that, as you were creating the personas, you were sort of documenting the scenarios of today. But it doesn’t sound like that’s what you’re doing.

Kim: Not in great detail. I mean, I think that you could spend a lot of time documenting that. I find that, when it comes to really getting the most value out of your research and design time, I find that’s not the most productive use of time, documenting everything that’s wrong. Because that doesn’t necessarily help us make a lot of progress.

Unless it’s something where you’re really focused on a detailed redesign and you have the FDA looking at everything for approval and so forth, that level of documentation doesn’t serve most teams well, I don’t find. Maybe there’s a white-board sketch of the current process and a highlight of some of the points where it’s broken in general, something like that.

Where you pull the scenarios from is out of your understanding of the users. Right? And it’s not a scientific process. Instead, you’re relying on your human intelligence and your understanding of the goals and the characteristics of those people and saying, “OK, in a magic world, what would this be like?”

And this is an uncomfortable step for some people who want design to be a science. But think about it. When you’re planning a party for a friend or buying a gift for someone, you’re imagining what would be awesome for somebody that you really care about and know very well. You can do that with a fair degree of reliability. Right? And that’s exactly what we’re trying to do for our users is to say, “We know what makes them tick. We know what’s going to make them very happy. So let’s imagine what that might be like.”

Now, that doesn’t mean that we’re not still going to do usability testing and things like that to make sure that we’re right, because we’re not perfect, but we’re going to trust our gut as a foundation to generate. Scenarios are fundamentally generative tools.

Jared: Right. And if you’ve gone out and done your research… When I go out with teams, and we go out and we actually see their customers and we’re in their homes or their offices, and we’re looking at them actually do the things they do every day, it’s pretty easy to see what gets them all excited and giddy and what is dragging them down. When you have that full context to put everything in, it becomes really easy to pull out the scenarios, I’ve found.

Kim: Yeah. When you know your users really well, scenarios are… Certainly, there are a few technique tricks to master. But once you’ve got those down, scenarios are the most natural tool in the world. Because it’s just storytelling, fundamentally. It’s storytelling based on your understanding of the data, but it’s something we’ve all been doing our whole lives, right?

We’ve all been telling stories since we were, what, two, maybe younger than that even? Almost as soon as kids learn to talk, they start to tell stories. And so, we’re not bogged down in drawing UML diagrams and these other things that are kind of alien to how we think and communicate. Instead, we’re using a tool that’s very deeply familiar to us.

Jared: Yeah. I was with a friend who has a two-and-a-half-year-old, and she was telling us this story. It was so rich in detail about everything that went on in the story and all the different pieces of it. We were laughing hysterically because of the details that were coming out. She had imagined this whole world and had no trouble producing that. It’s amazing that we sort of beat that out of ourselves, huh?

Kim: And I think that your use of the word “imagination” is one of the keys because… In our day-to-day work, how often do we get to just imagine what’s great, right?

And because storytelling is a tool that we use early in our lives… This is my personal theory, anyway, not proven scientifically. But I think what we’re doing is… OK, this is going to sound all touchy-feely, but we’re drawing on our childhood selves, in a way. We’re drawing on a time when we were more free to imagine, because we’re using this tool that we probably don’t use that much once we become adults.

Jared: It’s interesting. We’ve been talking a lot lately amongst ourselves about sketching and how we all sketch when we’re little, and then many of us get told that we’re not very good at it so we stop doing it.

Storytelling is sort of the same way. And I’m wondering how much of this idea of creating great experiences really is something that we’re more in-tuned with when we’re younger, because we’re all about play, we’re all about imagining a space that we’re not in that’s a better space for everybody, and then the harsh realities of the world sort of talk us out of that.

Kim: Yeah. Certainly, we’re still applying our adult skills. Right? I mean, we’re applying our knowledge as designers when we tell our scenarios. We’re drawing on design principles and synthesizing that as we go to imagine these things.

So it’s not pure imagination. There’s absolutely craft in there. There’s knowledge and expertise in there. It’s a frame of mind that is intentionally a little bit naive and optimistic to begin with.

This is one of the things that makes people a little crazy when they first try to use scenarios. Because, if you’re in an organization where people are accustomed to jumping to the constraints and the details very quickly, your first pass at scenarios is going to be very high level and optimistic. Let’s ignore the constraints for a little while to imagine what would be awesome to do.

We know that we’re going to throw some of it away. That has to be OK because, unless you’re free to imagine what’s desirable, you’re just going to focus on what’s possible and you’re never going to get to that really amazing solution that you actually could do more easily than you think.

Jared: I think that there is definitely something to this, because a lot of teams really get stuck trying to live inside their constraints. And of course, the really hard, wicked, gnarly problems are ones that are so rich with constraints that all they can do is, with any possible solution, list the 20 reasons why that solution couldn’t possibly work. They get really stuck in that world of what they can’t do and why they’ve tried it before, and why it hasn’t worked.

So now, how is it that you use scenarios to get beyond that space? What do you do with teams when they’re stuck there?

Kim: I think that who’s involved in scenarios is… It’s partly a temperament thing. Most of the time, when I’ve done scenarios, it’s been specifically with other designers who are willing to take on that optimistic mindset because it’s natural to them, or with members of a product team who… Maybe struggle with it a little bit, but they’re willing to try that on.

Occasionally, with people who aren’t used to it, you have to pull them up short in a discussion and say, “Look. We’re thinking optimistically here. Let’s write that down on the whiteboard as a concern we’ll deal with later. Let’s focus on what we love to do.”

I think the key is just that everybody knows the rules at the beginning of the discussion so that you’ve got a small set of people. I think it’s really hard to do scenarios with six or eight people in the room. I think it’s much easier with two or three people in the room because you just don’t have quite as much… You don’t have as much politics and compromise and things like that.

I think that generating is much easier in a small group. Evaluating makes a lot of sense in a bigger group. But I think that, if you can get together and say, “Look. These two or three people have knowledge of the data, they’ve got direct experience with the users,” let’s imagine what’s possible.

Jared: When you have six to eight people, have you ever experimented with breaking them into smaller groups for this scenario creation process and then coming together to share what they’ve got?

Kim: I haven’t really done that because I think that… Competition may work. It may not.

Jared: You could give them separate things to do scenarios on.

Kim: Yeah. I think you can. What I find challenging is it’s hard to actually have six or eight people all of whom have direct experience with the data. And so, then you get people coming up with scenarios based on what they think, rather than what the data really is. It’s a lot harder to get that sort of a group focused not on their own assumptions.

Jared: So it is a key element that you’ve really got to have some real solid data behind the scenarios you’re producing.

Kim: Ideally. Have I done scenarios in cases where we all have shared assumptions and not real data? Sure. They’re perfectly useful tools, even when you don’t have data. But they work a whole lot better when you’ve got some data to back them up.

Jared: What are some techniques that you’ve used with teams to get data?

Kim: To get data?

Jared: Yeah, to get the data for scenarios. Are there certain types of things that are better at pulling data that you’re going to use for scenarios than other things?

Kim: My go-to technique for this kind of stuff is interviews, observation, using techniques that are borrowed from ethnography. I won’t say that it is ethnography because it’s not, strictly speaking.

Going out into user’s context where they’re using competitive products or services, or where they’re using somewhat related tools, if you’re inventing something brand new. Observing what they do, asking them why they do it that way. Seeing what drives them crazy. Those kinds of things… I think that direct, in context, observation is irreplaceable for this kind of design.

Jared: I’ve found the same thing, too. The more direct exposure that the people who are involved in doing the design and creating the scenarios have, the easier a lot of that work comes. Because, like you said, you’re not having a lot of conversation about what you imagine. That makes perfect sense to me.

I’ve been working, lately, with a bunch of teams that seem to be in this mode where they’re focusing on the march of the endless feature list.

Kim: Oh, yes.

Jared: And they don’t seem to have any real direction in their design. They’ve just got this list of things that they’re doing, primarily because their competitors are doing it, or some big customer says they’ll buy 1,000 versions of the thing if they implement this one feature.

Everything that they do tends to add a piece of complexity on top of the complexity that they’ve already added the last 20,000 times they’ve done this. This sort of giant hairball of design emerges.

Is that a situation you’ve found yourself in? And if so, how do you work with the team to get out of that mode?

Kim: Oh, absolutely. I think almost every designer encounters that if they work in more than one place.

I think that the bolting things on feature by feature is a really common approach, I think partly because it’s a way to scope development. Developers think, “Well, this is a capability that’s going to take this kind of effort to code versus that kind of effort to code.” And in that respect, it makes sense.

On the other hand, users don’t really experience feature lists. They experience workflows. They experience start to finish processes. And so, if you really focus on feature lists, I think you get the Winchester Mystery House of products. Right?

There’s a place in San Jose where she was listening to voices and she just starting bolting on stairways that go nowhere and doors that open three stories up into nothingness, windows into closets and things like that.

Jared: I just worked with a product manager that was listening to voices. I think that’s exactly how they were designing.

Kim: It’s true. The thing is that those voices are probably executives or customers. Right?

Jared: I could only hope.

Kim: Yeah, that’s true.

And so, I think where personas and the data behind them come in handy, and where scenarios come in handy, is giving people a focal tool. Instead of looking at the feature list, you can say, “Look. Let’s imagine this better experience. Let’s start idealistic. Let’s take Persona A through these three major things that they’re going to do with our product or our service. Let’s take Persona B through these two things that they’re likely to do with our product or service.”

“Now, let’s compare that to what we think our feature list should be. Hey, look! Our stories uncovered two or three things we haven’t anticipated in our feature list that look pretty important. And look, these other things in our feature list… Hmm. Gosh, those didn’t show up in the scenarios at all. What does that tell us?”

So I think that they’re a great tool for product managers to help prioritize requirements. Product managers right now don’t have a lot of great tools for this. They can go out and try to play innovation games with their customers and give people $30 to spend on features and have them prioritize in the abstract.

I think all of that is much less useful than saying, “Look. Here’s the story of the experience we want to build. Here are the pieces that are integral to that story. Here are the pieces that could be left out and you’d still have a coherent story, and we’ll save those for later on.”

And so, it just gives everybody that decision making framework.

Jared: Now, a lot of what you’ve described sounds very familiar to what Indi Young does with her mental model stuff. She’s got these, basically, two aspects of the design and experience. On the top, she lists all the things people are trying to do. And on the bottom, she lists all the functionality within the product that does that.

You can quickly, using her technique once the chart is built… You can quickly see where you’ve got a lot of things people want to do and not much functionality to support it, and vice versa. Is this fundamentally the same thing, or is this a different spin? Or, is this completely different and I’ve just got it wrong?

Kim: I’d say they’re related. I think one is a very storytelling approach. I think that I tend to follow up the initial scenarios with something closer to what Indi is doing, which is, “Let’s break down the story into the stuff people are trying to do and what that means for our requirements.”

Because once you can agree on a story, you say, “OK. If this is the story we want to accomplish, here’s what we have to be able to do. We need to know this. We need to be able to pull this data from there. We need to be able to connect these two systems that aren’t talking right now. We need to completely overhaul how we do customer intake,” whatever.

And so it’s kind of like, “Here’s the story. Here are the implications of the story. Now which of these can we really take on in the near term?”

The story kind of sells the idea first. It says, “Can’t we all agree this is a great goal? Now let’s get pragmatic about how much of that we can take on and when.” And so, it’s similar.

Jared: And how much you’re already doing, and how far off are you for the things you’re not doing.

Kim: Sure.

Jared: Yeah, that makes a lot of sense.

Now, a lot of things about this story stuff is that folks eventually at some point have to translate this, particularly if they’re doing some sort of agile like process. I’ve come to learn that no one actually does agile, they just do things that they claim isn’t quite agile.

Whenever I talk to someone, I say, “So, are you guys doing any sort of agile development?”

They say, “Well, it’s sort of like agile.” Sometimes they’ll even say, “We don’t do agile the way you’re supposed to.” As far as I know, nobody does it the way you’re supposed to.

Somewhere in there there’s these user stories, often on cards, they’re part of the backlog. They’re this thing that, at some point, sort of drives the development effort and the design effort.

But those stories are not these scenarios, right? You have to get from your scenarios to those user stories in some way. Do I have that right?

Kim: Right. I think the great irony of the term “user stories” is that they’re really not stories.

Jared: And they’re not really about users.

Kim: Right. The thing is, a story has a character, somebody with skills, and goals, and feelings, and other real human characteristics. User stories employ roles which are real abstractions of users. A story has a plot, it has a beginning and an end. It has something that starts off the action and some logical conclusion that’s a satisfying ending to the story.

Whereas, user stories are just like… Use cases or scenarios, they’re sequential thinking, which is good. That’s a helpful way to approach interaction because it always happens over time. But they’re often fragments of complete stories.

So it’s very common to see a user story, something like, “User logs in.” That’s not a story. That user story might tell you, “User enters name, user enters password.” Well, who is that user and why are they logging in, and what is it they’re trying to get to after they log in? So that complete story needs to put that action in context.

Now, that’s not to say that you can’t use user stories. I think that scenarios are completely compatible with waterfall, agile, or whatever custom mix of development approaches you use. Because you can turn scenarios into chunked up user stories very easily. But by doing the scenario first, you get the overall context and the overall flow.

You can also turn them into really detailed use cases documented in 90 pages of UML if you want to. If you’re working on legacy systems and there’s tons of complexity and you want to make sure that you capture every detail, sure, great, do that, but use the scenarios first as a generative tool, as a discussion tool, as a persuasion tool. Get agreement on where you’re going and then break it down into these other ways of chunking that information.

Jared: So if you’re working with an agile team… People talk about this sprint zero activity, the stuff you do before everyone goes heads down and starts coding up their first iteration. So having those scenarios come out of that sprint zero and having those discussions with the team coming out of the sprint zero, that could be really valuable.

Kim: Well, not only the scenarios, but also an initial set of storyboards that illustrate the scenarios. In my experience, by the end of sprint zero, you at least want to be there if not starting to get into the detailed design for sprint one so that you’re a sprint ahead. Trying to do detailed design on something while it’s being coded, that’s still a real challenge.

I think the agile goal of bringing design and development closer together in time is great. Because, if you’re designing something that no developer is going to touch for another three months… Yeah, that doesn’t work so well. But it’s great if you can design it a week or two before it starts getting coded so that design isn’t playing catch up.

But yeah, definitely, at the end of sprint zero, having some storyboards that you can say, “Look, here’s what this ultimately is going to look like, here’s how all the pieces fit together so that everybody understands when they’re designing this piece or that piece how it fits into the whole, how it affects the other pieces,” that makes that whole process a lot less chaotic.

Jared: I’m wondering about this because in an agile world the reason that we want to get design and development closer together is because once we get that first iteration done we’re going to learn stuff and things are going to change.

Once our target users get a chance to play with what we’ve built for a prototype in that first iteration, we’re going learn all sorts of interesting stuff that we couldn’t possibly have predicted before we started that iteration. And that’s why we don’t do those big, heavy design documents where we lay out every radio button for everything in its final form.

But do the scenarios change over that period, or do they tend to stay the same? Have you found, in your experience, that you can have radically different scenarios emerge out of those first few iterations or do the scenarios basically stay the same and become sort of the grounding for what we’re learning about what the users need in there? Am I making any sense whatsoever?

Kim: Let me respond to something you said in the lead up to that question which is, we’re going to learn all kinds of things we couldn’t possibly predict. Really? I don’t know about that. I think that, if you don’t do any research… Oh, yeah, you’re going to learn all kinds of things you couldn’t possibly predict. If you do good research up front, you’re going to learn a few things you couldn’t predict.

Jared: OK, OK, I buy that, yeah.

Kim: Right? So if you’re diving straight into design and development without doing research first… Yeah, iteration is going to be really educational. If you do some research first and design thoughtfully… And I’m not talking about six months developing a detailed spec. But you get to a coherent set of storyboards that everybody can look at and say, “Yeah, we see where that’s going. That looks great,” perhaps we even do a little scenario walk through with some users and get a sense that, “Oh, yeah, that looks like it’ll work,” then you can start to get into spec level design later on if you want to.

But what you learn in those iterations really ought to be polishing and not undermining your fundamental assumptions, because you’ve already figured those out.

Jared: So it would be fair to say then that, if you don’t do good research up front, your first iterations are going to be your research, and they’re a very expensive way to do research.

Kim: Very expensive way to do research, yes.

Jared: So that’s a good argument for getting that good research done up front because you’re not trying to research through code.

Kim: Right. Code is a very expensive medium to throw away compared to a couple pieces of paper.

Jared: Exactly.

Kim: Right? I mean, yeah, great engineers can code really fast. I really love the agile ethic that we learn through coding and it accepts that we also learn through designing which is something that waterfall doesn’t quite culturally accept. Waterfall assumes that you can get the design exactly right before you code anything, and that’s not true. Design is an iterative process, it’s a learning process.

Scenarios will evolve over time. And the way that they’ll evolve is usually not to fundamentally change direction unless you’ve done them very badly and didn’t do your research well.

But what happens with scenarios as they iterate is, they get more detailed. Because your first scenarios… You’re not worried about what list box people are touching in those first scenarios. You’re just trying to get the big stuff right. You’re trying to get a sense of roughly what kind of information are they exchanging with the system. Approximately what are they getting back, and how is that all flowing and what’s the end result?

And so, at that stage, who knows what widget is on the screen. First you’re just trying to figure out what screens do we even have. And so, your scenarios will evolve over time, they’ll get gradually more detailed.

I like to document the handful of early scenarios if there’s time. Later scenarios, well, those are just completely throwaway. You’re sketching at the whiteboard and somebody says, “Well, what if our user wants to do this normal variation on what we’re drawing up here?” And the person with the whiteboard marker says, “Well, I think it would look like this and let’s test that out.”

Well, you just used a scenario. It was a totally throwaway scenario, it was very informal. It’s not something you’re going to waste your time documenting, but it was still a useful tool that you just pulled out and used almost invisibly.

Jared: Right, and if you got good research and you can say, “Well when we went and visited Mary, she was doing this,” that becomes a really powerful story. You can say, “OK, how is she going to do that with this new design, or is it something she still needs to do? Are we going to make it so she doesn’t need to do that anymore?”

Kim: Exactly.

Jared: Which will make her really happy because she hates doing that.

Kim: Right.

Jared: I learned that when I saw Mary.

[laughter]

So it really feels to me like this isn’t really something new, this scenario thing, as much as it is formalizing our activity about things we’re probably already doing in a clumsy, informal way.

I think of it like when you look at professional sports athletes, let’s say a baseball player, they start to deconstruct their stance and how they hold the bat, and how they swing, and at what point in the pitch they make their call and swing.

The coaches for these athletes have very specific language for describing each of these types of things and what the different ways to handle it are. In essence, they’ve formalized… And formalizing has a bad connotation for a lot of people. But they’ve really thought through and said, this is how we handle these types of situations in the way we’re going to do this activity.

It sounds like what you’re doing is saying, “Look, you’re probably already telling stories and you’re probably doing it in a sloppy way. Here is the difference between good stories and bad stories in terms of how effective they are in getting the work done. And here’s how to get your stories more on this direction where they’re working for you and not against you.” Is that a fair description of what we’re doing here with these scenarios?

Kim: I think so. I think it’s actually a fair description of any design technique. I think that something that successful design organizations do over time is they look at their more successful designs and at their more successful projects and compare them to their less successful ones and say, “What did we do differently in these, and where are the patterns in that?” And the stuff that works well, that’s the stuff you want to keep doing.

And so, that’s exactly what I’ll be teaching everybody in the workshop about scenarios is stuff that I’ve seen over time and my teams have seen over time that tends to work pretty well.

Jared: Yeah, you and I were going over the outline for the day and it’s really rich. You’re packing a lot of really fun stuff into this. And it really gets down deep into things like how to deal with different channels of delivery platforms in parts of your organization, how the scenarios go across that, and how much you can use the scenarios to do things like derive the important requirements and figure out what your storyboards should be. It’s really going to be a lot of fun.

Kim: Oh, I hope so.

Jared: I’m really excited about it, I can’t wait to do it. So this has been really an excellent chance for us to talk. I want to thank you for spending this time with us today.

Kim: Oh, thanks for having me.

Jared: And I want to thank all of our listeners. You guys can all hear Kim. She’s done some virtual seminars for us in a variety of things, but other podcasts too, we’ll publish a list of those. But I’ve got to tell you, you don’t want to miss this seminar that she’s going to be doing at the User Interface 16 conference. It’s a full day that dives deep into this area of using scenarios and bringing that out and getting the most out of that.

It’s going to be really a lot of fun and people are going to just walk out of there… You’re just going to go, “Oh, my gosh, this was just so awesome. I can use this right away.” You don’t want to miss that. That’s going to be November 7th through 9th in Boston, Massachusetts. You can find out more information at uiconf.com, that’s uiconf.com. Hope to see you there.

Kim, thank you again.

Kim: Thanks, Jared.

Jared: And thanks to everyone for listening. And as always, thank you for encouraging our behavior. Take care.

This entry was posted on Friday, August 5th, 2011 at 9:27 am and is filed under Design, Design Process, Events, Podcasts,Scenarios, SpoolCast, UI16, User Experience, UX . You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.