Friday, March 25, 2011

Testing Content: Early, Often, & Well

Insightful case study by my friend Colleen Jones...

Testing Content


Testing Content

Testing Content
Nobody needs to convince you that it’s important to test your website’s design and interaction with the people who will use it, right? But if that’s all you do, you’re missing out on feedback about the most important part of your site: the content.
Whether the purpose of your site is to convince people to do something, to buy something, or simply to inform, testing only whether they can find information or complete transactions is a missed opportunity: Is the content appropriate for the audience? Can they read and understand what you’ve written?

A tale of two audiences

Consider a health information site with two sets of fact sheets: A simplified version for the lay audience and a technical version for physicians. During testing, a physician participant reading the technical version stopped to say, “Look. I have five minutes in between patients to get the gist of this information. I’m not conducting research on the topic, I just want to learn enough to talk to my patients about it. If I can’t figure it out quickly, I can’t use it.” We’d made some incorrect assumptions about each audience’s needs and we would have missed this important revelation had we not tested the content.

You’re doing it wrong

Have you ever asked a user the following questions about your content?
How did you like that information?
Did you understand what you read?
It’s tempting to ask these questions, but they won’t help you assess whether your content is appropriate for your audience. The “like” question is popular—particularly in market research—but it’s irrelevant in design research because whether you like something has little to do with whether you understand it or will use it. Dan Formosa provides a great explanation about why you should avoid asking people what they like during user research. For what’s wrong with the “understand” question, it helps to know a little bit about how people read.

The reading process

Reading is a product of two simultaneous cognitive elements: decoding and comprehension.
When we first begin to read, we learn that certain symbols stand for concepts. We start by recognizing letters and associating the forms with the sounds they represent. Then we move to recognizing entire words and what they mean. Once we’ve processed those individual words, we can move on to comprehension: Figuring out what the writer meant by stringing those words together. It’s difficult work, particularly if you’re just learning to read or you’re one of the nearly 50% of the population who have low literacy skills.
While it’s tempting to have someone read your text and ask them if they understood it, you shouldn’t rely on a simple “yes” answer. It’s possible to recognize every word (decode), yet misunderstand the intended meaning (comprehend). You’ve probably experienced this yourself: Ever read something only to reach the end and realize you don’t understand what you just read? You recognize every word, but because the writing isn’t clear, or you’re tired, the meaning of the passage escapes you. Remember, too, that if someone misinterpreted what they read, there’s no way to know unless you ask questions to assess their comprehension.
So how do you find out whether your content will work for your users? Let’s look at how to predict whether it will work (without users) and test whether it does work (with users).

Estimate it

Readability formulas measure the elements of writing that can be quantified, such as the length of words and sentences, to predict the skill level required to understand them. They can be a quick, easy, and cheap way to estimate whether a text will be too difficult for the intended audience. The results are easy to understand: many state the approximate U.S. grade level of the text.
You can buy readability software. There are also free online tools from Added BytesJuicy Studio, and Edit Central; and there’s always the Flesch-Kincaid Grade Level formula in Microsoft Word.
But there is a big problem with readability formulas: Most features that make text easy to understand—like content, organization, and layout—can’t be measured mathematically. Using short words and simple sentences doesn’t guarantee that your text will be readable. Nor do readability formulas assess meaning. Not at all. For example, take the following sentence from A List Apart’s About page and plug it into a readability formula. The SMOG Indexestimates that you need a third grade education to understand it:
We get more mail in a day than we could read in a week.
Now, rearrange the words into something nonsensical. The result: still third grade.
In day we mail than a week get more in a could we read.
Readability formulas can help you predict the difficulty level of text and help you argue for funding to test it with users. But don’t rely on them as your only evaluation method. And don’t rewrite just to satisfy a formula. Remember, readability formulas estimate how difficult a piece of writing is. They can’t teach you how to write understandable copy.

Do a moderated usability test

To find out whether people understand your content, have them read it and apply their new knowledge. In other words, do a usability test! Here’s how to create task scenarios where participants interpret and use what they read:
  • Identify the issues that are critical to users and the business.
  • Create tasks that test user knowledge of these issues.
  • Tell participants that they’re not being tested; the content is.
Let’s say you’re testing SEPTA, a mass transit website. It offers several types of monthly passes that vary based on the mode of transportation used and distance traveled: For example, a TransPass lets you ride on the subway, bus or trolley. A TrailPass also lets you ride the train, etc. If you only wanted to test the interface, you might phrase the task like this:
Buy a monthly TrailPass.
But you want to test how well the content explains the difference between each pass so that people can choose the one that’s right for them. So phrase your task like this:
Buy the cheapest pass that suits your needs.
See the difference? The first version doesn’t require participants to consider the content at all. It just tells them what to choose. The second version asks them to use the content to determine which option is the best choice for them. Just make sure to get your participants to articulate what their needs are so you can judge whether they chose the right one.
Ask participants to think aloud while they read the content. You’ll get some good insight on what they find confusing and why. Ideally, you want readers to understand the text after a single reading. If they have to re-read anything, you must clarify the text. Also, ask them to paraphrase some sections; if they don’t get the gist, you’d better rewrite it.
To successfully test content with task scenarios and paraphrasing, you’ve got to know what the correct answer looks like. If you need to, work with a subject matter expert to create an answer key before you conduct the sessions. You can conduct live moderated usability tests either in person or remotely. But, there are also asynchronous methods you can use.

Do an unmoderated usability test

If you need a larger sample size, you’re on a small budget, or you’re squeezed for time, try a remote unmoderated study. Send people to the unmoderated user testing tool of your choice like Loop11 or OpenHallway, give them tasks, and record their feedback. You can even use something like SurveyMonkey and set up your study as a multiple-choice test: It takes more work up front than than open-ended questions because you must define the possible answers beforehand, but it will take less time for you to score.
The key to a successful multiple-choice test is creating strong multiple choice questions.
  • State the question in a positive, not negative, form.
  • Include only one correct or clearly best answer.
  • Come up with two–four incorrect answers (distractors) that would be plausible if you didn’t understand the text.
  • Keep the alternatives mutually exclusive.
  • Avoid giving clues in any of the answers.
  • Avoid “all of the above” and “none of the above” as choices.
  • Avoid using “never,” “always,” and “only.”
You may also want to add an option for “I don’t know” to reduce guessing. This isn't the SATafter all. A lucky guess won’t help you assess your content.
Task scenario:
You want to buy traveler's checks with your credit card. Which percentage rate applies to the purchase?
Possible answers:
  • The Standard APR of 10.99%
  • The Cash Advance APR of 24.24%*
  • The Penalty APR of 29.99%
  • I don’t know
(*This is the correct answer, based on my own credit card company’s cardmember agreement.)
As with moderated testing, make it clear to participants that they’re not being tested, the content is.

Use a Cloze test

Cloze test removes certain words from a sample of your text and asks users to fill in the missing words. Your test participants must rely on the context as well as their prior knowledge of the subject to identify the deleted words. It’s based on the Gestalt theory of closure—where the brain tries to fill in missing pieces—and applies it to written text.
It looks something like this:
If you want to __________ out whether your site __________ understand your content, you __________ test it with them.
It looks a lot like a Mad Lib, doesn’t it? Instead of coming up with a sentence that sounds funny or strange or interesting, participants must guess the exact word the author used. While Cloze tests are uncommon in the user experience field, educators have used them for decades to assess whether a text is appropriate for their students, particularly in English-as-an-additional-language instruction.
Here’s how to do it:
  • Take a sample of text—about 125-250 words or so.
  • Remove every fifth word, replacing it with a blank space.
  • Ask participants to fill in each space with the word they think was removed.
  • Score the answers by counting the number of correct answers and dividing that by the total number of blanks.
A score of 60% or better indicates the text is appropriate for the audience. Participants who score 40-60%, will have some difficulty understanding the original text. It’s not a deal breaker, but it does mean that the audience may need some additional help to understand your content. A score of less than 40% means that the text will frustrate readers and should be rewritten.
It might sound far fetched, but give this method a try before you dismiss it. In a government study on healthcare information readability, an expert panel categorized health articles as either easy or difficult. We ran a Cloze test using those articles with participants—who had low to average literacy skills—and found that the results reflected the expert panel’s findings. The average score for the “easy” version was 60, indicating the article was written at an appropriate level for these readers. The average score for the “difficult” version was 39: too hard for this audience.
Cloze tests are simple to create, administer, and score. They give you a good idea as to whether the content is right for the intended audience. If you use Cloze tests—either on their own or with more traditional usability testing methods—know that it takes a lot of cognitive effort to figure out those missing words. Aim for at least 25 blanks to get good feedback on your text; more than 50 can be very tiring.

When to test

Test your content at any point in your site development process. As long as you have content to test, you can test it. Need to convince your boss to budget for content testing? Run it through a readability formula. Got content but no wireframes or visual design? Run a Cloze test to evaluate content appropriateness. Is understanding the content key to a task or workflow? Display it in context during usability testing.

What to test

You can’t test every sentence on your site, nor do you need to. Focus on tasks that are critical to your users and your business. For example, does your help desk get calls about things the site should communicate? Test the content to find out if and where the site falls short.

So get to it

While usability testing watches what users do, not what they say they do, content testing determines what users understand, not what they say they understand.
Whatever your budget, timeline, and access to users, there’s a method to test whether your content is appropriate for the people reading it. So test! And then, either rest assured that your content works, or get cracking on that rewrite. 
Translations:
Italian
Russian

    Tuesday, March 8, 2011

    A Checklist for Content Work



    Link

    We are delighted to present an excerpt of The Elements of Content Strategy by Erin Kissane (A Book Apart, 2011), published today. —Ed.
    In content strategy, there is no playbook of generic strategies you can pick from to assemble a plan for your client or project. Instead, our discipline rests on a series of core principles about what makes content effective—what makes it work, what makes it good. Content may need to have other qualities to work within a particular project, but this list is limited to qualities shared across all sorts of content.
    If this looks like theory, don’t be fooled. It’s really entirely practical: if we consciously refer to principles like these as we go about our work as info-nerds of various kinds, we’ll have an easier time making good, useful content—and explaining our priorities when we’re called to do so.

    Good content is appropriate

    Publish content that is right for the user and for the business
    There’s really only one central principle of good content: it should be appropriate for your business, for your users, and for its context. Appropriate in its method of delivery, in its style and structure, and above all in its substance. Content strategy is the practice of determining what each of those things means for your project—and how to get there from where you are now.

    RIGHT FOR THE USER (AND CONTEXT)

    Let us meditate for a moment on James Bond. Clever and tough as he is, he’d be mincemeat a hundred times over if not for the hyper-competent support team that stands behind him. When he needs to chase a villain, the team summons an Aston Martin DB5. When he’s poisoned by a beautiful woman with dubious connections, the team offers the antidote in a spring-loaded, space-age infusion device. When he emerges from a swamp overrun with trained alligators, it offers a shower, a shave, and a perfectly tailored suit. It does not talk down to him or waste his time. It anticipates his needs, but does not offer him everything he might ever need, all the time.
    Content is appropriate for users when it helps them accomplish their goals.
    Content is perfectly appropriate for users when it makes them feel like geniuses on critically important missions, offering them precisely what they need, exactly when they need it, and in just the right form. All of this requires that you get pretty deeply into your users’ heads, if not their tailoring specifications.
    Part of this mind-reading act involves context, which encompasses quite a lot more than just access methods, or even a fine-grained understanding of user goals. Content strategist Daniel Eizans has suggested that a meaningful analysis of a user’s context requires not only an understanding of user goals, but also of their behaviors: What are they doing? How are they feeling? What are they capable of?
    Venn diagram of user's contexts
    Fig. 1. The user’s context includes actions, constraints, emotions, cognitive conditions, and more. And that in turn affects the ways in which the user interacts with content. (“Personal-Behavioral Context: The New User Persona.” © Daniel Eizans, 2010. Modified from a diagram by Andrew Hinton.)
    It’s a sensible notion. When I call the emergency room on a weekend, my context is likely to be quite different than when I call my allergy specialist during business hours. If I look at a subway map at 3:00 a.m., chances are that I need to know which trains are running now, not during rush hour tomorrow. When I look up your company on my phone, I’m more likely to need basic contact info than your annual report from 2006. But assumptions about reader context—however well researched—will never be perfect. Always give readers the option of seeing more information if they wish to do so.

    RIGHT FOR THE BUSINESS

    Content is appropriate for your business when it helps you accomplish your business goals in a sustainable way.
    Business goals include things like “increase sales,” “improve technical support service,” and “reduce printing costs for educational materials,” and the trick is to accomplish those goals using sustainable processes. Sustainable content is content you can create—and maintain—without going broke, without lowering quality in ways that make the content suck, and without working employees into nervous breakdowns. The need for this kind of sustainability may sound boneheadedly obvious, but it’s very easy to create an ambitious plan for publishing oodles of content without considering the long-term effort required to manage it.
    Fundamentally, though, “right for the business” and “right for the user” are the same thing. Without readers, viewers, and listeners, all content is meaningless, and content created without consideration for users’ needs harms publishers because ignored users leave.
    This principle boils down to enlightened self interest: that which hurts your users hurts you.

    Good content is useful

    Define a clear, specific purpose for each piece of content; evaluate content against this purpose
    Few people set out to produce content that bores, confuses, and irritates users, yet the web is filled with fluffy, purposeless, and annoying content. This sort of content isn’t neutral, either: it actively wastes time and money and works against user and business goals.
    To know whether or not you have the right content for a page (or module or section), you have to know what that content is supposed to accomplish. Greater specificity produces better results. Consider the following possible purposes for a chunk of product-related content:
    • “Sell products”—This is so vague as to be meaningless and is likely to produce buzzword-infested fluff.
    • “Sell this product”—Selling a product is a process made up of many smaller tasks, like discussing benefits, mapping them to features, demonstrating results and value, and asking people to buy. If your goal is this vague, you have no idea which of these tasks (if any) the content will perform.
    • “List and demonstrate the benefits of this product”—This is something a chunk of content can actually do. But if you don’t know who is supposed to benefit from the product, it’s difficult to be specific.
    • “Show how this product helps nurse practitioners”—If you can discover what nurse practitioners need, you can create content that serves this purpose. (And if you can’t find out what they need before trying to sell them a product, you have a lot more to worry about than your content.)
    Now do the same for every chunk of content in your project, and you’ll have a useful checklist of what you’re really trying to achieve. If that sounds daunting, think how much harder it would be to try to evaluate, create, or revise the content without a purpose in mind.

    Good content is user-centered

    Adopt the cognitive frameworks of your users
    On a web project, user-centered design means that the final product must meet real user needs and fulfill real human desires. In practical terms, it also means that the days of designing a site map to mirror an org chart are over.
    In The Psychology of Everyday Things, cognitive scientist Donald Norman wrote about the central importance of understanding the user’s mental model before designing products. In the user-centered design system he advocates, design should “make sure that (1) the user can figure out what to do, and (2) the user can tell what is going on.”
    When it comes to content, “user-centered” means that instead of insistently using the client’s internal mental models and vocabulary, content must adopt the cognitive frameworks of the user. That includes everything from your users’ model of the world to the ways in which they use specific terms and phrases. And that part has taken a little longer to sink in.
    Allow me to offer a brief illustrative puppet show.
    While hanging your collection of framed portraits of teacup poodles, you realize you need a tack hammer. So you pop down to the hardware store and ask the clerk where to find one. “Tools and Construction-Related Accessories,” she says. “Aisle five.”
    “Welcome to the Tools and Construction-Related Accessories department, where you will find many tools for construction and construction-adjacent activities. How can we help you?”
    “Hi. Where can I find a tack hammer?”
    “Did you mean an Upholstery Hammer (Home Use)?”
    “…yes?”
    “Hammers with heads smaller than three inches are the responsibility of the Tools for Home Use Division at the far end of aisle nine.”
    “Welcome to The Home Tool Center! We were established by the merger of the Tools for Home Use Division and the Department of Small Sharp Objects. Would you like to schedule a demonstration?”
    “I just need an upholstery hammer. For…the home?”
    “Do you require Premium Home Use Upholstery Hammer or Standard Deluxe Home Use Upholstery Hammer?”
    “Look, there’s a tack hammer right behind your head. That’s all I need.”
    DIRECTORY ACCESS DENIED. Please return to the front of the store and try your search again!”
    Publishing content that is self-absorbed in substance or style alienates readers. Most successful organizations have realized this, yet many sites are still built around internal org charts, clogged with mission statements designed for internal use, and beset by jargon and proprietary names for common ideas.
    If you’re the only one offering a desirable product or service, you might not see the effects of narcissistic content right away, but someone will eventually come along and eat your lunch by offering the exact same thing in a user-centered way.

    Good content is clear

    Seek clarity in all things
    When we say that something is clear, we mean that it works; it communicates; the light gets through. Good content speaks to people in a language they understand and is organized in ways that make it easy to use.
    Content strategists usually rely on others—writers, editors, and multimedia specialists—to produce and revise the content that users read, listen to, and watch. On some large projects, we may never meet most of the people involved in content production. But if we want to help them produce genuinely clear content, we can’t just make a plan, drop it onto the heads of the writers, and flee the building.
    Of course, clarity is also a virtue we should attend to in the production of our own work. Goals, meetings, deliverables, processes—all benefit from a love of clarity.

    Good content is consistent

    Mandate consistency, within reason
    For most people, language is our primary interface with each other and with the external world. Consistency of language and presentation acts as a consistent interface, reducing the users’ cognitive load and making it easier for readers to understand what they read. Inconsistency, on the other hand, adds cognitive effort, hinders understanding, and distracts readers.
    That’s what our style guides are for. Many of us who came to content strategy from journalistic or editorial fields have a very strong attachment to a particular style—I have a weakness for the Chicago Manual of Style—but skillful practitioners put internal consistency well ahead of personal preferences.
    Some kinds of consistency aren’t always uniformly valuable, either: a site that serves doctors, patients, and insurance providers, for example, will probably use three different voice/tone guidelines for the three audiences, and another for content intended to be read by a general audience. That’s healthy, reader-centric consistency. On the other hand, a company that permitted each of its product teams to create widely different kinds of content is probably breaking the principles of consistency for self-serving, rather than reader-serving, reasons.

    Good content is concise

    Omit needless content
    Some organizations love to publish lots of content. Perhaps because they believe that having an org chart, a mission statement, a vision declaration, and a corporate inspirational video on the About Us page will retroactively validate the hours and days of time spent producing that content. Perhaps because they believe Google will only bless their work if they churn out dozens of blog posts per week. In most cases, I think entropy deserves the blame: the web offers the space to publish everything, and it’s much easier to treat it like a hall closet with infinite stuffing-space than to impose constraints.
    So what does it matter if we have too much content? For one thing, more content makes everything more difficult to find. For another, spreading finite resources ever more thinly results in a decline in quality. It also often indicates a deeper problem—publishing everything often means “publishing everything we can,” rather than “publishing everything we’ve learned that our users really need.”
    There are many ways to discover which content is in fact needless; traffic analysis, user research, and editorial judgment should all play a role. You may also wish to begin with a hit list of common stowaways:
    • Mission statements, vision statements, and core values. If the people within your organization are genuinely committed to abstract principles, it will show in what they do. The exception is the small number of organizations for whom the mission is the product, as is the case with many charities. Even then, this kind of content should be supplemented with plentiful evidence of follow-through.
    • Press releases. These may work for their very narrow intended audience, but putting them undigested onto a website is a perfect example of the how-we’ve-always-done-it mistake.
    • Long, unreadable legal pages. Some legal awkwardness is acceptable, but if you want to demonstrate that you respect your readers, take the extra time to whittle down rambling legalese and replace needless circumlocutions with (attorney-vetted) plain language.
    • Endless feature lists. Most are not useful to readers. The few that are can usually be organized into subcategories that aid findability and comprehension.
    • Redundant documentation. Are you offering the same audience three different FAQs? Can they be combined or turned into contextual help?
    • Audiovisual dust bunnies. Do your videos or animations begin with a long flying-logo intro? Do they ramble on for 30 minutes to communicate ten minutes of important content? Trim, edit, and provide ways of skipping around.
    Once you’ve rooted out unnecessary content at the site-planning level, be prepared to ruthlessly eliminate (and teach others to eliminate) needless content at the section, page, and sentence level.

    Good content is supported

    Publish no content without a support plan
    If newspapers are “dead tree media,” information published online is a live green plant. And as we figured out sometime around 10,000 BC, plants are more useful if we tend them and shape their futures to suit our goals. So, too, must content be tended and supported.
    Factual content must be updated when new information appears and culled once it’s no longer useful; user-generated content must be nurtured and weeded; time-sensitive content like breaking news or event information must be planted on schedule and cut back once its blooming period ends. Perhaps most importantly, a content plan once begun must be carried through its intended growth cycle if it’s to bear fruit and make all the effort worthwhile.
    This is all easy to talk about, but the reason most content is not properly maintained is that most content plans rely on getting the already overworked to produce, revise, and publish content without neglecting other responsibilities. This is not inevitable, but unless content and publishing tasks are recognized as time-consuming and complex and then included in job descriptions, performance reviews, and resource planning, it will continue.
    Hoping that a content management system will replace this kind of human care and attention is about as effective as pointing a barn full of unmanned agricultural machinery at a field, going on vacation, and hoping it all works out. Tractors are more efficient than horse-drawn plows, but they still need humans to decide where and when and how to use them.

    Of theses and church doors

    One of the great images of the history of the Protestant Church is that of a German priest standing in the cold in front of the Castle Church in Wittenberg on All Saints Eve, nailing his manifesto to its wooden doors.
    The reality of the publication of Martin Luther’s 95 Theses is messier, and whether the church doors were really involved at all is the subject of academic dispute, but one thing is clear: Luther published his theses to begin an open, public conversation.
    Our industry doesn’t lack for manifestos, some of them even explicitly modeled after Luther’s. This article—and the book from which it’s extracted—is not one of the firebrand, nailed-to-the-door attempts at full-scale revolution.
    But it is intended to continue conversations we’ve been having for years, and to spark new ones, about the shared principles and assumptions that underlie our work, and the weird and interesting things we can build on top of them.
    I’ll bring the coffee and doughnuts. See you on the church steps? 

    Learn More

    Related Topics: Content Strategy

    Discuss

    Was it good for you, too?Join the discussion »

    About the Author

     Erin KissaneErin Kissane is a content strategist and editor based in New York City and Portland, Oregon. She currently leads projects for content strategy consultancyBrain Traffic, and was formerly editor of A List Apart magazine, editorial director of Happy Cog Studios, and a freelance writer and editor. She blogs at incisive.nu.