Cyd Harrell – Conducting Usability Research for Mobile

Go to article

By Sean Carmichael

January 25th, 2013

Podcast: Play in new window | Download (Duration: 25:46 — 13.4MB)

[ Transcript Available ]

Mobile changes everything about how we conduct usability research. Not only has the way we design and build websites and apps had to adapt, how we study them has to as well. Traditional research methods won’t translate to a mobile environment.

Cyd Harrell, formerly of Bolt | Peters, has been conducting mobile usability tests. The problem with traditional usability testing labs for mobile is that the users aren’t in their normal context of use. There’s this awkward interaction of holding the device in the right position, or operating while it’s attached to a sled or rig. Observing this unnatural interaction won’t give you the greatest results.

The context of use paradigm is only one important aspect. If you and your designers are all iPhone users, for instance, thinking of things in terms of iOS is natural. With the variety of operating systems on these devices, testing your app in the right context as far as both the user and device are concerned can be a challenge.

Cyd is presenting one of the daylong workshops at the UX Immersion Mobile conference April 22-24 in Seattle. For more information about Cyd’s and the other workshops, visit uxim.co.

Recorded: December, 2012
[ Subscribe to our podcast via  ←This link will launch the iTunes application.]
[ Subscribe with other podcast applications.]

Full Transcript.

---

Jared Spool: Hello there, everyone. It’s another episode of the SpoolCast here. I’m Jared Spool. I have today with me the amazing Cyd Harrell.

Cyd, up until very recently, was the VP of research for the UX design firm based in San Francisco called Bolt | Peters. Then suddenly they got sucked into the great Facebook acquisition vacuum cleaner.

Cyd’s miraculously escaped and is out doing all sorts of good for the world and sharing with us her expertise in usability testing, in particular, around that of mobile usability testing, which is exactly what we’re going to talk to her about today.

Hi, Cyd. How are you?

Cyd Harrell: Hi, Jared. I’m great. How are you?

Jared: I am doing fabulous. It’s a great day here.

Mobile usability testing. This requires a lot of hardware, now, doesn’t it?

Cyd: It can. I think that there’s a little bit of a perception that it requires more than it in fact does, because there’s an idea that if we’re going with the traditional lab model, and we want to take perfect videos and we want to get every single mobile device that our audience might use all lined up in the lab, then pretty soon, we are indeed talking about 20 pieces of hardware in addition to what we’d normally have for a test.

But there are ways to do it that minimize that a little bit, and there are ways to do it without adding on too much hardware at all. If you think a little bit more about how you might do a guerilla research study or how you might do a study in a home or even how you might do a remote study, which is just starting to become possible with some of the new developments. It’s pretty exciting what’s going on these days.

Jared: Let’s talk about this. If I was doing this in a lab, do I need to retrofit the lab with special cameras to capture things and stuff like that or are there tricks to this?

Cyd: There are some tricks. A lot of people will retrofit the lab by adding sleds for particular devices. I don’t think you need to. In fact, my favorite way to set up the lab for a lab mobile study is to have the participant sit in a comfortable armchair and place a little target on the arm of whichever hand they’re most comfortable using. Then have an over-the-shoulder camera, which can be something as simple as a flip or one of the small video cameras on a little boom that can focus on the mobile screen.

Then if you just do a little bit of tweaking to make sure that if it’s either a device you’re providing you can set the brightness so that it’s easy for the camera to catch it or that if the participant’s bringing it in, you just take that moment at the beginning to get it set right. You can get a really nice video without ten extra pieces of equipment.

Jared: That’s interesting, because I’ve gone to presentations and stuff. They’re always talking about molding these little plastic sled things that you mount the camera on and you mount the phone on. Or you have to have the phone, like they tape a little box on the table and the phone has to stay in the box so that the camera can hit it and stuff.

But you’re just doing this in a comfy chair.

Cyd: I think it’s better in a comfy chair, just because the participant is more comfortable. You can still have a target area on the arm of the comfy chair, but rather than necessarily laying the phone down, you can figure out in your pre-flight, which I certainly hope anybody does for a study, what’s the approximate angle that a couple different people hold their phone at. You can set up the boom so that you just need to do a little tweak to the angle as each person comes in and you’re good to go.

I do think sleds can be useful if you want someone to walk around in a lab environment, but they are hard to use out in a real world environment, which is, of course, the other environment where we really want to see people doing mobile testing because of the in-context nature of mobile use.

Jared: Now you’ve done some out in the field studies, right?

Cyd: Oh, yes.

Jared: What’s that like with the mobile stuff?

Cyd: It’s incredible fun, because you really can get embedded in peoples’ lives. It can be crazy technology-wise. I’ll tell you that the first mobile study I ever did was probably the most technically complex, but in some ways one of the most interesting. This was at Bolt | Peters, and one of the major car manufacturers wanted to create a concept car for the 2009 Geneva auto show that would take account of the way that drivers use mobile devices in their cars.

They asked us to do some ethnographic research on how people use various devices, including phones, iPods, and anything else that people actually happen to use. We proposed that we would actually accompany participants on trips that they planned to take for reasons of their own. Because of the location of the car company that was sponsoring this, we needed to broadcast it to a couple of different countries in real time so that they could send in questions.

This became quite an adventure, but it worked really well. We had a camera person along taking high-res video, and then we had a researcher with a laptop who also rode along and used the webcam on their laptop together with one of those Sprint devices that lets you connect pretty well from anywhere.

These days you’d use WiFi probably or something like that, but that was what was available at the time.

We got some incredibly rich information from seeing–I’m going to laugh for a minute here, because one of the things that people do tend to lie about is things they’re not supposed to do–like the way that they might use devices in a car.

By being there on trips that people actually cared about, they were going to pick up their kids. They were going to shop at the mall, they were going to take their dog to the dog park, perhaps. We could see what their real environment was like. We could see how did they manage with the dog in the car? Was it really true that they would never text while the care was moving? Not true for most people.

Jared: You were doing this in California?

Cyd: Yes.

Jared: Is it illegal to text in California?

Cyd: It wasn’t at the time.

Jared: OK. It is now, so if you were to do this, you’d actually have video evidence.

Cyd: Yeah. We have quite a consent form on this.

Jared: [laughs]

I bet you do.

Cyd: Yeah. But, you know, the interesting thing about this being the first mobile study that I did made the technical complexities of my future mobile studies really seem minor.

Jared: There’s a good recommendation. You just make your first one the most complicated one you’ll ever have to do, and it will make every other study seem so much easier.

Cyd: I think that once you sit down to technically plan something…The really tricky element there was the fact that we were broadcasting. If we had just been recording it it would have been significantly simpler.

But we did the same thing a number of other times with things like in-store walk-arounds, where we had a client who had an app that they wanted to enhance the in-store shopping experience. There really wasn’t any way to get at that besides actually going to the store and walking around with people and their phones. We set up something very similar, although we were able to take advantage of in-store WiFi that they had to make the broadcast a little more stable.

For me, I don’t feel like we have to start with the assumption that in-context mobile research isn’t possible. If we’re choosing to do it in a lab, then we should do that for good reasons. There are excellent ways to set that up, too. But it shouldn’t just be a choice because we assume that we can’t max out the realism of our research scenarios.

Jared: It feels almost like the lab is not necessarily the first choice for doing the research in a lot of these projects. That and the fact because doing them in context is not that much more difficult, we should be strongly considering that as we’re doing the studies.

Cyd: I think we should. I strongly encourage people to do it. Again, live broadcast is a complexity, but if you’re not doing live broadcast, you have a lot of ways that you can be with someone.

The other, bigger challenge in some ways is if what you’re studying is studying is an episodic activity, so it’s not something where a person walks into the store and then they might be using the app during a half-hour activity. If it’s something like checking directions that they might do throughout the day, it’s a lot harder to arrange and pay the incentives and make things work with a full-day session with a person.

What interests me with that is we’re getting better and better options for putting together sets of tools that can let us do things like diary studies or experience sampling studies using people’s mobile phones as essentially the research platform.

Sometimes, obviously, we’re talking today mainly about instances where the mobile phone is also the topic of the research, but it’s also possible to do it for completely different topics, where they might be doing an activity episodically that’s not specific to their mobile phone.

Jared: Say a little bit more about that.

Cyd: Well, there’s a famous example, actually, from back in 2008, which was called the “Happiness Project,” which I suspect a few of your listeners might have signed up for. It was a graduate student at Harvard who decided to survey people’s happiness and try to understand what activities actually make people happy and don’t. He set up a really fun incentive for this. He said, “If you participate in my project for two weeks, I’ll send you a graph of your happiness.”

The way that the study was engineered was you’d receive a text on your iPhone. This, of course, was on second generation iPhones, the 3Gs mainly and some of the first generation ones. When you receive the text, you are supposed to respond to a survey in a mobile website that asks, “What are you doing? Who are you with? And how happy are you right at the moment?”

Jared: I think I saw a TED talk on this.

Cyd: Yeah. It was a really interesting study model. The unfortunate thing was that with the iPhones of the time, filling out a mobile survey was enormously painful. You know, it was really hard to get through the survey that was only supposed to take 30 seconds, so a lot of people dropped before two weeks.

They ended up getting a lot of good data, I think, but now, in 2012 and heading into 2013, there are some really exciting developments in terms of mobile storytelling and mobile survey and mobile form creation applications, where you can really create a pretty pleasant experience for your participants of taking part in your research.

Jared: This gives us an opportunity to use mobile as a research tool to study things that are much broader than the phone itself. Like if I was trying to figure out when people are needing, for instance, my company’s intranet, I could have employees collect that data on the phone throughout the day in a diary-like study.

Cyd: Very much so, yes. I have heard of people…This is at the stage right now where a lot of researchers are duct taping together their own. There’s one platform that’s really designed for this which is e-Scout. They have a great team over there, and the app is developing along.

But I have heard of people using Survey Gizmo. I have used Blurb Mobile myself. There is a really nice new web and mobile form generation product that’s going into beta called “Typeform” that I haven’t seen yet, but I’m really excited about. It’s possible by the time we talk in April that it may be out. Things are really moving fast. That’s one thing in this space.

Jared: How did you find yourself, you know, doing a lot of mobile stuff? Was it just the projects just kept coming in, or were you guys pushing that as an expertise thing, or what was it that got you? Because at Bolt | Peters, you and the rest of the team there were doing some of the most advanced stuff I’d seen.

Cyd: Well, thank you, and, you know, it started out client-driven. The study that I described a little earlier was one of the first requests we got, and then we felt like it was part of who we were and what we were excited about to be able to, basically, take research technology limitations off the table, something that we did in our game research, and in our web research, and all of the development of things for remote research.

We started to get excited about mobile because of that because we wanted to push ourselves in a direction that people were thinking of exactly the way you mentioned at the beginning where, “Oh, no, this is a scary technical area.” It was always our answer to that at Bolt | Peters to say, “No, it’s not. Grab your duct tape and let’s do this.”

Jared: Is that approach of just challenging yourself to say, “Look. These things we believe are restrictions don’t necessarily have to be,” is that something you find yourself coaching other folks to think in terms of?

Cyd: I do, often. In fact, I’ve done a lot of coaching the last two years with Code For America fellows who are largely wonderful and technically-inclined people, but I think that as we think about research there are certain classical methods, and there are a lot of places that teach methods or techniques more than they teach theory and practice, if that makes sense if methods and techniques are a little different than theory and practice.

The most important practices of research are the design of research and asking the right questions and setting aside the technical limitations until you figure out what the right questions are and then seeing what you can put other to get those right questions answered.

That really is where I’m excited about the potential of mobile to let us answer some of those in-context questions about mobile behavior itself, which is, of course, one of the new things over the last few years that we’re all seeing a lot of, but also about other human behaviors.

Jared: I think this is really cool, this idea that mobile is a research tool in addition to just figuring out how to evaluate designs that we might be putting on the devices themselves. Have you been doing work that is looking at the questions of the apps that I’m building, let’s say, for mobile, “The apps that I’m building for mobile, are they a good idea versus have I got all the details right, right?”

It’s sort of a different process when you’re just looking at “Well, what ideas should we be focusing on?” versus, “Is this design we’ve come up with the best design we could have?”

Cyd: Yes, I think it’s really important to do that. I think there are a lot of clients right now who are concerned that they might be behind in mobile or concerned with competitive pressures in mobile and so who are developing an app. You know that’s really a choice now. That’s one strategy.

There are really good ways to build mobile websites that are not apps, as well. There are more strategic choices that need to be made, and there needs to be a reason to make any given one. Like any research you need to understand if you’re solving a real problem with something that you’re building.

There are a lot of interesting ways to do early prototyping on mobile. Mobile phones have nice photo capabilities so some of the simplest things are putting screenshots into a sequence much like you might do a click-through prototype for early testing on web or a desktop app.

I’ve seen a very cute paradigm where someone used regular-sized Post-It notes, which are about the width of an Android phone, which they had done nice sketches on and used that as a true paper prototype on a mobile setting so the person could demonstrate what they would do, and then they could peel off the Post-It note to get to the next “screen.”

So there are all kinds of methods to do what we would think of as kind of prototype concepts level tests, and then there is also just understanding the behaviors that people do with their phones, which where we started out with the concept car study was looking at that.

If I’m in my car, and I’m driving my dog to the dog park, what’s important for me to have within arm’s reach? What are the things that I either have taped together a way to do while I’m in the capabilities, or find that I can’t do and get irritated about, or never need to do? So that then they can take that and understand what capabilities they might want to provide in a very futuristic car that would serve people who are connected all the time.

Jared: Well, I think this is cool, this idea that you can do future-thinking prototyping, and idea generation, and even just sort of seeing the context that people have today and collect that data, and then at the same time go back and try out a design and try a prototype at various levels of fidelity, whether it’s a paper prototype that’s low fidelity or moving towards more higher fidelity screenshots.

Is there something that you think is still too difficult to do on mobile that you wish was a lot easier than it is right now?

Cyd: That’s a great question. I think that one thing that stumps a lot of teams is dealing with the proliferation of mobile devices and the different operating systems, and it’s easy to get in a bubble there. If you and your team are all iPhone people, then you can look at everything in iOS and consider it.

If you are dealing with the Android market there’s, of course, many, many different handsets, and there’s still, I think, a lot of question around…Some of them are almost sort of persona style questions, “Is there an android type user who is different from in iOS type of user who might be different from a Windows eight type of user?”

I think that that just plethora of platforms, to be a little bit alliterative, is something that we have some ways to deal with, but it doesn’t feel easy.

Jared: Yeah, I mean, I saw pictures that Facebook has posters up all over their offices. They’re paying employees a reward if they start using an Android phone so they can test the Android Facebook app, because I guess at some point they gave all the employees iPhones, and so now everybody’s got an iPhone and nobody has an Android phone.

So they’re resorting to paying people about to get this to happen so they can use it themselves. It would seem to me that’s problematic, plus do you have to worry about the different operating systems that Android runs on? I’m an iPhone guy so I actually know very little about Android so is it the case that the various…They’re all named after things I’d rather be eating.

Cyd: Absolutely.

Jared: Ice Cream Sandwich and Gingerbread House, and Peking Duck. I don’t know exactly what they are.

Cyd: I think that there are some differences in how things run on the operating systems, but that is going to get into the detail level when you’re really talking about usability and testing, which, obviously, is something that you should do as well as high-level in-context research.

The ways that people solve that question…There are a few firms out there that specialize in helping with specifically that testing in operating systems at the end. There’s one in San Francisco called OTIVO. There is an online crowdsourcing company called Mob4Hire that basically has a panel that has a broad diversity of different kinds of devices and operating systems.

What’s interesting to me about that…I’m not normally a big fan of panels. I think that for deeper research, you don’t want a lot of them, but just in terms of penetration of “Can I see someone use it on this operating system?” I think they might be onto something there.

I think, in general, if you are looking at behavior questions, they’re probably… [laughs] In some ways, even though I’ve done a bunch of these studies, I don’t feel like I have enough data yet in my layers of personal data to say, but I don’t feel like you need to account for five different Android operating systems in most user-research study designs.

It’s often a good idea to look at iPhone and Android. Of course, you want to consider, some people put tablet in with mobile research, some people don’t at this point, but it’s another platform that may be of interest.

Another thing, actually, that I would bring up is that once your app gets to a beta or even an alpha stage, there are starting to be platforms through which you can distribute it out to users and collect data on what they do with it.

I am beta-testing, actually, a civic app right now via a system called Test Flight that installs unfinished apps onto people’s mobile phones bypassing the app store so they can be tested. There’s another thing just for iOS called Delight.io that, I believe in that case, your app still goes through the app store. It’s for post release.

It’s an open question and apps make it more interesting, but much like we don’t test a website on a Mac and a PC and a Linux machine necessarily these days. If we are just doing usability research I think that you can restrict a little bit what you do. I think you can comfortably say, “I’m going to choose iOS. I’m going to choose an Android operating system that I have in the lab, or I’m going to allow users to use their own phones and I’m just going to record what they have.”

I think that’s part of what people feel overwhelmed by, honestly, is the notion that there are all these phones and, in addition, there are all the technical complexities of cameras and maybe, as you say, molded pieces of plastic and sleds and, “Oh, gosh. I’m not sure this is worth it.”

Jared: Yeah. It sounds like the fear of this is really just in our heads and that we just have to get over it and go off and do it and realize how easy it is.

Cyd: It’s easy, and furthermore, it’s really fun. A lot of the times when we’re talking about researching behavior on a computer we’re talking about a person sitting at a desk, but when we are research behavior on mobile we can be researching someone interacting with a city or interacting with a retail experience or a whole other range of human experiences.

I think it’s fun and I think getting over the technical hurdle can be fun if you enjoy duct tape, which I think most people should.

Jared: Yeah, I don’t get as much opportunity to enjoy duct tape in my life as I would like.

Cyd: That may be a little bit of my Bolt | Peters history sneaking back in there.

Jared: Well Cyd, thank you so much for taking time to talk to us about this. I’m really looking forward to your workshop at the conference. I think it’s going to be really cool because you’re going to give folks a chance to see what all the different pieces are, including how you use mobile technologies to research things that are beyond just what happens on the phone or on the tablet, right?

Cyd: Absolutely. Yes, I’m planning to do some work with walk-arounds and some exercises around how to get out in the world. I’m super excited for that, too.

Jared: Fabulous. Cyd’s going to be giving this full-day workshop on using mobile technology for usability research, “Conducting Usability Research for Mobile Apps,” we’re calling it, at the UX Immersion conference, April 22nd through 24th in Seattle, Washington.

We’ve already started seeing a lot of people register. Folks are signing up for Cyd’s session. It’s going to be great. If you’re interested, you want to get in there because I’m going to bet it’s probably going to sell out.

Cyd, thanks for taking the time today.

Cyd: Absolutely my pleasure.

Jared: I want to thank the audience one more time for listening in. As always, we love having you here. If you have a chance, please go to iTunes, if that’s how you download your podcasts, and just give us a little rating and let us know how you think about all these things that we’ve put together. We love the ratings and they help us make this better. We love to see what you have to say.

Thank you for listening to another episode of the “Spoolcast.” As always, thank you very much for encouraging our behavior. Take care. We’ll talk to you next time.