User Testing
How to find out what users want
6/15/98
There are no stock answers when it comes to web design. Two developers faced with the same problem will approach it in two very different ways. Two web consultants will give you two different (though equally expensive) opinions. So how can you really tell if your site is working?
The answer lies with your site's users. The most successful web sites are those that not only respond to user feedback, but actively seek it out. These sites understand that the most valuable resource a developer possesses is not a software tool or piece of equipment. It's an opinionated audience.
Why should you test?
As a developer, you probably know your site's territory like the back of your hand. And you should -- you built it, after all. But the same expertise that makes you an asset to a web project can also be a drawback when it comes to usability. Think of what it's like to use someone else's computer, drive their car, or stay at their house. They know exactly how things work and where to find things. You, on the other hand, don't.
Regardless of how web savvy you are, if you create an environment, you'll suffer from what designer Richard Saul Wurman calls the "disease of familiarity." Don't beat yourself up over problems -- you've probably done your best with what you know about the audience. Instead, get a fresh perspective. The best way to do this is to talk with your site's users. This article will help you get started.
When shouldn't you test?
Testing can be useful during initial development, before a redesign, or as a periodic checkup. But there is one time when you shouldn't test. If you are unwilling (for political or other reasons) to make needed changes based on testing results, there is no reason to waste your users' time or raise their expectations. Test if you are serious about listening to users. Don't test if you're simply looking for validation.
Planning the Setup
All you need to get started in user testing is a user, a computer, and a pencil and paper for notetaking. Some usability consultants make use of video cameras, tape recorders, one-way mirrors, or other high-tech tools. These are not only unnecessary, they can actually get in the way. Keep it simple and you can spend more time focusing on user behaviors and less time worrying about why the camcorder won't work.
Where should you test?
In choosing your setup, you'll need to decide whether to test at the user's site or at your offices. There are definite advantages to testing at the user's site, though it means more travel time. You'll be able to see how they really experience your site -- 16-color monitor, 14.4 modem and all. More importantly, though, you'll be able to see their environment. It's a opportunity for you to be part anthropologist, part tester. Visiting the user's site won't work in every case (home users, for example, won't be too keen on it), but in many instances it can provide valuable extra information.
How many users should you involve?
The best way to run a user test is one-on-one: one user with one facilitator. You don't need a large quantity of users for the tests to be effective. Twenty one-on-one tests will often show you what twenty group tests couldn't. Though you'll test with more users in the second case, there's no chance you'll be able to observe them all in any meaningful way. You simply won't be able to pay attention to the details, an essential part of testing.
Deciding on a Facilitator
Your choice of a facilitator (the person who will conduct the testing and record the results) depends to some degree on the resources you have available. You can hire an outside usability consultant to do your testing, which is the best method by far since it builds in maximum objectivity and experience. However, consultants are expensive. If you don't have the resources to hire one, you can still move ahead given certain conditions.
What are the qualities of a successful facilitator?
The most important quality of a successful facilitator is the ability to be objective and non-judgmental. Many developers simply cannot distance themselves enough from what they've created to be effective facilitators. It's a natural reaction, but it can negatively affect a user test. It's easy to get defensive or show hurt feelings, even to disagree with users during a test. Though it's possible for a developer to overcome this hurdle, it's not common. For that reason, web companies often turn to outside consultants or hire a full-time usability professional on staff.
Another area to consider is experience and knowledge. A usability consultant or on-staff usability professional will have the experience and knowledge to conduct the tests with ease. Other candidates will need to be trained if they are to avoid biasing users or leading them to certain conclusions. User testing isn't rocket science, but it does need to be done with careful methodology and a planned approach.
Personality and demeanor also have a lot to do with the facilitator's success. A facilitator who appears unfriendly, bullying, or anxious can affect the outcome of the test. An ideal candidate has a calm, open, and friendly personality and makes users feel that whatever they say is true and fascinating (regardless of what they really think). The best attitude is passive, friendly interest.
What else should you consider when choosing a facilitator?
It's also important to remember that every usability professional probably has his or her own particular style and preferred methods. If you're interviewing consultants or candidates for an on-staff usability position, ask about their preferred methods and approach. One approach is not necessarily better than another, but it's important that you understand what you might be getting yourself (and your users) into.
Finding Participants
This is the really tough part: finding users to participate in your tests. Depending on your project, there are different approaches to tracking down users. Your first step, which you should have already addressed during the early stages of your site's planning, involves targeting a specific audience. Once you've narrowed your scope that way, you can begin looking for participants.
Where can you find members of your target audience?
You can track down potential test participants through customer lists (including marketing mailing lists or records of sales), related organizations or associations, email discussion lists, conferences and events, and so on. If your audience is parents, you might consider visiting your local school system, finding out if there's a parenting magazine in which you can place a classified advertisement,or contacting a national parent advocacy organization. You might supplement that by including people you know, such as an acquaintance, neighbor, or coworker.
Some companies contact temp agencies when they need to hire users for product testing. For example, if you run a skiing site, you might request temporary workers with some computer experience who also were active skiiers. This approach is one viable option for companies with a general adult audience. However, it's fairly useless if your audience consists of kids or highly trained professional people such as doctors or lawyers.
Using your own staff, immediate family members, or close friends can be helpful, but they will always pull their punches. You're not likely to get completely honest answers from people you're very close to, because they won't want to hurt your feelings. Take the time to find participants who can be brutally honest if necessary.
Do you need to compensate participants?
There's not much agreement in the field on whether to compensate user testers or use volunteers. I'm a firm believer in compensation. It makes the selection process much easier if you provide an incentive. A cash stipend ($30-50 per person) seems to work nicely. You can also try product giveaways or service discounts. When compensating participants, make it clear you are paying for their time and honest reactions, not for validation or ego stroking.
Asking the Right Questions
So you've planned the setting, found the users and facilitator, and you're ready to go. Make the most of your pre-test planning by preparing your questions and approach as best you can. During a user test, your main goal is passive observation, but you still need to be prepared.
What should you ask?
Before you begin, you'll need to know a few basic facts about your participants. This doesn't mean handing them a 5-page survey about their habits and preferences. What you need to know can be covered in a few quick questions. It's important to get the user's name -- more for social reasons than anything else. It's usually also important to find out what their web skill level is. This becomes useful later when you sort results. If you decide to ask other preliminary questions, make sure they're tightly focused and essential to how you view results. Overwhelming users with too many personal questions up front is no way to begin a test.
Once you've gotten the initial information out of the way, you'll need to briefly reiterate what is expected of users. There's a lot you can do at this point to prejudice people. "Take a look at this site and tell me how we can fix it," for example, biases users toward the negative and provides no real instruction.
Instead, try to explain briefly that you are interested in discovering what they think of the site, both good and bad. Reassure them that you are an objective observer and won't be insulted by anything they say (whether this is true or not, say it anyway). Let them know that this is not a quiz and there are no wrong answers. And finally, tell them that you will not be touring them through the site; instead, it will be the reverse. Ask them to "talk out loud" about what they see and experience as they move through the site.
You may need to give users tasks to get them started or keep them focused during the test. If it's a personal banking site, you might begin by asking them what they would do to sign up for a new account. Watch where they go, when they feel unsure or take the wrong path, and what makes them react positively (or curse like a sailor). When they seem to stall or go around in circles, you can give them another task or ask them to explore another part of the site. Balancing tasks with free exploration is essential, since it increases opportunities for discovery.
When the test is over (30 minutes is usually ideal), you can ask users some more focused questions. If you're concerned that graphic design might be affecting company image, don't ask "What do you think of that red logo?" Instead ask something like, "How do you feel about this company?" or "What kind of company do you think this is? What words would you use to describe the people behind this site?" Once the test is done, you should ask for general impressions. If problems are mentioned, you may want to ask for suggestions on solving them. Avoid asking these more focused questions up front, since they can bias or confuse participants.
What can go wrong?
Some users will not react well to the test or to the site, and may be uncommunicative, belligerent, or distracted. If you've focused on tasks, try giving them more freedom. If you've focused on freedom, try giving them tasks. Different users will react better to different techniques. Avoid simply guiding them through the site, though some will plainly ask you to do so. If necessary, reiterate that you are here as an observer, and remain friendly and calm.
Some tests will be a wash despite all your best efforts, but don't feel bad. People have different personalities, and some users just won't take to the testing. There's nothing you can or should do to weed these people out, but be aware you may have an occasional uncommunicative or unpleasant tester.
Analyzing the Results
After each test you conduct, you will want to summarize your notes. Try to do this as soon as possible after the test, since unless you know shorthand, you'll probably need your short term memory to help you decipher what you've recorded. If you wait until all your tests are completed, you'll risk losing information or misinterpreting your notes.
What should you look for in the results?
Remember that user testing is not like science class. You don't have a hypothesis to prove. Approach it more like an anthropologist or detective, watching for clues in the results that will lead you to findings. The best clues are in patterns, the repetition of remarks or observations.
For example, if every person you test with rants about your navigation system, here's a news flash: you've got a problem with your navigation system. If only one person rants about it, that's a different story. Weed out individual reactions and focus on the shared issues that were raised, both specifics and generalities. This helps you to define where your site's weaknesses -- and strengths -- lie.
What other tools help in presenting results?
Entering your results into a database such as Microsoft Access or Filemaker Pro makes it easier to sort results by factors like skill level. The bulk of your "data" will not be data, necessarily. It may instead be insight, which is far more important. Still, archiving and organizing results this way can be a useful final step, particularly in a large organization where test results will need to be shared among departments.
Whoever conducted the tests should prepare a report presenting the trends in the test results and providing a summary of problem areas. He or she should also be prepared to discuss test findings with development team members, some of whom may need handholding. It can be a stressful experience for developers, but remember that user testing is a powerful tool -- and the only clear antidote to the "disease of familiarity."
Related Resources
Conducting Your First User Test
http://webreview.com/97/05/30/usability/index.html
Five-step approach outlining how to begin if you've never tried testing before.
User Test Your Web Site
http://webreview.com/97/04/25/usability/index.html
Web Review article by Keith Instone tells you how to get started.
How to Usability-Focused Companies Think?
http://world.std.com/~uieweb/focused.htm
What do "usability-focused" companies do right? This article from User Interface Engineering offers some thoughts.
Does Your Site Work? Ask a Usability Guru
http://www.thestandard.net/articles/recruitment_display/0,1267,444,00.html
Article from The Industry Standard features tips and quotes from usability experts Jakob Nielsen and Jared Spool.
Scheduling Hard-to-Find Users
http://world.std.com/~uieweb/
Tips from User Interface Engineering on pinning down those elusive users.
Test Your Designs -- on People!
http://www.hotwired.com/webmonkey/tools/97/39/index0a.html
Webmonkey article offers a common sense approach to testing with users.
Webmonkey: Why User Testing is Good
http://www.hotwired.com/webmonkey/98/14/index3a.html?tw=design
This Webmonkey article explains their upbeat approach to user testing.
useit.com: Usability Inspection Methods
http://www.useit.com/papers/heuristic/inspection_summary.html
Describes several methods for conducting a usability inspection, including heuristic evaluations and cognitive walkthroughs.
useit.com: Severity Ratings for Usability Problems
http://www.useit.com/papers/heuristic/severityrating.html
Essay by Jakob Nielsen outlining how to determine the seriousness of usability problems.
Jennifer Fleming is an Anchor producer and the author of the upcoming book Web Navigation: Designing the User Experience (O'Reilly, 1998; http://www.ora.com/catalog/navigation/). She runs Square Circle Solutions (http://www.squarecircle.com), a Boston-based web firm specializing in user experience consulting and idea generation.
© 1998-1999 Anchor Productions, Inc. All rights reserved.