Why does usability research requires only a few users?

Usability research is usually qualitative, or driven by insight (why users don't understand). Qualitative research follows different research rules to quantitative research and it is typical that sample size is low.

The end result of usability testing is not statistical validity per say but verification of insights and assumptions based on behavioral observation - quantitative vs. qualitative.

·       We are looking for behavioral based insight, like what people do.

·       Statistics tell half the story and often are devoid of context, like why did they fail?.

·       Research clearly shows that even with low numbers, you can gain valid data.

·       Usability testing is being used industry-wide and has been for past 25 years. Experts, authors and academics put their reputations and credentials behind the methodology.

As mentioned earlier, Usability research is behavior-driven: You observe what people do, not what they say.

In contrast, market research is largely opinion-driven: You ask people what they think and what they think they think. For this, big samples are needed for market research. This is why phone or web surveys require hundreds or thousands of responses.

Behavior-driven research is more predictable. Basically, if 10/15 users are confused you can assume that many more will also be confused as well.

How are Focus Groups different from Usability Testing?

This was originally written by Mini and I have again shamelessly copied this from Kern's Blog.

In today's fast paced life, people tend to avoid products that frustrate them. Satisfied customers would be ready to buy good products and services. This also helps in word-of-mouth marketing.

Used together, focus groups and usability testing help in building great products and services which are highly desirable, satisfying, fun and easy to use and also worth being recommended  to others. However, both, focus groups and usability testing have different focuses and must be used in conjunction with each other. Focus groups and usability testing must never replace each other.

A focus group speaks about the extent of promises that can be made to the customer while usability testing helps shaping the products based on those promises. Usability testing is special as it focuses on eliminating end-user frustration by turning it into satisfaction and pleasure.

This article is a continuation of a series of comparisons of various user research techniques. Here I've contrasted focus groups with usability testing.

Focus Groups	Usability Testing
What is the meaning of focus groups? Focus group is a brain storming session group administered  a moderator. The goal of a  focus group is usually to get a variety of  broad  opinions of what people want a product to do for them along  with what will drive the end users to buy the product?  This helps the moderator to begin considering the features of the product. Focus groups are  opinion based and their motivation is to find out what the desires and wants of the users are. .	What is the meaning of usability testing? Usability testing is a one-user-at-a-time testing by users but moderated by a seasoned usability professional. While the user interacts with the product in a structured manner, the usability professional notes down the stumbling blocks, hits, and reactions of the users while they use the product. This help usability professional to guide product development decisions. The key focus is what will make the end users of the product to learn about and use the product easily. Usability testing is observation based and tells  if or not something works.
What is the output of focus groups? The output of a focus group is a collection of various opinions, attitudes and preferences of the users.	What is the output of usability testing? The output of usability testing is observational and behavioural measures based on the ease with which the users handle the products.
When should focus groups be used? Focus groups are best used in the early stages of product or service development to provide wide-ranging information on opinions and attitudes of the target  user group. This is used primarily when important investment decisions are to be made. Focused on opinions: Focus groups have many voices at the same time, so it is impossible to observe the users in action and record the steps. The moderator can, at best, get opinions about their  intended usage With this lack of focus and precision, it only elicits vague opinions. Consider what to build: The purpose of a focus group is to decide what kind of product to consider building. Decide Costs: A company uses focus groups to decide the extent of investment of product development. This results in large near term costs in the hope of future profits.	When should usability testing be used? Usability testing is used throughout the development process to help in finely focused development decisions and it also guides in the realization of the big picture. During the early development stages, paper-prototype based Formative Usability Testing is used to test ideas, navigation, and task flow. Post development, Summative Usability Testing is used to benchmark navigation, task flow, measuring user performance, errors, and subjective satisfaction. Focused on tasks: In usability tests, the users are asked to complete many tasks one after another. The difficulty with which the users complete the task is carefully recorded. This clear focus helps them to get more specific and actionable results. Making products better: The purpose of a usability test is to improvise the product to make it work better. Increase profits: The usability test is used to improvise the products resulting directly in cost savings, greater sales, and increased profits.
How many people do I need for a focus group? Focus groups involve 6-10 people interviewed together in the same location at the same time.	How many people do I need for a usability testing? Ideally about 5-8 users are tested per sub-group individually with usability professional.
How much would  a focus group cost me? To conduct a focus group, expenses will be incurred on recruiting, incentivising, focus group facilities such as a place where a focus group can be conducted, a place where clients and others can view the proceeding, and audio-video equipment must be readily available. Also expenses have to be incurred to carefully prepare the focus group protocol and also to analyze the findings. Focus groups are  less expensive when compared to  usability testing.	How much would usability testing cost me? Preparation of testing protocol, hiring a usability lab with one way mirror, recruiting and incentivising people from a target user group, conducting tests one-by-one, and analysis is time consuming and expensive. Usability testing requires lot more time to plan and organize. Usability testing is one of the most expensive and most effective usability evaluation methods today.
How long does it take to conduct focus groups? A focus group involves a detailed protocol scripting and  participant recruiting. However, it is not time consuming when compared to usability testing. Ideally a focus group session should not exceed 1 ½ hours. For a couple of focus groups involving 14-20 people, Kern would ideally take 2 ½ weeks to recruit, conduct tests, and present analysis.	How long does it take to conduct usability tests? Usability testing method  involves detailed protocol scripting, participant recruiting, and conducting the test one by one, followed by data analysis and report generation, it is a time consuming process. For a usability testing involving 2 sub-groups (about 16 users), Kern will take about 3 ½ weeks for the entire process.
What are the advantages of focus groups? Focus group is one way to gather information about a target audience that you have very limited information about. They help us understand users' opinion, attitudes, preferences and initial reactions. Group discussions often spark new ideas.They are excellent if you are looking for multiple and a wide range of opinions, ideas and points of views in a short period. Focus groups are cheaper and faster to conduct.	What are the advantages of usability testing? The results of usability testing give a truer picture of the real problems people encounter as they're derived from observations of real users.. They help us to understand the different behavioural patterns of the users while using the products and services. Videos of user sessions provide a very powerful persuasion tool for reluctant stakeholders. It's hard to argue with recordings of people in tears of frustration. Understanding points of frustration usually trigger practical solutions.
What are the disadvantages of focus groups? The findings do not lead to statistically significant results. As there are more members, one or two members' opinions usually dominate  others and this does not help in arriving at  clear conclusions.  opinions.	What are the disadvantages of Usability testing? Usability testing is expensive and time consuming. The validity of test findings depends heavily on identifying the right target group, accuracy of usability testing protocol to recognize key user tasks, and accurate user recruiting.

References:

http://www.macadamian.com/index.php%3Foption%3Dcom_content%26task%3Dview%26id%3D54%26Itemid%3D0

http://www.usabilitynews.com/news/article4062.asp

http://jobfunctions.bnet.com/abstract.aspx%3Fdocid%3D242848

http://www.zanzara.com/pdf/UsabilityVsFocus.pdf

Usability Testing Versus Expert Usability Review

This article appeared first at Learnability Matters, a blog by Kern Communications. It was orignially written by Sneha Nair, and edited by yours truly. And, I have taken permission to shamelessly copy :)

Sometimes we want to do the right thing to increase our application’s usability but don’t know how and where to begin, what method is appropriate to maximize an optimal outcome.

We’ve heard of many methods from usability testing, focus group, expert review to contextual enquiry but what’s the difference and when do we use it? Contrasted below are two usability evaluation methods that are often used.

Usability Testing	Expert Usability Review
Who conducts them? Usability testing involves getting people from the target audience to evaluate your application while performing specific tasks. Usability experts watch the participants performing the tasks and analyse. Kern provides usability testing services in India. Kern helps companies around the world understand Indian users and their usage patterns.	An expert usability review is performed by usability experts. The usability experts inspect an application to identify potential usability problems. The experts review the applications based on published research, experience in watching user behavior in the usability lab and design heuristics. Kern follows a unique methodology for expert evaluations that gives a rigour and actionable results to clients. This methodology is a mixture of Heuristic Evaluation, GOMS, and Cognitive Walkthrough.
How many users do I need? Usually 5-15 respondents are enough to find key usability issues in a usability test. Kern helps clients understand how many users may be optimally needed for a test. Kern will also help clients understand where it would be ideal to conduct such a test.	Expert usability evaluation requires absolutely no users. Usability experts perform the evaluations. Using the traditional expert evaluations methodology, it is important to have more than one expert to perform the evaluation. Number of problems found will increase when the number of evaluator increase (Kantner & Rosenbaum, 1997). The ideal number of evaluators is 2-3. Using Kern’s unique evaluation methodology, one expert evaluator can perform the evaluation without missing problems.
How much would it cost me? Usability testing requires more time to plan and organize. Preparation of testing protocol, hiring a usability lab, recruiting and incentivising people from a target user group and conducting tests and analysis is time consuming and expensive. Usability testing is the most expensive usability evaluation method.	An expert usability review is much cheaper when compared to usability testing. It’ll only take a few days to evaluate an application and write a report. You neither need to hire expensive labs nor recruit users; costs are significantly lower. With Kern’s unique methodology, we reduce the time taken for an expert usability evaluation, therefore saves you significant costs.
How long does it take? Usability testing method involves detailed protocol scripting, participant recruiting, and conducting the test one by one, followed by data analysis and report generation, it is a time consuming process. For a 15-20 people usability test, it usually takes 4-6 weeks, however Kern requires 3 weeks time – start to finish.	Skilled evaluators can produce high-quality results in a limited time. At Kern, it usually takes about one to three weeks for an expert usability evaluation including a report of findings and recommendations. The time depends upon the complexity of the application/website.
What does it help me find? Usability testing is better at finding issues related to special domain knowledge of users and their task flows – what path do they follow while performing a task. If the usability testing protocol is designed well and covers key user tasks, user testing helps you find major usability problems.	Expert usability reviews tend to find high level breaches of design rules and consistency. Analysis of interface architecture and page flow efficiency including navigation bottlenecks. Heuristic evaluation helps you fine minor usability problems. Though heuristic evaluation helps identify some major problems, the number of minor problem still outweighs the major problem. With Kern’s unique methodology, evaluators find all key and many strategic and tactical usability issues. This helps clients’ schedule implementation.
What are the disadvantages of usability testing and expert evaluation? Usability testing is expensive and time consuming. The validity of test findings depends heavily on identifying the right target group, accuracy of usability testing protocol to recognize key user tasks, and accurate user recruiting. Kern guarantees its clients accuracy of usability testing protocol and identification & recruitment of right users.	False positives: Research (Kantner & Rosenbaum, 1997) shows that 43% of the “problems” that were identified by the heuristic evaluators did not turn out to be problems at all. A lot depends on the reviewer’s level of experience. No two usability experts will find exactly the same issues so the method is also open to practitioner variation. The biggest problem however is simply that they sometimes miss the real problems that cause users to fail tasks. This can be especially true when the target audience has a particular skill set. With Kern’s unique methodology, false positives are significantly reduced. Unlike other firms, Kern always uses experienced evaluators to perform evolutions.
What are the advantages of usability testing and expert usability review? The results give a truer picture of the real problems people encounter because they’re derived from real users in the first place. Another advantage of usability testing is that there’s less conjecture and feedback comes straight from the horse’s mouth. Videos of user sessions in particular, can provide a powerful persuasion tool for reluctant stakeholders. It’s hard to argue with recordings of people in tears of frustration.	Quick and cost effective. Evaluators are usually specialists and their results often match user testing. Evaluators find the almost the same issues found in user testing. Sometimes they spot minute usability problems, often missed by user testing.
When do I use usability testing and when do I use an expert usability review?Usability testing is recommended for use later in the development process, when the extra value of finding the problems real users may encounter justifies its cost and time. Kern recommends summative usability testing only: When the application is used by domain experts When the user groups and tasks are clear During later part of development after formative tests and expert evaluations are done The users (and evaluators) are from a completely different cultural group from the designers The management must be convinced by using a proof	People often use expert usability reviews early on to straighten up their design in preparation for usability testing. Kern not only helps evaluate the application but also helps them give design directions.

References:

http://www.useit.com/papers/heuristic/ http://www.usabilitynews.com/news/article3616.asp http://www.userfocus.co.uk/articles/expertreviews.html http://www.oclc.org/policies/usability/testing/default.htm http://kmvirtual.com/2008/05/26/heuristic-evaluation-and-user-testing/ http://www.teced.com/PDFs/sigdoc97.pdf