Usability Testing Versus Expert Usability Review

This article appeared first at Learnability Matters, a blog by Kern Communications. It was orignially written by Sneha Nair, and edited by yours truly. And, I have taken permission to shamelessly copy :)

Sometimes we want to do the right thing to increase our application’s usability but don’t know how and where to begin, what method is appropriate to maximize an optimal outcome.

We’ve heard of many methods from usability testing, focus group, expert review to contextual enquiry but what’s the difference and when do we use it? Contrasted below are two usability evaluation methods that are often used.

Usability Testing	Expert Usability Review
Who conducts them? Usability testing involves getting people from the target audience to evaluate your application while performing specific tasks. Usability experts watch the participants performing the tasks and analyse. Kern provides usability testing services in India. Kern helps companies around the world understand Indian users and their usage patterns.	An expert usability review is performed by usability experts. The usability experts inspect an application to identify potential usability problems. The experts review the applications based on published research, experience in watching user behavior in the usability lab and design heuristics. Kern follows a unique methodology for expert evaluations that gives a rigour and actionable results to clients. This methodology is a mixture of Heuristic Evaluation, GOMS, and Cognitive Walkthrough.
How many users do I need? Usually 5-15 respondents are enough to find key usability issues in a usability test. Kern helps clients understand how many users may be optimally needed for a test. Kern will also help clients understand where it would be ideal to conduct such a test.	Expert usability evaluation requires absolutely no users. Usability experts perform the evaluations. Using the traditional expert evaluations methodology, it is important to have more than one expert to perform the evaluation. Number of problems found will increase when the number of evaluator increase (Kantner & Rosenbaum, 1997). The ideal number of evaluators is 2-3. Using Kern’s unique evaluation methodology, one expert evaluator can perform the evaluation without missing problems.
How much would it cost me? Usability testing requires more time to plan and organize. Preparation of testing protocol, hiring a usability lab, recruiting and incentivising people from a target user group and conducting tests and analysis is time consuming and expensive. Usability testing is the most expensive usability evaluation method.	An expert usability review is much cheaper when compared to usability testing. It’ll only take a few days to evaluate an application and write a report. You neither need to hire expensive labs nor recruit users; costs are significantly lower. With Kern’s unique methodology, we reduce the time taken for an expert usability evaluation, therefore saves you significant costs.
How long does it take? Usability testing method involves detailed protocol scripting, participant recruiting, and conducting the test one by one, followed by data analysis and report generation, it is a time consuming process. For a 15-20 people usability test, it usually takes 4-6 weeks, however Kern requires 3 weeks time – start to finish.	Skilled evaluators can produce high-quality results in a limited time. At Kern, it usually takes about one to three weeks for an expert usability evaluation including a report of findings and recommendations. The time depends upon the complexity of the application/website.
What does it help me find? Usability testing is better at finding issues related to special domain knowledge of users and their task flows – what path do they follow while performing a task. If the usability testing protocol is designed well and covers key user tasks, user testing helps you find major usability problems.	Expert usability reviews tend to find high level breaches of design rules and consistency. Analysis of interface architecture and page flow efficiency including navigation bottlenecks. Heuristic evaluation helps you fine minor usability problems. Though heuristic evaluation helps identify some major problems, the number of minor problem still outweighs the major problem. With Kern’s unique methodology, evaluators find all key and many strategic and tactical usability issues. This helps clients’ schedule implementation.
What are the disadvantages of usability testing and expert evaluation? Usability testing is expensive and time consuming. The validity of test findings depends heavily on identifying the right target group, accuracy of usability testing protocol to recognize key user tasks, and accurate user recruiting. Kern guarantees its clients accuracy of usability testing protocol and identification & recruitment of right users.	False positives: Research (Kantner & Rosenbaum, 1997) shows that 43% of the “problems” that were identified by the heuristic evaluators did not turn out to be problems at all. A lot depends on the reviewer’s level of experience. No two usability experts will find exactly the same issues so the method is also open to practitioner variation. The biggest problem however is simply that they sometimes miss the real problems that cause users to fail tasks. This can be especially true when the target audience has a particular skill set. With Kern’s unique methodology, false positives are significantly reduced. Unlike other firms, Kern always uses experienced evaluators to perform evolutions.
What are the advantages of usability testing and expert usability review? The results give a truer picture of the real problems people encounter because they’re derived from real users in the first place. Another advantage of usability testing is that there’s less conjecture and feedback comes straight from the horse’s mouth. Videos of user sessions in particular, can provide a powerful persuasion tool for reluctant stakeholders. It’s hard to argue with recordings of people in tears of frustration.	Quick and cost effective. Evaluators are usually specialists and their results often match user testing. Evaluators find the almost the same issues found in user testing. Sometimes they spot minute usability problems, often missed by user testing.
When do I use usability testing and when do I use an expert usability review?Usability testing is recommended for use later in the development process, when the extra value of finding the problems real users may encounter justifies its cost and time. Kern recommends summative usability testing only: When the application is used by domain experts When the user groups and tasks are clear During later part of development after formative tests and expert evaluations are done The users (and evaluators) are from a completely different cultural group from the designers The management must be convinced by using a proof	People often use expert usability reviews early on to straighten up their design in preparation for usability testing. Kern not only helps evaluate the application but also helps them give design directions.

References:

http://www.useit.com/papers/heuristic/ http://www.usabilitynews.com/news/article3616.asp http://www.userfocus.co.uk/articles/expertreviews.html http://www.oclc.org/policies/usability/testing/default.htm http://kmvirtual.com/2008/05/26/heuristic-evaluation-and-user-testing/ http://www.teced.com/PDFs/sigdoc97.pdf

3C's to take care when blogging about work

"Publish or perish" is repackaged in a new wine bottle by many corporates by encouraging employees and associates to blog about work and life. Many people see this as a way to publish and distribute best practices, publish amateur papers which otherwise may not get published in established journals, or talk about experiences.

Many people see blogging a way to become famous overnight while others try to make it a living resume with all gory details about projects, clients, etc. While we all do have a right to publish, we need to be careful while we do so about work related stuff. Remember, all the work that you do in a company, the intellectual property either resides with your company or its clients.

You must be careful about the the 3 Cs before your publish: Company, Competitors, and Clients. I am putting together some questions within each category that people must ask before they write an blog entry or just before publishing.

Company

a. Does the article contain description about processes, methods, or workflows proprietary to my company?

b. Does the article contain information that can give away my company's future directions, future products, or services?

c. Does the article contain references to the financial information about your company?

d. Does the article contain any information that will embarrass the company in any way?

e. Does the article contain any information that you remotely think is confidential for any reason?

f. Are you criticizing your company or your colleagues in any way?

Competitors

a. Does the article contain information that your company's competitors may benefit from?

b. Does the article refers to named competitors or identifies competitors in an indirect way (like "large mobile handset company in Finland")?

c. Does the article talk about yours or your company's capabilities that may benefit the competitors?

Clients

a. Does the article contain specific information that identifies specific clients or projects?

b. Does the article contain references to any project done at any point of time with any client? These can be methods, processes, workflow, blueprints, images, etc.

c. Does the article contain any information that is proprietary to your client and not a public knowledge?

d. Does the article contain any information that reduces or eliminates your client's selling or competitive ability?

If the answer to any of the questions is Yes, please do not hesitate to either change the article or run it in writing by a director level person in your organization.

If you do want to become famous, do blog wisely -- success is seldom overnight!

Market Research Vs. User Research

Market research and user research are analogous to each other but have completely different goals. They sometimes share common goals and inform one another.

Both disciplines have different approaches, their own strengths, and weaknesses. Which method to use and when to use it depends on what problem you are trying to solve.

Marketing research may start this effort with demographic data, while user researchers find information that challenges and qualifies the segmentation. Both types of research have a role in innovation; both can find gaps that may drive new product ideas.

Here is quick comparison of each method:

Market Research	User Research
Goal - Buying Patterns: Aims to understand buying patterns: understands who will buy which product/services and peoples opinions about a product/service.	Goal - Usage Patterns: user research focuses on usage patterns: how and in what circumstances users use products and services, and observes what people actually do.
Markets: What products or services people are willing to buy?	Usage: What are the usage patterns of a specific product or service? In which environment(s) do they use?
Customers: Which segments of people are willing to buy a defined product or service?	Users: Will specified use a specific product or service? What are the key roadblocks for using a product or service?
Competitors: Who are the competitors? What are the key success factors in the given industry?	Comparison: In which way people use competitors products differently for doing same task?
Attributes: What features and attributes of a product or service is important for a buying decision?	Implementation: How is a feature or group of features implemented? Efficiency, Learnability, Memorability, Errors, Subjective Satisfaction.
Marketing Department: Conducted by marketing department.	User Research Department: Conducted by usability or user research department.
Opinion Based: Aims to understand what people's opinions about products and services are.	Observation Based: Observes what and how people interact with products and services not what they say.
Big Sample Size: Needs huge number of respondents to get statistically accurate results. 800 ~ 2000	Very Small Sample Size: Need only 5-15 participants in a particular group for statistically relevant as well as for uncovering most usability problems.
Geography Based: Buying patterns change across smaller geographies. So research is needed in many cities, town, and villages.	Culture Based: Usage patterns change with change in sub-cultures. So, research is not needed across similar cities. The results are going to be similar.

How Does Each Type of Research Help the Other?

a. Market research finds out which users may buy a service or product. User research focuses on product/service usage by specified users.

b. Market research narrows on which products/services people may buy. User research focuses on how easy is to use the important ones.

c. User research uncovers unusual usage patterns of users. Market research can then conduct studies to find out if that is rampant and if products need to be designed catering to certain usage patterns.

References:

a. Market Research and Usability

b. What is a usability professional's role in conducting traditional market research?

c. Marketing and User Research