Usability Testing Versus Expert Usability Review

This article appeared first at Learnability Matters, a blog by Kern Communications. It was orignially written by Sneha Nair, and edited by yours truly. And, I have taken permission to shamelessly copy :)

Sometimes we want to do the right thing to increase our application’s usability but don’t know how and where to begin, what method is appropriate to maximize an optimal outcome.

We’ve heard of many methods from usability testing, focus group, expert review to contextual enquiry but what’s the difference and when do we use it? Contrasted below are two usability evaluation methods that are often used.

Usability Testing	Expert Usability Review
Who conducts them? Usability testing involves getting people from the target audience to evaluate your application while performing specific tasks. Usability experts watch the participants performing the tasks and analyse. Kern provides usability testing services in India. Kern helps companies around the world understand Indian users and their usage patterns.	An expert usability review is performed by usability experts. The usability experts inspect an application to identify potential usability problems. The experts review the applications based on published research, experience in watching user behavior in the usability lab and design heuristics. Kern follows a unique methodology for expert evaluations that gives a rigour and actionable results to clients. This methodology is a mixture of Heuristic Evaluation, GOMS, and Cognitive Walkthrough.
How many users do I need? Usually 5-15 respondents are enough to find key usability issues in a usability test. Kern helps clients understand how many users may be optimally needed for a test. Kern will also help clients understand where it would be ideal to conduct such a test.	Expert usability evaluation requires absolutely no users. Usability experts perform the evaluations. Using the traditional expert evaluations methodology, it is important to have more than one expert to perform the evaluation. Number of problems found will increase when the number of evaluator increase (Kantner & Rosenbaum, 1997). The ideal number of evaluators is 2-3. Using Kern’s unique evaluation methodology, one expert evaluator can perform the evaluation without missing problems.
How much would it cost me? Usability testing requires more time to plan and organize. Preparation of testing protocol, hiring a usability lab, recruiting and incentivising people from a target user group and conducting tests and analysis is time consuming and expensive. Usability testing is the most expensive usability evaluation method.	An expert usability review is much cheaper when compared to usability testing. It’ll only take a few days to evaluate an application and write a report. You neither need to hire expensive labs nor recruit users; costs are significantly lower. With Kern’s unique methodology, we reduce the time taken for an expert usability evaluation, therefore saves you significant costs.
How long does it take? Usability testing method involves detailed protocol scripting, participant recruiting, and conducting the test one by one, followed by data analysis and report generation, it is a time consuming process. For a 15-20 people usability test, it usually takes 4-6 weeks, however Kern requires 3 weeks time – start to finish.	Skilled evaluators can produce high-quality results in a limited time. At Kern, it usually takes about one to three weeks for an expert usability evaluation including a report of findings and recommendations. The time depends upon the complexity of the application/website.
What does it help me find? Usability testing is better at finding issues related to special domain knowledge of users and their task flows – what path do they follow while performing a task. If the usability testing protocol is designed well and covers key user tasks, user testing helps you find major usability problems.	Expert usability reviews tend to find high level breaches of design rules and consistency. Analysis of interface architecture and page flow efficiency including navigation bottlenecks. Heuristic evaluation helps you fine minor usability problems. Though heuristic evaluation helps identify some major problems, the number of minor problem still outweighs the major problem. With Kern’s unique methodology, evaluators find all key and many strategic and tactical usability issues. This helps clients’ schedule implementation.
What are the disadvantages of usability testing and expert evaluation? Usability testing is expensive and time consuming. The validity of test findings depends heavily on identifying the right target group, accuracy of usability testing protocol to recognize key user tasks, and accurate user recruiting. Kern guarantees its clients accuracy of usability testing protocol and identification & recruitment of right users.	False positives: Research (Kantner & Rosenbaum, 1997) shows that 43% of the “problems” that were identified by the heuristic evaluators did not turn out to be problems at all. A lot depends on the reviewer’s level of experience. No two usability experts will find exactly the same issues so the method is also open to practitioner variation. The biggest problem however is simply that they sometimes miss the real problems that cause users to fail tasks. This can be especially true when the target audience has a particular skill set. With Kern’s unique methodology, false positives are significantly reduced. Unlike other firms, Kern always uses experienced evaluators to perform evolutions.
What are the advantages of usability testing and expert usability review? The results give a truer picture of the real problems people encounter because they’re derived from real users in the first place. Another advantage of usability testing is that there’s less conjecture and feedback comes straight from the horse’s mouth. Videos of user sessions in particular, can provide a powerful persuasion tool for reluctant stakeholders. It’s hard to argue with recordings of people in tears of frustration.	Quick and cost effective. Evaluators are usually specialists and their results often match user testing. Evaluators find the almost the same issues found in user testing. Sometimes they spot minute usability problems, often missed by user testing.
When do I use usability testing and when do I use an expert usability review?Usability testing is recommended for use later in the development process, when the extra value of finding the problems real users may encounter justifies its cost and time. Kern recommends summative usability testing only: When the application is used by domain experts When the user groups and tasks are clear During later part of development after formative tests and expert evaluations are done The users (and evaluators) are from a completely different cultural group from the designers The management must be convinced by using a proof	People often use expert usability reviews early on to straighten up their design in preparation for usability testing. Kern not only helps evaluate the application but also helps them give design directions.

References:

http://www.useit.com/papers/heuristic/ http://www.usabilitynews.com/news/article3616.asp http://www.userfocus.co.uk/articles/expertreviews.html http://www.oclc.org/policies/usability/testing/default.htm http://kmvirtual.com/2008/05/26/heuristic-evaluation-and-user-testing/ http://www.teced.com/PDFs/sigdoc97.pdf