Pass or fail? Binomialrelated hypothesis testing and confidence intervals using independent samples - Improving the User Experience through Practical Data Analytics - page 95

Table 4.1 Tasks and Number of Successful Completions

Successful
Completions
on Behemoth
Search Engine
(out of 10)

Successful
Completions on
Novix Search
Engine (out of 10) Task # Verbatim Task

1 Find a Java developer with at least

5 years' experience within 50 miles of
Tucson, Arizona.

3 9

2 Find a Web designer with skills using

Photoshop, Illustrator, and Flash within
25 miles of San Diego, California.

2 7

3 Find an electrical engineer with a mini-

mum of a Bachelor's degree within 25
miles of Boston, Massachusetts.

1 8

4 Find a business analyst with an MBA.

There is no location requirement.

5 6

5 Find a technical writer within 50 miles

of Chicago. The maximum your client
is willing to spend on the position is
$50,000 per year.

7 8

6 Find an intern for a marketing depart-

ment within 25 miles of New York
city. The minimum education level is a
Bachelor's degree.

7 9

7 Find a vice president of development

who is willing to travel over 50% of the
time.

1 9

4.3 HYPOTHESIS TESTING USING THE CHI-SQUARE TEST OF
INDEPENDENCE OR FISHER'S EXACT TEST
To put your dilemma into statistical terms, you want to know if there is a statistically
signiicant difference between the task success rates, comparing Novix with Behemoth.
Well, you came to the right place. By using either the chi-square test of indepen-
dence or Fisher's exact test (depending on your software), we can determine if the
results you gathered in your usability test are consistent with what you would theoreti-
cally expect to occur, if the task success rates were the same. More simply put, we want
to see if there is a signiicant difference between the search engines on any given task.
Please note that we have “independent samples,” as discussed in Chapter 2.
The 10 people who attempted the tasks with the Novix search engine and the 10
people who attempted the tasks with the Behemoth search engine are different sets
of 10 people, even though each group of 10 are picked from the same population
and are, in a sense, “nominally the same”!