Comparing two designs (or anything else!) using paired sample T-tests - Improving the User Experience through Practical Data Analytics - page 73

So, we know what you might be thinking: “How can I possibly use these advanced
statistical techniques with sample sizes this small?” Well, we acknowledge that hav-
ing only a small sample size is not ideal. But it's important to realize that, indeed,
lots can be done with small sample sizes. The smaller the sample size, the larger
the observed effect must be before we view it as a “real” effect, that is, indicative of
there truly being an effect. Thus, it follows that when you conduct a test with a small
sample size, you should not have an expectation of proving the existence of small,
subtle effects. Quite often, in UX work, that's OK, because, like in most all ields, it's
more important to ind the large effects—which often you can ind with the relatively
small sample size.
3.2 VIGNETTE: HOW FAST CAN YOU POST A JOB
AT BEHEMOTH.COM?
You're a usability researcher at Behemoth.com, an employment Web site. Behemoth.
com is aptly named, because it's one of the largest job search engines in the world,
with over 1 million job openings at any time, over 1 million resumes in the database,
and over 63 million job seekers per month. Behemoth has over 5000 employees
spread around the world.
Luckily for you, they realize the importance of usability testing and have built
a gleaming new usability lab with all the bells and whistles: dedicated observation
rooms, state-of-the-art software and hardware, and a dedicated recruiting team, so
you don't have to do your own recruiting (which, by the way, is a royal pain in the
neck). The company is located in an old nineteenth-century mill where the US Army
made Civil War blankets, but now it buzzes with new millennium dot.com energy.
It's a smarter, more pragmatic 2.0 Web; your execs learned the lessons of the late
1990s and aren't going to “get fooled again,” to quote the Who. And by the way, all
employees enjoy free lunch on Fridays.
Behemoth makes money in lots of ways, but one of the main sources of its income
is from employers who (1) post jobs on the site and (2) buy access to its enormous
database of resumes to search for good candidates to ill those jobs. (Conversely,
most job seekers go to the site to simply look for jobs, and submit resumes and appli-
cations to those jobs; they provide no revenue stream for Behemoth.)
As a consequence, you and your UX colleagues are constantly under the gun to
minimize the time it takes for job recruiters to post jobs on your job board. Quicker
job postings mean more postings. More postings mean more trafic, more resumes,
and more searching from paying employers.
The job posting process entails illing out a good old-fashioned form. And
despite consensus from everyone on the Design Team that the form is way too long,
the very nature of the company business mandates collection of the data. The insa-
tiable database needs to be fed so that the typical job seeker can easily ind jobs by
location, salary range, minimum requirements of salary, education, work experi-
ence, skills, languages, etc. And the company recruiter needs to search resumes by