Database Reference
In-Depth Information
“Eighty million?” you gasp. “Should I do a head-to-head usability test to see
if the new search does better than our current one before we close the deal with
Novix?”
“Too late. Done deal,” he says, taking a long slurp of his newly minted
cappuc-
cino
. “But, hopefully, we can make some tweaks before they roll it out. Go for it. Do
a head-to-head and see what we ind out.” “Then tell Hans.”
You get a hold of the Novix prototype and start to do some basic searches. You
must admit: the interface is slick. It begins with a simple open “title” ield that has a
type-ahead feature that even accommodates for misspellings. Then, there's a lengthy
page of open ields for years of experience, location, skills/keywords, education,
maximum salary, and job type. There's even “Willing to Travel” and “Relocation.”
But, Lou was right: no place for Boolean searching.
Your usability test includes seven recruiting tasks and several post-task Likert
scales. You decide to use two different independent groups of 10 participants, 20
participants in all, who try to complete the same seven tasks; group one will try to
complete the tasks on Behemoth's current candidate search while group two will try
to complete the tasks on the new Novix search.
The 4-day actual testing—ive participants per day—proceeds very well. No can-
cellations, zero no-shows, with articulate and thoughtful participants. (They should
be—they all got a $100 Amazon gift card!) Some trends become obvious right away,
but the other problematic areas are more nuanced. At the end of the twentieth partici-
pant, you're beat and can't really look at the data.
After a good night's sleep, you get to work on analyzing the data. And of course,
the irst thing you do is tally the task completions in
Table 4.1
.
The results are troubling, to say the least. Participants were much more successful
using the Behemoth search engine on at least four tasks, including pedestrian ones
that involved searching for skills, experience, location, and willingness to travel.
Other tasks involving searching by maximum salary and education level are much
closer. But, overall, it feels like the $80 million price tag is not producing the results
Behemoth top brass wanted.
The reasons for the different completion rates are complex, but one of the over-
arching qualitative indings of the tests is that most of the recruiters automatically
tried to use Boolean strings. Of course, on the Behemoth engine, it was business as
usual. On the new Novix search, recruiters tried in vain to use Boolean strings, usu-
ally in the title ield. When that failed, they tried combinations of the Novix ields,
with varying degrees of success. Groans of “no Boolean?” were heard during most
of the sessions; one recruiter with over 20 years of experience said, “This will make
my hair grayer than it already is.”
But, staring at the numbers, your analytic brain and statistical savvy kick in. With
relatively low sample sizes, is seven completions on task 2 for Behemoth versus two
for Novix really statistically signiicant? For that matter, how about the nine and
three difference for task 1? The terrifying question rumbles around your brain as you
continue to stare at the numbers: Did CEO Joey Velluci really blow 80 million bucks?
Search WWH ::
Custom Search