Information Technology Reference
In-Depth Information
This makes it clear that the largest drop-off rate, 21%, is associated with Page
#4. If you're going to redesign this multipage process, you would be well advised
to learn what's causing the drop-off at Page #4 and then try to address that in
the redesign.
9.1.4 A/B Tests
A/B tests are a special type of live-site study in which you manipulate elements
of the pages that are presented to the users. The traditional approach to A/B test-
ing on a website involves posting two alternative designs for a given page or ele-
ments of a page. Some visitors to the site see the “A” version whereas others see
the “B” version. In many cases, this assignment is random, so about the same
number of visitors sees each version. In some cases, the majority of visitors see
the existing page, and a smaller percentage see an experimental version that's
being tested. Although these studies are typically called A/B tests, the same con-
cept applies to any number of alternative designs for a page.
WHAT MAKES A GOOD A/B TEST?
A good A/B test requires careful planning. Here are some tips to keep in mind:
•
Makesurethemethodyou'reusingto“split”visitorsbetween“A”and“B”versions
really is random. If someone tells you it's good enough to just send all visitors in
the morning to version “A” and all visitors in the afternoon to version “B”, don't
believe it. There could be something different about morning visitors vs afternoon
visitors.
•
Testsmallchanges,especiallyatfirst.Itmightbetemptingtodesigntwocompletely
different versions of a page, but you'll learn much more by testing small differences.
If the two versions are completely different from each other, and one performs
significantly better than the other, you still don't know why that one was better. If
the only difference is, for example, wording of the call-to-action button, then you
know the difference is due to that wording.
•
Testforsignificance.Itmightlooklikeoneversionisbeatingtheotherone,butdo
a statistical test (e.g., χ
2
) to make sure.
•
Beagile.Whenyou'reconfidentthatoneversionisoutperformingtheother,then
“promote” the winning version (i.e., send all visitors to it) and move on to another
A/B test.
•
Believethedata,nottheHIPPO(HighestPaidPerson'sOpinion).Sometimes
the results of A/B tests are surprising and counterintuitive. One of the advantages
that UX researchers bring to the mix is that you can follow up on these surprising
findings using other techniques (e.g., surveys, lab, or online studies) to try to
understand them better.
Search WWH ::
Custom Search