Ten Keys to Success - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 284

makes it all the more important to get things right the first time you lay out your
benchmarking plans.
Benchmarking doesn't always have to be a special event. You can collect
benchmark data (anything that will allow you to compare across more than
one study) on a much smaller scale. For example, you could routinely collect
SUS data after each usability session, allowing you to easily compare SUS scores
across projects and designs. It isn't directly actionable, but at least it gives an
indication of whether improvements are being made from one design iteration
to the next and how different projects stack up against each other.
Running a competitive user experience study will put your data into perspec-
tive. What might seem like a high satisfaction score for your product might not
be quite as impressive when compared to the competition. Competitive metrics
around key business goals always speak volumes. For example, if your aban-
donment rates are much higher than your competition, this can be leveraged to
acquire budget for future design and user experience work.
11.6 EXPLORE YOUR DATA
One of the most valuable things you can do is to explore your data. Roll up your
shirt sleeves and dive into the raw data. Run exploratory statistics on your data
set. Look for patterns or trends that are not so obvious. Try slicing and dicing
your data in different ways. The keys to exploring your data are to give yourself
enough time and not to be afraid to try something new.
When we explore data, especially large data sets, the first thing we do is to make
sure we're working with a clean data set. We check for inconsistent responses and
remove outliers. We make sure all the variables are well labeled and organized.
After cleaning up the data, the fun begins. We start to create some new variables
based on the original data. For example, we might calculate top-2-box and bottom-
2-box scores for each self-reported question. We often calculate averages across
multiple tasks, such as total number of task successes. We might calculate a ratio to
expert performance or categorize time data according to different levels of accept-
able completion times. Many new variables could be created. In fact, many of our
most valuable metrics have come through data exploration.
You don't always have to be creative. One thing we often do is run basic
descriptive and exploratory statistics (explained in Chapter 2). This is easy to do
in statistical packages such as SPSS and even in Excel. By running some of the
basic statistics, you'll see the big patterns pretty quickly.
Also, try to visualize your data in different ways. For example, create different
types of scatterplots and plot regression lines, and even play with different types
of bar charts. Even though you might never be presenting these figures, it helps
give you a sense of what's going on.
Go beyond your data. Try to pull in data from other sources that confirm or
even conflict with your assertions. More data from several other sources lend