Information Technology Reference
In-Depth Information
•
Provide information.
Observers need to understand what's going on. Make sure that a
session schedule, moderator's guide, and any other relevant information are readily
available to the observers, both before and during the sessions.
•
Engage the observers.
Give the observers something to do besides just watching. Provide
whiteboards or sticky notes for them to record issues. If there are breaks between
sessions, have them do a quick review of the key takeaways from the last session.
11.2 DON'T WAIT TO BE ASKED TO MEASURE
Many years ago, one of the best things we ever did was to collect UX data with-
out being asked for it directly. At that time, we started to sense a certain level of
hesitancy or even skepticism about purely qualitative findings. Also, the project
teams started to ask more questions, specifically around design preferences and
the competitive landscape that we know could only really be answered with
quantitative data. As a result, we took it upon ourselves to start collecting UX
metrics central to the success of the design we were working on.
What is the best way to do this? We recommend starting off with something
small and manageable. It's critical that you be successful in your first uses of
metrics. If you're trying to incorporate metrics in routine formative testing, start
with categorizing types of issues and issue severity. By logging all the issues,
you'll have plenty of data to work with. Also, it's easy to collect System Usability
Scale (SUS) data at the conclusion of each usability session. It only takes a few
minutes to administer the survey, and it can provide valuable data in the long
run. That way you will have a quantitative measure across all of your tests and
you can show trends over time. As you get comfortable with some of the more
basic metrics, you can work your way up the metrics ladder.
A second phase might include some efficiency metrics such as completion
times and lostness. Consider some other types of self-reported metrics, such as
usefulness-awareness gaps or expectations. Also, explore different ways to repre-
sent task success, such as through levels of completion. Finally, start to combine
multiple metrics into an overall UX metric or even build your own UX scorecard.
Over time you'll build up a repertoire of different metrics. By starting off
small, you'll learn which metrics work for your situation and which don't. You'll
learn the advantages and disadvantages of each metric and start to reduce the
noise in the data collection process. In our work, it has taken us many years to
expand our metrics toolkit to where it is today so don't worry if you're not col-
lecting all the metrics you want at first; you'll get there eventually. Also, be aware
that your audience will have an adjustment period. If your audience is only used
to seeing qualitative findings, it may take them a while to get adjusted to seeing
metrics. If you throw too much at them too quickly, they may become resistant
or think you just got back from math camp.
Search WWH ::
Custom Search