Methodology III: Empirical Evaluation - Foundations for Designing User-Centered Systems - page 348

result that the delivered system is not acceptable, because it does not fit in with the
way that the users work. User testing during the early stages of development can
highlight potential problems: making developers watch users wrestle with their
product in an effort to make it do what they want is often a real eye-opener.
At least some part of systems development is based on the use of guidelines and
principles. There are many guidelines and principles available, and they are mostly
generic, which can make selecting the relevant ones for a specific project a major
task. If you do use guidelines and principles, user testing can help to highlight the
more subtle aspects of the system's context that need to be taken into account
during development, thereby providing evidence to support your particular choice.
13.2 Planning Your Evaluation Study
Collecting data is not difficult. Collecting the right data—data that is pertinent to
the questions that are being posed—is not so straightforward. Collecting the
appropriate data to answer strategically relevant questions requires careful plan-
ning. Here we identify some of the main issues that can help to improve the
chances of your evaluation study being a success. Many of the issues that we
highlight are central to good experimental design.
13.2.1 What Type of Data: Qualitative or Quantitative?
One of the first things you will need to consider is the type of data that you will
collect. The general rule of thumb is that you should think about collecting data to
help you understand the thing (system, product, application, and so on) before you
collect data to more precisely measure it. Typically the first type of data to collect
is qualitative data, in other words, statements or general behavior, rather than
precise numbers. Quantitative measures (e.g., times, number of clicks) can be used
to verify assumptions with some degree of confidence over the intended population
of users.
13.2.2 Selecting a Hypothesis
Creating hypotheses for your evaluation study helps to frame the study but also to
keep it focused and grounded. A hypothesis is a proposition of what you believe to
be the case (e.g., that a change you have made will cause a change in user behavior
in some way) and will check with data. The null hypothesis (H 0 ) normally states
that there is no difference between the things you are testing, e.g., there is no
difference between the usability of application A and of application B. The