Methodology III: Empirical Evaluation - Foundations for Designing User-Centered Systems - page 369

13.7 Summary
There is a lot more to evaluation than many people imagine. Carrying out an
evaluation requires careful thought and planning before you begin testing. In this
chapter we have highlighted the sorts of issues you need to think about during
planning. Most development is carried out using an iterative cycle in which a
formative evaluation is carried out during each cycle. The information that comes
out of the evaluation can then be used to inform development during the next cycle
of development. It is therefore important that you clearly understand what sort of
data you should collect, and why. You also need to think about whom you will
collect the data from, and the environment in which you will collect them.
Once you have the basic plan, you can start to think in more detail about how
you will collect the data (and how you will analyze them, although we have not
covered that issue here). There are many methods that you could use, and your
final choice may be determined by factors such as how much time is available,
what resources are available, and how many (potential) users are accessible to take
part in the tests. We have briefly discussed several evaluation methods that are
available, and touched)on the importance of making sure that you are aware of the
need to treat the participants in your evaluation in a way that is ethical.
Evaluation is important because it produces feedback on development as it is
progressing. If you can get real users to take part in your evaluations, their
feedback will help make sure that the system is more likely to be usable and
acceptable when it is delivered. In other words, it will reduce the risks that the final
system will be a failure when it is delivered.
13.8 Other Resources
There is a web site devoted to the subject of evaluating adaptive systems: EASy-
Hub (which stands for Evaluation of Adaptive Systems Hub) is available at http://
www.easy-hub.org .
If you are designing a usability laboratory, Jacob Nielsen edited a special issue
of the BIT journal about how to create and use usability laboratories. Although it is
now somewhat dated, it still contains several useful nuggets of information:
Nielsen, J. (Ed.). (1994). Special issue: Usability Laboratories. Behaviour & Information
Technology 13(1-2).
If you are not experienced working with studies with human participants, useful
guides include:
Ritter, F. E., Kim, J. W., Morgan, J. H., & Carlson, R. A. (2013). Running behavioral
studies with human participants: A practical guide. Thousand Oaks, CA: Sage.
Shadbolt, N. R., & Burton, A. M. (1995). Knowledge elicitation: A systematic approach.