Information Technology Reference
In-Depth Information
on a display. The measure depends on the size of the element and the
distance to it (see chapter 15 in [20] for more information).
Analytical evaluations are effective. They are quick to plan and
perform, require fewer participants than user studies, and can find usability
problems early in a design that users do not need to experience, and thus
not “have to waste time on.” However, methods such as cognitive
walkthroughs and heuristic evaluation should not replace validation by
evaluations with users, but be used as a complement [17]. For example,
they could be used prior to or in addition to user testing.
The next characteristic considers in what context (or environment) the
evaluation study is conducted. The nature of the context is important when
it comes to how results can be applied and generalized, and the more
generic the context the more flexibility there is with the applicability of the
outcome. Regarding the extent to which results can be applied and
generalized to other contexts, the level of constraints applied in the study
is also a determinant factor. It describes to what extent the researcher
imposes precision, structure, and control [21] on any part of the study.
This level ranges from low-level constraint evaluation (e.g. naturalistic
observation, field studies) that involve no or little control of the behaviour
of the participants or any equipment, to high-constraint studies conducted
in a strictly controlled setting (experimental studies), and where the nature
of the processing and analysis of the results is characterized by high
demands. For example, in statistical analysis, many assumptions must be
fulfilled to use a specific test for analysing data [21]. The more that is
controlled in a study, the more confident one can be that any results are
due to what is actually investigated and not to irrelevant, confounding
factors. Confounding factors can be failure or unexpected changes of the
equipment, fatigue due to long sessions, level of expertise that varies
between participants, sequence effects in the study between different parts
of it that may result in unwanted learning effects, etc. Although there are
advantages with a controlled setting, the disadvantage is that it can be
unrealistic and not representative for real use by real users. In most cases
there is a trade-off between level of control and the usefulness of results
that needs to be taken into careful consideration.
Finally, the third characteristic distinguishes between evaluation
methods in terms of their ability to generate either quantitative or
qualitative data. A quantitative evaluation usually consists of testing a
hypothesis through direct measurement: for example, user's performance
in terms of accuracy and response times on specific tasks. Data are
thereafter analysed by standardized statistical tests. A qualitative
evaluation, on the other hand, involves collecting non-numerical data by
Search WWH ::
Custom Search