Methodology III: Empirical Evaluation - Foundations for Designing User-Centered Systems - page 366

that has been recorded, and will also act as a back-up in case recording fails for
some reason.
Structured interviews are based around a fixed set of questions that the inter-
viewees must answer. These questions are often closed, i.e., the user is expected to
answer the question and no more. Typically these questions have ''Yes/No'' type
answers.
Unstructured interviews are generally more informal, and are a bit more like a
chat with the users. So you may start off with a small number of issues (perhaps as
few as one or two) that you want to discuss with the users, and then the direction
you take for the rest of the interview is determined by what they say.
Semi-structured interviews fall somewhere between structured and unstructured
interviews. Usually you will have a short standard list of questions, which may be
open, and then you direct the interview based on what the users say in response to
the questions you ask. Unstructured and semi-structured interviews tend to be
slightly harder to carry out because they will often require the interviewer to think
on their feet during the interview. Their big advantage, however, is that they can
uncover issues that may not previously have been thought of.
Whilst interviews tend to be carried out on a one-to-one basis, it can be useful
to have group discussions, which are often carried out as focus groups. Usually a
focus group is carried out with a small group of up to about ten users or stake-
holders. The basic aim is to get the focus group members to express their opinions
in a relatively friendly environment. To conduct a focus group successfully you
need to have a list of issues or questions for discussion, and to have an experienced
facilitator who can make sure that everybody gets a chance to air their opinions.
The sessions can produce lots of useful data, so it is often best to record them as
well as making notes (it may help to have separate people taking notes and
facilitating the discussions).
13.5.8 Workload Measures
Workload measures attempt to describe how much mental effort the user expends
in performing a particular task. They are generally used more often to evaluate
critical systems rather than web sites per se. The measures are hard to devise, but
can be useful in many contexts. The most common approach is to periodically ask
users to state (or rate) what they think their current workload is, although this can
be quite disruptive of performance and hence affect their perceived workload.
The NASA-TLX (Task Load indeX) workload measurement instrument (Hart
and Staveland 1988 ) is probably the most commonly used method. NASA-TLX
can be administered on paper or on-line. The NASA TLX is a multi-dimensional
rating procedure that provides an overall workload score based on a weighted
average of ratings on six workload dimensions: mental demands, physical
demands, temporal demands, own performance, effort, and frustration (NASA,
1987 ).