Methodology III: Empirical Evaluation - Foundations for Designing User-Centered Systems - page 355

• Observers feel less intrusive
• Observers can talk in the observation room
• Observers can move around, allowing brainstorming activities
• Participants are less aware of being watched (because there is no large mirror in
the room, and sound contamination is reduced)
• A visual area larger than the participant's screen can be used to observe what
they are doing
• There is greater flexibility in the way the observation room can be configured
• Costs are reduced
• It is easier to use
• The volume of what the participant is saying can be controlled within the
observation room.
Fluent Studios' laboratory is used to conduct usability tests in a creative
environment. Whilst one-to-one testing is carried out with users in the testing
room, it is regarded as very important to get the observers involved as well. So,
often when a new design or prototype is being tested, the tests will be designed to
try and identify usability issues early on in the testing, so that they can be resolved
at the earliest opportunity.
Usually the first step in testing is to develop a set of task scenarios that capture
the critical characteristics of the tasks that are likely to be carried out using the
system (Carroll 2000 ). These scenarios are usually descriptions of real-world tasks
that users can be expected to understand, but the scenario does not describe how the
task is done using this system. They are typically expressed as problems that the
user would normally be expected to solve using the system as part of their work.
For example, consider the following scenario:
You are a system administrator for a software system that schedules and allocates
resources ranging from company pool cars to meeting rooms. Unfortunately one of the
meeting rooms has unexpectedly been designated to be refurbished, which will take
2 months beginning in July. Your task is to notify those people who have booked the room
for July and August and to provide alternative resources.
You should be able to see that this scenario contains a goal, information about
that goal, and information about the context in which the task takes place. It does
not, however, contain instructions about how to use the system to achieve the
desired goal.
The focus of the scenarios determines the shape of the evaluation: everyday
usage scenarios, for example, will capture information about everyday usage of the
system. Similarly, for critical systems (safety critical, mission critical, business
critical, and so on) the scenarios would be designed to focus on critical (but
unusual) incidents. Ideally a more comprehensive evaluation could be carried out
using both types of scenarios.
An illustration of the benefits of usability testing occurred when new designs for
a national educational site were being tested in Fluent Studios. Some usability
problems were quickly observed with the method of global navigation: the first
four users who were tested all struggled to find the site's home page. Between