Information Technology Reference
In-Depth Information
HCI Evaluation Criteria
Perhaps the greatest challenge for HCI researchers is to select a meaningful measure of HCI suc-
cess. Expansion of the boundaries of the field implies less clarity regarding how HCI outcomes
should be evaluated. Usability, user satisfaction, and performance—the hallmarks of HCI evalua-
tion—remain important, but many other criteria become relevant when users consist of groups,
organizations, and communities. For groups, criteria such as mutual understanding, problem-
solving effectiveness, commitment, and creative thinking are often relevant (Hollingshead and
McGrath, 1995). For organizations and communities, criteria such as sustainability, competitive
advantage, growth, and flexibility are of interest today (Huber, 2004; Preece, 2000). At all levels,
matters of status, power, trust, leadership, network relations, time management, and learning are
important (Barge and Hirokowa, 1989; Bernard, 1973; Butler, 2001; Harasim, 1993; Scott, 1995).
Selection of evaluation criteria depends, of course, on the specific research question and the the-
oretical model used to drive the research. More than likely, multiple criteria are important and
integrated theoretical models are needed that provide explanatory power brought on by multi-
factor explanations of user behavior (see Trevino et al., 2002).
A major challenge in developing and applying theoretical models is incoherence of system
objectives and user needs in today's computing environment. Some groups or organizations may
have focused tasks and goals; but others interact without clear or specified reasons. The goals of
learning and entertainment may be mixed with those of efficiency or effectiveness. Greater diver-
sity within groups and communities compound the evaluation challenge (e.g., see Kim, this vol-
ume). Further, differing preferences and needs of individuals within a group or other collective
make selection of precise and meaningful evaluation criteria all the more difficult. In this research
context, researchers would do well to consider multiple theoretical lenses and multiple measures
of HCI success. Hubona, Straub, and Truex (in the second volume) describe how researchers can
integrate multiple theories to yield multiple measures for HCI evaluation.
CONCLUSION
The expanded conceptualization of computer user from individual to group, organization, and com-
munity implies significant design and research challenges. Major design challenges are to incorporate
the following capabilities into HCI systems: (1) virtuality, (2) dynamism, (3) background participa-
tion, (4) transparency, and (5) cultural transcendence. Major research challenges are to address ques-
tions such as: (1) What dimensions beyond structure (size and demographic makeup) can account for
variations in HCI needs and impacts within and between groups, organizations, and communities?
(2) How can boundaries between individual, group, organization, and community as levels of analy-
sis be meaningfully distinguished for research purposes? (3) What criteria should be used to evaluate
HCI success for groups, organizations, and communities? The HCI field is exploding in its scope and
ambition. Many more challenges wait on the horizon. To the extent that HCI design and research can
meet the unfolding challenges ahead, the field promises to take center stage in the ongoing study of
management information systems.
ACKNOWLEDGMENTS
This paper is based upon work supported by the National Science Foundation Grant No. SES-
0135602. The opinions, findings, and conclusions expressed here are those of the author and do
not necessarily reflect the views of the National Science Foundation.
Search WWH ::




Custom Search