ADVANCING THE THEORY OF DSS DESIGN FOR USER CALIBRATION - Human-Computer Interaction and Management Information Systems: Foundations - page 44

In fact, the focus on decision quality distinguishes DSS dialog (interface) design work from the
human-computer interface (HCI) research done by computer scientists. HCI research by com-
puter scientists has focused on efficient performance or on the minimization of time or errors
in completing a task (Shneiderman, 1992). By contrast, DSS researchers investigating interface
design issues focus on effective performance, typically in the form of improved decision quality,
with decision time and decision confidence as secondary outcomes (Speier and Morris, 2003).
In differentiating between confidence, trust, predictability, and decision accuracy, Muir (1987,
p. 1915) states:
Predictability is a basis for trust, which, in turn, is the basis for an operator [user/decision
maker] to make a prediction about the future behaviour of a referent. The accuracy of that
prediction may be assessed by comparing it with the actual behavioural outcome. In addi-
tion, an individual who makes a prediction may associate a particular level of confidence
with the prediction. Thus, confidence is a qualifier which is associated with a particular pre-
diction, it is not synonymous with trust.
Realism in confidence is essential for good decision making; the ruinous consequences of unre-
alistic confidence litter the business decision-making landscape (Russo and Schoemaker, 1992).
Indeed, the literature on “escalation to commitment” and the adage “throwing good money after bad”
testify to the recurrence and cost of overconfidence (Staw, 1981), as people are often unrealistically
confident in the quality of their decisions (Brown and Gould, 1987; Einhorn and Hogarth, 1978).
The best-known measure of the accuracy of one's confidence in a decision is calibration, the
correspondence between one's prediction of the quality of a decision and the actual quality of the
decision (Clemen and Murphy, 1990; Keller and Keller, 1993; Lichtenstein et al., 1982). Perfect
calibration exists when the ascribed confidence equals the accuracy of the predicted outcome; oth-
erwise, miscalibration is present, reflecting either underconfidence or overconfidence. Because
action precedes outcome, decision confidence plays an essential role in both selecting and imple-
menting a decision. When implementing a decision, overconfidence can be beneficial in helping
overcome the inevitable setbacks and challenges. When selecting a decision, however, perfect cal-
ibration is preferred so that an alternative can be chosen untainted by unrealistic enthusiasm or
trepidation (Fazlollahi and Vadihov, 2001).
Evidence from the DSS literature suggests that existing DSS can produce “illusory benefits”
(Aldag and Powers, 1986; Davis et al., 1991) resulting in miscalibration, thereby distorting the
decision selection process. Almost three decades had passed when Keen and Scott Morton (1978,
p. 1342) recognized this and wrote, “Even though the . . . [decision aided] subjects [in their study]
did better, their increased average time and reduced average confidence lead to the tentative con-
clusion that they did not have a 'handle' on the problem.” By now, almost everyone can recount
from personal experience a situation where computer-generated output produced an aura of exact-
ness and reliance bordering on blind acceptance, even in the presence of compelling evidence to
the contrary. For many, if the “computer says it's so,” it is taken to be fact, even if the output flies
in the face of logic. In these cases, user miscalibration may be amplified by the design of the DSS
artifact. If DSS are to improve decision making, DSS designers and researchers must be con-
cerned with user calibration.
The goal of DSS dialog design is to architect a two-way exchange of symbols and actions that pro-
duce holistic performance—performance that exceeds the sum of the parts. If this level of performance
is to be achieved, a DSS must not only improve decision quality, it must also facilitate the decision
maker's interpretation of the quality of decisions made using the DSS technology.