Information Technology Reference
In-Depth Information
- proficient, able to intuitively eliminate plans which have no prospect of
succeeding;
- expert, who intuitively moves in a space of reasoning.
Providing the users with these definitions, thus enabling them to determine
their own level of expertise can be slightly complicated, and has its own risks.
One solution is to ask them questions directly, such as “did you recognize a
usual communication situation?” and then deduce, bit by bit, their level of
expertise.
In the case of professional users such as soldiers who know their task
perfectly and have a more or less in-depth knowledge of the system, the level
of expertise is a crucial piece of information. We could imagine an operator
with a status of expert for a voice-controlled system but a novice status for the
multimodal system devoted to the same task. The analogy with the number of
hours logged by a pilot is immediate, since it matches the same issue: instead
of, or in addition to, a label such as competent or expert, it is interesting to use
the number of hours that the user has spent with the system. For a plane as for
a dialogue system, this number of hours implicitly includes a training and
learning period, and a single measurement can be enough. If the time spent in
training is considered to be a relevant measurement, for example if the user
tests have two sets of subjects: one which has accelerated training and the
other in-depth training, we can either keep two indicators (two numbers of
hours) or carry out a calculation with different weighting for the training and
the subsequent use. According to the same principle, we can draw a
distinction between the time spent with the real system and with the Wizard
of Oz. The analogy then relies on the number of hours that a pilot spends in a
simulator and the level of expertise includes a new form of weighting. And
again, according to this principle, we can dissociate the time potentially spent
training the speech recognition engine, time which allowed the user to get a
precise idea of the operational sentences. Or consider that repeating prepared
utterances is a type of training, and thus include it in the training period.
In any case, it is possible to go beyond the novice-expert dichotomy, as
long as we know how a more precise piece of information can be used. Indeed,
usingseverallevelsofexpertiseallowsustocompareseveraluserswithsimilar
levels (equal in case of a discrete measure, in the same interval in case of a
continuous measure as with previous calculations), as well as several users
of varied levels. The dialogue system assessment in that case includes new
indicators, such as:
Search WWH ::
Custom Search