Analyzing Personal Attribute Judgments - Modeling Users' Experiences with Interactive Systems - page 54

Figure 3.6 illustrates a comparison of the performance of the heuristic technique
versus the algorithmic technique which explicitly optimizes the accounted variance
R 2 . As expected, the algorithmic techniques provides an improvement in terms of
the accounted variance R 2 - in total 109 out of the 118 (92%) meet the explicit

criterium R 2

.5 as compared to the 87 attributes (74%) of the heuristic technique.
Nevertheless, the technique provides no improvement in terms of R k (38 attributes
meet R k >

>

6), which is somewhat expected as it does not explicitly take this criterion
into account in the optimization process. Thus, an ideal technique, should take into
account both R 2 and R k in optimizing the goodness of fit of a model for a given set
of attributes. While R 2 reflects the amount of variance in the attribute's ratings ac-
counted for by the model, some attributes might contain only limited variance, and
thus display high R 2 while providing limited meaningful differentiation between the
stimuli. While our heuristic technique explicitly took into account the R k ,ithasa
number of limitations that could be addressed in the future. First, while R k was used
as a criterion in the determining whether an attribute is adequately predicted by a
given model, it was not employed in the Multi-Dimensional Scaling optimization
procedure for determining the best configuration of stimuli when trying to model
a set of attributes. Thus, the majority of the attributes resulted in low R k values.
Second, as R k is the range in the predicted attribute ratings divided by the estimated
noise in the attribute's ratings, the denominator, i.e. estimated noise, will obviously
have a significant impact on the R k , and thus may render it as unreliable if we can-
not reliably estimate the noise. In other words, if noise is unrealistically low, this
will result in unrealistically high values of R k . This issue pertains to the nature of
the dataset as they lack repeated measures, thus noise cannot be directly estimated
within a single participant and for a given attribute. Eliciting repeated measures in a
repertory grid study would add substantial effort which might render the technique
infeasible in several contexts. Figure 3.5b depicts the range

A k , max − A k , min versus the

estimated noise ( σ k ) for the 38 attributes that are adequately modeled by the heuris-

tic technique. One may note that for the majority of the attributes

σ k <

1, which is
rather unrealistic given the quantization error when transforming the discrete data
into continuous. Thus, an ideal technique should not allow for σ k < 1:

A k , max − A k , min

R k = (3.5)

min ( 1 , σ k )

Given the limitations of the proposed R k measure one might then question the valid-
ity of our initial result, that only 18 out of all 118 attributes are adequately modeled
by the averaging analysis, since R k was critical for determining goodness of fit. We
earlier noted however a high correlation (r=.99) of the two configuration spaces,
one attempting to model all 118 attributes and one attempting to model only the 18
best predicted attributes. This high correlation means that when excluding the 100
least-fit attributes, there was virtually no change in the MDS configuration space. In
other words, when including these 100 least-fit attributes in the MDS optimization
procedure, they did not contribute in the creation of this space. We also saw earlier
that these attributes were not substantially different from the 18 best-fit in terms of
the variance in their ratings. Thus, the averaging analysis results in a configuration