Analyzing Personal Attribute Judgments - Modeling Users' Experiences with Interactive Systems - page 49

configuration space), they are grouped. Attributes that cannot be grouped (have
no replicates) are eliminated since no evidence exists that they contain reliable
information.

3.3.6 How Do the Diverse Views Compare to the Average View?

This question will be addressed in three ways. Firstly, it will be illustrated that the
average model predicts less than half of the attributes predicted by the three diverse
models together (attributes that are adequately explained by more than one model
are only counted once for the model that explains them best). Secondly, it will be
illustrated that, for the attributes that are predicted by the three diverse models, these
models provide a better fit than the average model, as demonstrated by the amount
of explained variance in the attribute data and the values of the well established
Akaike Information Criterion (AIC) for model selection. Thirdly, by exploring the
resulting views, it will be illustrated that the diverse models, not only account for
more attributes and with a better fit, but that they also result in semantically richer
insights, i.e., introduce more semantically different attributes.
Surprisingly enough, the average model could only predict 1/6th of all the at-
tributes from the ten participants, i.e. 18 out of the 118 attributes. This means, that
when deriving an average configuration to understand how individuals distinguish
between these websites, only 1/6th of the attributes are taken into account. This is
illustrated by the high correlation between the two resulting configurations ( R=.99 ),
the one derived using all 118 attributes and the one derived using only the 18 at-
tributes that are well predicted. Thus, the consequence of averaging is that we ac-
count only for 1/6th of the information available. The three diverse models predict
12, 10, and 16 attributes respectively (attributes predicted by more than one model
were excluded from the ones that displayed the least fit). Thus, by accounting for
diversity, even with our clearly heuristic and therefore sub-optimal procedure, we
account for more than double the number of attributes than in the case of the aver-
age model.
Table 3.5 illustrates the goodness of fit of the average and the three diverse models
for the 38 in total attributes resulting from models 1 to 3. As expected, a significant
increase in the accounted variance ( R 2 ) of the attribute data is observed as we the
move from the average to the specific (i.e. diverse) model. But, does this increase
in the goodness of fit of the model outweigh the increase in model complexity, i.e.
going from one 2D to three 2D models? One of the most widely used criteria for
model selection is the Akaike Information Criterion (AIC) (Burnham and Anderson,
2004) which is a function of the log likelihood value reflecting the goodness of fit
of the model and the M degrees of freedom in the model reflecting its complexity:

n

ˆ

AIC c = − 2log ( L ( θ ))+ 2 M (3.4)

n − M − 1

Burnham and Anderson (2004) proposed a set of heuristics when comparing the
AIC values of two models. Δ i reflects the difference between the AIC of the