Evaluating Information Visualizations - Information Visualization: Human-Centered Issues and Perspectives - page 39

• Are the social relationships of the observer free from associations that might in-
duce bias?
• Does the report appear to be self-serving? Does it benefit the experimenter to the
extent that it should be questioned?
• Is the report internally consistent? Do the facts within the report support each
other?
• Is the report externally consistent? Do the facts in the report agree with other in-
dependent reports?
As a result it is important to be explicit about data collection methods, the position of
the researcher with respect to the subject matter, analysis processes, and codes. These
details make it possible for other researchers to verify results.
In qualitative research it is acknowledged that the researcher's views, research con-
text, and interpretations are an essential part of the qualitative research method as
long as they are grounded in the collected data [3]. This does not, however, mean that
qualitative evaluations are less trustworthy compared to quantitative research. Auer-
bach suggests using the concept of 'transferability' rather than 'generalizability' when
thinking about the concepts of reliability and validity in qualitative research [3]. It is
more important that the theoretical understanding we have gained can also be found in
other research situations or systems and can be extended and developed further when
applied to other scenarios. This stands in contrast to the concept of generalizability in
quantitative research that wants to prove statistically that the results are universally
applicable within the population under study.
Sometimes the point has been raised that if results do not generalize how can they
be of use when designing software for general use. For example, qualitative methods
might be used to obtain a rich description of a particular situation perhaps only ob-
serving the processes of two or three people. The results of a study like this may or
may not generalize and the study itself provides no proof that they do. What we have
is existence proof: that such processes are in use in at least two or three instances.
Consider the worst case; that is that this rich description is an outlier that occurs only
rarely. For design purposes, outliers are also important and sensitive design for out-
liers has been often shown to create better designs for all. For example, motion sen-
sors to open doors may have been designed for wheelchairs but actually are useful
features for all.
5.3.3 Analyzing Qualitative Data
Qualitative data may be analyzed using qualitative, quantitative, or a combination of
both methods. Mixed methods research includes a qualitative phase and a quantitative
phase in the overall research study in order to triangulate results from different meth-
ods, to complement results from one method with another, or to increase the breadth
and range of inquiry by using different methods [28].
Many of the qualitative analysis methods can be grouped as types of thematic
analysis, in which analysis starts from observations, then themes are sensed through
review of the data, and finally coded [10]. Coding is the process of subdividing and
labeling raw data, then reintegrating collected codes to form a theory [70]. Moving
from the raw data into themes and a code set may proceed using one of three ap-