Process and Pitfalls in Writing Information Visualization Research Papers - Information Visualization: Human-Centered Issues and Perspectives - page 138

has a discussion of the performance, flexibility, and extensibility implications of
the ItemRegistry, Action, and ActionList data structure choices. Another good
example is the systems paper on design choices made in Rivet [37] and other
systems with similar goals, such as the tradeoffs of data granularity for trans-
formations.
A systems paper can be considered as a specialized kind of design study:
one about the choices made when building a library as opposed to the choices
made when solving a visual encoding problem. Like the design study category,
key aspects of a systems paper are the lessons learned from building the system,
and observing its use. I urge authors and reviewers of systems papers to peruse
Levin and Redell's classic on “How (and How Not) to Write a Good Systems
Paper” [22].
The category name might be a cause for confusion because the the term
system is often used interchangeably with application or implementation. The
original intent was to follow the distributed systems usage where there is a very
strong distinction between system-level and application-level work. Although a
name like To o l k i t might avert that confusion, the term 'systems paper' is such
a strong convention in computer science that I am reluctant to advocate this
change.
2.5 Evaluation
Evaluation papers focus on assessing how an infovis system or technique is used
by some target population. Evaluation papers typically do not introduce new
techniques or algorithms, and often use implementations described in previous
work. The most common approach in infovis thus far has been formal user stud-
ies conducted in laboratory setting, using carefully abstracted tasks that can be
quantitatively measured in terms of time and accuracy, and analyzed with sta-
tistical methods. A typical claim would be that the tested tasks are ecologically
valid; that is, they correspond to those actually undertaken by target users in
a target domain. A typical result would be a statistically significant main ef-
fect of an experimental factor, or interaction effect between factors. The work of
Yost and North on perceptual scalability is a good example of this subtype [44].
A different approach to studying user behavior is field studies, where a system
is deployed in a real-world setting with its target users. In these studies, the
number of participants is usually smaller, with no attempt to achieve statis-
tical significance, and the time span is usually weeks or months rather than
hours. However, the study design axes of field versus laboratory, short-term ver-
sus long-term, and size are all orthogonal. Both quantitative and qualitative
measurements may be collected. For example, usage patterns may be studied
through quantitative logging of mouse actions or eyegaze. The work of Hornbæk
and Hertzum on untangling fisheye menus is a good example of this subtype
[16]. Usage patterns can also be studied through qualitative observations during
the test itself or later via coding of videotaped sessions. Trafton et al. 's field
study of how meteorologists use visual representations is an excellent example
of the power of video coding [39].