From Annotation to Multimodal Behavior - Coverbal Synchrony in Human-Machine Interaction - page 207

(TRAINS, HCRC Map Task, Verbmobil, DIT, MUMIN, AMI, etc.) and
various initiatives towards domain-independent, inter-operative and
standardized sets of dialogue acts (DAMSL 1997; MATE 1999; DIT++
2005; LIRICS 2007), have produced the ISO DIS 24617-2 (January 2010)
standard for dialogue act annotation. This standard is accompanied
by the annotation language DiAML which supports abstract and
concrete syntax: semantics is defined for the abstract syntax in terms
of information-state update operators, whilst concrete syntax defines
XML representations. Similar standardization efforts are on-going
concerning purely linguistic annotations, and in the area of multimodal
annotations, the D-META Grand Challenge at ICMI 2012 (http://d-
meta.inrialpes.fr/) aimed at setting up a basis for comparison,
analysis, and further improvement of multimodal data annotations
and multimodal interactive systems.
2.2 Segmentation
Once an annotation element is defined, its temporal identification is
to be determined. This means that the annotator finds the element
they want to annotate, and establishes where the element starts and
ends. The exact start and end time may sometimes be difficult to
determine due to inaccuracy of the recording device, and there can
also be differences between the annotators' annotation accuracy. These
kinds of inaccuracies are usually taken care of by allowing a small
time-window within which the start and end times are considered
equivalent.
However, other segmentation incompatibilities are more serious in
that they may prevent direct comparison of the annotated data. One
such issue is the relation between the element's form and function.
For instance, gestures can be coded with features that describe their
function besides their shape and dynamics. Following Kendon (2005),
a gesture unit is identified with the help of the so-called gesture
phases: preparation, stroke, and retraction, but it may also contain
more specific phases if a detailed description is needed, or of a single
stroke span, if the occurrence of the gesture is important rather than
its internal structure.
Another issue, raised by the empirical studies, is the connection
of form and function: are they tightly connected so that a particular
gesture shape always corresponds to a specific function and vice
versa, or do they stand in a many-to-many relation. If the annotator
is expected to select gestures to be annotated only if the gesture has a
communicative function, not all gestures are annotated. For example,
a hand gesture related to lifting of a coffee cup may not be annotated,