Background: Corpora and Evaluation Methods - Methods for Mining and Summarizing Text Conversations - page 25

three sections could be empty; for example, some meetings might not involve any decisions being
made. Annotators were encouraged to listen to a meeting in its entirety before beginning to compose
the summary.
After authoring the abstractive summary, annotators were subsequently asked to create an
extractive summary, using a second GUI. As described in Chapter 1 , an extractive summary is
comprised of sentences, or, in this case, dialogue acts, taken from the document. With this GUI
they were able to view their abstract summary and the transcript of the meeting, with the topic
segments removed and with one dialogue act per line. They were given the pre-existing dialogue
act coding [ Shriberg et al. , 2004 ] and viewed only the dialogue act segments without the dialogue
act type labels. They were instructed to extract the dialogue acts that taken together could best
convey the information in the abstractive summary and could be used to support the correctness
of that abstract. They were not given any specific instructions or limitations about the number or
percentage of dialogue acts to extract, nor any instructions about extracting redundant dialogue acts.
They were then required to do a second pass of annotations, wherein for each extracted dialogue
act they chose the abstract sentences supported by that dialogue act. The result is a many-to-many
mapping between abstract sentences and extracted dialogue acts; that is, an abstract sentence can
be linked to more than one dialogue act, and vice versa. Although the expectation was that each
abstract sentence would be linked to at least one extracted dialogue act and each extracted dialogue
act linked to at least one abstract sentence, annotators were allowed to leave abstract sentences and
dialogue acts standing alone in some circumstances.
In addition to the annotation just described, Wilson [ 2008 ] annotated 20 AMI meetings for a
variety of subjective phenomena at the dialogue act level which fall into the broad classes of subjective
utterances , objective polar utterances and subjective questions . Two subclasses of subjective utterances are
positive subjective and negative subjective utterances. Such subjective utterances involve the expression
of a private state [ Quirk et al. , 1985 ] (an emotion or state of mind that is not always observable),
such as a positive/negative opinion, positive/negative argument, and agreement/disagreement. An
objective polar utterance is one that conveys positive or negative facts without indicating any private
state, e.g., “The camera broke the first time I used it” is a negative polar utterance [ Wilson , 2008 ].
Subjective questions are when a speaker enquires about the opinions or feelings of another person.
The 20 meetings were labeled by a single annotator, although Wilson [ 2008 ] did conduct a study of
annotator agreement on two meetings, reporting a κ of 0.56 for detecting subjective utterances.
2.1.2 EMAIL CORPORA
While emails have been a popular domain for mining and summarization work, a potential research
bottleneck is finding a sufficient amount of available email data. Research groups may have their
own email corpora, but privacy concerns often preclude their general release. However, in recent
years, two email corpora have been made publicly available and numerous researchers are now often
working on the same datasets, making their results easier to compare.