Background: Corpora and Evaluation Methods - Methods for Mining and Summarizing Text Conversations - page 33

￿ The group presented their final budget .

The first sentence is from a system summary and the second sentence is from a gold-standard
human summary of the document. We can see that the bigram final budget occurs in each sentence,
so we say that there is a bi-gram overlap between this sentence and the gold-standard. If we permit
intervening terms between the words of the bigram, we can identify further overlaps, which are
called skip bigram overlaps. The following pair of sentences illustrates skip bigram overlap:

￿ So let's look at the final revised budget .

￿ The group presented their final budget .

Here, there is a skip bigram overlap, where one intervening term occurs between final and
budget .
Lin [ 2004 ] provided evidence that the ROUGE-2 and ROUGE-SU4 metrics correlate well
with human evaluations for several years' worth of DUC data. Subsequent research has yielded
mixed results concerning ROUGE correlations with human evaluations [ Dorr et al. , 2004 , 2005 ,
Liu and Liu , 2010 , Murray et al. , 2005b , 2006 ], but ROUGE has become an official metric of the
Text Analysis Conference and is increasingly relied upon by researchers, allowing them to directly
compare summarization results on given datasets.
The creators of ROUGE have also developed the Basic Elements evaluation suite [ Hovy et al. ,
2006 ], which attempts to remedy the drawbacks of relying on n-gram units or sentence units
for comparing machine summaries to reference summaries. Instead of relying on n-grams like Basic

Elements

ROUGE does, this evaluation framework uses units called Basic Elements, which are defined in the
most simple case as either heads of major syntactic constituents (a single item) or relations between
BE-heads and their modifiers (a triple of head, modifier, and relation). The head of a syntactic
constituent, e.g., a Noun Phrase (NP) is the word which determines the syntactic category of the
constituent and typically conveys the core meaning of the constituent. In contrast, the modifiers
of the head simply modify that meaning by specifying values for relevant semantic relations. For
instance, topic is the head of the NP An interesting book , while interesting is a modifier of the
head, which specifies the “quality” of the topic. Here, we show an example sentence and some
of the associated BE triples < head, modifier, relation > , with semantic relations (in uppercase)
determined by a semantic role labeler [ Hovy et al. , 2005 ]:

Two Libyans were indicted for the Lockerbie bombing in 1991.

For the NP Two Libyans , with head Libyans , we have the BE:

| |

￿ < Libyans two CARDINAL >

For the whole sentence, with head indicted , we have three BEs:

￿ < indicted | Libyans | ACCUSED >