Database Reference
In-Depth Information
the World, etc.) published between October 2000 and January 2001, in three
languages - Arabic, English and Mandarin. Speech-recognized and machine-
translated versions of the non-English articles were provided as well.
LDC (21) has annotated the corpus with 100 topics, that correspond
to various news events in this time period. Out of these, we selected
a subset of 12 actionable events, and defined corresponding tasks for
them. 2 For the Texas prison break event, for example, we defined a hypo-
thetical task - ' Find information about the escape of convicts from
Texas prison, and information related to their recapture '. For each
task, we manually defined a profile consisting of an initial set of (5 to 10)
queries (e.g. 'number of escaped convicts,' 'their last known locations,'
'actions taken by police so far,' etc.), a free-text description of the user
history, i.e., what the user already knows about this event that should not be
repeated by the system, and a list of known on-topic and off-topic documents
(if available) as training examples.
For each query, we generated answer keys and corresponding nugget
matching rules using the procedure described in Section 9.4.1.2.
Thus we
had a total of 120 queries, with an average of 7 nuggets per query.
9.6 Experiments and Results
9.6.1 Baselines
We used Indri (20), a popular language-model based retrieval engine, as
a baseline for comparison with our system. Indri supports standard search
engine functionality, including pseudo-relevance feedback (PRF) (4; 7), and
is representative of a typical query-based retrieval system.
Indri does not
support any kind of novelty detection.
We compare Indri (System A) with PRF turned on and off, against our
system (system B) with user feedback, novelty detection and anti-redundant
ranking turned on and off.
9.6.2 Experimental Setup
We divided the TDT4 corpus spanning 4 months into 10 chunks ,each
defined as a period of 12 consecutive days. At any given point of time in
the distillation process, each system accesses the past data up to the current
point, and produces a ranked list of up 50 passages per query.
The 12 tasks defined on the corpus were divided into a training and test
2 URL: http://nyc.lti.cs.cmu.edu/downloads
 
Search WWH ::




Custom Search