Utility-Based Information Distillation - Text Mining: Classification, Clustering, and Applications - page 232

ranking, or ask the system to reveal the documents that were suppressed by
the system due to their redundancy to the current document they are viewing.
9.7 Concluding Remarks
In this chapter we presented the first investigation on utility-based
information distillation with a system that learns long-lasting information
needs from fine-grained user feedback over a sequence of ranked lists.
We focused on how to combine adaptive filtering, novelty detection, anti-
redundancy ranking and fine-grained feedback in a unified framework for
utility optimization. We developed a new scheme for automated evaluation
of such a system with simulated user feedback, which consists of 1) a semi-
automatic procedure for acquiring rules that allow automatically matching
nuggets against system responses, and 2) a modified NDCG metric for
assessing the utility of ranked passages as a weighted combination of relevance
and novelty. The importance of utility-based information dislillation is that
it combines relevance with novelty in a user-centric adaptive system.
Open challenges we have not included here but plan to address in future
work include: 1) modeling user's behavior (with uncertainty) in browsing
ranked lists as an extension of our current utility optimzation framework,
and 2) dynamic thresholding on system-produced ranked lists for utility
optimization over iterative user-system interactions.
Evaluation of utility-based information distillation with true users is
another important subject we did not include due to the space limitation.
Some work on distillation evaluation with real users is reported in a separate
paper (10).
9.8 Acknowledgments
Contributers to the presented work include Ni Lao, Abhay Harpale,
Bryan Kisiel, Monica Rogati, Jian Zhang and Jaime Carbonell at the
Carnegie Mellon University who participated in the method design, system
implementation and/or automated evaluations of the CAFE system, and Peter
Brusilowsky, Daqing He, Rosta Farzan, Jonathan Grady, Jaewook Ahn, and
Yefei Peng at the University of Pittsburgh who colloborated in generating
the extended TDT4 annotations and conducted user studies with CAFE .This
work is supported in parts by Defense Advanced Research Project Agency
(DARPA) under contracts NBCHD030010 and W0550432, and the National