Multicriteria Information Retrieval in Textual Corpora - Geographical Information Retrieval in Textual Corpora - page 83

Figure 3.12. Comparison of the PIV 3 CombMNZ and PIV 3 CMRP IRSs
3.5.3.3. Summary of the evaluation of PIV 3
The testing of PIV 3 endowed with the CombMNZ model, applied to the
MIDR_2010 test collection for the multicriteria GIR, validates our starting
hypothesis: “a totally compensatory aggregation of results (CombMNZ) from spatial,
temporal and thematic IRSs gives better results than a single thematic IRS or
two-by-two pairings of spatial, temporal and thematic IRSs” [PAL 12a].
The testing of PIV 3 endowed with the CMRP model, applied to the MIDR_2010
test collection for the multicriteria GIR, validates our second starting hypothesis: “a
partially compensatory aggregation of results from spatial, temporal and thematic
IRSs, offering more expressive power to the user, gives better results than a totally
compensatory aggregation” [PAL 12a]. The CMRP model allows the user to
associate specific roles with each search criterion, which explains the higher quality
results [PAL 12a].
Initscurrentstate,theproposedexperimentpresentsatleasttwolimits.First,with
its5,645paragraphstotaling3.7MB,theMIDR_2010testcollectionisofaverysmall
volume in comparison to the T REC collections. Second, we have performed analyzes
from 44 then from 10 topics. We continue the efforts of manual assessment of the
documents allowing us to obtain more topics to analyze.
Furthermore, we have also analyzed the contribution of expressive queries
integrating modal operators of requirement and preference in a single thematic
context. We have mainly studied the TREC-8 [VOO 99] collection to measure the
potential of improving the precision brought by the use of operators. The use of
different operators has been studied in [SIL 99], [SPI 01], [EAS 03], [WHI 07] and
[AUL 10] by analyzing the logs of such search engines as Altavista, Excite, Google,
MSN Search and Yahoo!. Globally, these studies have shown that only 20% of all
queries contain operators and that the use of operators improves relatively little the
quality of the results. Thus, independently of the users' profile, we have studied the
expected potential of improvement in a context of optimal usage of operators
[PAL 10b, HUB 11]. We have thus shown that the expressive topics prevail over the
standard topics in 68% of the cases. When the terms of an expressive topic are well
chosen, there is always a multicriteria query that could improve the results of the
search [PAL 10c].