Overview of ResPubliQA 2010: Question answering evaluation over European legislation

This paper describes the second round of ResPubliQA, a Question Answering (QA) evaluation task over European legislation, a LAB of CLEF 2010. Two tasks have been proposed this year: Paragraph Selection (PS) and Answer Selection (AS). The PS task consisted of extracting a relevant paragraph of text that satisfies completely the information need expressed by a natural language question. In the AS task, the exercise was to demarcate the shorter string of text corresponding to the exact answer supported by the entire paragraph. The general aims of this exercise are (i) to move towards a domain of potential users; (ii) to propose a setting which allows the direct comparison of performance across languages; (iii) to allow QA technologies to be evaluated against IR approaches; (iv) to promote validation technologies to reduce the amount of incorrect answers by leaving some questions unanswered. These goals are achieved through the use of parallel aligned document collections (JRC-Acquis and EUROPARL) and the possibility to return two different types of answers, either passages or exact strings. The paper describes the task in more detail, presenting the different types of questions, the methodology for the creation of the test sets and the evaluation measure, and analyzing the results obtained by systems and the more successful approaches. Thirteen groups participated in both PS and AS tasks submitting 49 runs in total.