QA4MRE 2011-2013: Overview of question answering for machine reading evaluation

Anselmo Peñas, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Richard Sutcliffe, Roser Morante

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

This paper describes the methodology for testing the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. This was the attempt of the QA4MRE challenge which was run as a Lab at CLEF 2011-2013. The traditional QA task was replaced by a new Machine Reading task, whose intention was to ask questions that required a deep knowledge of individual short texts and in which systems were required to choose one answer, by analysing the corresponding test document in conjunction with background text collections provided by the organization. Four different tasks have been organized during these years: Main Task, Processing Modality and Negation for Machine Reading, Machine Reading of Biomedical Texts about Alzheimer's disease, and Entrance Exams. This paper describes their motivation, their goals, their methodology for preparing the data sets, their background collections, their metrics used for the evaluation, and the lessons learned along these three years.

Original languageEnglish
Title of host publicationInformation Access Evaluation
Subtitle of host publicationMultilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings
Pages303-320
Number of pages18
DOIs
Publication statusPublished - 2013
Externally publishedYes
Event4th International Conference of the CLEF Initiative, CLEF 2013 - Valencia, Spain
Duration: 23 Sep 201326 Sep 2013

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8138 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference4th International Conference of the CLEF Initiative, CLEF 2013
Country/TerritorySpain
CityValencia
Period23/09/1326/09/13

Fingerprint

Dive into the research topics of 'QA4MRE 2011-2013: Overview of question answering for machine reading evaluation'. Together they form a unique fingerprint.

Cite this