QA4MRE 2011-2013: Overview of question answering for machine reading evaluation

Anselmo Peñas; Eduard Hovy; Pamela Forner; Álvaro Rodrigo; Richard Sutcliffe; Roser Morante

doi:10.1007/978-3-642-40802-1_29

QA4MRE 2011-2013: Overview of question answering for machine reading evaluation

Anselmo Peñas
, Eduard Hovy
, Pamela Forner
, Álvaro Rodrigo
, Richard Sutcliffe
, Roser Morante

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

This paper describes the methodology for testing the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. This was the attempt of the QA4MRE challenge which was run as a Lab at CLEF 2011-2013. The traditional QA task was replaced by a new Machine Reading task, whose intention was to ask questions that required a deep knowledge of individual short texts and in which systems were required to choose one answer, by analysing the corresponding test document in conjunction with background text collections provided by the organization. Four different tasks have been organized during these years: Main Task, Processing Modality and Negation for Machine Reading, Machine Reading of Biomedical Texts about Alzheimer's disease, and Entrance Exams. This paper describes their motivation, their goals, their methodology for preparing the data sets, their background collections, their metrics used for the evaluation, and the lessons learned along these three years.

Original language	English
Title of host publication	Information Access Evaluation
Subtitle of host publication	Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings
Pages	303-320
Number of pages	18
DOIs	https://doi.org/10.1007/978-3-642-40802-1_29
Publication status	Published - 2013
Externally published	Yes
Event	4th International Conference of the CLEF Initiative, CLEF 2013 - Valencia, Spain Duration: 23 Sep 2013 → 26 Sep 2013

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	8138 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	4th International Conference of the CLEF Initiative, CLEF 2013
Country/Territory	Spain
City	Valencia
Period	23/09/13 → 26/09/13

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 4 Quality Education

Access to Document

10.1007/978-3-642-40802-1_29

Cite this

Peñas, A., Hovy, E., Forner, P., Rodrigo, Á., Sutcliffe, R., & Morante, R. (2013). QA4MRE 2011-2013: Overview of question answering for machine reading evaluation. In Information Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings (pp. 303-320). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8138 LNCS). https://doi.org/10.1007/978-3-642-40802-1_29

Peñas, Anselmo ; Hovy, Eduard ; Forner, Pamela et al. / QA4MRE 2011-2013 : Overview of question answering for machine reading evaluation. Information Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings. 2013. pp. 303-320 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{0773bfe605c1400c969d666797799f09,

title = "QA4MRE 2011-2013: Overview of question answering for machine reading evaluation",

abstract = "This paper describes the methodology for testing the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. This was the attempt of the QA4MRE challenge which was run as a Lab at CLEF 2011-2013. The traditional QA task was replaced by a new Machine Reading task, whose intention was to ask questions that required a deep knowledge of individual short texts and in which systems were required to choose one answer, by analysing the corresponding test document in conjunction with background text collections provided by the organization. Four different tasks have been organized during these years: Main Task, Processing Modality and Negation for Machine Reading, Machine Reading of Biomedical Texts about Alzheimer's disease, and Entrance Exams. This paper describes their motivation, their goals, their methodology for preparing the data sets, their background collections, their metrics used for the evaluation, and the lessons learned along these three years.",

author = "Anselmo Pe{\~n}as and Eduard Hovy and Pamela Forner and {\'A}lvaro Rodrigo and Richard Sutcliffe and Roser Morante",

year = "2013",

doi = "10.1007/978-3-642-40802-1\_29",

language = "English",

isbn = "9783642408014",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

pages = "303--320",

booktitle = "Information Access Evaluation",

note = "4th International Conference of the CLEF Initiative, CLEF 2013 ; Conference date: 23-09-2013 Through 26-09-2013",

}

Peñas, A, Hovy, E, Forner, P, Rodrigo, Á, Sutcliffe, R & Morante, R 2013, QA4MRE 2011-2013: Overview of question answering for machine reading evaluation. in Information Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8138 LNCS, pp. 303-320, 4th International Conference of the CLEF Initiative, CLEF 2013, Valencia, Spain, 23/09/13. https://doi.org/10.1007/978-3-642-40802-1_29

QA4MRE 2011-2013: Overview of question answering for machine reading evaluation. / Peñas, Anselmo; Hovy, Eduard; Forner, Pamela et al.
Information Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings. 2013. p. 303-320 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8138 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - QA4MRE 2011-2013

T2 - 4th International Conference of the CLEF Initiative, CLEF 2013

AU - Peñas, Anselmo

AU - Hovy, Eduard

AU - Forner, Pamela

AU - Rodrigo, Álvaro

AU - Sutcliffe, Richard

AU - Morante, Roser

PY - 2013

Y1 - 2013

N2 - This paper describes the methodology for testing the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. This was the attempt of the QA4MRE challenge which was run as a Lab at CLEF 2011-2013. The traditional QA task was replaced by a new Machine Reading task, whose intention was to ask questions that required a deep knowledge of individual short texts and in which systems were required to choose one answer, by analysing the corresponding test document in conjunction with background text collections provided by the organization. Four different tasks have been organized during these years: Main Task, Processing Modality and Negation for Machine Reading, Machine Reading of Biomedical Texts about Alzheimer's disease, and Entrance Exams. This paper describes their motivation, their goals, their methodology for preparing the data sets, their background collections, their metrics used for the evaluation, and the lessons learned along these three years.

AB - This paper describes the methodology for testing the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. This was the attempt of the QA4MRE challenge which was run as a Lab at CLEF 2011-2013. The traditional QA task was replaced by a new Machine Reading task, whose intention was to ask questions that required a deep knowledge of individual short texts and in which systems were required to choose one answer, by analysing the corresponding test document in conjunction with background text collections provided by the organization. Four different tasks have been organized during these years: Main Task, Processing Modality and Negation for Machine Reading, Machine Reading of Biomedical Texts about Alzheimer's disease, and Entrance Exams. This paper describes their motivation, their goals, their methodology for preparing the data sets, their background collections, their metrics used for the evaluation, and the lessons learned along these three years.

UR - https://www.scopus.com/pages/publications/84886376548

U2 - 10.1007/978-3-642-40802-1_29

DO - 10.1007/978-3-642-40802-1_29

M3 - Conference contribution

AN - SCOPUS:84886376548

SN - 9783642408014

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 303

EP - 320

BT - Information Access Evaluation

Y2 - 23 September 2013 through 26 September 2013

ER -

Peñas A, Hovy E, Forner P, Rodrigo Á, Sutcliffe R, Morante R. QA4MRE 2011-2013: Overview of question answering for machine reading evaluation. In Information Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings. 2013. p. 303-320. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-642-40802-1_29

QA4MRE 2011-2013: Overview of question answering for machine reading evaluation

Abstract

Publication series

Conference

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this