Overview of QA4MRE main task at CLEF 2013

Richard Sutcliffe; Anselmo Peñas; Eduard Hovy; Pamela Forner; Álvaro Rodrigo; Corina Forascu; Yassine Benajiba; Petya Osenova

Overview of QA4MRE main task at CLEF 2013

Richard Sutcliffe
, Anselmo Peñas
, Eduard Hovy
, Pamela Forner
, Álvaro Rodrigo
, Corina Forascu
, Yassine Benajiba
, Petya Osenova

University of Essex
National Distance Education University
Carnegie Mellon University
CELCT
Alexandru Ioan Cuza University of Iaşi
Koninklijke Philips N.V.
Bulgarian Academy of Sciences

Research output: Contribution to journal › Conference article › peer-review

Abstract

This paper describes the Question Answering for Machine Reading (QA4MRE) Main Task at the 2013 Cross Language Evaluation Forum. In the main task, systems answered multiple-choice questions on documents concerned with four different topics. There were also two pilot tasks, Machine Reading on Biomedical Texts about Alzheimer's disease, and Japanese Entrance Exams. This paper describes the preparation of the data sets, the definition of the background collections, the metric used for the evaluation of the systems' submissions, and the results. We introduced two novelties this year: auxiliary questions to evaluate systems level of inference, and a portion of questions where none of the options were correct. Nineteen groups participated in the task submitting a total of 77 runs in five languages.

Original language	English
Journal	CEUR Workshop Proceedings
Volume	1179
Publication status	Published - 2013
Externally published	Yes
Event	2013 Cross Language Evaluation Forum Conference, CLEF 2013 - Valencia, Spain Duration: 23 Sept 2013 → 26 Sept 2013

Cite this

@article{d3e8bfbc46c44658be3f5f98bc98d4c9,

title = "Overview of QA4MRE main task at CLEF 2013",

abstract = "This paper describes the Question Answering for Machine Reading (QA4MRE) Main Task at the 2013 Cross Language Evaluation Forum. In the main task, systems answered multiple-choice questions on documents concerned with four different topics. There were also two pilot tasks, Machine Reading on Biomedical Texts about Alzheimer's disease, and Japanese Entrance Exams. This paper describes the preparation of the data sets, the definition of the background collections, the metric used for the evaluation of the systems' submissions, and the results. We introduced two novelties this year: auxiliary questions to evaluate systems level of inference, and a portion of questions where none of the options were correct. Nineteen groups participated in the task submitting a total of 77 runs in five languages.",

author = "Richard Sutcliffe and Anselmo Pe{\~n}as and Eduard Hovy and Pamela Forner and {\'A}lvaro Rodrigo and Corina Forascu and Yassine Benajiba and Petya Osenova",

year = "2013",

language = "English",

volume = "1179",

journal = "CEUR Workshop Proceedings",

issn = "1613-0073",

note = "2013 Cross Language Evaluation Forum Conference, CLEF 2013 ; Conference date: 23-09-2013 Through 26-09-2013",

}

TY - JOUR

T1 - Overview of QA4MRE main task at CLEF 2013

AU - Sutcliffe, Richard

AU - Peñas, Anselmo

AU - Hovy, Eduard

AU - Forner, Pamela

AU - Rodrigo, Álvaro

AU - Forascu, Corina

AU - Benajiba, Yassine

AU - Osenova, Petya

PY - 2013

Y1 - 2013

N2 - This paper describes the Question Answering for Machine Reading (QA4MRE) Main Task at the 2013 Cross Language Evaluation Forum. In the main task, systems answered multiple-choice questions on documents concerned with four different topics. There were also two pilot tasks, Machine Reading on Biomedical Texts about Alzheimer's disease, and Japanese Entrance Exams. This paper describes the preparation of the data sets, the definition of the background collections, the metric used for the evaluation of the systems' submissions, and the results. We introduced two novelties this year: auxiliary questions to evaluate systems level of inference, and a portion of questions where none of the options were correct. Nineteen groups participated in the task submitting a total of 77 runs in five languages.

AB - This paper describes the Question Answering for Machine Reading (QA4MRE) Main Task at the 2013 Cross Language Evaluation Forum. In the main task, systems answered multiple-choice questions on documents concerned with four different topics. There were also two pilot tasks, Machine Reading on Biomedical Texts about Alzheimer's disease, and Japanese Entrance Exams. This paper describes the preparation of the data sets, the definition of the background collections, the metric used for the evaluation of the systems' submissions, and the results. We introduced two novelties this year: auxiliary questions to evaluate systems level of inference, and a portion of questions where none of the options were correct. Nineteen groups participated in the task submitting a total of 77 runs in five languages.

UR - https://www.scopus.com/pages/publications/84922021564

M3 - Conference article

AN - SCOPUS:84922021564

SN - 1613-0073

VL - 1179

JO - CEUR Workshop Proceedings

JF - CEUR Workshop Proceedings

T2 - 2013 Cross Language Evaluation Forum Conference, CLEF 2013

Y2 - 23 September 2013 through 26 September 2013

ER -

Overview of QA4MRE main task at CLEF 2013

Abstract

Other files and links

Fingerprint

Cite this