Evaluating machine reading systems through comprehension tests

Anselmo Pens; Eduard Hovy; Pamela Forner; Alvaro Rodrigo; Richard Sutcliffe; Corina Forascu; Caroline Sporleder

Evaluating machine reading systems through comprehension tests

Anselmo Pens
, Eduard Hovy
, Pamela Forner
, Alvaro Rodrigo
, Richard Sutcliffe
, Corina Forascu
, Caroline Sporleder

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

This paper describes a methodology for testing and evaluating the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. The methodology is being used in QA4MRE (QA for Machine Reading Evaluation), one of the labs of CLEF. We report here the conclusions and lessons learned after the first campaign in 2011.

Original language	English
Title of host publication	Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012
Editors	Mehmet Ugur Dogan, Joseph Mariani, Asuncion Moreno, Sara Goggi, Khalid Choukri, Nicoletta Calzolari, Jan Odijk, Thierry Declerck, Bente Maegaard, Stelios Piperidis, Helene Mazo, Olivier Hamon
Publisher	European Language Resources Association (ELRA)
Pages	1143-1147
Number of pages	5
ISBN (Electronic)	9782951740877
Publication status	Published - 2012
Event	8th International Conference on Language Resources and Evaluation, LREC 2012 - Istanbul, Turkey Duration: 21 May 2012 → 27 May 2012

Publication series

Name	Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012

Conference

Conference	8th International Conference on Language Resources and Evaluation, LREC 2012
Country/Territory	Turkey
City	Istanbul
Period	21/05/12 → 27/05/12

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 4 Quality Education

Keywords

Evaluation
Machine Reading
Question Answering

Cite this

Pens, A., Hovy, E., Forner, P., Rodrigo, A., Sutcliffe, R., Forascu, C., & Sporleder, C. (2012). Evaluating machine reading systems through comprehension tests. In M. U. Dogan, J. Mariani, A. Moreno, S. Goggi, K. Choukri, N. Calzolari, J. Odijk, T. Declerck, B. Maegaard, S. Piperidis, H. Mazo, & O. Hamon (Eds.), Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012 (pp. 1143-1147). (Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012). European Language Resources Association (ELRA).

Pens, Anselmo ; Hovy, Eduard ; Forner, Pamela et al. / Evaluating machine reading systems through comprehension tests. Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012. editor / Mehmet Ugur Dogan ; Joseph Mariani ; Asuncion Moreno ; Sara Goggi ; Khalid Choukri ; Nicoletta Calzolari ; Jan Odijk ; Thierry Declerck ; Bente Maegaard ; Stelios Piperidis ; Helene Mazo ; Olivier Hamon. European Language Resources Association (ELRA), 2012. pp. 1143-1147 (Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012).

@inproceedings{58cec6e769c64fde941501aad34d6ccb,

title = "Evaluating machine reading systems through comprehension tests",

abstract = "This paper describes a methodology for testing and evaluating the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. The methodology is being used in QA4MRE (QA for Machine Reading Evaluation), one of the labs of CLEF. We report here the conclusions and lessons learned after the first campaign in 2011.",

keywords = "Evaluation, Machine Reading, Question Answering",

author = "Anselmo Pens and Eduard Hovy and Pamela Forner and Alvaro Rodrigo and Richard Sutcliffe and Corina Forascu and Caroline Sporleder",

year = "2012",

language = "English",

series = "Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012",

publisher = "European Language Resources Association (ELRA)",

pages = "1143--1147",

editor = "Dogan, \{Mehmet Ugur\} and Joseph Mariani and Asuncion Moreno and Sara Goggi and Khalid Choukri and Nicoletta Calzolari and Jan Odijk and Thierry Declerck and Bente Maegaard and Stelios Piperidis and Helene Mazo and Olivier Hamon",

booktitle = "Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012",

note = "8th International Conference on Language Resources and Evaluation, LREC 2012 ; Conference date: 21-05-2012 Through 27-05-2012",

}

Pens, A, Hovy, E, Forner, P, Rodrigo, A, Sutcliffe, R, Forascu, C & Sporleder, C 2012, Evaluating machine reading systems through comprehension tests. in MU Dogan, J Mariani, A Moreno, S Goggi, K Choukri, N Calzolari, J Odijk, T Declerck, B Maegaard, S Piperidis, H Mazo & O Hamon (eds), Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012. Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012, European Language Resources Association (ELRA), pp. 1143-1147, 8th International Conference on Language Resources and Evaluation, LREC 2012, Istanbul, Turkey, 21/05/12.

Evaluating machine reading systems through comprehension tests. / Pens, Anselmo; Hovy, Eduard; Forner, Pamela et al.
Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012. ed. / Mehmet Ugur Dogan; Joseph Mariani; Asuncion Moreno; Sara Goggi; Khalid Choukri; Nicoletta Calzolari; Jan Odijk; Thierry Declerck; Bente Maegaard; Stelios Piperidis; Helene Mazo; Olivier Hamon. European Language Resources Association (ELRA), 2012. p. 1143-1147 (Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Evaluating machine reading systems through comprehension tests

AU - Pens, Anselmo

AU - Hovy, Eduard

AU - Forner, Pamela

AU - Rodrigo, Alvaro

AU - Sutcliffe, Richard

AU - Forascu, Corina

AU - Sporleder, Caroline

PY - 2012

Y1 - 2012

N2 - This paper describes a methodology for testing and evaluating the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. The methodology is being used in QA4MRE (QA for Machine Reading Evaluation), one of the labs of CLEF. We report here the conclusions and lessons learned after the first campaign in 2011.

AB - This paper describes a methodology for testing and evaluating the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. The methodology is being used in QA4MRE (QA for Machine Reading Evaluation), one of the labs of CLEF. We report here the conclusions and lessons learned after the first campaign in 2011.

KW - Evaluation

KW - Machine Reading

KW - Question Answering

UR - https://www.scopus.com/pages/publications/84979830073

M3 - Conference contribution

AN - SCOPUS:84979830073

T3 - Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012

SP - 1143

EP - 1147

BT - Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012

A2 - Dogan, Mehmet Ugur

A2 - Mariani, Joseph

A2 - Moreno, Asuncion

A2 - Goggi, Sara

A2 - Choukri, Khalid

A2 - Calzolari, Nicoletta

A2 - Odijk, Jan

A2 - Declerck, Thierry

A2 - Maegaard, Bente

A2 - Piperidis, Stelios

A2 - Mazo, Helene

A2 - Hamon, Olivier

PB - European Language Resources Association (ELRA)

T2 - 8th International Conference on Language Resources and Evaluation, LREC 2012

Y2 - 21 May 2012 through 27 May 2012

ER -

Pens A, Hovy E, Forner P, Rodrigo A, Sutcliffe R, Forascu C et al. Evaluating machine reading systems through comprehension tests. In Dogan MU, Mariani J, Moreno A, Goggi S, Choukri K, Calzolari N, Odijk J, Declerck T, Maegaard B, Piperidis S, Mazo H, Hamon O, editors, Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012. European Language Resources Association (ELRA). 2012. p. 1143-1147. (Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012).

Evaluating machine reading systems through comprehension tests

Abstract

Publication series

Conference

UN SDGs

Keywords

Other files and links

Fingerprint

Cite this