Overview of the CLEF 2004 multilingual question answering track

Bernardo Magnini; Alessandro Vallin; Christelle Ayache; Gregor Erbach; Anselmo Peñas; Maarten De Rijke; Paulo Rocha; Kiril Simov; Richard Sutcliffe

doi:10.1007/11519645_38

Overview of the CLEF 2004 multilingual question answering track

Bernardo Magnini
, Alessandro Vallin
, Christelle Ayache
, Gregor Erbach
, Anselmo Peñas
, Maarten De Rijke
, Paulo Rocha
, Kiril Simov
, Richard Sutcliffe

ITC-IRST
ELDA/ELRA
German Research Center for Artificial Intelligence
National Distance Education University
University of Amsterdam
University of Minho
Bulgarian Academy of Sciences

Research output: Contribution to journal › Conference article › peer-review

Abstract

Following the pilot Question Answering Track at CLEF 2003, a new evaluation exercise for multilingual QA systems took place in 2004. This paper reports on the novelties introduced in the new campaign and on participants' results. Almost all the cross-language combinations between nine source languages and seven target languages were exploited to set up more than fifty different tasks, both monolingual and bilingual. New types of questions (How-questions and definition questions) were given as input to the participating systems, while just one exact answer per question was allowed as output. The evaluation exercise has highlighted some difficulties in assessing definition questions and can be improved in the future, but the overall analysis of submissions shows encouraging results.

Original language	English
Pages (from-to)	371-391
Number of pages	21
Journal	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	3491
DOIs	https://doi.org/10.1007/11519645_38
Publication status	Published - 2005
Event	5th Workshop of the Cross-Language Evaluation Forum, CLEF 2004: Multilingual Information Access for Text, Speech and Images - Bath, United Kingdom Duration: 15 Sep 2004 → 17 Sep 2004

Access to Document

10.1007/11519645_38

Cite this

Magnini, B., Vallin, A., Ayache, C., Erbach, G., Peñas, A., De Rijke, M., Rocha, P., Simov, K., & Sutcliffe, R. (2005). Overview of the CLEF 2004 multilingual question answering track. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 3491, 371-391. https://doi.org/10.1007/11519645_38

@article{897db24d546f4a5bbbb2cbb69a8123b6,

title = "Overview of the CLEF 2004 multilingual question answering track",

abstract = "Following the pilot Question Answering Track at CLEF 2003, a new evaluation exercise for multilingual QA systems took place in 2004. This paper reports on the novelties introduced in the new campaign and on participants' results. Almost all the cross-language combinations between nine source languages and seven target languages were exploited to set up more than fifty different tasks, both monolingual and bilingual. New types of questions (How-questions and definition questions) were given as input to the participating systems, while just one exact answer per question was allowed as output. The evaluation exercise has highlighted some difficulties in assessing definition questions and can be improved in the future, but the overall analysis of submissions shows encouraging results.",

author = "Bernardo Magnini and Alessandro Vallin and Christelle Ayache and Gregor Erbach and Anselmo Pe{\~n}as and \{De Rijke\}, Maarten and Paulo Rocha and Kiril Simov and Richard Sutcliffe",

year = "2005",

doi = "10.1007/11519645\_38",

language = "English",

volume = "3491",

pages = "371--391",

journal = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

issn = "0302-9743",

note = "5th Workshop of the Cross-Language Evaluation Forum, CLEF 2004: Multilingual Information Access for Text, Speech and Images ; Conference date: 15-09-2004 Through 17-09-2004",

}

Overview of the CLEF 2004 multilingual question answering track. / Magnini, Bernardo; Vallin, Alessandro; Ayache, Christelle et al.
In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 3491, 2005, p. 371-391.

Research output: Contribution to journal › Conference article › peer-review

TY - JOUR

T1 - Overview of the CLEF 2004 multilingual question answering track

AU - Magnini, Bernardo

AU - Vallin, Alessandro

AU - Ayache, Christelle

AU - Erbach, Gregor

AU - Peñas, Anselmo

AU - De Rijke, Maarten

AU - Rocha, Paulo

AU - Simov, Kiril

AU - Sutcliffe, Richard

PY - 2005

Y1 - 2005

N2 - Following the pilot Question Answering Track at CLEF 2003, a new evaluation exercise for multilingual QA systems took place in 2004. This paper reports on the novelties introduced in the new campaign and on participants' results. Almost all the cross-language combinations between nine source languages and seven target languages were exploited to set up more than fifty different tasks, both monolingual and bilingual. New types of questions (How-questions and definition questions) were given as input to the participating systems, while just one exact answer per question was allowed as output. The evaluation exercise has highlighted some difficulties in assessing definition questions and can be improved in the future, but the overall analysis of submissions shows encouraging results.

AB - Following the pilot Question Answering Track at CLEF 2003, a new evaluation exercise for multilingual QA systems took place in 2004. This paper reports on the novelties introduced in the new campaign and on participants' results. Almost all the cross-language combinations between nine source languages and seven target languages were exploited to set up more than fifty different tasks, both monolingual and bilingual. New types of questions (How-questions and definition questions) were given as input to the participating systems, while just one exact answer per question was allowed as output. The evaluation exercise has highlighted some difficulties in assessing definition questions and can be improved in the future, but the overall analysis of submissions shows encouraging results.

UR - https://www.scopus.com/pages/publications/24944457277

U2 - 10.1007/11519645_38

DO - 10.1007/11519645_38

M3 - Conference article

AN - SCOPUS:24944457277

SN - 0302-9743

VL - 3491

SP - 371

EP - 391

JO - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

JF - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

T2 - 5th Workshop of the Cross-Language Evaluation Forum, CLEF 2004: Multilingual Information Access for Text, Speech and Images

Y2 - 15 September 2004 through 17 September 2004

ER -

Overview of the CLEF 2004 multilingual question answering track

Abstract

Access to Document

Other files and links

Fingerprint

Cite this