Tinner, F;
Adelani, DI;
Emezue, C;
Hajili, M;
Goldman, O;
Adilazuarda, MF;
Al Kautsar, MD;
... Ataman, D; + view all
(2023)
Findings of the 1st Shared Task on Multi-lingual Multi-task Information Retrieval at MRL 2023.
In:
Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL).
(pp. pp. 310-323).
Association for Computational Linguistics: Singapore.
Preview |
PDF
2023.mrl-1.24.pdf - Published Version Download (193kB) | Preview |
Abstract
Large language models (LLMs) excel in language understanding and generation, especially in English which has ample public benchmarks for various natural language processing (NLP) tasks. Nevertheless, their reliability across different languages and domains remains uncertain. Our new shared task introduces a novel benchmark to assess the ability of multilingual LLMs to comprehend and produce language under sparse settings, particularly in scenarios with under-resourced languages, with an emphasis on the ability to capture logical, factual, or causal relationships within lengthy text contexts. The shared task consists of two subtasks crucial to information retrieval: Named Entity Recognition (NER) and Reading Comprehension (RC), in 7 data-scarce languages: Azerbaijani, Igbo, Indonesian, Swiss German, Turkish, Uzbek and Yorùbá, which previously lacked annotated resources in information retrieval tasks. Our evaluation of leading LLMs reveals that, despite their competitive performance, they still have notable weaknesses such as producing output in the non-target language or providing counterfactual information that cannot be inferred from the context. As more advanced models emerge, the benchmark will remain essential for supporting fairness and applicability in information retrieval systems.
| Type: | Proceedings paper |
|---|---|
| Title: | Findings of the 1st Shared Task on Multi-lingual Multi-task Information Retrieval at MRL 2023 |
| Event: | 3rd Workshop on Multi-lingual Representation Learning (MRL) |
| ISBN-13: | 9798891760561 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://aclanthology.org/2023.mrl-1.24/ |
| Language: | English |
| Additional information: | ©2023 Association for Computational Linguistics. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10188838 |
Archive Staff Only
 |
View Item |
