Fu, X;
Lipani, A;
Kando, N;
(2024)
An Evaluation Framework for Conversational Information Retrieval Using User Simulation.
In: Acharya, Praveen and Clarke, Charles LA and Crestani, Fabio and Fu, Xiao and Jones, Gareth JF and Kando, Noriko and Kato, Makoto P and Lipani, Aldo and Liu, Yiqun, (eds.)
CEUR Workshop Proceedings.
CEUR-WS.org: Tokyo, Japan.
Preview |
Text
um-cir-1.pdf - Published Version Download (1MB) | Preview |
Abstract
Recent advancements in the field of Conversational Information Retrieval (CIR) have increased the demand for more sophisticated modelling and evaluation approaches. This paper introduces a novel framework for user simulation in CIR, aimed at enhancing the modeling and evaluation of user interactions. Additionally, this study explores the potential integration of large language models (LLMs) within this domain. Furthermore, the paper anticipates future developments in CIR, particularly in the context of the widespread use of LLMs. The study emphasizes the necessity for robust evaluation paradigms that go beyond traditional methods to effectively measure the success of CIR systems.
| Type: | Proceedings paper |
|---|---|
| Title: | An Evaluation Framework for Conversational Information Retrieval Using User Simulation |
| Event: | Joint Proceedings of the SIGIR-AP 2024 Workshops EMTCIR 2024 and UM-CIR 2024 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://ceur-ws.org/Vol-3854 |
| Language: | English |
| Additional information: | © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). |
| Keywords: | Conversation Information Retrieval, Evaluation, User Simulation, Large Language Models |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Civil, Environ and Geomatic Eng |
| URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10203025 |
Archive Staff Only
 |
View Item |
