UCL Discovery Stage
UCL home » Library Services » Electronic resources » UCL Discovery Stage

An Evaluation Framework for Conversational Information Retrieval Using User Simulation

Fu, X; Lipani, A; Kando, N; (2024) An Evaluation Framework for Conversational Information Retrieval Using User Simulation. In: Acharya, Praveen and Clarke, Charles LA and Crestani, Fabio and Fu, Xiao and Jones, Gareth JF and Kando, Noriko and Kato, Makoto P and Lipani, Aldo and Liu, Yiqun, (eds.) CEUR Workshop Proceedings. CEUR-WS.org: Tokyo, Japan. Green open access

[thumbnail of um-cir-1.pdf]
Preview
Text
um-cir-1.pdf - Published Version

Download (1MB) | Preview

Abstract

Recent advancements in the field of Conversational Information Retrieval (CIR) have increased the demand for more sophisticated modelling and evaluation approaches. This paper introduces a novel framework for user simulation in CIR, aimed at enhancing the modeling and evaluation of user interactions. Additionally, this study explores the potential integration of large language models (LLMs) within this domain. Furthermore, the paper anticipates future developments in CIR, particularly in the context of the widespread use of LLMs. The study emphasizes the necessity for robust evaluation paradigms that go beyond traditional methods to effectively measure the success of CIR systems.

Type: Proceedings paper
Title: An Evaluation Framework for Conversational Information Retrieval Using User Simulation
Event: Joint Proceedings of the SIGIR-AP 2024 Workshops EMTCIR 2024 and UM-CIR 2024
Open access status: An open access version is available from UCL Discovery
Publisher version: https://ceur-ws.org/Vol-3854
Language: English
Additional information: © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Keywords: Conversation Information Retrieval, Evaluation, User Simulation, Large Language Models
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Civil, Environ and Geomatic Eng
URI: https://discovery-pp.ucl.ac.uk/id/eprint/10203025
Downloads since deposit
288Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item