Wang, Daphne; Sadrzadeh, Mehrnoosh; Stanojevic, Milos; Chow, Wing Yee; Breheny, Richard; (2024) How can large language models become more human? In: Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics. (pp. pp. 166-176). Association for Computational Linguistics (ACL)

Preview

Text 2024.cmcl-1.14.pdf - Published Version Download (533kB) | Preview

Abstract

Psycholinguistic experiments reveal that efficiency of human language use is founded on predictions at both syntactic and lexical levels. Previous models of human prediction exploiting LLMs have used an information theoretic measure called surprisal, with success on naturalistic text in a wide variety of languages, but under-performance on challenging text such as garden path sentences. This paper introduces a novel framework that combines the lexical predictions of an LLM with the syntactic structures provided by a dependency parser. The framework gives rise to an Incompatibility Fraction. When tested on two garden path datasets, it correlated well with human reading times, distinguished between easy and hard garden path, and outperformed surprisal.

Download activity - last month

Export as XMLCSVJSON

Download activity - last 12 months

Export as JSONXMLCSV

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item