Yordanov, Yordan; Camburu, Oana-Maria; Kocijan, Vid; Lukasiewicz, Thomas; (2020) Does the Objective Matter? Comparing Training Objectives for Pronoun Resolution. In: Webber, Bonnie and Cohn, Trevor and He, Yulan and Liu, Yang, (eds.) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). (pp. pp. 4963-4969). Association for Computational Linguistics: Online.

Preview

Text obj.pdf - Published Version Download (254kB) | Preview

Abstract

Hard cases of pronoun resolution have been used as a long-standing benchmark for commonsense reasoning. In the recent literature, pre-trained language models have been used to obtain state-of-the-art results on pronoun resolution. Overall, four categories of training and evaluation objectives have been introduced. The variety of training datasets and pre-trained language models used in these works makes it unclear whether the choice of training objective is critical. In this work, we make a fair comparison of the performance and seed-wise stability of four models that represent the four categories of objectives. Our experiments show that the objective of sequence ranking performs the best in-domain, while the objective of semantic similarity between candidates and pronoun performs the best out-of-domain. We also observe a seed-wise instability of the model using sequence ranking, which is not the case when the other objectives are used.

Download activity - last month

Export as JSONXMLCSV

Download activity - last 12 months

Export as JSONXMLCSV

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item