Vu, Thuy-Trang;
He, Xuanli;
Haffari, Gholamreza;
Shareghi, Ehsan;
(2023)
Koala: An Index for Quantifying Overlaps with Pre-training Corpora.
In: Feng, Yansong and Lefever, Els, (eds.)
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.
(pp. pp. 90-98).
Association for Computational Linguistics: Singapore, Singapore.
Preview |
PDF
2023.emnlp-demo.7.pdf - Published Version Download (374kB) | Preview |
Abstract
In very recent years more attention has been placed on probing the role of pre-training data in Large Language Models (LLMs) downstream behaviour. Despite the importance, there is no public tool that supports such analysis of pre-training corpora at large scale. To help research in this space, we launch Koala, a searchable index over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. In its first release we index the public proportion of OPT 175B, GPT-3, GPT-Neo, GPT-Neo, LLaMA, BERT, ELECTRA, RoBERTA, XLNet pre-training corpora. Koala provides a framework to do forensic analysis on the current and future benchmarks as well as to assess the degree of memorization in the output from the LLMs. Koala is available for public use at https://koala-index.erc.monash.edu/.
| Type: | Proceedings paper |
|---|---|
| Title: | Koala: An Index for Quantifying Overlaps with Pre-training Corpora |
| Event: | 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.18653/v1/2023.emnlp-demo.7 |
| Publisher version: | https://doi.org/10.18653/v1/2023.emnlp-demo.7 |
| Language: | English |
| Additional information: | © The Author(s), 2023. This is an Open Access article distributed under the terms of the Creative Commons Attribution Licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. https://creativecommons.org/licenses/by/4.0/ |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10188441 |
Archive Staff Only
 |
View Item |
