UCL Discovery Stage
UCL home » Library Services » Electronic resources » UCL Discovery Stage

Learning Meta Soft Prompt for Few-Shot Language Models

Chien, Jen-Tzung; Chen, Ming-Yen; Xue, Jing-Hao; (2023) Learning Meta Soft Prompt for Few-Shot Language Models. In: 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE: Taipei, Taiwan. Green open access

[thumbnail of APSIPA-ASC-2023-Jen-TzungChien-final.pdf]
Preview
Text
APSIPA-ASC-2023-Jen-TzungChien-final.pdf - Accepted Version

Download (571kB) | Preview

Abstract

Prompt-based learning is powerful to utilize the large-scaled pre-trained language model (PLM) for language understanding where the input sentences are augmented by either adding the hard prompt using word tokens or the soft prompt in a form of trainable tokens. However, the learned soft prompt in training domain may not really help a frozen PLM to handle domain shift in test domain. This paper presents an approach to incorporate meta learning into domain adaptation to train new soft prompt which sufficiently generalizes the frozen PLM to a number of domains. The meta soft prompt is then developed for few-shot unsupervised domain adaptation where a frozen PLM can be quickly adapted to a target domain. This soft prompt is optimized according to meta learning where the domain adaptation loss and the prompt-based classification loss are jointly minimized. The experiments on multi-domain natural language understanding show the benefits of the proposed meta soft prompt in pre-trained language model by using BERT under the few-shot setting.

Type: Proceedings paper
Title: Learning Meta Soft Prompt for Few-Shot Language Models
Event: 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)
Dates: 31 Oct 2023 - 3 Nov 2023
Open access status: An open access version is available from UCL Discovery
DOI: 10.1109/apsipaasc58517.2023.10317500
Publisher version: http://dx.doi.org/10.1109/apsipaasc58517.2023.1031...
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: Metalearning, Training, Adaptation models, Asia, Information processing, Natural language processing, Data models
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences > Dept of Statistical Science
URI: https://discovery-pp.ucl.ac.uk/id/eprint/10183788
Downloads since deposit
1,476Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item