Chien, Jen-Tzung;
Chen, Ming-Yen;
Xue, Jing-Hao;
(2023)
Learning Meta Soft Prompt for Few-Shot Language Models.
In:
2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC).
IEEE: Taipei, Taiwan.
Preview |
Text
APSIPA-ASC-2023-Jen-TzungChien-final.pdf - Accepted Version Download (571kB) | Preview |
Abstract
Prompt-based learning is powerful to utilize the large-scaled pre-trained language model (PLM) for language understanding where the input sentences are augmented by either adding the hard prompt using word tokens or the soft prompt in a form of trainable tokens. However, the learned soft prompt in training domain may not really help a frozen PLM to handle domain shift in test domain. This paper presents an approach to incorporate meta learning into domain adaptation to train new soft prompt which sufficiently generalizes the frozen PLM to a number of domains. The meta soft prompt is then developed for few-shot unsupervised domain adaptation where a frozen PLM can be quickly adapted to a target domain. This soft prompt is optimized according to meta learning where the domain adaptation loss and the prompt-based classification loss are jointly minimized. The experiments on multi-domain natural language understanding show the benefits of the proposed meta soft prompt in pre-trained language model by using BERT under the few-shot setting.
Type: | Proceedings paper |
---|---|
Title: | Learning Meta Soft Prompt for Few-Shot Language Models |
Event: | 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) |
Dates: | 31 Oct 2023 - 3 Nov 2023 |
Open access status: | An open access version is available from UCL Discovery |
DOI: | 10.1109/apsipaasc58517.2023.10317500 |
Publisher version: | http://dx.doi.org/10.1109/apsipaasc58517.2023.1031... |
Language: | English |
Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
Keywords: | Metalearning, Training, Adaptation models, Asia, Information processing, Natural language processing, Data models |
UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences > Dept of Statistical Science |
URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10183788 |
Archive Staff Only
![]() |
View Item |