Chien, Jen-Tzung; Chen, Ming-Yen; Xue, Jing-Hao; (2023) Learning Meta Soft Prompt for Few-Shot Language Models. In: 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE: Taipei, Taiwan.

Preview

Text APSIPA-ASC-2023-Jen-TzungChien-final.pdf - Accepted Version Download (571kB) | Preview

Abstract

Prompt-based learning is powerful to utilize the large-scaled pre-trained language model (PLM) for language understanding where the input sentences are augmented by either adding the hard prompt using word tokens or the soft prompt in a form of trainable tokens. However, the learned soft prompt in training domain may not really help a frozen PLM to handle domain shift in test domain. This paper presents an approach to incorporate meta learning into domain adaptation to train new soft prompt which sufficiently generalizes the frozen PLM to a number of domains. The meta soft prompt is then developed for few-shot unsupervised domain adaptation where a frozen PLM can be quickly adapted to a target domain. This soft prompt is optimized according to meta learning where the domain adaptation loss and the prompt-based classification loss are jointly minimized. The experiments on multi-domain natural language understanding show the benefits of the proposed meta soft prompt in pre-trained language model by using BERT under the few-shot setting.

Download activity - last month

Export as XMLJSONCSV

Download activity - last 12 months

Export as CSVJSONXML

Downloads by country - last 12 months

Export as CSVJSONXML

Archive Staff Only

View Item