UCL Discovery Stage
UCL home » Library Services » Electronic resources » UCL Discovery Stage

Mutual-learning sequence-level knowledge distillation for automatic speech recognition

Li, Z; Ming, Y; Yang, L; Xue, J-H; (2021) Mutual-learning sequence-level knowledge distillation for automatic speech recognition. Neurocomputing , 428 pp. 259-267. 10.1016/j.neucom.2020.11.025. Green open access

[thumbnail of NEUCOM-ZeruiLi-accepted.pdf]
Preview
Text
NEUCOM-ZeruiLi-accepted.pdf - Accepted Version

Download (5MB) | Preview

Abstract

Automatic speech recognition (ASR) is a crucial technology for man-machine interaction. End-to-end models have been studied recently in deep learning for ASR. However, these models are not suitable for the practical application of ASR due to their large model sizes and computation costs. To address this issue, we propose a novel mutual-learning sequence-level knowledge distillation framework enjoying distinct student structures for ASR. Trained mutually and simultaneously, each student learns not only from the pre-trained teacher but also from its distinct peers, which can improve the generalization capability of the whole network, through making up for the insufficiency of each student and bridging the gap between each student and the teacher. Extensive experiments on the TIMIT and large LibriSpeech corpuses show that, compared with the state-of-the-art methods, the proposed method achieves an excellent balance between recognition accuracy and model compression.

Type: Article
Title: Mutual-learning sequence-level knowledge distillation for automatic speech recognition
Open access status: An open access version is available from UCL Discovery
DOI: 10.1016/j.neucom.2020.11.025
Publisher version: http://dx.doi.org/10.1016/j.neucom.2020.11.025
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: Automatic speech recognition (ASR), Model compression, Knowledge distillation (KD), Mutual learning, Connectionist temporal classification (CTC)
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences > Dept of Statistical Science
URI: https://discovery-pp.ucl.ac.uk/id/eprint/10118118
Downloads since deposit
10,166Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item