Li, Z;
Ming, Y;
Yang, L;
Xue, J-H;
(2021)
Mutual-learning sequence-level knowledge distillation for automatic speech recognition.
Neurocomputing
, 428
pp. 259-267.
10.1016/j.neucom.2020.11.025.
Preview |
Text
NEUCOM-ZeruiLi-accepted.pdf - Accepted Version Download (5MB) | Preview |
Abstract
Automatic speech recognition (ASR) is a crucial technology for man-machine interaction. End-to-end models have been studied recently in deep learning for ASR. However, these models are not suitable for the practical application of ASR due to their large model sizes and computation costs. To address this issue, we propose a novel mutual-learning sequence-level knowledge distillation framework enjoying distinct student structures for ASR. Trained mutually and simultaneously, each student learns not only from the pre-trained teacher but also from its distinct peers, which can improve the generalization capability of the whole network, through making up for the insufficiency of each student and bridging the gap between each student and the teacher. Extensive experiments on the TIMIT and large LibriSpeech corpuses show that, compared with the state-of-the-art methods, the proposed method achieves an excellent balance between recognition accuracy and model compression.
Type: | Article |
---|---|
Title: | Mutual-learning sequence-level knowledge distillation for automatic speech recognition |
Open access status: | An open access version is available from UCL Discovery |
DOI: | 10.1016/j.neucom.2020.11.025 |
Publisher version: | http://dx.doi.org/10.1016/j.neucom.2020.11.025 |
Language: | English |
Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
Keywords: | Automatic speech recognition (ASR), Model compression, Knowledge distillation (KD), Mutual learning, Connectionist temporal classification (CTC) |
UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences > Dept of Statistical Science |
URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10118118 |
Archive Staff Only
![]() |
View Item |