Cao, L;
Wang, J;
Yang, B;
Su, D;
Yu, D;
(2023)
Trinet: Stabilizing Self-Supervised Learning From Complete or Slow Collapse.
In:
Proceedings of ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
IEEE: Rhodes, Greece.
Preview |
Text
2301.00656.pdf - Accepted Version Download (675kB) | Preview |
Abstract
Self-supervised learning (SSL) models confront challenges of abrupt informational collapse or slow dimensional collapse. We propose TriNet, which introduces a novel triple-branch architecture for preventing collapse and stabilizing the pretraining. TriNet learns the SSL latent embedding space and incorporates it to a higher level space for predicting pseudo target vectors generated by a frozen teacher. Our experimental results show that the proposed method notably stabilizes and accelerates pre-training and achieves a relative word error rate reduction (WERR) of 6.06% compared to the state-of- the-art (SOTA) Data2vec for a downstream benchmark ASR task. We will release our code at https://github.com/tencent-ailab/.
| Type: | Proceedings paper |
|---|---|
| Title: | Trinet: Stabilizing Self-Supervised Learning From Complete or Slow Collapse |
| Event: | ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) |
| Dates: | 4 Jun 2023 - 10 Jun 2023 |
| ISBN-13: | 9781728163277 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1109/ICASSP49357.2023.10094725 |
| Publisher version: | https://doi.org/10.1109/ICASSP49357.2023.10094725 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Self-supervised learning, collapse, pseudo label, self-learning, bootstrapping |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10183765 |
Archive Staff Only
 |
View Item |
