Sarao Mannelli, Stefano; Ivashynka, Yaraslau; Saxe, Andrew; Saglietti, Luca; (2024) Tilting the odds at the lottery: the interplay of overparameterisation and curricula in neural networks. Journal of Statistical Mechanics: Theory and Experiment , 2024 (11) , Article 114001. 10.1088/1742-5468/ad864b.

Text 4487_Tilting_the_Odds_at_the_L.pdf - Accepted Version Access restricted to UCL open access staff until 20 November 2025. Download (9MB)

Abstract

A wide range of empirical and theoretical works have shown that overparameterisation can amplify the performance of neural networks. According to the lottery ticket hypothesis, overparameterised networks have an increased chance of containing a sub-network that is well-initialised to solve the task at hand. A more parsimonious approach, inspired by animal learning, consists in guiding the learner towards solving the task by curating the order of the examples, i.e.  providing a curriculum. However, this learning strategy seems to be hardly beneficial in deep learning applications. In this work, we undertake an analytical study that connects curriculum learning and overparameterisation. In particular, we investigate their interplay in the online learning setting for a 2-layer network in the XOR-like Gaussian Mixture problem. Our results show that a high degree of overparameterisation—while simplifying the problem—can limit the benefit from curricula, providing a theoretical account of the ineffectiveness of curricula in deep learning.

Download activity - last month

Export as CSVXMLJSON

Download activity - last 12 months

Export as XMLCSVJSON

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item