Cruttwell, Geoffrey SH;
Gavranovic, Bruno;
Ghani, Neil;
Wilson, Paul;
Zanasi, Fabio;
(2022)
Categorical Foundations of Gradient-Based Learning.
In:
Programming Languages and Systems. ESOP 2022.
(pp. pp. 1-28).
Springer: Cham, Switzerland.
Preview |
Text
Zanasi_978-3-030-99336-8_1.pdf Download (858kB) | Preview |
Abstract
We propose a categorical semantics of gradient-based machine learning algorithms in terms of lenses, parametric maps, and reverse derivative categories. This foundation provides a powerful explanatory and unifying framework: it encompasses a variety of gradient descent algorithms such as ADAM, AdaGrad, and Nesterov momentum, as well as a variety of loss functions such as MSE and Softmax cross-entropy, shedding new light on their similarities and differences. Our approach to gradient-based learning has examples generalising beyond the familiar continuous domains (modelled in categories of smooth maps) and can be realized in the discrete setting of boolean circuits. Finally, we demonstrate the practical significance of our framework with an implementation in Python.
| Type: | Proceedings paper |
|---|---|
| Title: | Categorical Foundations of Gradient-Based Learning |
| Event: | European Symposium on Programming (ESOP 2022) |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1007/978-3-030-99336-8_1 |
| Publisher version: | https://doi.org/10.1007/978-3-030-99336-8_1 |
| Language: | English |
| Additional information: | Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10160499 |
Archive Staff Only
 |
View Item |
