eprintid: 10160773 rev_number: 7 eprint_status: archive userid: 699 dir: disk0/10/16/07/73 datestamp: 2022-11-29 16:33:49 lastmod: 2022-11-29 16:33:49 status_changed: 2022-11-29 16:33:49 type: proceedings_section metadata_visibility: show sword_depositor: 699 creators_name: Lee, S creators_name: Mannelli, SS creators_name: Clopath, C creators_name: Goldt, S creators_name: Saxe, AM title: Maslow’s Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation ispublished: pub divisions: UCL divisions: B02 divisions: C08 divisions: D76 note: This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third-party material in this article are included in the Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ abstract: Continual learning—learning new tasks in sequence while maintaining performance on old tasks—remains particularly challenging for artificial neural networks. Surprisingly, the amount of forgetting does not increase with the dissimilarity between the learned tasks, but appears to be worst in an intermediate similarity regime. In this paper we theoretically analyse both a synthetic teacher-student framework and a real data setup to provide an explanation of this phenomenon that we name Maslow’s Hammer hypothesis. Our analysis reveals the presence of a trade-off between node activation and node re-use that results in worst forgetting in the intermediate regime. Using this understanding we reinterpret popular algorithmic interventions for catastrophic interference in terms of this trade-off, and identify the regimes in which they are most effective. date: 2022-07-17 date_type: published publisher: Proceedings of Machine Learning Research (PMLR) official_url: https://proceedings.mlr.press/v162/lee22g.html oa_status: green full_text_type: pub language: eng primo: open primo_central: open_green verified: verified_manual elements_id: 1991406 lyricists_name: Saxe, Andrew lyricists_id: ASAXE99 actors_name: Flynn, Bernadette actors_id: BFFLY94 actors_role: owner full_text_status: public pres_type: paper publication: Proceedings of the 39th International Conference on Machine Learning, PMLR event_title: ICML 2022 event_dates: 17 Jul 2022 - 23 Jul 2022 book_title: Proceedings of the 39th International Conference on Machine Learning, PMLR citation: Lee, S; Mannelli, SS; Clopath, C; Goldt, S; Saxe, AM; (2022) Maslow’s Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation. In: Proceedings of the 39th International Conference on Machine Learning, PMLR. Proceedings of Machine Learning Research (PMLR) Green open access document_url: https://discovery-pp.ucl.ac.uk/id/eprint/10160773/1/lee22g.pdf