eprintid: 10160773
rev_number: 7
eprint_status: archive
userid: 699
dir: disk0/10/16/07/73
datestamp: 2022-11-29 16:33:49
lastmod: 2022-11-29 16:33:49
status_changed: 2022-11-29 16:33:49
type: proceedings_section
metadata_visibility: show
sword_depositor: 699
creators_name: Lee, S
creators_name: Mannelli, SS
creators_name: Clopath, C
creators_name: Goldt, S
creators_name: Saxe, AM
title: Maslow’s Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation
ispublished: pub
divisions: UCL
divisions: B02
divisions: C08
divisions: D76
note: This work is licensed under a Creative Commons Attribution 4.0 International License. The images
or other third-party material in this article are included in the Creative Commons license,
unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license,
users will need to obtain permission from the license holder to reproduce the material. To view a copy of this
license, visit http://creativecommons.org/licenses/by/4.0/
abstract: Continual learning—learning new tasks in sequence while maintaining performance on old tasks—remains particularly challenging for artificial neural networks. Surprisingly, the amount of forgetting does not increase with the dissimilarity between the learned tasks, but appears to be worst in an intermediate similarity regime. In this paper we theoretically analyse both a synthetic teacher-student framework and a real data setup to provide an explanation of this phenomenon that we name Maslow’s Hammer hypothesis. Our analysis reveals the presence of a trade-off between node activation and node re-use that results in worst forgetting in the intermediate regime. Using this understanding we reinterpret popular algorithmic interventions for catastrophic interference in terms of this trade-off, and identify the regimes in which they are most effective.
date: 2022-07-17
date_type: published
publisher: Proceedings of Machine Learning Research (PMLR)
official_url: https://proceedings.mlr.press/v162/lee22g.html
oa_status: green
full_text_type: pub
language: eng
primo: open
primo_central: open_green
verified: verified_manual
elements_id: 1991406
lyricists_name: Saxe, Andrew
lyricists_id: ASAXE99
actors_name: Flynn, Bernadette
actors_id: BFFLY94
actors_role: owner
full_text_status: public
pres_type: paper
publication: Proceedings of the 39th International Conference on Machine Learning, PMLR
event_title: ICML 2022
event_dates: 17 Jul 2022 - 23 Jul 2022
book_title: Proceedings of the 39th International Conference on Machine Learning, PMLR
citation:        Lee, S;    Mannelli, SS;    Clopath, C;    Goldt, S;    Saxe, AM;      (2022)    Maslow’s Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation.                     In:  Proceedings of the 39th International Conference on Machine Learning, PMLR.    Proceedings of Machine Learning Research (PMLR)       Green open access   
 
document_url: https://discovery-pp.ucl.ac.uk/id/eprint/10160773/1/lee22g.pdf