eprintid: 10188016 rev_number: 6 eprint_status: archive userid: 699 dir: disk0/10/18/80/16 datestamp: 2024-02-28 10:17:24 lastmod: 2024-02-28 10:17:24 status_changed: 2024-02-28 10:17:24 type: article metadata_visibility: show sword_depositor: 699 creators_name: Xu, Mengya creators_name: Islam, Mobarakol creators_name: Bai, Long creators_name: Ren, Hongliang title: Privacy-Preserving Synthetic Continual Semantic Segmentation for Robotic Surgery ispublished: inpress divisions: UCL divisions: B04 divisions: C05 divisions: F42 note: This version is the author accepted manuscript. For information on re-use, please refer to the publisher's terms and conditions. abstract: Deep Neural Networks (DNNs) based semantic segmentation of the robotic instruments and tissues can enhance the precision of surgical activities in robot-assisted surgery. However, in biological learning, DNNs cannot learn incremental tasks over time and exhibit catastrophic forgetting, which refers to the sharp decline in performance on previously learned tasks after learning a new one. Specifically, when data scarcity is the issue, the model shows a rapid drop in performance on previously learned instruments after learning new data with new instruments. The problem becomes worse when it limits releasing the dataset of the old instruments for the old model due to privacy concerns and the unavailability of the data for the new or updated version of the instruments for the continual learning model. For this purpose, we develop a privacy-preserving synthetic continual semantic segmentation framework by blending and harmonizing (i) open-source old instruments foreground to the synthesized background without revealing real patient data in public and (ii) new instruments foreground to extensively augmented real background. To boost the balanced logit distillation from the old model to the continual learning model, we design overlapping class-aware temperature normalization (CAT) by controlling model learning utility. We also introduce multi-scale shifted-feature distillation (SD) to maintain long and short-range spatial relationships among the semantic objects where conventional short-range spatial features with limited information reduce the power of feature distillation. We demonstrate the effectiveness of our framework on the EndoVis 2017 and 2018 instrument segmentation dataset with a generalized continual learning setting. Code is available at https://github.com/XuMengyaAmy/Synthetic_CAT_SD. date: 2024-02-21 date_type: published publisher: Institute of Electrical and Electronics Engineers (IEEE) official_url: http://dx.doi.org/10.1109/tmi.2024.3364969 oa_status: green full_text_type: other language: eng primo: open primo_central: open_green verified: verified_manual elements_id: 2252205 doi: 10.1109/tmi.2024.3364969 lyricists_name: Islam, Mobarakol lyricists_id: MISLB53 actors_name: Flynn, Bernadette actors_id: BFFLY94 actors_role: owner full_text_status: public publication: IEEE Transactions on Medical Imaging citation: Xu, Mengya; Islam, Mobarakol; Bai, Long; Ren, Hongliang; (2024) Privacy-Preserving Synthetic Continual Semantic Segmentation for Robotic Surgery. IEEE Transactions on Medical Imaging 10.1109/tmi.2024.3364969 <https://doi.org/10.1109/tmi.2024.3364969>. (In press). Green open access document_url: https://discovery-pp.ucl.ac.uk/id/eprint/10188016/1/Privacy-Preserving_Synthetic_Continual_Semantic_Segmentation_for_Robotic_Surgery.pdf