Joshi, Jitesh;
Agaian, Sos S;
Cho, Youngjun;
(2024)
FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing.
In:
Advances in Neural Information Processing Systems 37 (NeurIPS 2024).
NeurIPS: Vancouver, Canada.
(In press).
Preview |
Text
FactorizePhys_NeurIPS-2024.pdf - Published Version Download (44MB) | Preview |
Abstract
Remote photoplethysmography (rPPG) enables non-invasive extraction of blood volume pulse signals through imaging, transforming spatial-temporal data into time series signals. Advances in end-to-end rPPG approaches have focused on this transformation where attention mechanisms are crucial for feature extraction. However, existing methods compute attention disjointly across spatial, temporal, and channel dimensions. Here, we propose the Factorized Self-Attention Module (FSAM), which jointly computes multidimensional attention from voxel embeddings using nonnegative matrix factorization. To demonstrate FSAM's effectiveness, we developed FactorizePhys, an end-to-end 3D-CNN architecture for estimating blood volume pulse signals from raw video frames. Our approach adeptly factorizes voxel embeddings to achieve comprehensive spatial, temporal, and channel attention, enhancing performance of generic signal extraction tasks. Furthermore, we deploy FSAM within an existing 2D-CNN-based rPPG architecture to illustrate its versatility. FSAM and FactorizePhys are thoroughly evaluated against state-of-the-art rPPG methods, each representing different types of architecture and attention mechanism. We perform ablation studies to investigate the architectural decisions and hyperparameters of FSAM. Experiments on four publicly available datasets and intuitive visualization of learned spatial-temporal features substantiate the effectiveness of FSAM and enhanced cross-dataset generalization in estimating rPPG signals, suggesting its broader potential as a multidimensional attention mechanism. The code is accessible at https://github.com/PhysiologicAILab/FactorizePhys.
| Type: | Proceedings paper |
|---|---|
| Title: | FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing |
| Event: | Advances in Neural Information Processing Systems |
| Location: | Vancouver, Canada |
| Dates: | 10 Dec 2024 - 15 Dec 2024 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://openreview.net/forum?id=qrfp4eeZ47 |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Remote photo-plethysmography, Multidimensional Attention Mechanism, Non-negative Matrix Factorization, Remote Physiological Sensing |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10200710 |
Archive Staff Only
 |
View Item |
