loading page

Continual Egocentric Activity Recognition via Foreseeable-Generalized Multi-Modal Representations
  • +4
  • Chiyuan He ,
  • Shaoxu Cheng ,
  • Zihuan Qiu ,
  • Linfeng Xu ,
  • Fanman Meng ,
  • Qingbo Wu ,
  • Hongliang Li
Chiyuan He
University of Electronic Science and Technology of China, University of Electronic Science and Technology of China

Corresponding Author:[email protected]

Author Profile
Shaoxu Cheng
Author Profile
Zihuan Qiu
Author Profile
Linfeng Xu
Author Profile
Fanman Meng
Author Profile
Qingbo Wu
Author Profile
Hongliang Li
Author Profile

Abstract

The rapid development of wearable sensors promotes convenient data collection in human daily life. Human Activity Recognition (HAR), as a prominent research direction for wearable applications, has made remarkable progress in recent years. However, existing efforts mostly focus on improving recognition accuracy, paying limited attention to the model’s functional scalability, specifically its ability for continual learning. This limitation greatly restricts its application in open-world scenarios. Moreover, due to storage and privacy concerns, it is often impractical to retain the activity data of different users for subsequent tasks, especially egocentric visual information. Furthermore, the imbalance between visual-based and inertial-measurement-unit (IMU) sensing modality introduces challenges of lack of generalization when employing conventional continual learning techniques. In this paper, we propose a motivational learning scheme to address the limited generalization caused by the modal imbalance, enabling foreseeable generalization in a visual-IMU multimodal network. To overcome forgetting, we introduce a robust representation estimation technique and a pseudo-representation generation strategy for continual learning. Experimental results on the egocentric multimodal activity dataset UESTC-MMEA-CL demonstrate the effectiveness of our proposed method. Furthermore, our method effectively leverages the generalization capabilities of IMU-based modal representations, outperforming general and state-of-the-art continual learning methods in various task settings.
21 Mar 2024Submitted to TechRxiv
29 Mar 2024Published in TechRxiv
2024Published in IEEE Sensors Journal on pages 1-1. 10.1109/JSEN.2024.3371975