CMI-Net: a Unified Framework for Physiological Time Series Classification with Incomplete Modalities

In recent years, health assessment and aided diagnosis based on physiological time series (PTSs) have become a research hotspot. To achieve reliable assessment and diagnosis, multimodal PTSs are excellent choices for manipulation in deep learning pipelines. However, existing methods are not effective in solving the problem of missing modalities. In addition, the exploration of temporal information within PTSs is also a serious challenge. To this end, we propose a unified framework, named contrastive modal imagination network (CMI-Net), for the PTSs classification with incomplete modalities. Specifically, CMI-Net handles the problem of arbitrary modal missing through the combination of modal awareness imagination module (MAIM) and semantic & modal calibration contrastive learning (SMCCL). Among them, MAIM can capture the interaction among modalities by learning the shared representation distribution of all modalities. Meanwhile, SMCCL introduces prior information of semantics and modality to check semantic consistency while maintaining the uniqueness of each modality. Utilizing the calibration of SMCCL, the data distribution recovered by MAIM is aligned with the real data distribution. We further design a novel context-aware architecture, which can learn the intra- and inter-segment temporal dependencies of PTSs and facilitate the mining of cross-scale temporal context information. To the best of our knowledge, this is one of the first works to formally study the multimodal missing problem in PTSs. Extensive experiments on four real-world multimodal PTSs datasets demonstrate that CMI-Net remarkably outperforms other competitive methods.