loading page

Locating X-ray coronary angiogram keyframes via long short-term spatiotemporal attention with image-to-patch contrastive learning
  • +3
  • Ruipeng Zhang ,
  • Binjie Qin ,
  • Song Ding ,
  • Yueqi Zhu ,
  • Xu Chen ,
  • Yisong Lv
Ruipeng Zhang
Author Profile
Binjie Qin
School of Biomedical Engineering, School of Biomedical Engineering

Corresponding Author:[email protected]

Author Profile
Song Ding
Author Profile
Yueqi Zhu
Author Profile
Yisong Lv
Author Profile

Abstract

Locating the start, apex and end keyframes of moving contrast agents for keyframe counting during X-ray coronary angiography (XCA) is very important in the diagnosis and treatment of cardiovascular diseases. To locate these keyframes from the class-imbalanced and boundary-agnostic foreground vessel actions that overlap complex backgrounds, we propose long short-term spatiotemporal attention by integrating a convolutional long short-term memory (CLSTM) network into a multiscale Transformer to learn the segment- and sequence-level dependences in the consecutive-frame-based deep features. Image-to-patch contrastive learning is further embedded between the CLSTM-based long-term spatiotemporal attention and Transformer-based short-term attention modules. The imagewise contrastive module reuses the long-term attention to contrast image-level foreground/background of XCA sequence, while patchwise contrastive projection selects the random patches of backgrounds as convolution kernels to project foreground/background frames into different latent spaces. A new XCA video dataset is collected to evaluate the proposed neural network. The experimental results show that the proposed method achieves a mAP of 70.51\% and an F1-score of 0.8188, considerably outperforming the state-of-the-art methods. The source code and dataset are available at https://github.com/Binjie-Qin/STA-IPCon.
2023Published in IEEE Transactions on Medical Imaging on pages 1-1. 10.1109/TMI.2023.3286859