Recent studies have shown that subtle changes in human face color due to heartbeats can be captured by regular RGB digital video cameras. It is possible, though challenging, to track one’s pulse rate when a video contains significant subject’s body motions in a fitness setting. The robustness gain in the recently proposed systems is often achieved by adding or changing certain modules in the system’s pipeline. Most existing works, however, only evaluate the performance of the pulse rate estimation at the system level of particular pipeline configurations, whereas the contribution from each module remains unclear. To gain a better understanding of the performance at the module level and facilitate future research in explainable learning and artificial intelligence (AI) in physiological monitoring, this paper conducts an in-depth comparative study for video-based pulse rate tracking algorithms; a special focus is placed on challenging fitness scenarios involving significant movement. The representative efforts over the past decade in the field are reviewed, upon which a reconfigurable rPPG framework/pipeline is constructed comprising of major processing modules. For performance attribution, different candidates for each module are evaluated while having the rest of modules fixed. The performance evaluation is based on a signal quality metric and four pulse-rate estimation metrics and uses the simultaneously recorded ECG-based heart rate measurement as a reference. Experimental results using a challenging fitness dataset reveals the synergy between pulse color mapping and adaptive motion filtering in obtaining accurate pulse rate estimates. The results also suggest the importance of robust frequency tracking for accurate PR estimation in low signal-to-noise ratio fitness scenarios.

Recent studies have shown that physiological signals such as heart beat and breathing can be remotely captured from human faces using a regular color camera under ambient light. This technology, referred to as remote photoplethysmography (rPPG), can be used to collect the physiological status of users who are in front of a camera, which may raise privacy concerns. To avoid the privacy abuse of the rPPG technology, this paper develops PulseEdit, a novel and efficient algorithm that can edit the physiological signals in facial videos without affecting visual appearance and thus protect the user’s physiological signal from disclosure. PulseEdit can either remove the trace of the physiological signal in a video or transform the video to contain a target physiological signal chosen by a user. Experimental results show that PulseEdit can effectively edit physiological signals in facial videos and prevent heart rate measurement based on rPPG. It is possible to utilize PulseEdit in adversarial scenarios against rPPG-based visual security algorithms. We present analyses on the performance of PulseEdit against rPPG-based liveness detection and rPPG-based deepfake detection, and demonstrate its ability to circumvent these visual security algorithms and its important role in supporting the design of attack-resilient systems.

Camera-based remote photoplethysmography (rPPG) technology has shown a promising future in contact-free cardiac and other smart health applications. The rPPG technology typically requires facial videos as a source input, which may lead to identity-privacy concerns. Facial videos are sensitive and contain subjects’ identifiable appearance features. Coupled with the health information potentially revealed by rPPG techniques, the compounding sensitivity has been a major obstacle to encouraging the sharing of facial rPPG video datasets in the research community to foster the advancement of the field. This paper investigates a suite of anonymization transforms that remove the identifiable appearance features in facial videos and retain the physiological signals for rPPG analysis. After the transformation, the facial videos are de-identified and may be shared in public with little risk of identity-privacy leakage. The proposed algorithm offers tunable options to balance the physiological fidelity and the identity-protecting strength to meet different levels of privacy requirements. A human subject study has been carried out to understand – both qualitatively and quantitatively – the perceived strength and efficacy of privacy protection by these anonymization techniques in de-identifying the facial videos and maintaining the physiological signals.

Submitted to IEEE Transactions on Information Forensics and Security (T-IFS). Under revision.

Published in IEEE Transactions on Information Forensics and Security (T-IFS) on 23 June 2022. This version on TechRxiv contains supplemental material.

Blood oxygen saturation (SpO2) is an important indicator for pulmonary and respiratory functionalities. Clinical findings on COVID-19 show that many patients had dangerously low blood oxygen levels not long before conditions worsened. It is therefore recommended, especially for the vulnerable population, to regularly monitor the blood oxygen level for precaution. Recent works have investigated how ubiquitous smartphone cameras can be used to infer SpO2. Most of these works are contact-based, requiring users to cover a phone’s camera and its nearby light source with a finger to capture reemitted light from the illuminated tissue. Contact-based methods may lead to skin irritation and sanitary concerns, especially during a pandemic. In this paper, we propose a noncontact method for SpO2 monitoring using hand videos acquired by smartphones. Considering the optical broadband nature of the red (R), green (G), and blue (B) color channels of the smartphone cameras, we exploit all three channels of RGB sensing to distill the SpO2 information beyond the traditional ratio-of-ratios (RoR) method that uses only two wavelengths. To further facilitate an accurate SpO2 prediction, we design adaptive narrow bandpass filters based on accurately estimated heart rate to obtain the most cardiac-related AC component for each color channel. Experimental results show that our proposed blood oxygen estimation method can reach a mean absolute error of 1.26% when a pulse oximeter is used as a reference, outperforming the traditional RoR method by 25%.

Blood oxygen saturation (SpO2) is an essential indicator of respiratory functionality and is receiving increasing attention during the COVID-19 pandemic. Clinical findings show that it is possible for COVID-19 patients to have significantly low SpO2 before any obvious symptoms. The prevalence of cameras has motivated researchers to investigate methods for monitoring SpO2 using videos. Most prior schemes involving smartphones are contact-based: They require a fingertip to cover the phone’s camera and the nearby light source to capture re-emitted light from the illuminated tissue. In this paper, we propose the first convolutional neural network based noncontact SpO2 estimation scheme using smartphone cameras. The scheme analyzes the videos of a participant’s hand for physiological sensing, which is convenient and comfortable, and can protect their privacy and allow for keeping face masks on. We design our neural network architectures inspired by the optophysiological models for SpO2 measurement and demonstrate the explainability by visualizing the weights for channel combination. Our proposed models outperform the state-of-the-art model that is designed for contact-based SpO2 measurement, showing the potential of our proposed method to contribute to public health. We also analyze the impact of skin type and the side of a hand on SpO2 estimation performance.

The Electric Network Frequency (ENF) is a signature of power distribution networks that can be captured by multimedia recordings made in areas where there is electrical activity. This has led to an emergence of several forensic applications based on the use of the ENF signature. Examples of such applications include estimating or verifying the time-of-recording of a media signal and inferring the power grid associated with the location in which the media signal was recorded. In this paper, we carry out a feasibility study to examine the possibility of using embedded ENF traces to pinpoint the location-of-recording of a signal within a power grid. In this study, we demonstrate that it is possible to pinpoint the location-of-recording to a certain geographical resolution using power signal recordings containing strong ENF traces. To this purpose, a high-passed version of an ENF signal is extracted and it is demonstrated that the correlation between two such signals, extracted from recordings made in different geographical locations within the same grid, decreases as the distance between the recording locations increases. We harness this property of correlation in the ENF signals to propose trilateration based localization methods, which pinpoint the unknown location of a recording while using some known recording locations as anchor locations. We also discuss the challenges that need to be overcome in order to extend this work to using ENF traces in noisier audio/video recordings for such fine localization purposes.