WACV 2027Main Track

Pixels-to-Phase

Reliability-Aware Latent State-Space Modeling
for Contactless Respiratory Estimation

Shadman Sakib Nirmalya Roy

University of Maryland, Baltimore County

Papersoon Supplementarysoon arXivsoon Codesoon Datasetsoon Video BibTeX

Pixels-to-Phase treats breathing as a hidden phase on the unit circle. Each frame, it predicts the phase from learned dynamics, then corrects it with the video only as much as the video can be trusted.

Abstract

Contactless respiratory estimation from video is challenging because respiratory motion is weak, spatially localized, and easily corrupted by nuisance motion and poor visual conditions. We introduce Pixels-to-Phase, a reliability-aware latent state-space model that represents respiration as a partially observed phase process and separates temporal state propagation from observation-dependent correction. The method localizes persistent respiratory support, constructs visual phase observations from complementary mechanical and semantic cues, and uses estimated observation reliability to control how strongly each visual observation updates the latent respiratory state. Persistent motion evidence and semantic observability therefore contribute to measurement construction, while temporal consistency and motion-quality cues determine when that measurement should be trusted. This formulation preserves short-term respiratory continuity when visual evidence is unreliable while allowing the state to re-anchor when informative observations return. Across RGB, NIR, and thermal benchmarks, Pixels-to-Phase reduces average respiratory-rate MAE from 1.69 BPM for the strongest baseline to 1.51 BPM. Held-out-dataset evaluation further demonstrates consistent improvements across changes in sensing modality, subject population, and acquisition conditions. Targeted analyses show that reliability-aware correction is particularly beneficial under degraded visual evidence while the latent state remains responsive to genuine respiratory transitions.

Motivation

Breathing shows up in video as a few pixels of chest, shoulder and clothing motion. Camera shake, a moving hand or a dark room can swamp it for seconds at a time. A model that maps each frame straight to a waveform cannot tell an unreliable frame from a real change in breathing, so it follows the noise. Pixels-to-Phase keeps a latent respiratory phase, carries it through unreliable intervals and re-anchors when reliable evidence returns.

Timeline with input frames, a reliability bar, a direct prediction that oscillates during motion and low light, and the Pixels-to-Phase estimate that propagates its latent phase through the unreliable interval and re-anchors afterwards.
Figure 1. Motivation for reliability-aware respiratory state estimation. Unreliable visual observations can disrupt direct prediction, while latent state propagation preserves respiratory continuity until reliable evidence returns.

Method

Respiration is modeled as a partially observed phase process whose state lies on the unit circle. The model propagates this state with learned dynamics, builds a visual observation of it from motion and semantic cues, and uses an estimated reliability to decide how much that observation counts. Select a component to highlight it in the figure.

Pixels-to-Phase architecture: support localization from RAFT flow and point tracks, visual evidence from PhaseFlow, a motion encoder and CLIP semantic observability, a GRU with phase rotation for the prior, a reliability head, and the reliability-aware state update on the unit circle.
Figure 2. Overview of Pixels-to-Phase. Respiratory support and visual observations are constructed from motion and semantic observability cues, while a causal latent phase prior is propagated independently. Observation reliability controls the update between the predicted state and current visual observation to estimate waveform, phase, respiratory rate, and frame-wise reliability.

Four components, one causal loop

Support localization decides where to look. Visual evidence construction turns that region into a phase observation . Phase dynamics rotate the previous state into a prior . A reliability head scores the current observation, and the update blends prior and observation by that score. All updates are causal. On a single NVIDIA H200 the full pipeline processes about 207 frames per second, and the recurrent update takes about 1.8 ms per frame.

Interactive demo

Corrupt the video or set the reliability yourself, and see how the update in Eq. 17 balances the propagated prior against the visual observation.

  • ground truth
  • direct prediction
  • prior
  • Pixels-to-Phase
  • reliability ρ
0.92
Rate, Pixels-to-Phase
16.2 bpm
Rate, direct prediction
16.2 bpm
True rate
16.2 bpm

Results

We evaluate on five respiratory video benchmarks spanning RGB, NIR and thermal sensing: AIR-400, MPSC-RR, Sleep, OVRM and our In-House Dataset v1.0. Pixels-to-Phase achieves the lowest average respiratory-rate error, and its gain over FLOW, the strongest baseline, holds when the test dataset is never seen in training.

Average respiratory-rate MAE (breaths/min, lower is better), FLOW versus Pixels-to-Phase.
Paired MAE reduction over FLOW across five runs, with 95% bootstrap intervals and Holm-adjusted p-values.

Table 1. Main results.

Statistical tests, support validation and ablations
Table 2. Comparison with FLOW across five independent runs.
DatasetOurs MAEFLOW MAEΔMAE95% CIpadj
AIR-4002.18 ± 0.062.36 ± 0.070.18[0.06, 0.31]0.018
MPSC-RR1.32 ± 0.041.58 ± 0.050.26[0.15, 0.38]0.003
Sleep1.73 ± 0.051.88 ± 0.060.15[0.03, 0.28]0.031
OVRM1.05 ± 0.031.26 ± 0.040.21[0.11, 0.32]0.006
In-house1.28 ± 0.041.36 ± 0.040.08[0.01, 0.16]0.041
Overall1.51 ± 0.041.69 ± 0.050.18[0.09, 0.27]0.004
Table 3. Validation of respiratory-support localization (mean ± std over runs).
Support strategyMAE ↓R ↑Band conc. ↑
Random matched2.61 ± 0.100.68 ± 0.020.39 ± 0.03
Maximum motion2.08 ± 0.080.74 ± 0.020.48 ± 0.03
Fixed torso1.79 ± 0.060.81 ± 0.010.61 ± 0.02
Persistent tracked support1.36 ± 0.040.90 ± 0.010.76 ± 0.02
Table 5. Per-dataset component ablation, MAE in BPM (mean ± std over runs).
VariantAIR-400MPSC-RRSleepOVRMIn-house
Full model2.18 ± 0.061.32 ± 0.041.73 ± 0.051.05 ± 0.031.28 ± 0.04
w/o point tracking3.04 ± 0.102.21 ± 0.082.49 ± 0.091.89 ± 0.072.12 ± 0.08
w/o temporal consensus2.57 ± 0.081.91 ± 0.062.12 ± 0.071.71 ± 0.061.89 ± 0.06
w/o PhaseFlow2.44 ± 0.081.72 ± 0.062.05 ± 0.071.55 ± 0.051.69 ± 0.06
w/o semantic context2.31 ± 0.071.68 ± 0.051.96 ± 0.061.39 ± 0.051.54 ± 0.05
w/o phase dynamics2.66 ± 0.091.96 ± 0.072.30 ± 0.081.61 ± 0.061.92 ± 0.07
w/o reliability gate2.27 ± 0.071.63 ± 0.051.91 ± 0.061.42 ± 0.051.52 ± 0.05
Δ w/o reliability gate+0.09+0.31+0.18+0.37+0.24

Analysis

The benefit of reliability-aware correction is small on clean video and grows under weak motion, occlusion, motion artifacts and low-light NIR. Estimated reliability is also calibrated: RR error rises from 1.08 BPM in high-reliability intervals to 1.57 and 2.46 BPM at intermediate and low reliability.

Bar chart of MAE under clean, weak motion, occlusion, motion artifact and low-light NIR conditions for four model variants, and a calibration plot where RR error falls as the reliability score rises.
Figure 3. Reliability-aware phase inference. (a) Condition-wise MAE under clean and degraded observations. (b) Reliability–error calibration, where lower corresponds to larger RR error.
Waveform, prior–observation disagreement and reliability over 60 seconds. During a corrupted interval from about 20 to 32 seconds, disagreement rises, reliability drops, and the predicted waveform stays close to the reference.
Figure 4. Temporal response to visual corruption. Prior–observation disagreement rises and falls during the corrupted interval, while the estimate stays stable and recovers as reliable observations return.
Normal breathing, a 20-second breath hold and forced breathing. The latent phase stops advancing during the hold and resumes afterwards; reliability stays mostly above the threshold.
Figure 5. Normal breathing, breath hold and forced breathing. During the 20-s hold the model produces 0.3 ± 0.5 spurious cycles (1.4 ± 0.7 without the gate) and re-anchors within 2.1 ± 0.8 s (3.0 ± 1.0 s without).

Motion evidence on real recordings

How visible breathing is depends on the sensor and on the distance to the subject. These examples show the respiratory region and its motion fields under different acquisition conditions.

Respiratory region crops and RAFT-Large flow for RGB, thermal, infant NIR and low-light NIR recordings.
Respiratory-centered appearance and RAFT-Large motion support across RGB, thermal, infant NIR, and low-light/occluded NIR recordings. Respiratory-motion visibility changes substantially with the modality.

In-House Dataset v1.0 · MPSC-RespView

Release in preparation

A versioned RGB corpus collected under an IRB-approved protocol with informed consent. It varies the factors that make breathing hard to see: posture, viewpoint, subject–camera distance, illumination, clothing texture and body motion. In the paper it serves as an additional evaluation resource alongside the public benchmarks. It is now released to the research community as MPSC-RespView on Hugging Face, with a data card, statistics and a preview.

Indoor lying-posture recording with a torso region of interest, the cropped respiratory region, and band-pass filtered traces from a chest point and a control point.
Indoor, lying posture. The chest point carries clear respiratory modulation; the control point mostly captures unrelated variation.
Outdoor long-distance recording where the subject is small in the frame, with the cropped region and chest and control point traces.
Outdoor, long distance. The torso covers few pixels, so the visual observation is weaker.
Recordings
73 RGB videos: 50 indoor and 23 outdoor.
Indoor
Mostly Full HD at 30 fps, about 1.5–5 min each, recorded with a synchronized respiratory belt (belt reference signals are not part of the public release). Spontaneous and paced breathing (slow, regular, fast) in frontal seated, top-angular, backward-view and lying/supine views.
Outdoor
One-minute recordings at 4, 5, 6, 7, 9, 11 and 13 m, each with about 20 s of normal breathing, 20 s of breath holding and 20 s of forced breathing.
Access
Openly available on Hugging Face under CC BY-NC 4.0 (non-commercial, with attribution). Faces are anonymized and the videos contain no audio or names. Use is subject to the terms of use in the data card, including no attempts to identify participants. The dataset is intended for respiratory-monitoring research, not clinical diagnosis or apnea assessment.
Hugging Facesoon Data cardsoon Downloadsoon
Evaluation protocols. Split manifests for the public benchmarks will be released with the code.
DatasetProtocolSplit
AIR-4006-fold subject-wise12 / 3 / 3
MPSC-RR5-fold subject-wiseFold-based
Sleep6-fold subject-wise8 / 2 / 2
OVRM5-fold group-wiseGroup-based
In-House v1.05-fold subject-wiseFold-based

Citation

@inproceedings{sakib2027pixels,
  title     = {Pixels-to-Phase: Reliability-Aware Latent State-Space
               Modeling for Contactless Respiratory Estimation},
  author    = {Sakib, Shadman and Roy, Nirmalya},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on
               Applications of Computer Vision (WACV)},
  year      = {2027}
}