Thermal-depth fusion for occluded body skeletal posture estimation

Reliable occluded skeletal posture estimation is a fundamentally challenging problem for vision-based monitoring techniques. This is due to several imaging related challenges introduced by existing depth-based pose estimation techniques that fail to provide accurate joint position estimates when the line of sight between the imaging device and the patient is obscured by an occluding material. In this work, we present a new method of estimating skeletal posture in occluded applications using both depth and thermal imaging through volumetric modeling and introduce a new occluded ground-truth tracking method inspired by modern motion capture solutions. Using this integrated volumetric model, we utilize Convolutional Neural Networks to characterize and identify volumetric thermal distributions that match trained skeletal posture estimates which includes disconnected skeletal definitions and allows correct posture estimation in highly ambiguous cases. We demonstrate this approach by correctly identifying common sleep postures that present challenging cases for current skeletal joint estimations, obtaining an average classification accuracy of ∼94.45%.