Joint Alignment and Modeling of Correlated Behavior Streams

The Variable Time-Shift Hidden Markov Model (VTS-HMM) is proposed for learning and modeling pairs of correlated streams. Unlike previous coupled models for time series, the VTS-HMM accounts for varying time shifts between correlated events in pairs of streams having different properties. The VTS-HMM is learned on a set of pairs of unaligned streams and, thus, learning entails simultaneous estimation of the varying time shifts and of the parameters of the model. The formulation is demonstrated in the analysis of videos of dyadic social interactions between children and adults in the Multimodal Dyadic Behavior Dataset (MMDB). In dyadic social interactions, an agent starts an interaction with one or more "initiating behaviors" that elicit one or more "responding behaviors" from the partner within a temporal window. The proposed VTS-HMM explicitly accounts for varying time shifts between initiating and responding behaviors in these behavior streams. The experiments confirm that modeling of these varying time shifts in the VTS-HMM can yield improved estimation of the level of engagement of the child and adult and more accurate discrimination among complex activities.