Multi-feature and Multi-instance Learning with Anti-overfitting Strategy for Engagement Intensity Prediction

This paper proposes a novel engagement intensity prediction approach, which is also applied in the EmotiW Challenge 2019 and resulted in good performance. The task is to predict the engagement level when a subject student is watching an educational video in diverse conditions and various environments. Assuming that the engagement intensity has a strong correlation with facial movements, upper-body posture movements and overall environmental movements in a time interval, we extract and incorporate these motion features into a deep regression model consisting of layers with a combination of LSTM, Gated Recurrent Unit (GRU) and a Fully Connected Layer. In order to precisely and robustly predict the engagement level in a long video with various situations such as darkness and complex background, a multi-features engineering method is used to extract synchronized multi-model features in a period of time by considering both the short-term dependencies and long-term dependencies. Based on the well-processed features, we propose a strategy for maximizing validation accuracy to generate the best models covering all the model configurations. Furthermore, to avoid the overfitting problem ascribed to the extremely small database, we propose another strategy applying a single Bi-LSTM layer with only 16 units to minimize the overfitting, and splitting the engagement dataset (train + validation) with 5-fold cross validation (stratified k-fold) to train the conservative model. By ensembling the above models, our methods finally win the second place in the challenge with MSE of 0.06174 on the testing set.