Abstract
Micro-expression recognition (MER) is a pivotal research area within human emotion analysis. However, the fleeting, subtle, and complex nature of micro-expressions poses challenges for accurate and efficient recognition. To address this, this paper proposes a Dual-Stream Network with Cross-layer Attention and Similarity Constraint (DSN-CASC) for MER. The network is designed with two parallel branches, each dedicated to learning features of stacked optical flow maps and independent micro-expression optical flow components. The network amplifies its focus on representing features at different hierarchical levels via incorporating a cross-layer attention module, thereby enhancing the capture of key features associated with micro-expression variations. Furthermore, a similarity constraint strategy is introduced to ensure that the micro-expression features extracted by the dual branches exhibit similar representations, improving the network’s overall representation capability. Finally, a simple feature fusion approach is employed for micro-expression classification. Extensive experiments on the composite database validate the effectiveness of DSN-CASC under leave-one-subject-out cross-validation and composite database evaluation protocol. The results demonstrate that our proposed approach achieves promising performance improvements, which provide new insights and effective solutions for MER research.
Similar content being viewed by others
Data availibility
No datasets were generated or analysed during the current study.
References
Ekman, P.: Lie catching and microexpressions. Philos. Decep. 1(2), 5 (2009)
Yan, W.-J., Wu, Q., Liang, J., Chen, Y.-H., Fu, X.: How fast are the leaked facial expressions: the duration of micro-expressions. J. Nonverbal Behav. 37(4), 217–230 (2013)
Li, Y., Wei, J., Liu, Y., Kauttonen, J., Zhao, G.: Deep learning for micro-expression recognition: a survey. IEEE Trans. Affect. Comput. 13(4), 2028–2046 (2022)
Ben, X., Ren, Y., Zhang, J., Wang, S.-J., Kpalma, K., Meng, W., Liu, Y.-J.: Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 44(9), 5826–5846 (2021)
Li, J., Dong, Z., Lu, S., Wang, S.-J., Yan, W.-J., Ma, Y., Liu, Y., Huang, C., Fu, X.: Cas (me) 3: a third generation facial spontaneous micro-expression database with depth information and high ecological validity. IEEE Trans. Pattern Anal. Mach. Intell. 45(3), 2782–2800 (2022)
Frank, M., Herbasz, M., Sinuk, K., Keller, A., Nolan, C.: I see how you feel: training laypeople and professionals to recognize fleeting emotions. In: The Annual Meeting of the International Communication Association. Sheraton New York, New York City, pp. 1–35 (2009)
Rosenberg, E.L., Ekman, P.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, Oxford (2020)
Liong, S.-T., Gan, Y.S., See, J., Khor, H.-Q., Huang, Y.-C.: Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5 (2019). IEEE
Zhou, L., Mao, Q., Xue, L.: Dual-inception network for cross-database micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5 (2019). IEEE
Wang, G., Huang, S., Tao, Z.: Shallow multi-branch attention convolutional neural network for micro-expression recognition. Multimedia Syst. 29(4), 1967–1980 (2023)
Zhou, H., Huang, S., Xu, Y.: Inceptr: micro-expression recognition integrating inception-cbam and vision transformer. Multimedia Syst. 29(6), 3863–3876 (2023)
Zhai, Z., Zhao, J., Long, C., Xu, W., He, S., Zhao, H.: Feature representation learning with adaptive displacement generation and transformer fusion for micro-expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22086–22095 (2023)
Gan, Y., Lien, S.-E., Chiang, Y.-C., Liong, S.-T.: Laenet for micro-expression recognition. Vis. Comput. 40(2), 585–599 (2024)
Wang, C., Peng, M., Bi, T., Chen, T.: Micro-attention for micro-expression recognition. Neurocomputing 410, 354–362 (2020)
Xia, Z., Peng, W., Khor, H.-Q., Feng, X., Zhao, G.: Revealing the invisible with model and data shrinking for composite-database micro-expression recognition. IEEE Trans. Image Process. 29, 8590–8605 (2020)
Liu, D., Liang, Z., Sun, Y.: Micro-expression recognition method based on spatial attention mechanism and optical flow features. J. Comput. Aid. Des. Comput. Gr. 33, 1541–1552 (2021)
Chaudhry, R., Ravichandran, A., Hager, G., Vidal, R.: Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1932–1939 (2009)
Pfister, T., Li, X., Zhao, G., Pietikäinen, M.: Recognising spontaneous facial micro-expressions. In: 2011 International Conference on Computer Vision, pp. 1449–1456 (2011)
Happy, S., Routray, A.: Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans. Affect. Comput. 10(03), 394–406 (2019)
Huang, X., Wang, S.-J., Liu, X., Zhao, G., Feng, X., Pietikainen, M.: Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans. Affect. Comput. 10(01), 32–47 (2019)
Wang, G., Huang, S., Dong, Z.: Haphazard cuboids feature extraction for micro-expression recognition. IEEE Access 10, 110149–110162 (2022)
Wei, J., Lu, G., Yan, J., Zong, Y.: Learning two groups of discriminative features for micro-expression recognition. Neurocomputing 479, 22–36 (2022)
Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G., Fu, X.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2015)
Dong, Z., Wang, G., Lu, S., Yan, W.-J., Wang, S.-J.: A brief guide: Code for spontaneous expressions and micro-expressions in videos. In: Proceedings of the 1st Workshop on Facial Micro-Expression: Advanced Techniques for Facial Expressions Generation and Spotting, pp. 31–37 (2021)
Dong, Z., Wang, G., Lu, S., Li, J., Yan, W., Wang, S.-J.: Spontaneous facial expressions and micro-expressions coding: from brain to face. Front. Psychol. 5808 (2022)
Liong, S.-T., See, J., Wong, K., Phan, R.C.-W.: Less is more: micro-expression recognition from video using apex frame. Signal Process. Image Commun. 62, 82–92 (2018)
Peng, M., Wang, C., Chen, T., Liu, G., Fu, X.: Dual temporal scale convolutional neural network for micro-expression recognition. Front. Psychol. 8, 1745 (2017)
Khor, H.-Q., See, J., Liong, S.-T., Phan, R.C., Lin, W.: Dual-stream shallow networks for facial micro-expression recognition. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 36–40 (2019). IEEE
Gao, J., Zhang, T., Xu, C.: Learning to model relationships for zero-shot video classification. IEEE Trans. Pattern Anal. Mach. Intell. 43(10), 3476–3491 (2020)
Gao, J., Xu, C.: Learning video moment retrieval without a single annotated video. IEEE Trans. Circuits Syst. Video Technol. 32(3), 1646–1657 (2021)
Gan, Y.S., Liong, S.-T., Yau, W.-C., Huang, Y.-C., Tan, L.-K.: Off-apexnet on micro-expression recognition system. Signal Process. Image Commun. 74, 129–139 (2019)
Nie, X., Takalkar, M.A., Duan, M., Zhang, H., Xu, M.: Geme: dual-stream multi-task gender-based micro-expression recognition. Neurocomputing 427, 13–28 (2021)
Zhou, H., Huang, S., Li, J., Wang, S.-J.: Dual-atme: dual-branch attention network for micro-expression recognition. Entropy 25(3), 460 (2023)
Hu, Y., Gao, J., Dong, J., Fan, B., Liu, H.: Exploring rich semantics for open-set action recognition. IEEE Trans. Multimedia (2023)
Gao, J., Chen, M., Xu, C.: Vectorized evidential learning for weakly-supervised temporal action localization. IEEE Trans. Pattern Anal. Mach. Intell. (2023)
Van Quang, N., Chun, J., Tokuyama, T.: Capsulenet for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–7 (2019). IEEE
Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., Fu, X.: Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1), 86041 (2014)
Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: Samm: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(01), 116–129 (2018)
Li, X., Pfister, T., Huang, X., Zhao, G., Pietikäinen, M.: A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (fg), pp. 1–6 (2013). IEEE
Liong, S.-T., See, J., Wong, K., Le Ngo, A.C., Oh, Y.-H., Phan, R.: Automatic apex frame spotting in micro-expression database. In: 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pp. 665–669 (2015). IEEE
Hou, Q., Zhou, D., Feng, J.: Coordinate attention for efficient mobile network design. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13713–13722 (2021)
See, J., Yap, M.H., Li, J., Hong, X., Wang, S.-J.: Megc 2019—the second facial micro-expressions grand challenge. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5 (2019). IEEE
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2018)
Hong, T., Longjiao, F.S.Z., Hongmei, L.: Micro-expression recognition based on optical flow method and pseudo three-dimensional residual network. J. Signal Process. 38(05), 1075–1087 (2022)
Shu, X., Li, J., Shi, L., Huang, S.: Res-capsnet: an improved capsule network for micro-expression recognition. Multimedia Syst. 29(3), 1593–1601 (2023)
Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grants 62276118 and 61772244.
Author information
Authors and Affiliations
Contributions
GW wrote the main manuscript text. All authors reviewed the manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no Conflict of interest to declare that are relevant to the content of this article.
Additional information
Communicated by J. Gao.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, G., Huang, S. Dual-stream network with cross-layer attention and similarity constraint for micro-expression recognition. Multimedia Systems 30, 147 (2024). https://doi.org/10.1007/s00530-024-01352-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s00530-024-01352-6