Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Evaluating facial recognition services as interaction technique for recommender systems

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Recommender systems are tools and techniques to assist users in the content selection process thereby coping with the problem of information overload. For recommender systems, user authentication and feedback gathering are of crucial importance. However, the typical user authentication with username / password and feedback method with a star rating system are not user friendly and often bypassed. This article proposes an alternative method for user authentication based on facial recognition and an automatic feedback gathering method by detecting various face characteristics such as emotions. We studied the use case of video watching. Photos made with the front-facing camera of a tablet, smartphone, or smart TV are used as input of a facial recognition service. The persons in front of the screen can be identified. During video watching, implicit feedback for the video content is automatically gathered through emotion recognition, attention measurements, and behavior analysis. An evaluation with a test panel showed that the recognized emotions are correlated with the user’s star ratings and that happiness can be most accurately detected. So as the main contribution, this article indicates that emotion recognition might be used as an alternative feedback mechanism for recommender systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Adomavicius G, Sankaranarayanan R, Sen S, Tuzhilin A (2005) Incorporating contextual information in recommender systems using a multidimensional approach. ACM Transactions on Information Systems (TOIS) 23(1):103–145

    Article  Google Scholar 

  2. Arapakis I, Moshfeghi Y, Joho H, Ren R, Hannah D, Jose JM (2009) Enriching user profiling with affective features for the improvement of a multimodal recommender system. In: Proceedings of the ACM international conference on image and video retrieval, pp 1–8

  3. Arapakis I, Moshfeghi Y, Joho H, Ren R, Hannah D, Jose JM (2009) Integrating facial expressions into user profiling for the improvement of a multimodal recommender system. In: 2009 IEEE international conference on multimedia and expo. IEEE, pp 1440–1443

  4. Baltrunas L, Makcinskas T, Ricci F (2010) Group recommendations with rank aggregation and collaborative filtering. In: Proceedings of the fourth ACM conference on Recommender systems, RecSys ’10. ACM, New York, pp 119–126

  5. Bi H, Li N, Guan H, Lu D, Yang L (2019) A multi-scale conditional generative adversarial network for face sketch synthesis. In: 2019 IEEE international conference on image processing (ICIP). IEEE, pp 3876–3880

  6. Bimbot F, Bonastre JF, Fredouille C, Gravier G, Magrin-Chagnolleau I, Meignier S, Merlin T, Ortega-garcía J, Petrovska-Delacrétaz D, Reynolds DA (2004) A tutorial on text-independent speaker verification. EURASIP Journal on Advances in Signal Processing 2004(4):101,962

    Article  Google Scholar 

  7. Buthpitiya S, Zhang Y, Dey AK, Griss M (2011) N-gram geo-trace modeling. In: International conference on pervasive computing. Springer, pp 97–114

  8. Chauhan M, Sakle M (2014) Study & analysis of different face detection techniques. Int J Comput Sci Inform Technol 5(2):1615–1618

    Google Scholar 

  9. Cho M, Kim T, Kim IJ, Lee S (2020) Relational deep feature learning for heterogeneous face recognition. arXiv:2003.00697

  10. CommonSenseMedia: You know your kids. we know media and tech. together we can build a digital world where our kids can thrive. (2019). Available at https://www.commonsensemedia.org/about-us/our-mission

  11. De Pessemier T, Dooms S, Martens L (2014) Comparison of group recommendation algorithms. Multimed Tools Appl 72(3):2497–2541

    Article  Google Scholar 

  12. De Pessemier T, Verlee D, Martens L (2016) Enhancing recommender systems for tv by face recognition. In: 12th international conference on web information systems and technologies (WEBIST 2016), vol 2, pp 243–250

  13. Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. Journal of Personality and Social Psychology 17(2):124

    Article  Google Scholar 

  14. Ekstrand MD (2018) The lkpy package for recommender systems experiments. Computer Science Faculty Publications and Presentations 147, Boise State University. https://doi.org/10.18122/cs_facpubs/147/boisestate. https://md.ekstrandom.net/pubs/lkpy

  15. Face++: Cognitive services - leading facial recognition technology (2019). Available at https://www.faceplusplus.com/

  16. Fan DP, Zhang S, Wu YH, Liu Y, Cheng MM, Ren B, Rosin PL, Ji R (2019) Scoot: a perceptual metric for facial sketches. In: Proceedings of the IEEE international conference on computer vision, pp 5612–5622

  17. Feng T, Yang J, Yan Z, Tapia EM, Shi W (2014) Tips: Context-aware implicit user identification using touch screen in uncontrolled environments. In: Proceedings of the 15th workshop on mobile computing systems and applications, pp 1–6

  18. Hassan MM, Alam MGR, Uddin MZ, Huda S, Almogren A, Fortino G (2019) Human emotion recognition using deep belief network architecture. Information Fusion 51:10–18

    Article  Google Scholar 

  19. Hossain MS, Muhammad G (2019) Emotion recognition using deep learning approach from audio–visual emotional big data. Information Fusion 49:69–78

    Article  Google Scholar 

  20. Huang Y, Wang Y, Tai Y, Liu X, Shen P, Li S, Li J, Huang F (2020) Curricularface: adaptive curriculum learning loss for deep face recognition. arXiv:2004.00288

  21. IMDb: Ratings and reviews for new movies and tv shows. (2019). Available at https://www.imdb.com/

  22. Jain AK, Bolle R, Pankanti S (2006) Biometrics: personal identification in networked society, vol 479. Springer Science & Business Media, Berlin

    Google Scholar 

  23. Joho H, Jose JM, Valenti R, Sebe N (2009) Exploiting facial expressions for affective video summarisation. In: Proceedings of the ACM international conference on image and video retrieval, vol 31. ACM

  24. Kairos: Serving businesses with face recognition (2019). Available at https://www.kairos.com/

  25. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings Fourth IEEE international conference on automatic face and gesture recognition (Cat. No. PR00580). IEEE, pp 46–53

  26. Li Y, Hu H, Zhou G (2018) Using data augmentation in continuous authentication on smartphones. IEEE Internet of Things Journal 6(1):628–640

    Article  Google Scholar 

  27. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops. IEEE, pp 94–101

  28. Masthoff J (2011) Group recommender systems: combining individual models. In: Recommender systems handbook. Springer, Berlin, pp 677–702

  29. Microsoft-Azure: Face api - facial recognition software (2019). Available at https://azure.microsoft.com/en-us/services/cognitive-services/face/

  30. Nickel C, Wirtl T, Busch C (2012) Authentication of smartphone users based on the way they walk using k-nn algorithm. In: 2012 eighth international conference on intelligent information hiding and multimedia signal processing. IEEE, pp 16–20

  31. Noldus: Facereader - facial expression recognition software (2019). Available at https://www.noldus.com/human-behavior-research/products/facereader

  32. Qi M, Lu Y, Li J, Li X, Kong J (2008) User-specific iris authentication based on feature selection. In: 2008 international conference on computer science and software engineering, vol 1. IEEE, pp 1040–1043

  33. Schein AI, Popescul A, Ungar LH, Pennock DM (2002) Methods and metrics for cold-start recommendations. In: Proceedings of the 25th annual international ACM SIGIR conference on research and development in information retrieval, SIGIR ’02. ACM, New York, pp 253–260

  34. Shabrina N, Isshiki T, Kunieda H (2016) Fingerprint authentication on touch sensor using phase-only correlation method. In: 2016 7th international conference of information and communication technology for embedded systems (IC-ICTES). IEEE, pp 85–89

  35. Soleymani M, Pantic M, Pun T (2011) Multimodal emotion recognition in response to videos. IEEE Transactions on Affective Computing 3(2):211–223

    Article  Google Scholar 

  36. Tkalc̆ic̆ M, Kos̆ir A, Tasic̆ J (2011) Affective recommender systems: the role of emotions in recommender systems. In: Joint proceedings of the RecSys 2011 workshop on human decision making in recommender systems (Decisions@RecSys’11) and user-centric evaluation of recommender systems and their interfaces-2 (UCERSTI 2) affiliated with the 5th ACM conference on recommender, pp 9–13

  37. Wang J, Zhang J, Luo C, Chen F (2017) Joint head pose and facial landmark regression from depth images. Computational Visual Media 3(3):229–241

    Article  Google Scholar 

  38. Yang MH, Kriegman DJ, Ahuja N (2002) Detecting faces in images: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(1):34–58

    Article  Google Scholar 

  39. Yu Z, Zhou X, Hao Y, Gu J (2006) Tv program recommendation for multiple viewers based on user profile merging. User Model User-Adap Inter 16:63–82

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Imec - WAVES - Ghent University, iGent - Technologiepark 126, Ghent, 9052, Belgium

    Toon De Pessemier, Ine Coppens & Luc Martens

Authors
  1. Toon De Pessemier
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ine Coppens
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luc Martens
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Toon De Pessemier.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

De Pessemier, T., Coppens, I. & Martens, L. Evaluating facial recognition services as interaction technique for recommender systems. Multimed Tools Appl 79, 23547–23570 (2020). https://doi.org/10.1007/s11042-020-09061-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09061-8

Keywords

Search

Navigation