Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

EmoSnaps: a mobile application for emotion recall from facial expressions

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

We introduce EmoSnaps, a mobile application that captures unobtrusively pictures of one’s facial expressions throughout the day and uses them for later recall of her momentary emotions. We describe two field studies that employ EmoSnaps in an attempt to investigate if and how individuals and their relevant others infer emotions from self-face and familiar face pictures, respectively. Study 1 contrasted users’ recalled emotions as inferred from EmoSnaps’ self-face pictures to ground truth data as derived from Experience Sampling. Contrary to our expectations, we found that people are better able to infer their past emotions from a self-face picture the longer the time has elapsed since capture. Study 2 assessed EmoSnaps’ ability to capture users’ experiences while interacting with different mobile apps. The study revealed systematic variations in users’ emotions while interacting with different categories of mobile apps (such as productivity and entertainment), social networking services, as well as direct social communications through phone calls and instant messaging, but also diurnal and weekly patterns of happiness as inferred from EmoSnaps’ self-face pictures. All in all, the results of both studies provided us with confidence over the validity of self-face pictures captured through EmoSnaps as memory cues for emotion recall, and the effectiveness of the EmoSnaps tool in measuring users’ momentary experiences.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. Z-transformation was applied to normalize the distance Δ between ESM and reconstruction ratings.

References

  1. Larson R, Csikszentmihalyi M (1983) The experience sampling method. New Direct Methodol Soc Behav Sci

  2. Ekman P (1993) Facial expression and emotion. Am Psychol 48:384

    Article  Google Scholar 

  3. Ortony A, Turner TJ (1990) What’s basic about basic emotions? Psychol Rev 97:315

    Article  Google Scholar 

  4. Russell JA (1994) Is there universal recognition of emotion from facial expressions? A review of the cross-cultural studies. Psychol Bull 115:102

    Article  Google Scholar 

  5. Pantic M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22:1424–1445

    Article  Google Scholar 

  6. Cohen I, Sebe N, Garg A, Chen LS, Huang TS (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 91:160–187

    Article  Google Scholar 

  7. Azcarate A, Hageloh F, van de Sande K, Valenti R (2005) Automatic facial emotion recognition. Universiteit van Amsterdam

  8. Teeters A, El Kaliouby R, Picard R (2006) Self-Cam: feedback from what would be your social partner. p 138

  9. Gruebler A, Suzuki K (2010) Measurement of distal EMG signals using a wearable device for reading facial expressions. pp 4594–4597

  10. Bartlett FC (1995) Remembering: a study in experimental and social psychology. Cambridge University Press, Cambridge

    Book  Google Scholar 

  11. Tulving E (1984) Precis of elements of episodic memory. Behav Brain Sci 7:223–268

    Article  Google Scholar 

  12. Robinson MD, Clore GL (2002) Belief and feeling: evidence for an accessibility model of emotional self-report. Psychol Bull 128:934–960. doi:10.1037//0033-2909.128.6.934

    Article  Google Scholar 

  13. Barsalou LW (1988) The content and organization of autobiographical memories. Cambridge University Press, Cambridge

    Google Scholar 

  14. Lee ML, Dey AK (2008) Lifelogging memory appliance for people with episodic memory impairment. In: Proceedings of the 10th international conference on Ubiquitous computing. ACM, New York, NY, USA, pp 44–53

  15. Hodges S, Williams L, Berry E, Izadi S, Srinivasan J, Butler A, Smyth G, Kapur N, Wood K (2006) SenseCam: a retrospective memory aid. UbiComp 2006: Ubiquitous Computing 177–193

  16. Gouveia R, Karapanos E (2013) Footprint tracker: supporting diary studies with Lifelogging. In: Proceedings of CHI 2013

  17. Sas C, Fratczak T, Rees M, Gellersen H, Kalnikaite V, Coman A, Höök K (2013) AffectCam: arousal-augmented sensecam for richer recall of episodic memories. CHI’13 extended abstracts on human factors in computing systems. ACM, pp 1041–1046

  18. Conway MA (2009) Episodic memories. Neuropsychologia 47:2305–2313

    Article  Google Scholar 

  19. Kalnikaite V, Sellen A, Whittaker S, Kirk D (2010) Now let me see where i was: understanding how lifelogs mediate memory. In: Proceedings of the 28th international conference on human factors in computing systems, pp 2045–2054

  20. Whittaker S, Kalnikaitė V, Petrelli D, Sellen A, Villar N, Bergman O, Clough P, Brockmeier J (2012) Socio-technical lifelogging: deriving design principles for a future proof digital past. Human Comput Interact 27:37–62

    Google Scholar 

  21. Karapanos E, Martens JB, Hassenzahl M (2009) Reconstructing experiences through sketching. arXiv preprint arXiv:0912.5343

  22. Young AW, McWeeny KH, Hay DC, Ellis AW (1986) Matching familiar and unfamiliar faces on identity and expression. Psychol Res 48:63–68

    Article  Google Scholar 

  23. Kenny DA, Acitelli LK (2001) Accuracy and bias in the perception of the partner in a close relationship. J Pers Soc Psychol 80:439

    Article  Google Scholar 

  24. Anderson SJ, Conway MA (1993) Investigating the structure of autobiographical memories. J Exp Psychol Learn Mem Cogn 19:1178

    Article  Google Scholar 

  25. Kahneman D, Krueger AB, Schkade DA, Schwarz N, Stone AA (2004) A survey method for characterizing daily life experience: the day reconstruction method. Science 306:1776–1780. doi:10.1126/science.1103572

    Article  Google Scholar 

  26. Conner TS, Reid KA (2012) Effects of intensive mobile happiness reporting in daily life. Soc Psychol Pers Sci 3:315–323

    Article  Google Scholar 

  27. MacKerron G, Mourato S (2010) LSE’s mappiness project may help us track the national mood: but how much should we consider happiness in deciding public policy? British Politics and Policy at LSE

  28. Killingsworth MA, Gilbert DT (2010) A wandering mind is an unhappy mind. Science 330:932. doi:10.1126/science.1192439

    Article  Google Scholar 

  29. Isaacs E, Konrad A, Walendowski A, Lennig T, Hollis V, Whittaker S (2013) Echoes from the past: how technology mediated reflection improves well-being. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 1071–1080

  30. Sanches P, Höök K, Vaara E, Weymann C, Bylund M, Ferreira P, Peira N, Sjölinder M (2010) Mind the body!: designing a mobile stress management application encouraging personal reflection. In: Proceedings of the 8th ACM conference on designing interactive systems. ACM, pp 47–56

  31. Guan Z, Lee S, Cuddihy E, Ramey J (2006) The validity of the stimulated retrospective think-aloud method as measured by eye tracking. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 1253–1262

  32. Stone AA, Smyth JM, Pickering T, Schwartz J (1996) Daily mood variability: form of diurnal patterns and determinants of diurnal patterns. J Appl Soc Psychol 26:1286–1305

    Article  Google Scholar 

  33. Ryan RM, Bernstein JH, Brown KW (2010) Weekends, work, and well-being: psychological need satisfactions and day of the week effects on mood, vitality, and physical symptoms. J Soc Clin Psychol 29:95–122

    Article  Google Scholar 

  34. Rystrom DS, Benson ED (1989) Investor psychology and the day-of-the-week effect, pp 75–78

  35. Jin B, Peña JF (2010) Mobile communication in romantic relationships: mobile phone use, relational uncertainty, love, commitment, and attachment styles. Commun Rep 23:39–51. doi:10.1080/08934211003598742

    Article  Google Scholar 

  36. Green MC, Hilken J, Friedman H, Grossman K, Gasiewskj J, Adler R, Sabini J (2005) Communication via instant messenger: short-and long-term effects. J Appl Soc Psychol 35:445–462

    Article  Google Scholar 

  37. Oulasvirta A, Rattenbury T, Ma L, Raita E (2012) Habits make smartphone use more pervasive. Pers Ubiquit Comput 16:105–114

    Article  Google Scholar 

  38. Böhmer M, Hecht B, Schöning J, Krüger A, Bauer G (2011) Falling asleep with angry birds, Facebook and kindle: a large scale study on mobile application usage. In: Proceedings of the 13th international conference on human computer interaction with mobile devices and services. pp 47–56

  39. Ickin S, Wac K, Fiedler M, Janowski L, Hong J-H, Dey AK (2012) Factors influencing quality of experience of commonly used mobile applications. Commun Mag 50:48–56

    Article  Google Scholar 

  40. Verkasalo H (2008) Contextual patterns in mobile service usage. Pers Ubiquit Comput 13:331–342. doi:10.1007/s00779-008-0197-0

    Article  Google Scholar 

  41. De Guzman ES, Sharmin M, Bailey BP (2007) Should I call now? Understanding what context is considered when deciding whether to initiate remote communication via mobile devices. Proc Graph Interface 2007:143–150

    Google Scholar 

  42. Larson R (1989) Is feeling “in control” related to happiness in daily life? Psychol Rep 64:775–784

    Article  Google Scholar 

  43. Karapanos E (2011) Experience sampling, day reconstruction, what’s next? Towards technology-assisted reconstruction. M-ITI, Funchal

    Google Scholar 

Download references

Acknowledgments

This work was conducted in the frames of Logica Service Design Lab with support of Knowledge + incentive system in Madeira, Portugal. The authors also acknowledge the financial support of the Future and Emerging Technologies (FET) programme within the 7th Framework Programme for Research of the European Commission, under FET Grant Number: 612933.

Author information

Authors and Affiliations

  1. Faculty of Informatics, University of Lugano, Lugano, Switzerland

    Evangelos Niforatos

  2. Madeira Interactive Technologies Institute, Funchal, Portugal

    Evangelos Karapanos

Authors
  1. Evangelos Niforatos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Evangelos Karapanos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Evangelos Niforatos.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Niforatos, E., Karapanos, E. EmoSnaps: a mobile application for emotion recall from facial expressions. Pers Ubiquit Comput 19, 425–444 (2015). https://doi.org/10.1007/s00779-014-0777-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-014-0777-0

Keywords

Search

Navigation