Abstract
In this paper, we describe a wearable first-person video (FPV) analysis system for evaluating the skill levels of caregivers. This is a part of our project that aims to quantize and analyze the tender-care technique known as Humanitude by using wearable sensing and AI technology devices. Using our system, caregivers can evaluate and elevate their care levels by themselves. From the FPVs of care sessions taken by wearable cameras worn by caregivers, we obtained the 3D facial distance, pose and eye-contact states between caregivers and receivers by using facial landmark detection and deep neural network (DNN)-based eye contact detection. We applied statistical analysis to these features and developed algorithms that provide scores for tender-care skill. In experiments, we first evaluated the performance of our DNN-based eye contact detection by using eye contact datasets prepared from YouTube videos and FPVs that assume conversational scenes. We then performed skill evaluations by using Humanitude training scenes involving three novice caregivers, two Humanitude experts and seven middle-level students. The results showed that our eye contact detection outperformed existing methods and that our skill evaluations can estimate the care skill levels.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Acton, G.J., Kang, J.: Interventions to reduce the burden of caregiving for an adult with dementia: a meta-analysis. Res. Nurs. Health 24(5), 349–360 (2001)
Adelman, R.D., Tmanova, L.L., Delgado, D., Dion, S., Lachs, M.S.: Caregiver burden: a clinical review. Jama 311(10), 1052–1060 (2014)
Alzheimer’s Society: Factsheet: Communicating. https://www.alzheimers.org.uk/site/scripts/documents_info.php?documentID=130, [Online; accessed 18-Nov-2016] (2016)
Baltrusaitis, T., Robinson, P., Morency, L.P.: (2016) OpenFace: An Open source facial behavior analysis toolkit. 2016 IEEE Winter Conference on Applications of Computer Vision WACV. https://doi.org/10.1109/WACV.2016.7477553 (2016)
Bertasius, G., Park, H.S., Stella, X.Y., Shi, J.: Am I a Baller? Basketball Performance Assessment from First-Person Videos. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2196–2204. IEEE (2017)
Binetti, N., Harrison, C., Coutrot, A., Johnston, A., Mareschal, I.: Pupil dilation as an index of preferred mutual gaze duration. Royal Society Open Science 3(7). https://doi.org/10.1098/rsos.160086, http://rsos.royalsocietypublishing.org/content/3/7/160086.full.pdf (2016)
Biquand, S., Zittel, B.: Care giving and nursing, work conditions and humanitude®;. Work 41(Supplement 1), 1828–1831 (2012)
Boseley, S.: Dementia research funding to more than double to £66m by 2015. The Guardian (2012)
Campbell, L.C., Keefe, F.J., Scipio, C., McKee, D.C., Edwards, C.L., Herman, S.H., Johnson, L.E., Colvin, O.M., McBride, C.M., Donatucci, C.: Facilitating research participation and improving quality of life for african american prostate cancer survivors and their intimate partners: a pilot study of telephone-based coping skills training. Cancer: Interdiscip. Int. J. Amer. Cancer Soc. 109(S2), 414–424 (2007)
Casado, B., Sacco, P.: Correlates of caregiver burden among family caregivers of older korean americans. J. Gerontol. Ser. B: Psychol. Sci. Soc. Sci. 67(3), 331–336 (2011)
Chong, E., Chanda, K., Ye, Z., Southerland, A., Ruiz, N., Jones, R.M., Rozga, A., Rehg, J.M.: Detecting gaze towards eyes in natural social interactions and its use in child assessment. Proc. ACM Interact. Mob. Wearab. Ubiquit. Technol. 1(3), 43 (2017)
Clyburn, L.D., Stones, M.J., Hadjistavropoulos, T., Tuokko, H., et al.: Predicting caregiver burden and depression in alzheimer’s disease. J. Gerontol. Ser. B 55(1), S2–S13 (2000)
Coon, D.W., Thompson, L., Steffen, A., Sorocco, K., Gallagher-Thompson, D.: Anger and depression management: psychoeducational skill training interventions for women caregivers of a relative with dementia. Gerontologist 43(5), 678–689 (2003)
Dozat, T.: Incorporating nesterov momentum into adam. http://cs229.stanford.edu/proj2015/054_report.pdf, [Online; accessed 25-Aug-2018] (2016)
Dunkin, J.J., Anderson-Hanley, C.: Dementia caregiver burden: a review of the literature and guidelines for assessment and intervention. Neurology 51(1 Suppl 1), S53–S60 (1998)
Etters, L., Goodall, D., Harrison, B.E.: Caregiver burden among dementia patient caregivers: a review of the literature. J. Am. Acad. Nurse Pract. 20(8), 423–428 (2008)
Fathi, A., Hodgins, J.K., Rehg, J.M.: Social Interactions: a First-Person Perspective. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1226–1233. IEEE (2012)
Gineste, Y., Pellissier, J.: Humanitude: comprendre la vieillesse, prendre soin des hommes vieux. A. Colin (2007)
Given, B., Sherwood, P.R., Given, C.W.: What knowledge and skills do caregivers need? J. Soc. Work. Educ. 44(sup3), 115–123 (2008)
Honda, M., Ito, M., Ishikawa, S., Takebayashi, Y., Tierney, L.: Reduction of behavioral psychological symptoms of dementia by multimodal comprehensive care for vulnerable geriatric patients in an acute care hospital: a case series. Case reports in medicine 2016 (2016)
Ishikawa, S., Ito, M., Honda, M., Takebayashi, Y.: The skill representation of a multimodal communication care method for people with dementia. JJAP Conf. Proc. 011616, 4 (2016)
Ito, M., Honda, M.: An examination of the influence of humanitude caregiving on the behavior of older adults with dementia in japan. In: Proceedings of the 8th International Association of Gerontology and Geriatrics European Region Congress (2015)
Kim, H., Chang, M., Rose, K., Kim, S.: Predictors of caregiver burden in caregivers of individuals with dementia. J. Adv. Nurs. 68(4), 846–855 (2012)
Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., Torralba, A.: Eye Tracking for Everyone. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
Larson, E.B., Yaffe, K., Langa, K.M.: New insights into the dementia epidemic. Engl. J. Med. 369(24), 2275–2277 (2013)
Law, H., Ghani, K., Deng, J.: Surgeon technical skill assessment using computer vision based analysis. In: Machine Learning for Healthcare Conference, pp. 88–99 (2017)
Lu, F., Sugano, Y., Okabe, T., Sato, Y.: Gaze estimation from eye appearance: a head pose-free method via eye image synthesis. IEEE Trans. Image Process. 24(11), 3680–3693 (2015)
Ma, M., Fan, H., Kitani, K.M.: Going deeper into first-person activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1894–1903 (2016)
Ministry of Health, Labour and Welfare, Japan: Supply and demand estimation for nursing care personnel for 2025. https://www.mhlw.go.jp/stf/houdou/0000088998.html, [Online; accessed 11-Aug-2018] (2015)
Mitsuzumi, Y., Nakazawa, A., Nishida, T.: Deep Eye Contact Detector: Robust Eye Contact Bid Detection Using Convolutional Neural Network. In: Proceedings of the British Machine Vision Conference (BMVC) (2017)
Northouse, L.L., Katapodi, M.C., Song, L., Zhang, L., Mood, D.W.: Interventions with family caregivers of cancer patients: meta-analysis of randomized trials. CA: Cancer J. Clin. 60(5), 317–339 (2010)
Ostwald, S.K., Hepburn, K.W., Caron, W., Burns, T., Mantell, R.: Reducing caregiver burden: a randomized psychoeducational intervention for caregivers of persons with dementia. Gerontologist 39(3), 299–309 (1999)
Papastavrou, E., Kalokerinou, A., Papacostas, S.S., Tsangari, H., Sourtzi, P.: Caring for a relative with dementia: family caregiver burden. J. Adv. Nurs. 58(5), 446–457 (2007)
Petric, F., Miklić, D., kovačić, Z.: Probabilistic eye contact detection for the robot-assisted asd diagnostic protocol. In: Lončarić S., Cupec, R. (eds.) Proceedings of the Croatian Compter Vision Workshop, Year 4, Center of Excellence for Computer Vision, pp 3–8. University of Zagreb, Osijek (2016)
Povithead: Pivothead KUDU. http://www.pivothead.com/, [Online; accessed 29-Aug-2016] (2016)
Pupil Labs: Pupil labs camera system. https://pupil-labs.com/pupil/, [Online; accessed 25-Aug-2018] (2018)
Singh, S., Arora, C., Jawahar, C.: First person action recognition using deep learned descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2620–2628 (2016)
Smith BA, Yin Q, Feiner SK, Nayar SK: Gaze locking: passive eye contact detection for human-object interaction. In: Proceedings of the 26th annual ACM symposium on User interface software and technology, pp. 271–280. ACM (2013)
Soo Park, H., Shi, J., et al.: Force from motion: decoding physical sensation in a first person video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3834–3842 (2016)
Su, S., Hong, J.P., Shi, J., Park, H.S.: Predicting Behaviors of Basketball Players from First Person Videos. In: CVPR, vol. 2, pp. 3 (2017)
Win, K.K., Chong, M.S., Ali, N., Chan, M., Lim, W.S.: Burden among family caregivers of dementia in the oldest-old: an exploratory study. Front. Med. 4, 205 (2017)
World Health Organization, et al.: Dementia: Fact sheet N 362 (2012)
Ye, Z., Li, Y., Fathi, A., Han, Y., Rozga, A., Abowd, G.D., Rehg, J.M.: Detecting eye contact using wearable eye-tracking glasses. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pp. 699–704. ACM (2012a)
Ye, Z., Li, Y., Fathi, A., Han, Y., Rozga, A., Abowd, G.D., Rehg, J.M.: Detecting eye contact using wearable eye-tracking glasses. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, UbiComp’12, pp 699–704. ACM, New York, https://doi.org/10.1145/2370216.2370368 (2012b)
Ye, Z., Li, Y., Liu, Y., Bridges, C., Rozga, A., Rehg, J.M.: Detecting Bids for Eye Contact Using a Wearable Camera. In: 2015 11Th IEEE International Conference and Workshops On Automatic Face and Gesture Recognition (FG), vol. 1, pp. 1–8. IEEE (2015)
Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: Appearance-based gaze estimation in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4511–4520 (2015)
Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: It’s written all over your face: Full-face appearance-based gaze estimation. arXiv:1611.08860, https://perceptual.mpi-inf.mpg.de/wp-content/blogs.dir/12/files/2016/11/zhang16_arxiv.pdf (2016)
Zhang, X., Sugano, Y., Bulling, A.: Everyday eye contact detection using unsupervised gaze target discovery. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 193–203. ACM (2017)
Acknowledgements
All the experiments were conducted in compliance with the protocol which was reviewed and approved by the ethical committee of Unit for Advanced Studies of the Human Mind, Kyoto University (Permit Number: 30-P-4). This work was supported by JST CREST Grant Number JPMJCR17A5 and JSPS KAKENHI 17H01779, Japan.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Nakazawa, A., Mitsuzumi, Y., Watanabe, Y. et al. First-person Video Analysis for Evaluating Skill Level in the Humanitude Tender-Care Technique. J Intell Robot Syst 98, 103–118 (2020). https://doi.org/10.1007/s10846-019-01052-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10846-019-01052-8