Abstract
Absence of human in-charge in fully autonomous vehicles (FAVs) impose multi-pronged in-cabin monitoring in real-time to monitor any irregular situation. Moreover, occupant’s visual information (such as video, images) transmission from in-cabin of vehicle to the data center and control-room is required for various in-cabin monitoring tasks. However, this information transfer may cause a substantial threat to the occupants. Though applying some patch on face of the occupants protects personal information however, simultaneously it deteriorates the important facial information which is very crucial in monitoring various tasks such as emotion detection. Therefore, a human-computer-interaction (HCI) is required to manage this trade-off between facial information and facial anonymization without any important information loss. Also, this HCI based in-cabin monitoring system should be implicit to retain the users trust in the intelligent transportation system (ITS). In this paper, we have proposed generative adversarial network (GAN) based in-cabin monitoring approach for emotion detection in FAV. In this method, we proposed to generate an artificial (virtual) face having real-facial expressions to provide facial anonymity to the occupants while monitoring FAV cabin. Therefore, this method provides anonymous facial information which is essentially required in detection and monitoring tasks to avoid irregular situations. We have used both publicly available and our own in-cabin dataset to assess our proposed approach. Our experiments show satisfactory results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Koopman, P., Wagner, M.: Autonomous vehicle safety: An interdisciplinary challenge. IEEE Intell. Transp. Syst. Mag. 9(1), 90–96 (2017)
Yurtsever, E., Lambert, J., Carballo, A., Takeda, K.: A survey of autonomous driving: common practices and emerging technologies. IEEE Access 8, 58443–58469 (2020)
Skrickij, V., Šabanovič, E., Žuraulis, V.: Autonomous road vehicles: recent issues and expectations. IET Intel. Transport Syst. 14(6), 471–479 (2020)
Marcondes, F., Durães, D., Gonçalves, F., Fonseca, J., Machado, J., Novais, P.: In-vehicle violence detection in carpooling: a brief survey towards a general surveillance system. In: Dong, Y. (ed.) DCAI 2020. AISC, vol. 1237, pp. 211–220. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-53036-5_23
Khan, M.Q., Lee, S.: A comprehensive survey of driving monitoring and assistance systems. Sensors 19(11), 2574 (2019)
Bell, J.L., Taylor, M.A., Chen, G.X., Kirk, R.D., Leatherman, E.R.: Evaluation of an in-vehicle monitoring system (IVMS) to reduce risky driving behaviors in commercial drivers: comparison of in-cab warning lights and supervisory coaching with videos of driving behavior. J. Saf. Res. 60, 125–136 (2017)
Szawarski, H., Le, J., Rao, M.K.: Monitoring a vehicle cabin. U.S. Patent 10,252,688, issued April 9 (2019)
Song, X.: Safety and clean vehicle monitoring system. U.S. Patent 10,196,070, issued February 5 (2019)
Fridman, L., et al.: MIT advanced vehicle technology study: large scale naturalistic driving study of driver behavior and interaction with automation. IEEE Access 7, 102021–102038 (2019)
Bosch, E., et al.: Emotional GaRage: a workshop on in-car emotion recognition and regulation. In: Adjunct Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 44–49 (2018)
Qin, Z., Weng, J., Cui, Y., Ren, K.: Privacy-preserving image processing in the cloud. IEEE Cloud Comput. 5(2), 48–57 (2018)
Xia, Z., Zhu, Y., Sun, X., Qin, Z., Ren, K.: Towards privacy-preserving content-based image retrieval in cloud computing. IEEE Trans. Cloud Comput. 6(1), 276–286 (2015)
Taeihagh, A., Lim, H.S.M.: Governing autonomous vehicles: emerging responses for safety, liability, privacy, cybersecurity, and industry risks. Transp. Rev. 39(1), 103–128 (2019)
Glancy, D.J.: Privacy in autonomous vehicles. Santa Clara L. Rev. 52, 1171 (2012)
Collingwood, L.: Privacy implications and liability issues of autonomous vehicles. Inf. Commun. Technol. Law 26(1), 32–45 (2017)
Lim, H.S.M., Taeihagh, A.: Autonomous vehicles for smart and sustainable cities: an in-depth exploration of privacy and cybersecurity implications. Energies 11(5), 1062 (2017)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples (2014)
Braunegg, A., et al.: APRICOT: A Dataset of Physical Adversarial Attacks on Object Detection. arXiv preprint arXiv:1912.08166 (2019)
Fox, E., Lester, V., Russo, R., Bowles, R.J., Pichler, A., Dutton, K.: Facial expressions of emotion: are angry faces detected more efficiently? Cogn. Emot. 14(1), 61–92 (2000)
Blanz, V., Scherbaum, K., Vetter, T., Seidel, H.P.: Exchanging faces in images. In: Computer Graphics Forum, vol. 23, no. 3, pp. 669–676. Blackwell Publishing, Inc., Oxford and Boston (2004)
Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., Nayar, S.K.: Face swapping: automatically replacing faces in photographs. In: ACM SIGGRAPH Papers, pp. 1–8 (2008)
Lin, Y., Wang, S., Lin, Q., Tang, F.: Face swapping under large pose variations: a 3D model based approach. In: IEEE International Conference on Multimedia and Expo, pp. 333–338. IEEE (2012)
Zhang, Y., Zheng, L., Thing, V.L.: Automated face swapping and its detection. In: IEEE 2nd International Conference on Signal and Image Processing (ICSIP), pp. 15–19. IEEE (2017)
Nirkin, Y., Masi, I., Tuan, A.T., Hassner, T., Medioni, G.: On face segmentation, face swapping, and face perception. In: 13th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 98–105. IEEE (2018)
Natsume, R., Yatagawa, T., Morishima, S.: Rsgan: face swapping and editing using face and hair representation in latent spaces. arXiv preprint arXiv:1804.03447 (2018)
Korshunov, P., Marcel, S.: Deepfakes: a new threat to face recognition? Assessment and detection. arXiv preprint arXiv:1812.08685 (2018)
Natsume, R., Yatagawa, T., Morishima, S.: Fsnet: an identity-aware generative model for image-based face swapping. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11366, pp. 117–132. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20876-9_8
Bailer, W.: Face swapping for solving collateral privacy issues in multimedia analytics. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, W.-H., Vrochidis, S. (eds.) MMM 2019. LNCS, vol. 11295, pp. 169–177. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05710-7_14
Nirkin, Y., Keller, Y., Hassner, T.: FSGAN: subject agnostic face swapping and reenactment. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7184–7193 (2019)
Naruniec, J., Helminger, L., Schroers, C., Weber, R.M.: High‐resolution neural face swapping for visual effects. In: Computer Graphics Forum, vol. 3, no. 4, pp. 173–184 (2020)
Mishra, A., Kim, J., Kim, D., Cha, J., Kim, S.: An intelligent in-cabin monitoring system in fully autonomous vehicles. In: 17th International SoC Conference (ISOCC 2020), South Korea (2020)
Kim, S., Shrestha, R.: Automotive Cyber Security: Introduction, Challenges, and Standardization. Springer Singapore, Singapore (2020). https://doi.org/10.1007/978-981-15-8053-6
Shrestha, R., Kim, S.: Integration of IoT with blockchain and homomorphic encryption: challenging issues and opportunities. In: Advances in Computers, vol. 115, pp. 293–331. Elsevier (2019)
Acknowledgment
This research was supported by Korea Research Fellowship program funded by the Ministry of Science and ICT through the National Research Foundation of Korea (NRF-2019H1D3A1A01071115).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Mishra, A., Cha, J., Kim, S. (2021). HCI Based In-Cabin Monitoring System for Irregular Situations with Occupants Facial Anonymization. In: Singh, M., Kang, DK., Lee, JH., Tiwary, U.S., Singh, D., Chung, WY. (eds) Intelligent Human Computer Interaction. IHCI 2020. Lecture Notes in Computer Science(), vol 12616. Springer, Cham. https://doi.org/10.1007/978-3-030-68452-5_40
Download citation
DOI: https://doi.org/10.1007/978-3-030-68452-5_40
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68451-8
Online ISBN: 978-3-030-68452-5
eBook Packages: Computer ScienceComputer Science (R0)