research-article

Real-Time System for Driver Fatigue Detection by RGB-D Camera

Authors:

Jinhui TangAuthors Info & Claims

ACM Transactions on Intelligent Systems and Technology (TIST), Volume 6, Issue 2

Article No.: 22, Pages 1 - 17

https://doi.org/10.1145/2629482

Published: 31 March 2015 Publication History

Abstract

Drowsy driving is one of the major causes of fatal traffic accidents. In this article, we propose a real-time system that utilizes RGB-D cameras to automatically detect driver fatigue and generate alerts to drivers. By introducing RGB-D cameras, the depth data can be obtained, which provides extra evidence to benefit the task of head detection and head pose estimation. In this system, two important visual cues (head pose and eye state) for driver fatigue detection are extracted and leveraged simultaneously. We first present a real-time 3D head pose estimation method by leveraging RGB and depth data. Then we introduce a novel method to predict eye states employing the WLBP feature, which is a powerful local image descriptor that is robust to noise and illumination variations. Finally, we integrate the results from both head pose and eye states to generate the overall conclusion. The combination and collaboration of the two types of visual cues can reduce the uncertainties and resolve the ambiguity that a single cue may induce. The experiments were performed using an inside-car environment during the day and night, and theyfully demonstrate the effectiveness and robustness of our system as well as the proposed methods of predicting head pose and eye states.

References

[1]

Kircher Albert, Marcus Uddman, and Jesper Sandin. 2002. Vehicle control and drowsiness. Retrieved February 16, 2015, from http://www.vti.se/en/publications/pdf/vehicle-control-and-drowsiness.pdf.

[2]

Jose M. Alonso, Luis Magdalena, and Serge Guillaume. 2004. KBCT: A knowledge extraction and representation tool for fuzzy logic based systems. In Proceedings of the Conference on Fuzzy Systems. 989--994.

[3]

Anon. 1999. Perclos and Eyetracking: Challenge and Opportunity. Technical Report. Applied Science Laboratories, Bedford, MA.

[4]

Luis Miguel Bergasa, Jesús Nuevo, Miguel Ángel Sotelo, Rafael Barea, and María Elena López Guillén. 2006. Real-time system for monitoring driver vigilance. IEEE Transactions on Intelligent Transportation Systems 7, 1, 63--77.

Digital Library

[5]

Christoph Bregler and Jitendra Malik. 1998. Tracking people with twists and exponential maps. In Proceedings of the Conference on Computer Vision and Pattern Recognition. 8--15.

Digital Library

[6]

Michael D. Breitenstein, Daniel Küttel, Thibaut Weise, Luc J. Van Gool, and Hanspeter Pfister. 2008. Real-time face pose estimation from single range images. In Proceedings of the Conference on Computer Vision and Pattern Recognition.

[7]

Qin Cai, David Gallup, Cha Zhang, and Zhengyou Zhang. 2010. 3D deformable face tracking with a commodity depth camera. In Proceedings of the European Conference on Computer Vision: Part III. 229--242.

Digital Library

[8]

Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2, 3, 27.

Digital Library

[9]

Jie Chen, Shiguang Shan, Chu He, Guoying Zhao, Matti Pietikäinen, Xilin Chen, and Wen Gao. 2010. WLD: A robust local image descriptor. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 9, 1705--1720.

Digital Library

[10]

Jyh-Yuan Deng and Feipei Lai. 1997. Region-based template deformation and masking for eye-feature extraction and description. Pattern Recognition 30, 3, 403--419.

[11]

Gabriele Fanelli, Juergen Gall, and Luc J. Van Gool. 2011. Real time head pose estimation with random regression forests. In Proceedings of the Conference on Computer Vision and Pattern Recognition. 617--624.

Digital Library

[12]

Jennifer Healey and Rosalind W. Picard. 2000. SmartCar: Detecting driver stress. In Proceedings of the Conference on Pattern Recognition. 4218--4221.

Digital Library

[13]

Ueno Hiroshi, Masayuki Kaneda, and Masataka Tsukino. 1994. Development of drowsiness detection system. In Proceedings of the Vehicle Navigation and Information Systems Conference.

[14]

Berthold K. P. Horn and Brian G. Schunck. 1981. Determining optical flow. Artificial Intelligence 17, 1--3, 185--203.

Digital Library

[15]

Qiang Ji and Xiaojie Yang. 2002. Real-time eye, gaze, and face pose tracking for monitoring driver vigilance. Real-Time Imaging 8, 5, 357--377.

Digital Library

[16]

Farid Abedan Kondori, Shahrouz Youse, Haibo Li, Samuel Sonning, and Sabina Sonning. 2011. 3D head pose estimation using the Kinect. In Proceedings of the Conference on Wireless Communications and Signal Processing. 1--4.

[17]

Peilin Lan, Qiang Ji, and Carl G. Looney. 2002. Information fusion with Bayesian networks for monitoring human fatigue. In Proceedings of the Conference on Information Fusion. 535--542.

[18]

Fan Liu, Zhenmin Tang, and Jinhui Tang. 2013a. WLBP: Weber local binary pattern for local image description. Neurocomputing 120, 325--335.

[19]

Wu Liu, Yongdong Zhang, Sheng Tang, Jinhui Tang, Richang Hong, and Jintao Li. 2013b. Accurate estimation of human body orientation from RGB-D sensors. IEEE Transactions on Cybernetics 43, 5, 1442--1452.

[20]

Hieu Tat Nguyen and Arnold W. M. Smeulders. 2006. Robust tracking using foreground-background texture discrimination. International Journal of Computer Vision 69, 3, 277--293.

Digital Library

[21]

Timo Ojala, Matti Pietikäinen, and Topi Mäenpää. 2002. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 7, 971--987.

Digital Library

[22]

Edgar Seemann, Kai Nickel, and Rainer Stiefelhagen. 2004. Head pose estimation using stereo vision for human-robot interaction. In Proceedings of the Conference on Automatic Face and Gesture Recognition. 626--631.

Digital Library

[23]

Rajinda Senaratne, David Hardy, Bill Vanderaa, and Saman K. Halgamuge. 2007. Driver fatigue detection by fusing multiple cues. In Advances in Neural Networks—ISNN 2007. Lecture Notes in Computer Science, Vol. 4492. Springer, 801--809.

Digital Library

[24]

Paul Smith, Mubarak Shah, and Niels da Vitoria Lobo. 2003. Determining driver visual attention with one camera. IEEE Transactions on Intelligent Transportation Systems 4, 4, 205--218.

Digital Library

[25]

Sheng Tang, Yan-Tao Zheng, Yu Wang, and Tat-Seng Chua. 2012. Sparse ensemble learning for concept detection. IEEE Transactions on Multimedia 14, 1, 43--54.

Digital Library

[26]

Yingli Tian, Takeo Kanade, and Jeffrey F. Cohn. 2000. Eye-state action unit detection by Gabor wavelets. In Advances in Multimodal Interfaces—ICMI 2000. Lecture Notes in Computer Science, Vol. 1948. Springer, 143--150.

Digital Library

[27]

Angelos Tzotsos and Demetre Argialas. 2006. A support vector machine approach for object based image analysis. In Proceedings of the Conference on Object-Based Image Analysis.

[28]

Vladimir Vapnik. 1999. The Nature of Statistical Learning Theory. Springer.

Digital Library

[29]

Peng Wang and Qiang Ji. 2005. Learn discriminant features for multi-view face and eye detection. In Proceedings of the Conference on Computer Vision and Pattern Recognition. 373--379.

Digital Library

[30]

Qiong Wang and Jingyu Yang. 2006. Eye location and eye state detection in facial images with unconstrained background. Journal of Information and Computing Science 1, 5, 284--289.

[31]

Yu-Shan Wu, Ting-Wei Lee, Quen-Zong Wu, and Heng-Sung Liu. 2010. An eye state recognition method for drowsiness detection. In Proceedings of the Vehicular Technology Conference (Spring). 1--5.

[32]

Cui Xu, Ying Zheng, and Zengfu Wang. 2008. Efficient eye states detection in real-time for drowsy driving monitoring system. In Proceedings of the Conference on Information and Automation. 170--174.

[33]

Liyan Zhang, Dmitri V. Kalashnikov, and Sharad Mehrotra. 2013a. A unified framework for context assisted face clustering. In Proceedings of the Conference on Multimedia Retrieval. 9--16.

Digital Library

[34]

Liyan Zhang, Dmitri V. Kalashnikov, Sharad Mehrotra, and Ronen Vaisenberg. 2013b. Context-based person identification framework for smart video surveillance. Machine Vision and Applications 25, 7, 1711--1725.

Digital Library

[35]

Tao Zhao and Ramakant Nevatia. 2004. Tracking multiple humans in complex situations. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 9, 1208--1221.

Digital Library

Cited By

Misir OCelik M(2025)Visual-based obstacle avoidance method using advanced CNN for mobile robotsInternet of Things10.1016/j.iot.2025.10153831(101538)Online publication date: May-2025
https://doi.org/10.1016/j.iot.2025.101538
Sun ZMiao YJeon JKong YPark G(2023)Facial feature fusion convolutional neural network for driver fatigue detectionEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.106981126:PCOnline publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.engappai.2023.106981
Verma RPargal SDas DParbat TKambalapalli SMitra BChakraborty S(2022)Impact of Driving Behavior on Commuter’s Comfort during Cab Rides: Towards a New Perspective of Driver RatingACM Transactions on Intelligent Systems and Technology10.1145/3523063Online publication date: 22-Mar-2022
https://doi.org/10.1145/3523063
Show More Cited By

Index Terms

Real-Time System for Driver Fatigue Detection by RGB-D Camera
1. Computer systems organization
  1. Architectures
    1. Parallel architectures
      1. Multiple instruction, multiple data

Recommendations

Driver's drowsiness inference based on hidden Markov model using head pose and eye-blink tracking
HCIK '15: Proceedings of HCI Korea

This paper proposes the system to decide the drowsiness drive by Hidden Markov Model (HMM) using the head-pose-estimation and eye blink. Head-Pose-Estimation is using the Corner detection, Optical flow and POSIT algorithm. Eye blinked is detected by ...
A biological and real-time framework for hand gestures and head poses
UAHCI'13: Proceedings of the 7th international conference on Universal Access in Human-Computer Interaction: design methods, tools, and interaction techniques for eInclusion - Volume Part I

Human-robot interaction is an interdisciplinary research area that aims at the development of social robots. Since social robots are expected to interact with humans and understand their behavior through gestures and body movements, cognitive psychology ...
Gaze Estimation in the 3D Space Using RGB-D Sensors

We address the problem of 3D gaze estimation within a 3D environment from remote sensors, which is highly valuable for applications in human---human and human---robot interactions. To the contrary of most previous works, which are limited to screen ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Intelligent Systems and Technology

ACM Transactions on Intelligent Systems and Technology Volume 6, Issue 2

Special Section on Visual Understanding with RGB-D Sensors

May 2015

381 pages

ISSN:2157-6904

EISSN:2157-6912

DOI:10.1145/2753829

Editor:
Huan Liu
Arizona State University

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 March 2015

Accepted: 01 March 2014

Revised: 01 December 2013

Received: 01 June 2013

Published in TIST Volume 6, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

973 Program
National Nature Science Foundation of China
Program for New Century Excellent Talents in University

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
722
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)3

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Misir OCelik M(2025)Visual-based obstacle avoidance method using advanced CNN for mobile robotsInternet of Things10.1016/j.iot.2025.10153831(101538)Online publication date: May-2025
https://doi.org/10.1016/j.iot.2025.101538
Sun ZMiao YJeon JKong YPark G(2023)Facial feature fusion convolutional neural network for driver fatigue detectionEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.106981126:PCOnline publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.engappai.2023.106981
Verma RPargal SDas DParbat TKambalapalli SMitra BChakraborty S(2022)Impact of Driving Behavior on Commuter’s Comfort during Cab Rides: Towards a New Perspective of Driver RatingACM Transactions on Intelligent Systems and Technology10.1145/3523063Online publication date: 22-Mar-2022
https://doi.org/10.1145/3523063
Guria MBhowmik B(2022)IoT-Enabled Driver Drowsiness Detection Using Machine Learning2022 Seventh International Conference on Parallel, Distributed and Grid Computing (PDGC)10.1109/PDGC56933.2022.10053235(519-524)Online publication date: 25-Nov-2022
https://doi.org/10.1109/PDGC56933.2022.10053235
Dziuda ŁBaran PZieliński PMurawski KDziwosz MKrej MPiotrowski MStablewski RWojdas AStrus WGasiul HKosobudzki MBortkiewicz A(2021)Evaluation of a Fatigue Detector Using Eye Closure-Associated Indicators Acquired from Truck Drivers in a Simulator StudySensors10.3390/s2119644921:19(6449)Online publication date: 27-Sep-2021
https://doi.org/10.3390/s21196449
Chaabene SBouaziz BBoudaya AHökelmann AAmmar AChaari L(2021)Convolutional Neural Network for Drowsiness Detection Using EEG SignalsSensors10.3390/s2105173421:5(1734)Online publication date: 3-Mar-2021
https://doi.org/10.3390/s21051734
El Khatib AOu CKarray F(2020)Driver Inattention Detection in the Context of Next-Generation Autonomous Vehicles Design: A SurveyIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2019.294087421:11(4483-4496)Online publication date: Nov-2020
https://doi.org/10.1109/TITS.2019.2940874
Kolodziej MTarnowski PSawicki DMajkowski ARak RBala APluta A(2020)Fatigue Detection Caused by Office Work With the Use of EOG SignalIEEE Sensors Journal10.1109/JSEN.2020.301240420:24(15213-15223)Online publication date: 15-Dec-2020
https://doi.org/10.1109/JSEN.2020.3012404
Murugan SSelvaraj JSahayadhas A(2019)Analysis of Different Measures to Detect Driver States: A Review2019 IEEE International Conference on System, Computation, Automation and Networking (ICSCAN)10.1109/ICSCAN.2019.8878844(1-6)Online publication date: Mar-2019
https://doi.org/10.1109/ICSCAN.2019.8878844
Liu FLi XLv TXu F(2019)A Review of Driver Fatigue Detection: Progress and Prospect2019 IEEE International Conference on Consumer Electronics (ICCE)10.1109/ICCE.2019.8662098(1-6)Online publication date: Jan-2019
https://doi.org/10.1109/ICCE.2019.8662098
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents