4D: A Real-Time Driver Drowsiness Detector Using Deep Learning
<p>Media Research Lab (MRL) Eye dataset.</p> "> Figure 2
<p>Diagram of the proposed model.</p> "> Figure 3
<p>Facial extraction.</p> "> Figure 4
<p>Eye extraction.</p> "> Figure 5
<p>Convolutional neural network.</p> "> Figure 6
<p>Proposed 4D model.</p> "> Figure 7
<p>Architecture of the VGG16 model.</p> "> Figure 8
<p>Architecture of the VGG19 model.</p> "> Figure 9
<p>Training and testing accuracy curves for different models.</p> "> Figure 10
<p>Training and testing precision curves for different models.</p> "> Figure 11
<p>Training and testing recall curves for different models.</p> "> Figure 12
<p>Confusion matrices of different models.</p> "> Figure 13
<p>Activation map for open and closed eyes.</p> "> Figure 14
<p>Flow diagram of the real-time implementation.</p> "> Figure 15
<p>Real-time implementation of drowsiness detection (with and without glasses).</p> ">
Abstract
:1. Introduction
- A convolutional-neural-network-based novel classification model was developed for drowsiness detection on the basis of eye state.
- The results show that the model is capable of classifying eyes as either open or closed.
- A class activation map (CAM) for the proposed model is shown to visualize the learning area of the images that is used to make predictions.
2. Related Work
3. Materials and Methods
3.1. Dataset Selection
3.2. Data Preprocessing
3.3. Data Augmentation
3.4. Region of Interest Selection
3.5. Eye Aspect Ratio Calculation
3.6. Convolutional Neural Network
- Convolution layer: The convolution operation was carried out with a stride of 1 by sliding a convolutional filter with a size of 33 and 5 across the input data matrix. We experimented with various kernels, ranging from 64 to 1024, and varying step sizes before settling on a combination of kernels that maximized the validation accuracy.
- Activation function: A weighted total was calculated by the activation function, and then bias was added to it to decide whether or not to activate a neuron. We used the nonlinear activation function of ReLU [29], which converted negative elements into 0, and it may be written as Relu(x) = max(0,x), where x indicates a neuron’s input.
- Batch normalization: The batch normalization layer allowed each layer to increase its independent learning. It made the outputs of the previous layers more normal. Every activation function followed the batch normalization layer. It aided in the speeding up of the learning process [30] and reduces the sensitivity to fluctuations in the input data. As a result, the neural network was stabilized. Batch normalization was also used to keep the data’s distribution consistent.
- Dropout: A model is considered to be over-fitted when it learns and performs poorly on the test dataset, but gives good accuracy on the training dataset. By randomly setting the activation to zero, the dropout layer is responsible for preventing over-fitting and improving performance [31].
- Maxpooling layer: The process of choosing the largest element from a feature map is known as maxpooling. A feature map is produced from the output of the maxpooling layer.
- Fully connected layer: When an input has been multiplied by a weight matrix, a bias vector is then produced in a fully connected layer. One or more fully connected layers are added after the convolution layers.
- Output layer: Calculating the probability of each class occurring with a given input image is the responsibility of the output layer. With a sigmoid activation function, it produces a two-dimensional vector (3). The sigmoid activation function can be presented as follows:
3.7. VGG16 and VGG19
4. Results and Discussion
4.1. Accuracy Evaluation
4.2. Model Learning Visualization
4.3. Real-Time Implementation of Drowsiness Detection
5. Conclusions
6. Future Work
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Jabbar, R.; Shinoy, M.; Kharbeche, M.; Al-Khalifa, K.; Krichen, M.; Barkaoui, K. Driver Drowsiness Detection Model Using Convolutional Neural Networks Techniques for Android Application. In Proceedings of the ICIoT 2020, Doha, Qatar, 2–5 February 2020; pp. 237–242. [Google Scholar] [CrossRef]
- Ahsan, M.M.; Li, Y.; Zhang, J.; Ahad, M.T.; Yazdan, M.M.S. Face recognition in an unconstrained and real-time environment using novel BMC-LBPH methods incorporates with DJI vision sensor. J. Sens. Actuator Netw. 2020, 9, 54. [Google Scholar] [CrossRef]
- Ramzan, M.; Khan, H.U.; Awan, S.M.; Ismail, A.; Ilyas, M.; Mahmood, A. A survey on state-of-the-art drowsiness detection techniques. IEEE Access 2019, 7, 61904–61919. [Google Scholar] [CrossRef]
- Abtahi, S.; Omidyeganeh, M.; Shirmohammadi, S.; Hariri, B. YawDD: A yawning detection dataset. In Proceedings of the 5th ACM Multimedia Systems Conference, Singapore, 19–21 March 2014; pp. 24–28. [Google Scholar]
- Weng, C.H.; Lai, Y.H.; Lai, S.H. Driver drowsiness detection via a hierarchical temporal deep belief network. In Proceedings of the Asian Conference on Computer Vision, Taipei, Taiwan, 20–24 November 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 117–133. [Google Scholar]
- Revelo, A.; Álvarez, R.; Grijalva, F. Human drowsiness detection in real time, using computer vision. In Proceedings of the 2019 IEEE Fourth Ecuador Technical Chapters Meeting (ETCM), Guayaquil, Ecuador, 13–15 November 2019; pp. 1–6. [Google Scholar]
- Adhikary, A.; Murad, S.A.; Munir, M.S.; Hong, C.S. Edge Assisted Crime Prediction and Evaluation Framework for Machine Learning Algorithms. In Proceedings of the 2022 International Conference on Information Networking (ICOIN), Jeju-si, Republic of Korea, 12–15 January 2022; pp. 417–422. [Google Scholar]
- Fan, X.; Yin, B.; Sun, Y. Yawning detection based on gabor wavelets and LDA. J. Beijing Univ. Technol. 2009, 35, 409–413. [Google Scholar]
- Zhang, Z.; Zhang, J. A new real-time eye tracking based on nonlinear unscented Kalman filter for monitoring driver fatigue. J. Control Theory Appl. 2010, 8, 181–188. [Google Scholar] [CrossRef]
- Philip, P.; Sagaspe, P.; Moore, N.; Taillard, J.; Charles, A.; Guilleminault, C.; Bioulac, B. Fatigue, sleep restriction and driving performance. Accid. Anal. Prev. 2005, 37, 473–478. [Google Scholar] [CrossRef] [PubMed]
- Tremaine, R.; Dorrian, J.; Lack, L.; Lovato, N.; Ferguson, S.; Zhou, X.; Roach, G. The relationship between subjective and objective sleepiness and performance during a simulated night-shift with a nap countermeasure. Appl. Ergon. 2010, 42, 52–61. [Google Scholar] [CrossRef] [PubMed]
- Savaş, B.K.; Becerikli, Y. Real time driver fatigue detection based on SVM algorithm. In Proceedings of the 2018 6th International Conference on Control Engineering & Information Technology (CEIT), Istanbul, Turkey, 25–27 October 2018; pp. 1–4. [Google Scholar]
- Jalilifard, A.; Pizzolato, E.B. An efficient K-NN approach for automatic drowsiness detection using single-channel EEG recording. In Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA, 16–20 August 2016; pp. 820–824. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- Awais, M.; Badruddin, N.; Drieberg, M. A hybrid approach to detect driver drowsiness utilizing physiological signals to improve system performance and wearability. Sensors 2017, 17, 1991. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Chellappa, A.; Reddy, M.S.; Ezhilarasie, R.; Suguna, S.K.; Umamakeswari, A. Fatigue detection using raspberry pi 3. Int. J. Eng. Technol. 2018, 7, 29–32. [Google Scholar] [CrossRef] [Green Version]
- Bhandarkar, S.; Naxane, T.; Shrungare, S.; Rajhance, S. Neural Network Based Detection of Driver’s Drowsiness. TechRxiv 2021. [Google Scholar] [CrossRef]
- Khushaba, R.N.; Kodagoda, S.; Lal, S.; Dissanayake, G. Driver drowsiness classification using fuzzy wavelet-packet-based feature-extraction algorithm. IEEE Trans. Biomed. Eng. 2010, 58, 121–131. [Google Scholar] [CrossRef] [PubMed]
- Mardi, Z.; Ashtiani, S.N.M.; Mikaili, M. EEG-based drowsiness detection for safe driving using chaotic features and statistical tests. J. Med. Signals Sens. 2011, 1, 130. [Google Scholar] [PubMed] [Green Version]
- Noori, S.M.R.; Mikaeili, M. Driving drowsiness detection using fusion of electroencephalography, electrooculography, and driving quality signals. J. Med. Signals Sens. 2016, 6, 39. [Google Scholar] [PubMed]
- Picot, A.; Charbonnier, S.; Caplier, A. On-line detection of drowsiness using brain and visual information. IEEE Trans. Syst. Man Cybern.-Part A Syst. Hum. 2011, 42, 764–775. [Google Scholar] [CrossRef]
- Krajewski, J.; Sommer, D.; Trutschel, U.; Edwards, D.; Golz, M. Steering wheel behavior based estimation of fatigue. In Proceedings of the Driving Assesment Conference, Big Sky, MT, USA, 22–25 June 2009; Volume 5. [Google Scholar]
- Mandal, B.; Li, L.; Wang, G.S.; Lin, J. Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE Trans. Intell. Transp. Syst. 2016, 18, 545–557. [Google Scholar] [CrossRef]
- Yang, C.; Yang, Z.; Li, W.; See, J. FatigueView: A Multi-Camera Video Dataset for Vision-Based Drowsiness Detection. IEEE Trans. Intell. Transp. Syst. 2022. [Google Scholar] [CrossRef]
- Islam, M.S.; Hasan, M.M.; Abdullah, S.; Akbar, J.U.M.; Arafat, N.; Murad, S.A. A deep Spatio-temporal network for vision-based sexual harassment detection. In Proceedings of the 2021 Emerging Technology in Computing, Communication and Electronics (ETCCE), Dhaka, Bangladesh, 21–23 December 2021; pp. 1–6. [Google Scholar]
- Fusek, R. Pupil localization using geodesic distance. In Proceedings of the International Symposium on Visual Computing, Las Vegas, NV, USA, 19–21 November 2018; Springer: Berlin/Heidelberg, Germany, 2018; pp. 433–444. [Google Scholar]
- King, D.E. Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res. 2009, 10, 1755–1758. [Google Scholar]
- LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 1998, 86, 2278–2324. [Google Scholar] [CrossRef]
- Zheng, H.; Yang, Z.; Liu, W.; Liang, J.; Li, Y. Improving deep neural networks using softplus units. In Proceedings of the 2015 International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland, 12–17 July 2015; pp. 1–4. [Google Scholar]
- Ioffe, S.; Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning (PMLR), Lille, France, 6–11 July 2015; pp. 448–456. [Google Scholar]
- Park, S.; Kwak, N. Analysis on the dropout effect in convolutional neural networks. In Proceedings of the Asian Conference on Computer Vision, Taipei, Taiwan, 20–24 November 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 189–204. [Google Scholar]
- Kongcharoen, W.; Nuchitprasitchai, S.; Nilsiam, Y.; Pearce, J.M. Real-Time Eye State Detection System for Driver Drowsiness Using Convolutional Neural Network. In Proceedings of the 2020 17th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), Phuket, Thailand, 24–27 June 2020; pp. 551–554. [Google Scholar]
- Suresh, Y.; Khandelwal, R.; Nikitha, M.; Fayaz, M.; Soudhri, V. Driver Drowsiness Detection using Deep Learning. In Proceedings of the 2021 2nd International Conference on Smart Electronics and Communication (ICOSEC), Trichy, India, 20–22 October 2021; pp. 1526–1531. [Google Scholar]
- Walizad, M.E.; Hurroo, M.; Sethia, D. Driver Drowsiness Detection System using Convolutional Neural Network. In Proceedings of the 2022 6th International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India, 28–30 April 2022; pp. 1073–1080. [Google Scholar]
- Tibrewal, M.; Srivastava, A.; Kayalvizhi, R. A deep learning approach to detect driver drowsiness. Int. J. Eng. Res. Technol. 2021, 10, 183–189. [Google Scholar]
Network Model | Accuracy | Precision | Recall |
---|---|---|---|
VGG16 | 95.93% | 93.15% | 93.87% |
VGG19 | 95.03% | 94.82% | 95.47% |
4D model | 97.53% | 97.35% | 97.06% |
Network Model | Training Time | Prediction Time |
---|---|---|
VGG16 | 1036.07 s | 16.17 s |
VGG19 | 1144.900 s | 19.69 s |
4D model | 1205.379 s | 19.35 s |
Network Model | Total Flops () | Total MACCs () |
---|---|---|
VGG16 | 5.93124 | 2.9650 |
VGG19 | 7.51787 | 3.7583 |
4D model | 6.19578 | 3.0967 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Jahan, I.; Uddin, K.M.A.; Murad, S.A.; Miah, M.S.U.; Khan, T.Z.; Masud, M.; Aljahdali, S.; Bairagi, A.K. 4D: A Real-Time Driver Drowsiness Detector Using Deep Learning. Electronics 2023, 12, 235. https://doi.org/10.3390/electronics12010235
Jahan I, Uddin KMA, Murad SA, Miah MSU, Khan TZ, Masud M, Aljahdali S, Bairagi AK. 4D: A Real-Time Driver Drowsiness Detector Using Deep Learning. Electronics. 2023; 12(1):235. https://doi.org/10.3390/electronics12010235
Chicago/Turabian StyleJahan, Israt, K. M. Aslam Uddin, Saydul Akbar Murad, M. Saef Ullah Miah, Tanvir Zaman Khan, Mehedi Masud, Sultan Aljahdali, and Anupam Kumar Bairagi. 2023. "4D: A Real-Time Driver Drowsiness Detector Using Deep Learning" Electronics 12, no. 1: 235. https://doi.org/10.3390/electronics12010235
APA StyleJahan, I., Uddin, K. M. A., Murad, S. A., Miah, M. S. U., Khan, T. Z., Masud, M., Aljahdali, S., & Bairagi, A. K. (2023). 4D: A Real-Time Driver Drowsiness Detector Using Deep Learning. Electronics, 12(1), 235. https://doi.org/10.3390/electronics12010235