Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article
Public Access

iLid: Low-power Sensing of Fatigue and Drowsiness Measures on a Computational Eyeglass

Published: 30 June 2017 Publication History

Abstract

The ability to monitor eye closures and blink patterns has long been known to enable accurate assessment of fatigue and drowsiness in individuals. Many measures of the eye are known to be correlated with fatigue including coarse-grained measures like the rate of blinks as well as fine-grained measures like the duration of blinks and the extent of eye closures. Despite a plethora of research validating these measures, we lack wearable devices that can continually and reliably monitor them in the natural environment. In this work, we present a low-power system, iLid, that can continually sense fine-grained measures such as blink duration and Percentage of Eye Closures (PERCLOS) at high frame rates of 100fps. We present a complete solution including design of the sensing, signal processing, and machine learning pipeline; implementation on a prototype computational eyeglass platform; and extensive evaluation under many conditions including illumination changes, eyeglass shifts, and mobility. Our results are very encouraging, showing that we can detect blinks, blink duration, eyelid location, and fatigue-related metrics such as PERCLOS with less than a few percent error.

Supplementary Material

rostaminia (rostaminia.zip)
Supplemental movie, appendix, image and software files for, iLid: Low-power Sensing of Fatigue and Drowsiness Measures on a Computational Eyeglass

References

[1]
2007. SensoMotoric Instruments (SMI): Eye and gaze tracking systems. (28 Nov. 2007). http://www.smivision.com/en.html.
[2]
2014. Current Centeye Vision Chips. http://www.centeye.com/. (22 Feb. 2014).
[3]
2014. OmniVision: CMOS VGA (640x480) Image Sensor. (27 Feb. 2014). http://www.ovt.com/products/sensor.php?id=146.
[4]
2015. JINS MEME Eyewear. (14 Oct. 2015). https://jins-meme.com/en/.
[5]
2015. STM3232-bitARMCortex MCUs - STMicroelectronics. (24 Mar. 2015). Retrieved September 10, 2016 from http://www.st.com/stm32.
[6]
2015. Tobii Glasses: Mobile Eye Tracker for real world research. (15 Oct. 2015). http://www.tobii.com.
[7]
2016. Research on Drowsy Driving. (23 Jan. 2016). http://www.nhtsa.gov/Driving+Safety/Drowsy+Driving/scope-of-the-problem.
[8]
2016. SMI Eye Tracking Glasses. (2016). http://www.eyetracking-glasses.com/products/eye-tracking-glasses-2-wireless/technology/.
[9]
2017. Center of Excellence for Mobile Sensor Data-to-Knowledge. (2017). https://md2k.org.
[10]
Saeed Abdullah, Elizabeth L Murnane, Mark Matthews, Matthew Kay, Julie A Kientz, Geri Gay, and Tanzeem Choudhury. 2016. Cognitive rhythms: unobtrusive and continuous sensing of alertness using a mobile phone. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 178--189.
[11]
Christer Ahlstrom, Marcus Nyström, Kenneth Holmqvist, Carina Fors, David Sandberg, Anna Anund, Göran Kecklund, and Torbjörn Åkerstedt. 2013. Fit-for-duty test for estimation of drivers? sleepiness level: eye movements improve the sleep/wake predictor. Transportation research part C: emerging technologies 26 (2013), 20--32.
[12]
Rizwan Ahmad. 2016. Understanding the Language of the Eye: Detecting and Identifying Eye Events in Real Time via Electrooculography. (2016). http://www.escholarship.org/uc/item/53n1p2z5
[13]
Tobias Appel, Thiago Santini, and Enkelejda Kasneci. 2016. Brightness-and motion-based blink detection for head-mounted eye trackers. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. ACM, 1726--1735.
[14]
Ioana Bacivarov, Mircea Ionita, and Peter Corcoran. 2008. Statistical models of appearance for eye tracking and eye-blink detection and measurement. IEEE transactions on consumer electronics 54, 3 (2008).
[15]
P. L. Benitez, G. H. Kamimori, T. J. Balkin, A. Greene, and M. L. Johnson. 2009. Modeling fatigue over sleep deprivation, circadian rhythm, and caffeine with a minimal performance inhibitor model. Meth. Enzymol. 454 (2009), 405--421.
[16]
Luis Miguel Bergasa, Jesús Nuevo, Miguel A Sotelo, Rafael Barea, and María Elena Lopez. 2006. Real-time system for monitoring driver vigilance. IEEE Transactions on Intelligent Transportation Systems 7, 1 (2006), 63--77.
[17]
Serge Boverie and Alain Giralt. 2008. Driver vigilance diagnostic based on eyelid movement observation. IFAC Proceedings Volumes 41, 2 (2008), 12831--12836.
[18]
J. E. Bower. 2014. Cancer-related fatigue--mechanisms, risk factors, and treatments. Nat Rev Clin Oncol 11, 10 (Oct 2014), 597--609.
[19]
Andreas Bulling, Daniel Roggen, and Gerhard Tröster. 2008. It's in your eyes: towards context-awareness and mobile HCI using wearable EOG goggles. In Proceedings of the 10th international conference on Ubiquitous computing. ACM, 84--93.
[20]
Andreas Bulling, Daniel Roggen, and Gerhard Tröster. 2009. Wearable EOG goggles: Seamless sensing and context-awareness in everyday environments. Journal of Ambient Intelligence and Smart Environments 1, 2 (2009), 157--171.
[21]
Andreas Bulling, Jamie A Ward, Hans Gellersen, and Gerhard Troster. 2011. Eye movement analysis for activity recognition using electrooculography. IEEE transactions on pattern analysis and machine intelligence 33, 4 (2011), 741--753.
[22]
Michael Chau and Margrit Betke. 2005. Real time eye tracking and blink detection with usb cameras. Technical Report. Boston University Computer Science Department.
[23]
Elizabeth J. Corwin, Laura Cousino Klein, and Kristin Rickelman. 2002. Predictors of Fatigue in Healthy Young Adults: Moderating Effects of Cigarette Smoking and Gender. Biological Research For Nursing 3, 4 (2002), 222--233. 12184665.
[24]
Anirban Dasgupta, Anjith George, SL Happy, and Aurobinda Routray. 2013. A Vision-Based System for Monitoring the Loss of Attention in Automotive Drivers. IEEE Trans. Intelligent Transportation Systems 14, 4 (2013), 1825--1838.
[25]
Tomas Drutarovsky and Andrej Fogelton. 2014. Eye blink detection using variance of motion vectors. In European Conference on Computer Vision. Springer, 436--448.
[26]
John D. Fisk, Amanda Pontefract, Paul G. Ritvo, Catherine J. Archibald, and T.J. Murray. 1994. The Impact of Fatigue on Patients with Multiple Sclerosis. Canadian Journal of Neurological Sciences / Journal Canadien des Sciences Neurologiques 21, 1 (001 002 1994), 9--14.
[27]
J. Friedman, T. Hastie, and R. Tibshirani. 2001. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics, Vol. 1.
[28]
Wolfgang Fuhl, Thiago Santini, David Geisler, Thomas Kübler, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2016. Eyes wide open? eyelid location and eye aperture estimation for pervasive eye tracking in real-world scenarios. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. ACM, 1656--1665.
[29]
Kyosuke Fukuda, John A Stern, Timothy B Brown, and Michael B Russo. 2005. Cognition, blinks, eye-movements, and pupillary movements during performance of a running memory task. Aviation, space, and environmental medicine 76, 7 (2005), C75--C85.
[30]
X. Y. Gao, Y. F. Zhang, W. L. Zheng, and B. L. Lu. 2015. Evaluating driving fatigue detection algorithms using eye tracking glasses. In 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER). 767--770.
[31]
I Garcia, Sebastian Bronte, Luis Miguel Bergasa, Javier Almazán, and J Yebes. 2012. Vision-based drowsiness detector for real driving conditions. In Intelligent Vehicles Symposium (IV), 2012 IEEE. IEEE, 618--623.
[32]
N. Goel, M. Basner, H. Rao, and D. F. Dinges. 2013. Circadian rhythms, sleep deprivation, and human performance. Prog Mol Biol Transl Sci 119 (2013), 155--190.
[33]
Kristen Grauman, Margrit Betke, James Gips, and Gary R Bradski. 2001. Communication via eye blinks-detection and duration analysis in real time. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, Vol. 1. IEEE, I--I.
[34]
Martijn Haak, Steven Bos, Sacha Panic, and LJM Rothkrantz. 2009. Detecting stress using eye blinks and brain activity from EEG signals. Proceeding of the 1st driver car interaction and interface (DCII 2008) (2009), 35--60.
[35]
E. Havlikova, J. Rosenberger, I. Nagyova, B. Middel, T. Dubayova, Z. Gdovinova, J. P. Van Dijk, and J. W. Groothoff. 2008. Impact of fatigue on quality of life in patients with Parkinsonfis disease. European Journal of Neurology 15, 5 (2008), 475--480.
[36]
Tianyi Hong and Huabiao Qin. 2007. Drivers drowsiness detection in embedded system. In Vehicular Electronics and Safety, 2007. ICVES. IEEE International Conference on. IEEE, 1--5.
[37]
Chin-Shun Hsieh and Cheng-Chi Tai. 2013. An improved and portable eye-blink duration detection system to warn of driver fatigue. Instrumentation Science 8 Technology 41, 5 (2013), 429--444.
[38]
Michael Ingre, Torbjörn Ákerstedt, Björn Peters, Anna Anund, and Göran Kecklund. 2006. Subjective sleepiness, simulated driving performance and blink duration: examining individual differences. Journal of Sleep Research 15, 1 (2006), 47--53.
[39]
Amir-Homayoun Javadi, Zahra Hakimi, Morteza Barati, Vincent Walsh, and Lili Tcheang. 2015. SET: a pupil detection method using sinusoidal approximation. Frontiers in neuroengineering 8 (2015).
[40]
Xianta Jiang, Geoffrey Tien, Da Huang, Bin Zheng, and M Stella Atkins. 2013. Capturing and evaluating blinks from video-based eyetrackers. Behavior research methods 45, 3 (2013), 656--663.
[41]
Hughes JR and Hatsukami D. 1986. Signs and symptoms of tobacco withdrawal. Archives of General Psychiatry 43, 3 (1986), 289--294.
[42]
Hughes JR, Higgins ST, Bickel WK, and et al. 1991. Caffeine self-administration, withdrawal, and adverse effects among coffee drinkers. Archives of General Psychiatry 48, 7 (1991), 611--617.
[43]
Won Oh Lee, Eui Chul Lee, and Kang Ryoung Park. 2010. Blink detection robust to various facial poses. Journal of neuroscience methods 193, 2 (2010), 356--372.
[44]
I. S. Lobentanz, S. Asenbaum, K. Vass, C. Sauter, G. Klsch, H. Kollegger, W. Kristoferitsch, and J. Zeitlhofer. 2004. Factors influencing quality of life in multiple sclerosis patients: disability, depressive mood, fatigue and sleep quality. Acta Neurologica Scandinavica 110, 1 (2004), 6--13.
[45]
Amol M Malla, Paul R Davidson, Philip J Bones, Richard Green, and Richard D Jones. 2010. Automated video-based measurement of eye closure for detecting behavioral microsleep. In Engineering in Medicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE. IEEE, 6741--6744.
[46]
R Martins and JM Carvalho. 2015. Eye blinking as an indicator of fatigue and mental load-a systematic review. Occupational Safety and Hygiene III (2015), 231--235.
[47]
Addison Mayberry, Pan Hu, Benjamin Marlin, Christopher Salthouse, and Deepak Ganesan. 2014. iShadow: Design of a Wearable, Real-time Mobile Gaze Tracker. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys ‘14). ACM, New York, NY, USA, 82--94.
[48]
Addison Mayberry, Yamin Tun, Pan Hu, Duncan Smith-Freedman, Deepak Ganesan, Benjamin M. Marlin, and Christopher Salthouse. 2015. CIDER: Enabling Robustness-Power Tradeoffs on a Computational Eyeglass. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom ‘15). ACM, New York, NY, USA, 400--412.
[49]
Lindsey K McIntire, John P McIntire, R Andy McKinley, and Chuck Goodyear. 2014. Detection of vigilance performance with pupillometry. In Proceedings of the Symposium on Eye Tracking Research and Applications. ACM, 167--174.
[50]
T. R. Mendoza, X. S. Wang, C. S. Cleeland, M. Morrissey, B. A. Johnson, J. K. Wendt, and S. L. Huber. 1999. The rapid assessment of fatigue severity in cancer patients: use of the Brief Fatigue Inventory. Cancer 85, 5 (Mar 1999), 1186--1196.
[51]
Inga Meyhöfer, Veena Kumari, Antje Hill, Nadine Petrovsky, and Ulrich Ettinger. 2016. Sleep deprivation as an experimental model system for psychosis: Effects on smooth pursuit, prosaccades, and antisaccades. Journal of Psychopharmacology (2016), 0269881116675511.
[52]
Tim Morris, Paul Blenkhorn, and Farhan Zaidi. 2002. Blink detection for real-time eye tracking. Journal of Network and Computer Applications 25, 2 (2002), 129--143.
[53]
TL Morris and James C Miller. 1996. Electrooculographic and performance indices of fatigue during simulated flight. Biological psychology 42, 3 (1996), 343--360.
[54]
M Patel, SKL Lal, Diarmuid Kavanagh, and Peter Rossiter. 2011. Applying neural network analysis on heart rate variability data to assess driver fatigue. Expert systems with Applications 38, 6 (2011), 7235--7242.
[55]
Leo Pauly and Deepa Sankar. 2015. Detection of drowsiness based on hog features and svm classifiers. In Research in Computational Intelligence and Communication Networks (ICRCICN), 2015 IEEE International Conference on. IEEE, 181--186.
[56]
Marco Pedrotti, Shengguang Lei, Jeronimo Dzaack, and Matthias Rötting. 2011. A data-driven algorithm for offline pupil signal preprocessing and eyeblink detection in low-speed eye-tracking protocols. Behavior Research Methods 43, 2 (2011), 372--383.
[57]
A. Picot, A. Caplier, and S. Charbonnier. 2009. Comparison between EOG and high frame rate camera for drowsiness detection. In Applications of Computer Vision (WACV), 2009 Workshop on. 1--6.
[58]
Antoine Picot, Alice Caplier, and Sylvie Charbonnier. 2009. Comparison between EOG and high frame rate camera for drowsiness detection. In Applications of Computer Vision (WACV), 2009 Workshop on. IEEE, 1--6.
[59]
R. Schleicher, N. Galley, S. Briest, and L. Galley. 2008. Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? Ergonomics 51, 7 (2008), 982--1010. 18568959.
[60]
Robert Schleicher, Niels Galley, Susanne Briest, and Lars Galley. 2008. Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? Ergonomics 51, 7 (2008), 982--1010.
[61]
Paul Smith, Mubarak Shah, and Niels da Vitoria Lobo. 2003. Determining driver visual attention with one camera. IEEE transactions on intelligent transportation systems 4, 4 (2003), 205--218.
[62]
M. D. Stein and P. D. Friedmann. 2005. Disturbed sleep and its relationship to alcohol use. Subst Abus 26, 1 (Mar 2005), 1--13.
[63]
J. A. Stern, D. Boyer, David J. Schroeder, United States., and Civil Aeromedical Institute. 1994. Blink rate as a measure of fatigue {microform}: a review / J.A. Stern, D. Boyer, D.J. Schroeder. U.S. Dept. of Transportation, Federal Aviation Administration, Office of Aviation Medicine; Available to the public through the National Technical Information Service Washington, D.C.: Springfield, Va. i, 12 p.; pages. http://purl.access.gpo.gov/GPO/LPS84339
[64]
Federico M Sukno, Sri-Kaushik Pavani, Constantine Butakoff, and Alejandro F Frangi. 2009. Automatic assessment of eye blinking patterns through statistical shape models. In International Conference on Computer Vision Systems. Springer, 33--42.
[65]
Louis Tijernia, Stoltzfus Duane Gleckler, Mark nad, Scott Johnston, Michael J. Goodman, and Walter W. Wierwille. 1999. A Preliminary Assessment of Algorithms for Drowsy and Inattentive Driver Detection on the Road. Technical Report. Springfield VA.
[66]
Marc Tonsen, Xucong Zhang, Yusuke Sugano, and Andreas Bulling. 2016. Labelled pupils in the wild: a dataset for studying pupil detection in unconstrained environments. In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research 8 Applications. ACM, 139--142.
[67]
W. W. Wierwille, S. S. Wreggit, C. L. Kirn, L. A. Ellsworth, and R. J. Fairbanks. 1994. Research on Vehicle-Based Driver Status/Performance Monitoring; Development, Validation, and Refinement of Algorithms For Detection of Driver Drowsiness. Technical Report. Springfield VA.
[68]
Ann Williamson. 2007. Predictors of Psychostimulant Use by Long-Distance Truck Drivers. American Journal of Epidemiology 166, 11 (2007), 1320.
[69]
Junwen Wu and Mohan M Trivedi. 2007. Simultaneous eye tracking and blink detection with interactive particle filters. EURASIP Journal on Advances in Signal Processing 2008, 1 (2007), 823695.
[70]
Cui Xu, Ying Zheng, and Zengfu Wang. 2008. Efficient eye states detection in real-time for drowsy driving monitoring system. In Information and Automation, 2008. ICIA 2008. International Conference on. IEEE, 170--174.
[71]
Chuang-Wen You, Nicholas D Lane, Fanglin Chen, Rui Wang, Zhenyu Chen, Thomas J Bao, Martha Montes-de Oca, Yuting Cheng, Mu Lin, Lorenzo Torresani, and others. 2013. Carsafe app: Alerting drowsy and distracted drivers using dual cameras on smartphones. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services. ACM, 13--26.

Cited By

View all
  • (2025)Wearable biosensorsFundamentals of Biosensors in Healthcare10.1016/B978-0-443-21658-9.00018-8(211-229)Online publication date: 2025
  • (2024)A Framework for Selecting and Assessing Wearable Sensors Deployed in Safety Critical ScenariosSensors10.3390/s2414458924:14(4589)Online publication date: 15-Jul-2024
  • (2024)ERUDITE: Human-in-the-Loop IoT for an Adaptive Personalized Learning SystemIEEE Internet of Things Journal10.1109/JIOT.2023.334346211:8(14532-14550)Online publication date: 15-Apr-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 1, Issue 2
June 2017
665 pages
EISSN:2474-9567
DOI:10.1145/3120957
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 June 2017
Accepted: 01 June 2017
Revised: 01 March 2017
Received: 01 February 2017
Published in IMWUT Volume 1, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Blinks
  2. Drowsiness
  3. Eyeglasses
  4. Eyelid
  5. Fatigue
  6. PERCLOS

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)183
  • Downloads (Last 6 weeks)55
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Wearable biosensorsFundamentals of Biosensors in Healthcare10.1016/B978-0-443-21658-9.00018-8(211-229)Online publication date: 2025
  • (2024)A Framework for Selecting and Assessing Wearable Sensors Deployed in Safety Critical ScenariosSensors10.3390/s2414458924:14(4589)Online publication date: 15-Jul-2024
  • (2024)ERUDITE: Human-in-the-Loop IoT for an Adaptive Personalized Learning SystemIEEE Internet of Things Journal10.1109/JIOT.2023.334346211:8(14532-14550)Online publication date: 15-Apr-2024
  • (2024)Comparative Study: Physiological-based Driver Drowsiness Detection Utilizing Traditional and Hybrid Methods2024 8th International Conference on Image and Signal Processing and their Applications (ISPA)10.1109/ISPA59904.2024.10536845(1-8)Online publication date: 21-Apr-2024
  • (2024)Carry-forward effect: providing proactive scaffolding to learning processesBehaviour & Information Technology10.1080/0144929X.2024.2411592(1-40)Online publication date: 16-Oct-2024
  • (2023)An Electro-Oculogram (EOG) Sensor’s Ability to Detect Driver Hypovigilance Using Machine LearningSensors10.3390/s2306294423:6(2944)Online publication date: 8-Mar-2023
  • (2023)Carry-Forward Effect: Early scaffolding learning processesProceedings of the 2023 Symposium on Learning, Design and Technology10.1145/3594781.3594786(43-52)Online publication date: 23-Jun-2023
  • (2023)Physiological Indices to Predict Driver Situation Awareness in VRAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610687(40-45)Online publication date: 8-Oct-2023
  • (2022)Exploring Gaze Movement Gesture Recognition Method for Eye-Based Interaction Using Eyewear with Infrared Distance Sensor ArrayElectronics10.3390/electronics1110163711:10(1637)Online publication date: 20-May-2022
  • (2022)Keep Calm and Do Not Carry-Forward: Toward Sensor-Data Driven AI Agent to Enhance Human LearningFrontiers in Artificial Intelligence10.3389/frai.2021.7131764Online publication date: 12-Jan-2022
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Full Access

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media