Open AccessArticle

IoT-Based Assessment of a Driver’s Stress Level

Veronica Mattioli

^1,2

Luca Davoli

Laura Belli

Sara Gambetta

³,

Luca Carnevali

Andrea Sgoifo

Riccardo Raheli

and

Gianluigi Ferrari

^2,*

Multimedia Laboratory, Department of Engineering and Architecture, University of Parma, 43124 Parma, Italy

Internet of Things (IoT) Laboratory, Department of Engineering and Architecture, University of Parma, 43124 Parma, Italy

Stress Physiology Laboratory, Department of Chemistry, Life Sciences and Environmental Sustainability, University of Parma, 43124 Parma, Italy

Author to whom correspondence should be addressed.

Sensors 2024, 24(17), 5479; https://doi.org/10.3390/s24175479

Submission received: 29 June 2024 / Revised: 19 August 2024 / Accepted: 22 August 2024 / Published: 23 August 2024

(This article belongs to the Special Issue Robust Multimodal Sensing for Automated Driving Systems)

Download

Browse Figures

Versions Notes

Abstract

Driver Monitoring Systems (DMSs) play a key role in preventing hazardous events (e.g., road accidents) by providing prompt assistance when anomalies are detected while driving. Different factors, such as traffic and road conditions, might alter the psycho-physiological status of a driver by increasing stress and workload levels. This motivates the development of advanced monitoring architectures taking into account psycho-physiological aspects. In this work, we propose a novel in-vehicle Internet of Things (IoT)-oriented monitoring system to assess the stress status of the driver. In detail, the system leverages heterogeneous components and techniques to collect driver (and, possibly, vehicle) data, aiming at estimating the driver’s arousal level, i.e., their psycho-physiological response to driving tasks. In particular, a wearable sensorized bodice and a thermal camera are employed to extract physiological parameters of interest (namely, the heart rate and skin temperature of the subject), which are processed and analyzed with innovative algorithms. Finally, experimental results are obtained both in simulated and real driving scenarios, demonstrating the adaptability and efficacy of the proposed system.

Keywords:

Driver Monitoring System; Internet of Things; IoT; arousal; wearable; thermal camera

1. Introduction

Over the last five decades, research activities focusing on the impact of driving tasks on the psycho-physiological status of a driver have gained an increasing scientific interest. Indeed, driving can be considered a major experience in many people’s life. Instances of early works on this topic mainly focus on the relationship between traffic congestion and stress [1,2,3], demonstrating that conditions of high congestion are associated with higher levels of perceived stress. However, driver stress may also be influenced by other external factors (e.g., roadway conditions, weather, visibility, etc.) and internal factors (e.g., gender, age, personality, etc.) [4]. Moreover, it has been demonstrated that the driving context (e.g, road objects, signs, vehicles, etc.) has a significant impact on the driver’s status [5]. Attempts to characterize the driving context from driver behavior have been carried out in [6], where car trajectory segmentation is exploited to derive meaningful characteristics of the driving context. Hence, both external and internal factors may negatively affect the driving performance by altering the psycho-physiological status of the driver, possibly causing road accidents and dangerous situations.

This further motivates the development of advanced monitoring systems able to extract and process physiological parameters to evaluate the psycho-physiological activation of a subject as a response to the driving task. Indeed, the perception of stressful stimuli causes a psycho-physiological reaction in the human body by altering its functions. In particular, physiological processes are regulated by the Autonomic Nervous System (ANS), which is composed of the sympathetic and parasympathetic systems [7]. A psycho-physiological activation corresponds to the activation of the sympathetic system, which regulates body functions in stressful conditions, and a simultaneous deactivation of the parasympathetic system, which regulates body functions at rest. In this work, we refer to the psycho-physiological activation of the driver with the term arousal.

Among different indices that might be observed, an important informative physiological index of the stress level of a subject is the heart rate variability (HRV), which can be defined as the beat-to-beat temporal fluctuation in heart rate (HR) [8]. In particular, the HRV reflects the activity of the ANS, which may be influenced by stressful stimuli and/or mental efforts. Moreover, HRV can be inferred from an ElectroCardioGram (ECG) and has been considered in various research works in the context of driver stress detection [9,10,11,12,13]. An extensive review of works where the HRV is considered as the main physiological parameter used to detect stress in different contexts can be found in [14]. This review provides evidence to support the use of the HRV for the objective assessment of physiological stress. Further examples of works, which highlight the reliability of the HRV as stress indicator, are [15,16], where it is demonstrated that mental stress can be inferred from HRV analysis.

Given the importance of the HRV as a stress indicator, several attempts at developing driver assistance systems (DASs) based on the monitoring of physiological parameters of interest have been proposed in the literature. As an example, in [17] a DAS equipped with a stress monitoring system was designed to provide assistance in electric vehicles by intervening with the vehicle speed in case of anomalies, whereas in [18], a Machine Learning (ML)-oriented stress detection system to be possibly integrated with a DAS is presented. Both systems presented in [17,18] are based on the analysis of ECG signals and other physiological data, including skin temperature (among others). Indeed, variations in the skin temperature of a subject can be considered another important stress indicator. In particular, the skin temperature on the nasal tip region is regulated by the ANS and has been observed to decrease in the case of high levels of stress [19]. Therefore, thermal imaging-based systems able to detect facial skin temperature variations have been developed in simulated driving environments [20,21] and in real driving scenarios [22]. In general, the extensive analysis carried out in the literature on the use of HRV and facial skin temperature to estimate the stress status of a subject proves how these two physiological parameters are considered significant indicators of stress by the research community.

On the basis of these remarks, in this work, the design and implementation of an advanced in-vehicle Driver Monitoring System (DMS) able to collect and jointly process physiological data, including HRV signals and facial skin temperature variations, to evaluate the arousal level of the driver, are presented. More in detail, the proposed DMS is composed of interconnected heterogeneous sensors properly interacting according to different Internet of Things (IoT)-oriented technologies. In particular, wireless communication protocols, such as Wi-Fi and Bluetooth Low Energy (BLE), and standard messaging protocols, such as the Message Queuing Telemetry Transport (MQTT) protocol and Transmission Control Protocol (TCP)-based sockets, are exploited to collect physiological data extracted by a wearable sensor and a thermal camera integrated in the system. Recent progress on wearable device technologies makes them effective for a wide range of applications, including those in the healthcare sector [23]. In particular, flexibility and stretchability are conferred to the most advanced wearable devices by innovative electronic materials, which make them comfortable and adaptable. To this end, recent examples of the successful development of innovative wearable sensors can be found in [24,25,26]. The effectiveness of the DMS was evaluated in both simulated and real driving scenarios according to specific operational protocols. This work expands upon previous contributions preliminarily presented in [27,28], where the first versions of the system architecture and of a thermal imaging system were introduced, respectively.

More in detail, the main purpose of this work was to demonstrate the feasibility of the developed DMS, in terms of data collection (through commercial IoT devices) considering a realistic driving scenario. More thorough comparative analyses with other existing systems are left as a future research activity, as they are out of the scope of the present study. Moreover, the proposed IoT-based architecture differentiates itself from other approaches described in the literature (e.g., [29,30]), where ML models are considered to estimate the driver’s mental workload and distractions only in simulated scenarios. It also differs from other approaches mentioned in this paper (namely, the work in [9]), since the number of employed sensors and extracted physiological signals is different. In particular, in [9], five wearable sensors were employed and needed to be attached to different parts of the subject’s body, making the system more invasive and uncomfortable than a DMS, which requires only one wearable sensor and a contactless thermal camera.

Finally, we highlight that the experimental setup in the real vehicle scenario does not actually consider the integration of Engine Control Unit (ECU) data, since access to this information is generally forbidden by manufacturers on commercial vehicles. Despite this, the proposed DMS can, in principle, be improved by including data fusion with in-vehicle-related information (provided that access to it is granted). This paper is organized as follows. In Section 2, the architecture of the proposed IoT monitoring system, together with a brief description of each component, is provided. The operational procedure for collecting data obtained through the involved sensors is detailed in Section 3, where the interactions between the system components are described. In Section 4, we present innovative algorithms to process and further analyze the collected data. The performance results obtained in simulated and real environments are discussed in Section 5. Finally, in Section 6, conclusions and possible future extensions are summarized.

2. In-Vehicle Monitoring Architecture

The proposed DMS is composed of heterogeneous sensing and processing devices that should properly be positioned inside the vehicle cabin and connected with each other. For the sake of clarity, the experimental system setup is shown in Figure 1, where the various components and their interactions are shown. In particular, a wearable sensor and a thermal camera are employed to gather physiological parameters of interest and record facial skin temperature variations in the driver, respectively. Additional vehicular data might be extracted by embedded inertial sensors, i.e., an ECU located on board the vehicle. The processing component in the proposed DAS is an Intel Next Unit of Computing (NUC), acting as a Wi-Fi Access Point (AP), providing a private Wi-Fi network, and also providing networking, storage and processing functionalities for the entire system.

2.1. Wearable Sensor

In order to collect physiological data from the driver, an Equivital EQ02 Life Monitor [31] device (manufactured by Equivital, Cambridge, UK) is employed. In detail, the Equivital EQ02 Life Monitor is a wireless wearable sensing device capable of recording multiple vital signs related to a subject wearing the sensor, and is composed of two elements: (i) a textile bodice embedding electrode, i.e., the EQ02 Sensor Belt, and (ii) a EQ02 Sensor Electronics Module (SEM). The SEM can capture biological signals in real time, including ECG and respiratory signals and other indices such as HR, respiratory rate (RR) and skin temperature. Additional information on the body position and motion can be collected from a 3-axis accelerometer embedded in the SEM. Moreover, the SEM is equipped with a Class-1 Bluetooth interface for real-time data transmission. Alternatively, the acquired data may be logged on the sensor and a posteriori downloaded. Pictures of the EQ02 Sensor Belt and SEM are shown in Figure 2a and Figure 2b, respectively, whereas their correct on-body positioning is illustrated in Figure 2c. As can be seen, the SEM needs to be inserted in a specific pocket on the left side of the belt itself. Thus, this wearable device can be considered non-invasive since it is made of a flexible fabric that can easily adapt to the body of the driver. Owing to its wearing comfort, this device is also employed by athletes and workers during their regular activities, as described in [32,33].

2.2. Thermal Camera

The second component integrated in the architecture of the proposed monitoring system is a FLIR One Pro LT thermal camera [34] (manufactured by Teledyne FLIR LLC, Wilsonville, OR, USA), which allows us to collect thermal information from specific regions on the body of the driver (namely, the face and nose) and from the surrounding environment. This device is composed of an RGB sensor and an infrared sensor, allowing us to simultaneously capture visible and thermal images or videos, respectively. Data may also be acquired according to a blended modality, providing a combination of visible and thermal outputs.

On the practical side, the FLIR One Pro LT thermal camera has to be connected to a smartphone as an external USB Type-C “dongle” and may be managed by either Android- or iOS-like mobile applications. The smartphone and connected thermal camera dongle need to be accurately installed inside the vehicle cabin to guarantee optimal positioning for data acquisition. To this end, a fair trade-off between recording quality and positioning obtrusiveness should be considered, since further video processing and analysis tasks require a frontal perspective. Then, similarly to the wearable sensor, this device can also be considered non-invasive since it does not require direct contact with the subject and is positioned unobtrusively, without limiting the road view. For the sake of illustration, the connection of the FLIR dongle to the smartphone and their positioning inside the cabin are shown in Figure 3a and Figure 3b, respectively. An illustrative example of a captured infrared video frame is shown in Figure 3c. Finally, we highlight that the thermal camera has small dimensions (namely,

68 \times 34 \times 14

mm), making it easily positionable by means of a standard phone car holder attached to the front windscreen of the vehicle, as shown in Figure 3b. Hence, thanks to this simple positioning, the thermal camera can be considered as not distracting and/or stressful for the driver.

3. Data Acquisition

3.1. Data Acquisition Architecture

The different modules detailed in Section 2 and composing the proposed driver monitoring architecture transmit their collected data to the central hub, namely, the Intel NUC running Windows 10, located on board the vehicle and intended to work as a multi-interface gateway for data storage, fusion and processing. A pictorial representation of the data acquisition architecture of the proposed DMS is shown in Figure 4, where the connections between the different components are highlighted and the central hub is denoted as the Joint Driver–Vehicle Status (JDVS) module.

With regard to the interconnection of the sub-modules, standard communication protocols are employed. In particular, the EQ02 SEM sends physiological data (i.e., the HR, HRV and RR of the subject wearing the wearable bodice) through its Class-1 Bluetooth interface to the JDVS module, where the Equivital eqView Pro [35] desktop application is installed and collects the physiological data. These data are then forwarded to a TCP socket toward an internal Python application (running on board the JDVS module), hosting a TCP server and listening for incoming packets for further processing. On the other hand, thermal data recorded by the FLIR One Pro LT thermal camera are transmitted (by the smartphone hosting the FLIR dongle), exploiting the MQTT protocol, to the JDVS module through the private Wi-Fi network hosted and advertised by the JDVS itself. To this purpose, an Android-based mobile application, denoted as MoniDrive (as shown in Figure 4) and running on a Huawei P20 Lite smartphone, was developed to acquire RGB, thermal images and additional frame and camera information (i.e., temperature scale, framed hottest and coldest points, etc.) through the FLIR camera connected on its USB-C interface. Finally, serial communication channels, such as the Controller Area Network (CAN) bus, might also be exploited to transmit vehicular data from an ECU. In this study, we did not investigate this aspect. Moreover, the proposed DMS was designed to support this decision.

We remark that vehicle vibrations do not affect the data acquisition phase. In fact, the employed wearable bodice is not sensitive to small vibrations. Moreover, the proposed video processing techniques allow us to detect a framed face regardless of the camera position and orientation, with the subject’s facial temperature not being affected by vehicle vibrations in any way. As a consequence, in-vehicle vibrations occurring during the driving activities do not represent a noise source in the experimental data collection phase. The JDVS module asynchronously processes the data received from the various components (every 1 s from the wearable sensor and every 5 s from the thermal camera, respectively) in order to produce an arousal index (denoted as

φ

) in the range

[0, 1]

, where 0 and 1 indicate null and maximum physiological activations.

To summarize, the operational tasks performed by the JDVS module are the following:

Acting as a Wi-Fi AP (advertising a private Wi-Fi network) and as a MQTT broker;
Processing the information received from the MoniDrive app, through innovative image and video processing algorithms that will be detailed later;
Processing the data received from the sensorized belt;
Performing data fusion to estimate the arousal $φ$ and transmit it to external interested entities (e.g., the vehicular ECU through a parallel Ethernet or Wi-Fi interface).

Finally, with regard to the communication protocols adopted in the proposed IoT-based monitoring architecture, it should be highlighted that, even if the use of BLE, Bluetooth and, more in general, wireless technologies in a vehicle could lead to interference problems [36,37], this issue has not been experienced in our experimental data collection campaigns. However, this aspect would deserve full attention, should the proposed system be integrated, as an on-board functionality, in a real vehicle.

3.2. Driving Protocol

With regard to a realistic deployment and the validation of the proposed data acquisition architecture introduced in the previous section, specific experimental scenarios were designed to evaluate the driver’s response to different external stimuli, associated with different amounts of perceived stress. To this end, the driving sessions monitored through the proposed system were run in both a controlled environment (namely, a driving simulator), and in realistic scenarios (namely, urban and beltway driving with smooth and heavy traffic), in order to analyze the variations in the driver’s psycho-physiological status in both smooth and fast driving conditions. As an example, in [38], an experimental route including urban and beltway driving was defined to collect and analyze data on the stress status of the driver in different driving contexts.

3.2.1. Real Driving

In this work, a first example of a driving protocol defined for acquiring data in real driving situations is shown in Figure 5. In detail, the driving protocol consists of specific epochs—with their duration shown in Figure 5—associated with different amounts of perceived stress induced by various external factors, e.g., road type, traffic conditions and co-driving. The face icons in Figure 5 delimit the time interval when additional stress is induced on the driver by the co-driver. To this end, it is useful to highlight how, in the specific context of real driving, the considered stress is a mental stress. Indeed, the stress induced by road and traffic conditions and by the presence of a co-driver is aimed at increasing the driver’s mental workload.

Before starting each driving test, the driver and vehicle are equipped with the proposed IoT monitoring system during an initial arrangement phase. In particular, the driver is asked to wear the Equivital EQ02 Life Monitor sensor, while the FLIR One Pro LT thermal camera (attached to a smartphone) is properly positioned inside the vehicle cabin according to the guidelines provided in Section 2.1 and Section 2.2. Moreover, during this phase the stress surveys are also administrated.

The various phases of the 45 min driving tests in realistic scenarios (as shown in Figure 5) are the following.

BASELINE: During this phase, reference data from the driver are collected for 10 min. In particular, during this phase the driver is required to sit still inside the vehicle with the car engine turned off, with the resulting data recorded as “reference” and further compared with the data collected during the whole driving test.
BELTWAY: The driver is asked to drive on a beltway for 5 min.
CO-DRIVER: The driver is asked to drive on urban roads for 15 min. During the first and last 5 min intervals of this phase, road indications are given to the driver by a co-driver sitting in the back of the vehicle, and during the central 5 min interval, the co-driver provides stressful stimuli to the driver by simply talking to the driver and providing some rude comments to raise a stress reaction. Moreover, it should be highlighted that the co-driver is the same for all participants and is unrelated to all of them—in terms of familiarity—and the traffic condition during this phase was slow to heavy due to urban roads conditions.
RECOVERY: During this phase, data are collected for 10 min more in order to check the driver’s response after a recovery phase. The driver is required to sit still inside the vehicle with the car engine turned off (as in the BASELINE phase). Finally, additional surveys about the perceived stress are also administered to investigate the impact of the driving test on the subject’s status.

The experimental route followed during the aforementioned driving tests lies in the city of Parma, Italy, as shown in Figure 6. The maps in Figure 6a,c refer to beltway routes, whereas the map in Figure 6b refers to an urban route. For all maps, the precise latitude and longitude coordinates are shown.

3.2.2. Simulated Driving

The driving protocol defined for acquiring experimental data in a simulated environment (i.e., a driving simulator) is defined in [39], where the designed experimental scenario is described. In particular, during the simulated driving sessions, the participants are required to perform a series of distracting tasks in order to increase their level of perceived stress. For the sake of completeness, the various phases of a 14 min simulated driving test are shown in Figure 7, where, in detail, (i) reference data were acquired during the baseline phases and (ii) cognitive, visual and emotional distractions are emulated during the corresponding phases (as highlighted in Figure 7) in order to acquire data in stressful conditions. In particular, during the first cognitive distraction phase, the driver is asked to perform a memory exercise, named n-back (with

n = 2

), where they have to listen to a sequence of letters read by an external voice and declare a match if and only if the last heard letter corresponds to the second-to-last heard letter. Then, the driver is asked to perform a second task to emulate both cognitive and visual distraction, during which they have to write on a cellphone a sequence of city names read by an external voice. During the last emotional distraction phase, the driver listens to an audio track and is asked to write on a cellphone the emotions that they are feeling. Finally, data at rest are collected during the recovery phase. It is finally noteworthy to highlight that none of the participants took part in both real and simulated driving scenarios, and that since cognitive, visual and emotional distractions are induced in the simulated driving tests, the type of stress in this specific scenario is considered both mental end emotional.

4. Data Analysis

In order to extract useful information from the data collected according to the protocols detailed in Section 3.2, different dedicated processing techniques are implemented. In particular, physiological and thermal data are processed separately in order to compute specific indices related to the stress status of the driver, as detailed and described in the following.

4.1. Physiological Data Analysis

Physiological indices of interest, i.e., HR and ECG signals, are acquired every 40 ms by the Equivital EQ02 Life Monitor sensor described in Section 2.1. Then, raw ECG signals are properly processed with the LabChartPro 5.0 software [40] to correctly detect R wave peaks—the R wave is an upward deflection in the ECG signal and represents early ventricular depolarization [41]. These data are of interest to evaluate the time interval between consecutive R peaks, also referred to as an R-R interval. This interval represents the time interval between normal heart beats, and its variations are expedient to quantifying the HRV, which is considered to evaluate the stress level. In particular, the Root Mean Square of Successive Difference (RMSSD, dim: [ms]) between consecutive R-R intervals is computed over a window of N samples, for each phase of the considered driving protocol, as follows:

RMSSD = \sqrt{\frac{\sum_{n = 2}^{N} {({RR}_{n} - {RR}_{n - 1})}^{2}}{n - 1}}

(1)

where

{RR}_{n}

represents the duration in ms of the n-th R-R interval, with

n \in {2, \dots, N}

, and the number of samples N is given by

D / 40

, where D is the duration in ms of the considered phase and data are acquired every 40 ms. The RMSSD reflects the activity of the parasympathetic system, which is part of the ANS and regulates body functions at rest [42]. In particular, the vagus nerve is the main component of the parasympathetic system and is responsible for the heart rate reduction under relaxed conditions. On the other hand, the activity of the vagus nerve is reduced under stress conditions causing heart rate accelerations. Thus, reduced values of RMSSD are associated with a reduced activity of the vagus nerve as, for example, under higher levels of stress.

4.2. Thermal Data Analysis

Thermal images and additional information of interest—namely, temperature ranges detected by the thermal camera and the pixel coordinates of the framed hottest and coldest points—were acquired by the FLIR One Pro Lt device described in Section 2.2 every 5 s and processed according to a dedicated algorithm, for which its block diagram is shown in Figure 8.

More in detail, the proposed algorithm retrieves the temperature information related to the considered thermal images, which are coded according to the RGB standard. Hence, a simple temperature mapping is performed in order to associate the RGB values of each pixel in a considered thermal image with their corresponding actual temperature values (dim: [°C]). For the sake of mapping, each thermal image, coded according to the RGB standard, is first indexed according to a specific color map, i.e., the temperature bar provided by the FLIR One Pro Lt device as additional information, in order to associate each RGB value within the image (representing the pixel intensity) to a unique integer index. This indexed image is then converted to grayscale to reduce the computational complexity of the mapping operation. The derived mapping rule is the following:

I_{t} = t (l_{1}, l_{2}) + [t (h_{1}, h_{2}) - t (l_{1}, l_{2})] I_{i}

(2)

where

I_{t}

and

I_{i}

are two-dimensional matrices (dim: [pixel × pixel]) representing the thermal image with restored actual temperature values and the original RGB thermal image after being converted to a grayscale indexed image, respectively;

t (l_{1}, l_{2})

and

t (h_{1}, h_{2})

are the coldest and hottest temperature values (dim: [°C]) related to the pixels at position

(l_{1}, l_{2})

and

(h_{1}, h_{2})

within the indexed image

I_{i}

, respectively.

The RGB frames collected along with the thermal frames are also processed in order to extract two Regions Of Interest (ROIs) on the subject’s face. In particular, the MediaPipe Face Detector framework [43] is exploited to detect the face and nose of the driver in each RGB image. Then, two rectangular ROIs (centered at the nose and face) are extracted from the corresponding thermal image, and the mean temperatures are computed within the selected areas as follows:

\bar{t} = \frac{1}{W H} \sum_{w = 0}^{W - 1} \sum_{h = 0}^{H - 1} t (w, h)

(3)

where W and H represent the width and height (in pixels) of the considered ROI, respectively, and

t (w, h)

is the temperature value associated with the pixel at position

(w, h)

Finally, a check procedure is introduced in order to discard incorrectly detected ROIs. In detail, a nose ROI is ignored if the top-left corner

y_{0}^{(n - roi)} \in {0, \dots, H - 1}

of the corresponding nasal area falls above

80 %

or below

25 %

of the height

h^{(f - roi)} \in {0, \dots, H - 1}

of the face region, i.e.,

y_{0}^{(n - roi)} < 0.25 h^{(f - roi)}

y_{0}^{(n - roi)} > 0.8 h^{(f - roi)}

4.3. Arousal Extraction

Finally, in order to quantify the real-time psycho-physiological activation of the subject as a response to the driving task, the arousal index

φ

is obtained on the basis of the RMSSD value computed on the time interval corresponding to the current phase C (denoted as

{RMSSD}_{C}

) with respect to the RMSSD value obtained on the time interval corresponding to the baseline phase (denoted as

{RMSSD}_{B}

). A pseudo-code representation of the algorithm defined to compute the arousal

φ

is shown in Algorithm 1. In detail, with reference to Algorithm 1, E denotes the total number of driving phases in the considered protocol (excluding the baseline phase) and is set to 4 and 6 for real and simulated driving, respectively (as shown in Figure 5 and Figure 7); s represents the granularity of the arousal and is set to

0.1

, in order to let the algorithm provide 10 different levels of arousal (namely:

φ \in {0.1, 0.2, 0.3, \dots, 1}

); and

Δ_{{RMSSD}_{rat}}^{(\min)}

and

Δ_{{RMSSD}_{rat}}^{(\max)}

represent the minimum and maximum variation in the value

Δ_{{RMSSD}_{rat}}

that can be measured during the driving task, and they are set to 10 and 120, respectively, on the basis of an experimental values tuning phase.

Algorithm 1 Pseudo-code of the arousal

φ

extraction.

1:: $φ \leftarrow 0$ , $e \leftarrow 1$
2:: $s \leftarrow granularity of the arousal$
3:: $Δ_{{RMSSD}_{rat}}^{(\min)} \leftarrow minimum threshold$
4:: $Δ_{{RMSSD}_{rat}}^{(\max)} \leftarrow maximum threshold$
5:: ${RMSSD}_{B} \leftarrow \sqrt{\frac{\sum_{n = 2}^{N_{B}} {({RR}_{n} - {RR}_{n - 1})}^{2}}{n - 1}}$
6:: while $e < E$ do
7:: ${RMSSD}_{C} \leftarrow \sqrt{\frac{\sum_{n = 2}^{N_{C}} {({RR}_{n} - {RR}_{n - 1})}^{2}}{n - 1}}$
8:: $Δ_{RMSSD} \leftarrow {RMSSD}_{B} - {RMSSD}_{C}$
9:: $Δ_{{RMSSD}_{rat}} \leftarrow abs (\frac{Δ_{RMSSD} \cdot 100}{{RMSSD}_{B}})$
10:: if $Δ_{RMSSD} < 0$ then
11:: if $Δ_{{RMSSD}_{rat}} < Δ_{{RMSSD}_{rat}}^{(\min)}$ then
12:: $φ \leftarrow 0$
13:: else if $Δ_{{RMSSD}_{rat}} < Δ_{{RMSSD}_{rat}}^{(\max)}$ then
14:: $φ \leftarrow \frac{1}{10} ⌈ \frac{Δ_{{RMSSD}_{rat}} - Δ_{{RMSSD}_{rat}}^{(\min)}}{s} ⌉$
15:: end if
16:: else
17:: $φ \leftarrow 0$
18:: end if
19:: $e \leftarrow e + 1$
20:: end while

5. Experimental Results

In order to demonstrate the feasibility of the proposed IoT-oriented DMS, several driving sessions were performed both in simulated and realistic scenarios. To this end, the psycho-physiological data of interest were collected according to the acquisition protocols described in Section 3.2 and processed according to the algorithms detailed in Section 4. More in detail, the driving tests were administered to 28 healthy subjects (13 women and 15 men) and 40 healthy subjects (20 women and 20 men), aged 20 to 50, in the case of simulated and realistic scenarios, respectively. Each participant was required to sign an informed consent, to hold a driving license from at least 3 years and to own a car (which was used as the mobile vehicle during their driving session and equipped as shown in Figure 1).

5.1. Simulated Scenarios

The results of two simulated driving sessions, denoted as

S_{1}

and

S_{2}

, are detailed hereafter, with examples of RGB and thermal frames extracted from the corresponding video sequences by the FLIR One Pro Lt sensor and properly processed by the algorithm described in Section 4.2, as shown in Figure 9 and Figure 10, respectively.

The physiological data of interest collected during

S_{1}

and

S_{2}

are shown in Figure 11a and Figure 11b, respectively. In particular, the mean HR and RMSSD are shown for each phase of the driving protocol detailed in Section 3.2 (and depicted in Figure 7). To this end, the RMSSD values are computed according to Equation (1) with

N = D / 40 = 3000

, considering that each phase has a duration of 2 min (namely,

D = 1200

ms) and data have been acquired every 40 ms. The various phases are labeled as Baseline, Baseline Driving (BD), Cognitive Distraction (CD), Visual Distraction (VD), Emotion (E), Emotion and Visual Distraction (E+VD), and Recovery.

Therefore, through carefully analyzing the results shown in Figure 11a,b, a psycho-physiological activation of the drivers can be observed during both sessions. In detail, the RMSSD values acquired during the epochs of the protocol corresponding to stressful conditions (i.e., CD, VD, E, E + VD) are indeed lower than the baseline values acquired during Baseline and BD epochs.

Considering Figure 11a, the most significant activation of the subject is observed during the CD phase, where a very small value of RMSSD (namely,

41.49

ms) is computed, and a very high value of mean HR (namely,

87.27

bpm) is recorded, indicating a reduced activity of the vagus nerve, as typically observed in stressful circumstances. A similar condition is observed during the E+VD phase, where the value of the RMSSD index is

40.41

ms, and the mean HR is

76.34

bpm. Likewise, considering Figure 11b, the subject is mainly activated during the CD and E+VD phases, where the values of the RMSSD are

17.99

ms and

17.41

ms, respectively, while the values of the mean HR are

76.89

bpm and

78.23

bpm, respectively.

Moreover, it can be observed that during the final Recovery phase (in both Figure 11a,b), RMSSD increases, and the mean HR decreases with respect to the values obtained during the previous E+VD phase, indicating that both subjects are experiencing a relaxed condition once the driving session is over.

As a further analysis, the mean temperatures extracted from the thermal frames acquired during

S_{1}

and

S_{2}

are shown in Figure 12 and Figure 13, respectively. In particular, the total number of analyzed frames was equal to 168 for both sessions, corresponding to a 14 min total duration of the simulated driving protocol, considering that thermal data are acquired every 5 s.

For Figure 12a and Figure 13a, the mean temperature values (dim: [°C]) were computed by applying Equation (3) on the drivers’ face and nose ROIs reported for each acquired frame, along with the mean temperature extracted from the whole frame (detected and extracted through the processing performed by means of the MediaPipe Face Detector framework). However, at some time instants, artifacts, possibly due to movements of the subjects, may lead to incorrect detections, as visible in Figure 12 and Figure 13, at those frame indices where the temperature curves are not defined.

In order to neutralize the effects of environmental factors, e.g., air conditioning and/or heating inside the vehicle’s cabin, the mean temperature values extracted from face and nose ROIs are also normalized, for each acquired frame, with respect to the mean temperature obtained over the whole frame. In fact, this normalization allows us to compensate the effects of undesired temperature oscillations inside the vehicle’s cabin, which are unrelated to the stress perceived by the drivers. To this end, the normalized temperature curves associated with

S_{1}

and

S_{2}

are shown in Figure 12b and Figure 13b, respectively.

In order to fairly compare physiological and thermal data, the normalized mean temperatures shown in Figure 12b and Figure 13b were temporally averaged over each phase of the simulated driving protocol, lasting 2 min, obtaining the normalized temporally averaged temperature shown in Figure 14a and Figure 14b, associated with

S_{1}

and

S_{2}

, respectively.

In observing these results, it can be noticed the decreasing trend in the temperature curves in Figure 14a, which suggests a potential stress-related alteration in the psycho-physiological status of the driver during

S_{1}

. The same conclusion may be drawn by observing the temperature curve related to the face ROI shown in Figure 14b. On the basis of these observations, it can be claimed that the temperature data are in agreement with the physiological data highlighted in Figure 11. Indeed, from CD to E+VD phases, the temperature curves are descending, whereas the RMSSD curves are ascending: both trends indicate a psycho-psychological activation of the subject during the phases associated with stressful conditions. On the other hand, the temperature curve related to the nose ROI shown in Figure 14b exhibits a slightly increasing trend from the CD to E phases that may be due to partially incorrect ROI detection. However, a deeper analysis of the thermal data is yet under consideration.

Finally, in order to further strengthen the agreement between the experimental physiological data and arousal

φ

calculated on the basis of Algorithm 1, in Figure 15, the trends related to session

S_{2}

are shown.

In detail, with reference to Figure 15, there is a clear interest in evaluating the mean arousal

\hat{φ}

, computed as the arithmetic average of the arousal values

φ

(obtained each 5 s on the basis of Algorithm 1) over each phase of the driving protocol. It can be observed that

\hat{φ}

shows a clear “inverse” behavior with respect to the RMSSD processed over each phase, thus confirming how the arousal estimation might be effective in these scenarios. Finally, it should be mentioned that the current version of the arousal algorithm only considers HRV data, whereas the integration of thermal data is yet under investigation. In this first experimental analysis, we limited the investigation to the IoT-based system behavior evaluated in terms of thermal data acquisition from specific regions of interest (namely, face and nose).

5.2. Real Scenarios

Finally, the results of a driving session run in a realistic scenario, in terms of collected physiological data, are shown in Figure 16. In particular, the mean HR and RMSSD values have been computed according to Equation (1) with

N = 7500

, considering phases with a fixed duration of 5 min and data acquired every 40 ms. To this purpose, the last 5 min of the Baseline and Recovery phase recordings are taken into account for the physiological data analysis described in Section 4.1. The obtained mean HR and RMSSD values are plotted in Figure 16 for each phase of the considered protocol. In detail, the different phases were labeled as Baseline, Beltway 1 (B1), Co-Driver 1 (C1), Co-Driver 2 (C2), Co-Driver 3 (C3), Beltway 2 (B2) and Recovery, where C1, C2 and C3 correspond to the CO-DRIVER phases detailed in Section 3.2.1—in fact, during C1 and C3, only road indications are given to the driver, whereas during C3 phase, stressful stimuli are also induced.

Hence, it can be observed that in stressful conditions, i.e., from phases C1 to C3 in Figure 16, the driver is subject to a psycho-physiological activation, with the mean HR exhibiting an increasing trend. At the opposite, RMSSD decreases, suggesting a reduction in the vagus nerve activity and an increase in the perceived stress.

Since the feasibility of the proposed system in real scenarios is a much more difficult to achieve compared to in controlled environments, i.e., the simulator, it was not possible to exploit the thermal camera during this driving session. The installation of the thermal camera is, indeed, highly sensitive to road and vehicle conditions, which may cause vibrations, possibly interfering with its correct functioning. Moreover, the vehicle cabin is more likely to be subject to temperature variations, which may be hardly detectable, unless more sensors, such as thermometers, are installed. Hence, thermal data are not reported for this illustrative example.

6. Conclusions and Future Activities

In this work, an IoT-based monitoring system that assesses the psycho-physiological status of a driver was developed and presented. Heterogeneous sensors, i.e., a wearable bodice and a thermal camera, are integrated in the proposed architecture and used to sense various physiological indices of interest, including HR, ECG signals and skin temperature. Data are collected according to well-defined protocols and properly processed by dedicated algorithms, specifically implemented for this purpose. In particular, the HRV of the driver is evaluated by computing the RMSSD index, which reflects the activity of the parasympathetic system and is considered a physiological index of stress. Variations in the skin temperature on the subject’s face and nose regions are also analyzed, since they are associated with stress-related physiological activation. Finally, experimental results were obtained in both simulated and real driving scenarios, and analyzed in order to demonstrate the relationship between the driving task and increased levels of perceived stress. The feasibility of the system was also demonstrated.

As future activities, the extraction of vehicular data from embedded inertial sensors and on-board units (OBUs) could be considered part of the proposed JDVS module. Also, the correlation between vehicular data and physiological parameters could be further investigated in order to evaluate the relationship between the driving behavior and the detected level of stress. Moreover, an optimization of the whole system could be foreseen by better integrating the thermal camera sensor in realistic driving scenarios. Finally, the use of different devices could be investigated, such as a smartwatch used to extract physiological measurements in a less-invasive way.

Author Contributions

Conceptualization, V.M., L.D., L.B., S.G., L.C., A.S., R.R. and G.F.; methodology, V.M., L.D., L.B., S.G., L.C., A.S., R.R. and G.F.; software, V.M., L.D. and L.B.; validation, V.M., L.D., L.B., S.G., L.C., A.S., R.R. and G.F.; formal analysis, V.M., L.D., L.B., S.G., L.C., A.S., R.R. and G.F.; investigation, V.M., L.D., L.B., S.G., L.C., A.S., R.R. and G.F.; resources, V.M., L.D., L.B., S.G., L.C., A.S., R.R. and G.F.; data curation, V.M., L.D., L.B., S.G., L.C., A.S., R.R. and G.F.; writing—original draft preparation, V.M., L.D., L.B., S.G., L.C., A.S., R.R. and G.F.; writing—review and editing, V.M., L.D., L.B., S.G., L.C., A.S., R.R. and G.F.; visualization, V.M., L.D. and L.B.; supervision, L.C., A.S., R.R. and G.F.; project administration, A.S., R.R. and G.F.; funding acquisition, G.F. All authors have read and agreed to the published version of the manuscript.

Funding

This research received funding from the European Union’s Horizon 2020 research and innovation program ECSEL Joint Undertaking (JU) under grant agreement No. 876487, NextPerception project—“Next Generation Smart Perception Sensors and Distributed Intelligence for Proactive Human Monitoring in Health, Wellbeing, and Automotive Systems”. The JU received support from the European Union’s Horizon 2020 research and innovation program and the nations involved in the mentioned projects. This work reflects only the authors’ views; the European Commission is not responsible for any use that may be made of the information it contains.

Institutional Review Board Statement

This study was conducted in accordance with the Declaration of Helsinki, and approved by the Research Ethics Board (REB) for the Ethics of Non-Medical Research on the Person of the University of Parma (protocol no. 85795, approved on 29 March 2022).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study. Written informed consent was obtained from the subjects to publish this paper.

Data Availability Statement

Data are contained within the article.

Conflicts of Interest

The authors declare no conflicts of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Abbreviations

The following abbreviations are used in this manuscript:

ANS	Autonomic Nervous System;
AP	Access Point;
BLE	Bluetooth Low Energy;
CAN	Controller Area Network;
DAS	Driver Assistance System;
DMS	Driver Monitoring System;
ECG	ElectroCardioGram;
ECU	Engine Control Unit;
HR	Heart Rate;
HRV	Heart Rate Variability;
IoT	Internet of Things;
JDVS	Joint Driver–Vehicle Status;
ML	Machine Learning;
MQTT	Message Queuing Telemetry Transport;
OBU	On-Board Unit;
RMSSD	Root Mean Square of Successive Difference;
ROI	Region Of Interest;
RR	Respiratory Rate;
SEM	Sensor Electronics Module;
TCP	Transmission Control Protocol.

References

Stokols, D.; Novaco, R.W.; Stokols, J.; Campbell, J. Traffic Congestion, Type A Behavior, and Stress. J. Appl. Psychol. 1978, 63, 467–480. [Google Scholar] [CrossRef]
Gulian, E.; Matthews, G.; Glendon, A.I.; Davies, D.R.; Debney, L.M. Dimension of Driver Stress. Ergonomics 1989, 32, 585–602. [Google Scholar] [CrossRef]
Hennessy, D.; Wiesenthal, D. The Relationship Between Traffic Congestion, Driver Stress and Direct Versus Indirect Coping Behaviours. Ergonomics 1997, 40, 348–361. [Google Scholar] [CrossRef]
Hill, J.D.; Boyle, L.N. Driver Stress as Influenced by Driving Maneuvers and Roadway Conditions. Transp. Res. Part Traffic Psychol. Behav. 2007, 10, 177–186. [Google Scholar] [CrossRef]
Tavakoli, A.; Lai, N.; Balali, V.; Heydarian, A. How are Drivers’ Stress Levels and Emotions Associated with the Driving Context? A Naturalistic Study. J. Transp. Health 2023, 31, 101649. [Google Scholar] [CrossRef]
Moosavi, S.; Omidvar-Tehrani, B.; Craig, R.B.; Nandi, A.; Ramnath, R. Characterizing Driving Context from Driver Behavior. In Proceedings of the 25th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, Redondo Beach, CA, USA, 7–10 November 2017; SIGSPATIAL ’17. pp. 1–4. [Google Scholar] [CrossRef]
Waxenbaum, J.A.; Reddy, V.; Varacallo, M. Anatomy, Autonomic Nervous System; StatPearls Publishing: Treasure Island, FL, USA, 2023. Available online: https://pubmed.ncbi.nlm.nih.gov/30969667/ (accessed on 1 August 2024).
van Ravenswaaij-Arts, C.M.; Kollee, L.A.; Hopman, J.C.; Stoelinga, G.B.; van Geijn, H.P. Heart Rate Variability. Ann. Intern. Med. 1993, 118, 436–447. [Google Scholar] [CrossRef]
Healey, J.; Picard, R. Detecting Stress During Real-World Driving Tasks using Physiological Sensors. IEEE Trans. Intell. Transp. Syst. 2005, 6, 156–166. [Google Scholar] [CrossRef]
Riener, A.; Ferscha, A.; Aly, M. Heart on the Road: HRV Analysis for Monitoring a Driver’s Affective State. In Proceedings of the International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Essen, Germany, 21–22 September 2009; pp. 99–106. [Google Scholar] [CrossRef]
Munla, N.; Khalil, M.; Shahin, A.; Mourad, A. Driver Stress Level Detection Using HRV Analysis. In Proceedings of the 2015 International Conference on Advances in Biomedical Engineering (ICABME), Beirut, Lebanon, 16–18 September 2015; pp. 61–64. [Google Scholar] [CrossRef]
Liu, K.; Jiao, Y.; Du, C.; Zhang, X.; Chen, X.; Xu, F.; Jiang, C. Driver Stress Detection Using Ultra-Short-Term HRV Analysis under Real World Driving Conditions. Entropy 2023, 25, 194. [Google Scholar] [CrossRef]
Ahmed, N.; Rony, R.J. Understanding self-reported stress among drivers and designing stress monitor using heart rate variability. Qual. User Exp. 2021, 6, 1–21. [Google Scholar] [CrossRef]
Kim, H.G.; Cheon, E.J.; Bai, D.S.; Lee, Y.H.; Koo, B.H. Stress and Heart Rate Variability: A Meta-Analysis and Review of the Literature. Psychiatry Investig. 2018, 15, 235–245. [Google Scholar] [CrossRef]
Desok, K.; Seo, Y.; Kim, S.; Jung, S. Short Term Analysis of Long Term Patterns of Heart Rate Variability in Subjects under Mental Stress. In Proceedings of the 2008 International Conference on BioMedical Engineering and Informatics, Sanya, China, 27–30 May 2008; Volume 2, pp. 487–491. [Google Scholar] [CrossRef]
Chen, C.; Li, C.; Tsai, C.W.; Deng, X. Evaluation of Mental Stress and Heart Rate Variability Derived from Wrist-Based Photoplethysmography. In Proceedings of the 019 IEEE Eurasia Conference on Biomedical Engineering, Healthcare and Sustainability, Okinawa, Japan, 31 May–3 June 2019; pp. 65–68. [Google Scholar] [CrossRef]
Athanasiou, G.; Tsotoulidis, S.; Mitronikas, E.; Lymberopoulos, D. Stress-Oriented Driver Assistance System for Electric Vehicles. In Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA, 26–30 August 2014; pp. 5936–5939. [Google Scholar] [CrossRef]
Siam, A.; Gamel, S.; Talaat, F. Automatic Stress Detection in Car Drivers Based on Non-Invasive Physiological Signals using Machine Learning Techniques. Neural Comput. Appl. 2023, 35, 12891–12904. [Google Scholar] [CrossRef]
Matsuno, S.; Mizuno, T.; Asano, H.; Mito, K.; Itakura, N. Estimating Autonomic Nerve Activity Using Variance of Thermal Face Images. Artif. Life Robot. 2018, 23, 367–372. [Google Scholar] [CrossRef]
Engert, V.; Merla, A.; Grant, J.A.; Cardone, D.; Tusche, A.; Singer, T. Exploring the Use of Thermal Infrared Imaging in Human Stress Research. PLoS ONE 2014, 9, e90782. [Google Scholar] [CrossRef]
Aloui, S.; Morvillier, R.; Prat, C.; Sodnik, J.; Diaz-Piedra, C.; Angioi, F.; Di Stasi, L.L. Driver Monitoring Systems in Automated Interactions: A Realtime, Thermographic-based Algorithm. In Proceedings of the CEUR Workshop Proceedings, Ljubljana, Slovenia, 29 November 2022; Volume 3300, pp. 1–7. [Google Scholar] [CrossRef]
Diaz-Piedra, C.; Gomez-Milan, E.; Di Stasi, L.L. Nasal Skin Temperature Reveals Changes in Arousal Levels Due to Time on Task: An Experimental Thermal Infrared Imaging Study. Appl. Ergon. 2019, 81, 102870. [Google Scholar] [CrossRef]
Banitaba, S.N.; Khademolqorani, S.; Jadhav, V.V.; Chamanehpour, E.; Mishra, Y.K.; Mostafavi, E.; Kaushik, A. Recent progress of bio-based smart wearable sensors for healthcare applications. Mater. Today Electron.s 2023, 5, 100055. [Google Scholar] [CrossRef]
Wu, P.; He, H. Integrating High-Performance Flexible Wires with Strain Sensors for Wearable Human Motion Detection. Sensors 2024, 24, 4795. [Google Scholar] [CrossRef]
Feng, T.; Ling, D.; Li, C.; Zheng, W.; Zhang, S.; Li, C.; Emel’yanov, A.; Pozdnyakov, A.S.; Lu, L.; Mao, Y. Stretchable on-skin touchless screen sensor enabled by ionic hydrogel. Nano Res. 2024, 17, 4462–4470. [Google Scholar] [CrossRef]
Zhu, P.; Mu, S.; Huang, W.; Sun, Z.; Lin, Y.; Chen, K.; Pan, Z.; Haghighi, M.G.; Sedghi, R.; Wang, J.; et al. Soft multifunctional neurological electronic skin through intrinsically stretchable synaptic transistor. Nano Res. 2024, 17, 6550–6559. [Google Scholar] [CrossRef]
Davoli, L.; Mattioli, V.; Gambetta, S.; Belli, L.; Carnevali, L.; Martalo, M.; Sgoifo, A.; Raheli, R.; Ferrari, G. Non-Invasive Psycho-Physiological Driver Monitoring through IoT-Oriented Systems. In The Internet of Medical Things: Enabling technologies and emerging applications; Institution of Engineering and Technology: London, UK, 2021; Chapter 2; pp. 19–33. [Google Scholar] [CrossRef]
Mattioli, V.; Davoli, L.; Belli, L.; Ferrari, G.; Raheli, R. Thermal Camera-based Driver Monitoring in the Automotive Scenario. In Proceedings of the 2023 AEIT International Conference on Electrical and Electronic Technologies for Automotive (AEIT AUTOMOTIVE), Modena, Italy, 17–19 July 2023; pp. 1–6. [Google Scholar] [CrossRef]
Cardone, D.; Perpetuini, D.; Filippini, C.; Mancini, L.; Nocco, S.; Tritto, M.; Rinella, S.; Giacobbe, A.; Fallica, G.; Ricci, F.; et al. Classification of Drivers’ Mental Workload Levels: Comparison of Machine Learning Methods Based on ECG and Infrared Thermal Signals. Sensors 2022, 22, 7300. [Google Scholar] [CrossRef]
Gjoreski, M.; Gams, M.Ž.; Luštrek, M.; Genc, P.; Garbas, J.U.; Hassan, T. Machine Learning and End-to-End Deep Learning for Monitoring Driver Distractions From Physiological and Visual Signals. IEEE Access 2020, 8, 70590–70603. [Google Scholar] [CrossRef]
Equivital Ltd. Equivital Life Monitor. 2023. Available online: https://equivital.com/products/eq02-lifemonitor (accessed on 1 August 2024).
Buller, M.J.; Atkinson, E.; Driver, K.; Tharion, W.J.; Ely, B.R.; Cheuvront, S.N.; Charkoudian, N. Individualized monitoring of heat illness risk: Novel adaptive physiological strain index to assess exercise-heat strain from athletes to fully encapsulated workers. Physiol. Meas. 2023, 44, 10NT01. [Google Scholar] [CrossRef]
Konrad, J.; Gagnon, D.; Serresse, O.; Oddson, B.; Leduc, C.; Dorman, S.C. Effect of a Simulated Mine Rescue on Physiological Variables and Heat Strain of Mine Rescue Workers. J. Occup. Environ. Med. 2019, 61, 251–261. [Google Scholar] [CrossRef] [PubMed]
Teledyne FLIR LLC. FLIR ONE Pro LT, 2023. Available online: https://www.flir.it/products/flir-one-pro-lt/ (accessed on 1 August 2024).
Equivital Ltd. Equivital eq02+ Lifemonitor–Data Monitoring That Makes a Difference, 2023. Available online: https://equivital.com/wp-content/uploads/2023/02/EQ-000708-BR-3p0-eq02-LifeMonitor-Datasheet-.pdf (accessed on 1 August 2024).
Mourad, A.; Muhammad, S.; Al Kalaa, M.O.; Refai, H.H.; Hoeher, P.A. On the performance of WLAN and Bluetooth for in-car infotainment systems. Veh. Commun. 2017, 10, 1–12. [Google Scholar] [CrossRef]
Tavakoli, R.; Nabi, M.; Basten, T.; Goossens, K. An Experimental Study of Cross-Technology Interference in In-Vehicle Wireless Sensor Networks. In Proceedings of the 19th ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, Malta, 13–17 November 2016; MSWiM ’16. pp. 195–204. [Google Scholar] [CrossRef]
Bitkina, O.V.; Kim, J.; Park, J.; Park, J.; Kim, H.K. Identifying Traffic Context Using Driving Stress: A Longitudinal Preliminary Case Study. Sensors 2019, 19, 2152. [Google Scholar] [CrossRef] [PubMed]
Plomp, J. D4.10 Final Pilot Evaluation Reports. Technical Report, Next Perception. 2023. Available online: https://www.nextperception.eu/wp-content/uploads/2023/10/NextPerception_D4.10_v1.2_public.pdf (accessed on 1 August 2024).
ADInstruments Ltd. LabChart. 2023. Available online: https://www.adinstruments.com/products/labchart (accessed on 1 August 2024).
Lederer, W. Cardiac Electrophysiology and the Electrocardiogram. In Medical Physiology; Elsevier Health Sciences: Philadelphia, PA, USA, 2012; Chapter 21; pp. 504–528. [Google Scholar]
Shaffer, F.; Ginsberg, J.P. An Overview of Heart Rate Variability Metrics and Norms. Front. Public Health 2017, 5, 1–17. [Google Scholar] [CrossRef]
Google. MediaPipe–on-Device Machine Learning for Everyone. 2023. Available online: https://developers.google.com/mediapipe (accessed on 1 August 2024).

Figure 1. Experimental setup of the proposed DMS.

Figure 2. Equivital EQ02 Life Monitor sensor: (a) belt, (b) SEM, and (c) positioning.

Figure 3. FLIR One Pro LT thermal camera: (a) sensor connection, (b) positioning, and (c) recorded infrared frame.

Figure 4. Data acquisition architecture of proposed DMS.

Figure 5. Driving protocol adopted in realistic scenarios.

Figure 6. Beltway (a,c) and urban (b) roads crossed in the city of Parma, Italy, during driving tests.

Figure 7. Driving protocol adopted in simulated scenarios.

Figure 8. Dedicated algorithm for thermal data processing.

Figure 9. Samples of (a) original RGB and (b) processed thermal frames extracted during driving session

S_{1}

Figure 9. Samples of (a) original RGB and (b) processed thermal frames extracted during driving session

S_{1}

Figure 10. Samples of (a) original RGB and (b) processed thermal frames extracted during driving session

S_{2}

Figure 10. Samples of (a) original RGB and (b) processed thermal frames extracted during driving session

S_{2}

Figure 11. Physiological data extracted during (a)

S_{1}

and (b)

S_{2}

Figure 11. Physiological data extracted during (a)

S_{1}

and (b)

S_{2}

Figure 12. (a) Mean temperatures and (b) normalized mean temperatures extracted during

S_{1}

Figure 12. (a) Mean temperatures and (b) normalized mean temperatures extracted during

S_{1}

Figure 13. (a) Mean temperatures and (b) normalized mean temperatures extracted during

S_{2}

Figure 13. (a) Mean temperatures and (b) normalized mean temperatures extracted during

S_{2}

Figure 14. Mean temperatures averaged over time windows corresponding to the epochs of the driving protocol for (a)

S_{1}

and (b)

S_{2}

Figure 14. Mean temperatures averaged over time windows corresponding to the epochs of the driving protocol for (a)

S_{1}

and (b)

S_{2}

Figure 15. Experimental RMSSD (purple), experimental arousal (green, calculated each 5 s) and mean arousal (orange, calculated over each epoch) for scenario

S_{2}

Figure 15. Experimental RMSSD (purple), experimental arousal (green, calculated each 5 s) and mean arousal (orange, calculated over each epoch) for scenario

S_{2}

Figure 16. Physiological data extracted during a real driving session.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Mattioli, V.; Davoli, L.; Belli, L.; Gambetta, S.; Carnevali, L.; Sgoifo, A.; Raheli, R.; Ferrari, G. IoT-Based Assessment of a Driver’s Stress Level. Sensors 2024, 24, 5479. https://doi.org/10.3390/s24175479

AMA Style

Mattioli V, Davoli L, Belli L, Gambetta S, Carnevali L, Sgoifo A, Raheli R, Ferrari G. IoT-Based Assessment of a Driver’s Stress Level. Sensors. 2024; 24(17):5479. https://doi.org/10.3390/s24175479

Chicago/Turabian Style

Mattioli, Veronica, Luca Davoli, Laura Belli, Sara Gambetta, Luca Carnevali, Andrea Sgoifo, Riccardo Raheli, and Gianluigi Ferrari. 2024. "IoT-Based Assessment of a Driver’s Stress Level" Sensors 24, no. 17: 5479. https://doi.org/10.3390/s24175479

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Menu

IoT-Based Assessment of a Driver’s Stress Level

Abstract

1. Introduction

2. In-Vehicle Monitoring Architecture

2.1. Wearable Sensor

2.2. Thermal Camera

3. Data Acquisition

3.1. Data Acquisition Architecture

3.2. Driving Protocol

3.2.1. Real Driving

3.2.2. Simulated Driving

4. Data Analysis

4.1. Physiological Data Analysis

4.2. Thermal Data Analysis

4.3. Arousal Extraction

5. Experimental Results

5.1. Simulated Scenarios

5.2. Real Scenarios

6. Conclusions and Future Activities

Author Contributions

Funding

Institutional Review Board Statement

Informed Consent Statement

Data Availability Statement

Conflicts of Interest

Abbreviations

References

Share and Cite

Article Metrics

Article Access Statistics

Further Information

Guidelines

MDPI Initiatives

Follow MDPI