JP7578021B2 - Driving state detection method - Google Patents

Description

The technology disclosed herein belongs to the technical field of detecting the driving state of a driver who drives a vehicle.

In recent years, technologies have been developed to detect the driving state of a driver who is driving a vehicle, for example, from an image of the driver captured by a driver monitor camera or from output data of various sensors mounted on the vehicle. For example, Patent Document 1 discloses a technology for evaluating the wakefulness of a driver of a vehicle by using an error value of the driver's control of the vehicle.

Patent No. 6492096

Here, it is possible to detect a driver's decreased alertness or inattentiveness from images captured by the driver monitor camera. However, when a driver is daydreaming or thinking about something and in a distracted state (conscious inattentiveness), this is indistinguishable from a normal state from external appearances, making it difficult to detect from images captured by the driver monitor camera.

The technology disclosed here aims to detect a driver's decreased alertness or inattentiveness, and to make it possible to detect when the driver is in a distracted state.

In order to solve the above problem, the technology disclosed herein is a driving state detection method for detecting the state of a vehicle driver, comprising: a step (a) of estimating a predicted load of the driver using information indicating the surrounding conditions of the vehicle; a step (b) of evaluating the performance of the driver's response behavior using information indicating the surrounding conditions of the vehicle, information indicating the driving operation of the driver, and information indicating the driver's behavior; a step (c) of evaluating the current driving ability of the driver based on the predicted load estimated in step (a) and the performance of the response behavior evaluated in step (b); a step (d) of comparing the driving ability evaluated in step (c) with the normal driving ability of the driver to determine the degree of driving ability; a step (e) of determining whether the driver is driving while drowsy or looking away from the road using information indicating the driver's behavior; and a step (f) of determining that the driver is in a distracted state when the degree of driving ability determined in step (d) is equal to or lower than a predetermined level and it is determined in step (e) that the driver is not driving while drowsy or looking away from the road.

According to this configuration, the driver's current driving ability is evaluated based on the predicted load estimated from the vehicle's surrounding conditions and the driver's performance of the corresponding behavior. Here, the corresponding behavior refers to the driving behavior the driver performs in response to the conditions outside the vehicle. The degree of driving ability is then determined by comparing the evaluated current driving ability with the driver's normal driving ability. Meanwhile, using information indicating the driver's behavior, it is determined whether the driver is driving while drowsy or looking away. Then, when it is determined that the degree of driving ability is below a predetermined level and the driver is not driving while drowsy or looking away, the driver is determined to be in a distracted state. Therefore, it is possible to comprehensively determine whether the driver's inherent driving ability is not being exercised and the behavior observed when driving while drowsy or looking away, and to determine whether the driver is in a distracted state where he or she is daydreaming or thinking about something, or whether he or she is driving while drowsy or looking away.

In the driving state detection method, step (e) may include a step of determining that the driver is driving while drowsy or while looking away when the distance between the driver's upper and lower eyelids becomes smaller than a predetermined threshold value compared to a reference value, when the position of the driver's head deviates from a reference position by a predetermined threshold value or more, or when the deviation between the driver's line of sight and the traveling direction of the vehicle becomes larger than a predetermined threshold value.

According to this, when the eyelids are slightly closed, the head is not in a normal position, or the driver's gaze is not in the direction of travel, the driver is judged to be drowsy or distracted. On the other hand, when the driver is absent-minded, the eyelids are open, the head is in a normal position, and the driver's gaze is in the direction of travel. Therefore, absent-minded driving can be distinguished from drowsy driving or inattentive driving.

As explained above, the technology disclosed herein can detect the driver's decreased alertness and inattentiveness, and can also detect when the driver is in a distracted state.

Functional diagram of a system for implementing the driving state detection method according to this embodiment Method for assessing driving ability Flow of processing in the driving state detection method according to the present embodiment Flow of processing in the driving state detection method according to the present embodiment An example of a system configuration for executing the driving state detection method according to the present embodiment

An exemplary embodiment will be described in detail below with reference to the drawings.

Figure 1 is a functional image diagram of a system that implements a driving state detection method according to this embodiment. The system in Figure 1 receives external sensing information, information on driving operations for the vehicle, and information from a driver monitor camera that captures images of the driver, etc., as inputs. The external sensing information is information obtained by external detection means such as a camera mounted on the vehicle or an on-board sensor such as an Advanced Driver Assistance Systems (ADAS), and includes information on targets in the external environment. Driving operation information is obtained by on-board sensors such as an accelerator opening sensor and a brake pressure sensor. Information on the driver's behavior, such as eye movement, head behavior, and facial expressions, is obtained from the driver monitor camera.

The system in Figure 1 outputs the following: determination of distracted driving, drowsy driving, and inattentive driving. Inattentive driving refers to a state in which the driver is daydreaming or thinking about something, resulting in "mental inattention" and driving in a distracted state. "Inattentiveness" refers to a state in which the amount of attention paid to driving is reduced due to thinking about something, and falls below the required amount.

The processing of the system in Figure 1 will now be explained in detail.

In block A, the predicted load is calculated using external sensing information. Here, a person's cognitive load consists of three components: attention allocation load, perceptual load, and predictive load. Attention allocation load is the load applied when directing the eyes to a place that should be checked in order to obtain information. Perceptual load is the load applied when recognizing and interpreting the distance and movement of something seen. Predictive load is the load applied when predicting a future situation from perceptual information. In block A, a predictive load model is constructed, and the predictive load model is used to estimate the current predicted load from external sensing information.

Here, we explain the predictive load model. It is thought that people recall memories and predict the future from clue information. For example, it has been confirmed from public road driving data that clue information for the event of a leading vehicle slowing down includes the vehicle slowing down, an uphill slope, a red light, a sharp curve, a blinker, congestion in the surrounding area, and a mistake in speed adjustment by the leading vehicle. For example, when a leading vehicle turns on its blinker in a driving environment, the driver is thought to search for the memory that "vehicles often slow down after turning on their blinker" and predict that "the leading vehicle may slow down."

The inventors of this application therefore hypothesized that prediction load is composed of margin time and encounter rate. Margin time is the time from the appearance of a clue (the blinker in the above example) to the occurrence of an event (the deceleration of the preceding vehicle in the above example), and encounter rate is the ratio of clues to an event (blinker 20%, red light 25%, deceleration of the preceding vehicle 25%, etc.). In other words, the idea is that the longer the time available for recall, and the more frequently an encounter is made and the more firmly it is remembered, the lower the prediction load should be.

However, according to experiments conducted by the inventors of this application, the results were not always as expected. Further investigation revealed that it was necessary to consider the uncertainty of events in the real world when developing the predictive load. For example, while a leading vehicle will almost certainly slow down when the light is red, the leading vehicle will not necessarily slow down when the vehicle ahead of it slows down. In other words, the likelihood of an event occurring in response to a clue varies depending on the clue.

The predicted load model is therefore defined, for example, by the following formula:

In the above formula, uncertainty is reflected in the predictive load model. Through research by the present inventors, it has been confirmed that this predictive load model that reflects uncertainty can be applied to driving situations.

In Block B, the performance of the driver's risk reduction behavior is evaluated. Risk reduction behavior is a behavior for reducing the risk of an accident occurring, such as taking one's foot off the accelerator to slow down the vehicle ahead. The performance of risk reduction behavior is evaluated, for example, based on the time from the appearance of a clue such as a turn signal to taking one's foot off the accelerator. Other examples of risk reduction behaviors include stepping on the brakes, looking at the mirrors, etc.

Blocks C, D, and E judge the degree to which the driver is demonstrating his or her driving ability. Block C combines the predicted load calculated in Block A with the performance of risk reduction actions evaluated in Block B and outputs it as the driver's current driving ability. Block D accumulates the data on the driver's driving ability output from Block C. Then, based on the accumulated data, it performs individual learning of the driver's normal driving ability, i.e., his or her original driving ability. Block E compares the driver's current driving ability output from Block C with the driver's normal driving ability output from Block D.

Figure 2 is a diagram for explaining a method for determining the degree of driving ability. In Figure 2, driving ability is represented on a two-dimensional coordinate system with the horizontal axis representing predicted load and the vertical axis representing the performance of risk reduction behavior. Figure 2(a) shows the driver's current driving ability obtained in block C. Figure 2(b) shows the driver's normal driving ability obtained in block D. A line representing normal driving ability is obtained from the accumulated driving ability data. Then, as shown in Figure 2(c), the current driving ability shown in Figure 2(a) is compared with the normal driving ability shown in Figure 2(b) to determine the degree of driving ability. In Figure 2(c), the performance of risk reduction behavior has deteriorated under the same predicted load, so it is determined that the driving ability has decreased, that is, the degree of driving ability is low. The degree of performance may be evaluated, for example, by the distance between the coordinate position of the current driving ability and the line representing normal driving ability.

Returning to FIG. 1, on the other hand, in block X, it is determined whether the driver is drowsy or distracted while driving based on the image of the driver captured by the driver monitor camera. In block X, the determination is made using, for example, the distance between the driver's upper and lower eyelids, the head position, the line of sight, etc.

In block F, if it is not determined in block X that the driver is not drowsy or not looking away from the road, and the comparison result in block E indicates that the driver is not demonstrating driving ability to a predetermined level or below, it is determined that the driver is driving absent-mindedly.

It is preferable that the data accumulated in block D does not include data when the driver is driving absent-mindedly or while drowsy/distracted. In addition, in the system of FIG. 1, noises affecting the processing performance include the driver's driving characteristics (safety sensitivity, acceptable risk, etc.), cognitive ability (young, elderly, etc.), and driving experience (beginner, veteran, etc.).

Figures 3 and 4 are flowcharts showing an example of the process flow in the driving state detection method according to this embodiment. Also, Figure 5 is a block diagram showing an example of a system configuration for executing the driving state detection method of Figures 3 and 4. The controller 1 is, for example, configured with an IC chip equipped with a processor and memory, or multiple IC chips equipped with a processor and memory. The controller 1 may also be equipped with a dedicated chip for AI (Artificial Intelligence), a GPU (Graphic Processing Unit), etc.

The system in FIG. 5 includes a camera and a sensor mounted on the vehicle as input means for the controller 1. The forward camera 10 is installed, for example, at the front of the vehicle and captures the situation in front of the vehicle. The in-vehicle camera 11 is installed, for example, in the passenger compartment of the vehicle and captures the situation inside the vehicle, and functions as a driver monitor camera. The vehicle speed sensor 12, the acceleration sensor 13, and the yaw rate sensor 14 are sensors that detect the speed, acceleration, and yaw rate of the vehicle, respectively. The steering angle sensor 15, the steering torque sensor 16, the accelerator opening sensor 17, and the brake pressure sensor 18 are sensors that detect the steering angle, steering torque, accelerator opening, and brake pressure of the vehicle, respectively. The GPS (Global Positioning Systems) unit 19 is a unit that detects the position of the vehicle. The forward camera 10 and the GPS unit 19 are examples of means for detecting the external situation. The steering angle sensor 15, the steering torque sensor 16, the accelerator opening sensor 17, and the brake pressure sensor 18 are examples of means for detecting the driving operation.

The controller 1 sends image signals to a center display 21 and a HUD (Head-Up Display) 22, sends audio signals to an audio output device 23, and sends control signals to a steering control device 24, a brake control device 25, and an engine control device 26.

In the processing of FIG. 3 and FIG. 4, the controller 1 outputs a distracted driving judgment flag and a drowsiness/distracted driving judgment flag. When the controller 1 judges that the driver is driving distractedly, it sets the distracted driving judgment flag to "1". When the controller 1 judges that the driver is driving drowsy/distractedly, it sets the drowsiness/distracted driving flag to "1". When the distracted driving judgment flag or the drowsiness/distracted driving flag is set to "1", the controller 1 sends, for example, a signal to output a warning image to the center display 21 or HUD 22, and a signal to output a warning sound to the audio output device 23. Alternatively, it sends a control signal to the brake control device 25 and the engine control device 26 to reduce the speed of the vehicle.

In the process of FIG. 3, the controller 1 acquires information about the vehicle's surroundings (S11a), acquires driving operation information of the driver (S11b), and acquires information about the driver's line of sight and head behavior (S11c). The vehicle's surroundings information is acquired, for example, based on images captured by the front camera 10 and position information acquired by the GPS unit 19. The driving operation information is acquired, for example, based on various data detected by the steering angle sensor 15, steering torque sensor 16, accelerator opening sensor 17, and brake pressure sensor 18. The driver's line of sight and head behavior information is acquired, for example, based on images captured by the in-vehicle camera 11.

The controller 1 calculates the driver's predicted load based on the vehicle's surrounding information (S12). Here, the predicted load is calculated using the predicted load model described above for clues obtained from the vehicle's surrounding information, such as the blinker of the preceding vehicle. The controller 1 then evaluates the driver's performance of risk reduction behavior (S13). For example, the controller 1 evaluates the time it takes to release the accelerator in response to the blinker of the preceding vehicle. The controller 1 then evaluates the driver's current driving ability based on the predicted load calculated in S12 and the performance of risk reduction behavior evaluated in S13 (S14).

The controller 1 determines whether the driver's current driving ability obtained in S14 is below the driver's normal driving ability (S15). As described above, the driver's normal driving ability is learned from the accumulated data of the driver's past driving ability under normal circumstances. If it is not below the normal driving ability, it is determined that there is no problem and the process ends. On the other hand, if it is below the normal driving ability, the process proceeds to the next step S16.

In S16, the controller 1 determines whether the drowsiness/inattentive driving judgment flag is "0" (S16). The flow for setting the drowsiness/inattentive driving judgment flag is shown in FIG. 4 and will be described later. When the drowsiness/inattentive driving judgment flag is set to "1", the controller 1 outputs the drowsiness/inattentive driving judgment flag (S19) and ends the process. On the other hand, when the drowsiness/inattentive driving judgment flag is "0", the controller 1 sets the absentminded driving judgment flag to "1" (S17). The controller 1 then outputs the absentminded driving judgment flag (S18) and ends the process.

Figure 4 shows the flow of setting the drowsiness/inattentive driving judgment flag. The controller 1 acquires information on the driver's posture, facial expression, etc. from the image captured by the in-vehicle camera 11 (S21). From the information acquired in S21, the difference between the distance between the driver's upper and lower eyelids and a reference value, for example, the distance in the initial state, is calculated (S22). If the calculated difference is equal to or greater than a predetermined threshold (No in S23), the drowsiness/inattentive driving judgment flag is set to "1" (S28). If the calculated difference is smaller than the predetermined threshold, the deviation of the driver's head position from the reference position is calculated from the information acquired in S21 (S24). If the calculated deviation is equal to or greater than a predetermined threshold (No in S25), the drowsiness/inattentive driving judgment flag is set to "1" (S28). If the calculated deviation is smaller than the predetermined threshold, the deviation between the line of sight and the vehicle's traveling direction is calculated from the information acquired in S21 (S26). If the calculated deviation is equal to or greater than the predetermined threshold (No in S27), the drowsiness/inattentive driving judgment flag is set to "1" (S28). If the calculated deviation is smaller than the predetermined threshold, the drowsiness/inattentive driving judgment flag is not set to "1".

As described above, according to this embodiment, the driver's current driving ability is evaluated based on the predicted load estimated from the vehicle's surrounding conditions and the driver's performance of risk reduction behavior. The degree of driving ability is then determined by comparing the evaluated current driving ability with the driver's normal driving ability. Meanwhile, using information indicating the driver's behavior, it is determined whether the driver is driving while drowsy or looking away. When it is determined that the degree of driving ability is below a predetermined level and the driver is not driving while drowsy or looking away, the driver is determined to be in a distracted state. Therefore, it is possible to comprehensively determine whether the driver is in a distracted state where he or she is daydreaming or thinking about something, or whether the driver is driving while drowsy or looking away.

In addition, if the driver's eyelids are slightly closed, the head is not in the normal position, or the driver's gaze is not in the direction of travel, the driver is judged to be drowsy or distracted. On the other hand, if the driver is absent-minded, the eyelids are open, the head is in the normal position, and the driver's gaze is in the direction of travel. Therefore, absent-minded driving can be distinguished from drowsy driving or inattentive driving.

In the above-described embodiment, the driver's driving ability is determined using the performance of risk-reducing actions. However, the present disclosure is not limited to this, and the driver's driving ability may be evaluated using, for example, the performance of actions to maintain ride comfort or actions to comply with laws and regulations. That is, in the present disclosure, the driving ability is evaluated using the performance of the driver's response actions. Here, "response actions" refer to driving actions that the driver performs in response to the situation outside the vehicle, and risk-reducing actions, actions to maintain ride comfort, and actions to comply with laws and regulations are examples of the driver's response actions.

For comfort maintenance behavior and regulatory compliance behavior, the predicted load and performance can be evaluated, for example, as follows:

(Actions to maintain ride comfort)
Regarding the load (pedal operation related) for maintaining ride comfort in the longitudinal direction of the vehicle, it can be said that the load is higher the more unstable the speed of the preceding vehicle, the more drastic the gradient change, and the lower the traveling speed. Specifically, when the preceding vehicle is traveling at a constant speed, the vehicle itself only needs to travel at a constant speed, so no control that causes acceleration/deceleration that affects the ride comfort is generated. In other words, the load for maintaining ride comfort is low. On the other hand, when the preceding vehicle is repeatedly accelerating and decelerating, it is necessary to predict the situation and operate the pedals early in order to maintain a smooth ride. In other words, the load for maintaining ride comfort is high.

Also, on flat roads, it is sufficient to keep the speed constant, so there is no need for control that causes acceleration or deceleration that affects the ride comfort, and the burden of maintaining a comfortable ride is low. On the other hand, in scenes with frequent uphill and downhill sections, if the driver does not operate the accelerator or brake in advance, the speed will increase or decrease too much, and additional operation of the accelerator or brake will be required. In other words, the burden of maintaining a comfortable ride is high. Also, when the driving speed is high, the front and rear gravitational acceleration generated by accelerator and brake operations is small, making adjustments easy. Therefore, the burden of maintaining a comfortable ride is low. On the other hand, when the driving speed is low, the front and rear gravitational acceleration generated by accelerator and brake operations is large, so careful operation is required to maintain a comfortable ride. Therefore, the burden of maintaining a comfortable ride is high.

In addition, with regard to the load to maintain ride comfort in the lateral direction (related to steering operations), the load is higher the faster the road shape changes (the sharper the curve, the higher the driving speed). Specifically, when driving on a straight road, there is no need to make large steering operations. In other words, the load to maintain ride comfort is low. On the other hand, the sharper the curve, the larger the steering operation is required, and careful operation is required to maintain ride comfort. Therefore, the load to maintain ride comfort is high. Furthermore, the faster you drive around a curve, the faster you need to operate the steering wheel, so careful operation is required to maintain ride comfort. Therefore, the load to maintain ride comfort is high.

Performance can be evaluated, for example, based on the smoothness of the gravitational acceleration acting on the vehicle. For example, when the jerk (the derivative of acceleration/deceleration) is small, the performance is evaluated as high. Alternatively, it can be evaluated based on the smoothness of the connection between the front, rear, left and right G forces when driving around a curve. For example, it can be evaluated based on whether the G-G diagram is close to a circle. If the connection is smooth, the performance is evaluated as high.

(Legal Compliance Actions)
- In the case of traffic lights When considering the timing when a traffic light turns red when passing through an intersection, the closer the vehicle is to the intersection the closer the traffic light turns red, the higher the load. Specifically, for example, if the traffic light turns red 30 seconds before the vehicle approaches the intersection, the vehicle only needs to see the red light and stop slowly. Therefore, the load of compliance with regulations is low. On the other hand, if the traffic light turns red 3 seconds before the vehicle approaches the intersection, the vehicle must see the red light and operate immediately. Therefore, the load of compliance with regulations is high. Also, for traffic lights that change from green to yellow to red, the load is low because there is a sufficient amount of time before the light turns red, but for traffic lights that suddenly change from flashing yellow to red, the load is high.

Performance can be evaluated, for example, based on how many seconds it takes for the car to pass through an intersection after the light turns red (or yellow). For example, if the car stops, performance can be evaluated as high, and if the car passes through an intersection 10 seconds after the light turns red or yellow, performance can be evaluated as low.

- In the case of signs: If the sign's color, visibility, and location make it easy to recognize that it exists, the burden is low. Specifically, for example, if the sign is faded and the contents are difficult to understand even when looking closely, or if a tree branch is in front of the sign and it is difficult to see, the compliance burden is high. Also, if the sign has fallen over and is in an unusual location, the compliance burden can be said to be high.

For example, in the case of speed signs, performance can be evaluated based on how much the driver exceeds the speed limit. For example, performance can be evaluated as high if the driver adheres to the speed limit, and the greater the degree of exceedance, the lower the performance.

- In the case of driving lanes, if the lane is clearly visible, the load is low, and if it is blurred and difficult to see, the load is high.

Performance can be evaluated, for example, based on how far the vehicle is protruding from the lane. For example, if the vehicle is not protruding from the lane, the performance can be evaluated as high, and the more the vehicle protrudes from the lane, the lower the performance can be evaluated.

The application of the present invention is not limited to automobiles. It is also effective for detecting the driving state of moving objects other than automobiles, such as trains.

The technology disclosed herein may also be realized in a form other than a single information processing device. For example, part of the processing performed by the controller 1 may be realized by another information processing device. For example, part of the processing may be performed by an information processing device not mounted on the vehicle, such as a smartphone or tablet carried by the driver. Alternatively, the technology may be realized in a form in which the cloud performs part or all of the processing.

The above-described embodiments are merely examples and should not be interpreted as limiting the scope of the present disclosure. The scope of the present disclosure is defined by the claims, and all modifications and variations that fall within the scope of the claims are within the scope of the present disclosure.

The technology disclosed here can detect a driver's decreased alertness or inattentiveness, and can also detect when the driver is in a distracted state, making it useful for improving the safety of automobile driving, for example.

1 Controller 10 Front camera 11 In-vehicle camera (driver monitor camera)
12 vehicle speed sensor 13 acceleration sensor 14 yaw rate sensor 15 steering angle sensor 16 steering torque sensor 17 accelerator opening sensor 18 brake pressure sensor 19 GPS unit 21 center display 22 HUD
23 Audio output device 24 Steering control device 25 Brake control device 26 Engine control device

Claims (2)

A driving state detection method for detecting a state of a driver of a vehicle, comprising:
(a) estimating a predicted load of the driver using information indicative of a surrounding situation of the vehicle;
(b) evaluating the performance of the driver's response action using information indicating a surrounding situation of the vehicle, information indicating the driving operation of the driver, and information indicating the behavior of the driver;
(c) evaluating the current driving ability of the driver based on the predicted load estimated in step (a) and the performance of the corresponding behavior evaluated in step (b);
A step (d) of comparing the driving ability evaluated in the step (c) with the driver's normal driving ability to determine the degree of driving ability exercised;
A step (e) of determining whether the driver is drowsy or distracted while driving using the information indicating the driver's behavior;
and a step (f) of determining that the driver is in a distracted state when the degree of driving ability determined in step (d) is below a predetermined level and when it is determined in step (e) that the driver is not driving while drowsy or looking away from the road. 2. The driving condition detection method according to claim 1,
The step (e)
A driving condition detection method comprising a step of determining that a driver is driving drowsy or looking away when the distance between the driver's upper and lower eyelids becomes smaller than a predetermined threshold value compared to a reference value, when the driver's head position deviates from a reference position by more than a predetermined threshold value, or when the deviation between the driver's line of sight and the vehicle's traveling direction becomes larger than a predetermined threshold value.

Images