JP2022120344A - Driver state estimation device - Google Patents

Abstract

An object of the present invention is to detect, as early as possible, a sign that a driver of a moving object will be unable to drive. A head behavior detection unit 21 for detecting the behavior of the driver's head, an eyeball behavior detection unit 22 for detecting the behavior of the driver's eyeballs, and only the head behavior detected by the head behavior detection unit 21. Then, the first detection unit 110 that detects an abnormality sign of the driver and the correlation between the head behavior detected by the head behavior detection unit 21 and the eyeball behavior detected by the eyeball behavior detection unit 22 detects an abnormality sign of the driver. and a second detection unit 120 for detecting. [Selection drawing] Fig. 15

Description

The technology disclosed herein belongs to the technical field related to a driver state estimation device that estimates the state of a driver driving a mobile object.

Recently, the development of automatic driving systems has been promoted nationally. The applicant of the present application believes that, at present, there are roughly two directions for automatic driving systems.

The first direction is a system in which an automobile plays a central role in transporting passengers to a destination without the need for driver's operation, that is, so-called fully automatic driving of an automobile. On the other hand, the second direction is an automatic driving system based on the premise that a human being will drive the car, for example, to enjoy driving the car.

In the automatic driving system of the second direction, for example, when the driver becomes ill and it becomes difficult to drive normally, it is assumed that the car automatically replaces the passenger and performs automatic driving. be done. For this reason, the ability to quickly and accurately detect the occurrence of an abnormality in the driver, especially the occurrence of a functional disorder or disease in the driver, will improve the survival rate of the driver and ensure the safety of the surrounding area. very important from this point of view.

In Patent Document 1, the timing of the driver's eye movement and the timing of the head movement are specified, the order of these specified timings is determined, and the state of the driver is estimated based on the determination result. .

In addition, Patent Document 2 discloses a state detection device that detects a driver's inoperable state based on the swing width of the driver's head and detects a driver's inoperable state based on the degree of the whites of the driver's eyes. is disclosed.

Japanese Patent No. 6068964 Japanese Patent No. 6361312

T. Nakamura, et al., “Multiscale Analysis of Intensive Longitudinal Biomedical Signals and its Clinical Applications”, Proceedings of the IEEE, Institute of Electrical and Electronics Engineers, 2016, vol.104, pp.242-261 Mizuta et al., "Fractal analysis for center of gravity sway", Equilibrium Research, Japanese Society of Equilibrium Medicine, 2016, Vol.75(3), pp.154-161

As disclosed in Patent Literatures 1 and 2, techniques for determining whether a driver is unable to drive or whether there is an abnormality based on the movement of the driver's head and the state of the eyeballs are already known. However, in Patent Literature 1, even if a decrease in concentration and physical fatigue can be detected, a disease occurring in the driver cannot be predicted in advance. Further, Patent Document 2 detects a state in which the driver becomes unable to drive after a disease develops. In order to make an emergency stop or the like more safely when an abnormality occurs in the driver, it is preferable to catch a sign of the problem as soon as possible before the driver becomes unable to drive.

The technology disclosed herein has been made in view of this point, and its purpose is to detect as early as possible a sign that the driver of the moving object will be unable to drive.

In order to solve the above-described problems, the technology disclosed herein is targeted at a driver state estimation device that estimates the state of a driver driving a mobile object, and a head behavior detection unit that detects the behavior of the driver's head. an eyeball behavior detection unit that detects the behavior of the eyeballs of the driver; a first detection unit that detects an abnormality sign of the driver only from the head behavior detected by the head behavior detection unit; a second detector that detects an abnormality sign of the driver from a correlation between head behavior detected by the behavior detector and eyeball behavior detected by the eyeball behavior detector; the first detector and the second detector; and a determination unit that determines whether or not an abnormality has occurred in the driver based on the detection result of at least one of the units.

The inventors of the present application focused on the human homeostatic function and conducted research on the relationship between the deterioration of the homeostatic function and the driver's head behavior and eyeball behavior. Consistency refers to the ability to maintain a constant state against disturbances, and head behavior refers to the tendency of the driver to maintain the head posture while driving. As a result of intensive research by the inventors of the present application, it was found that the coordination between the eyeball behavior and the head behavior is disturbed as the driver's homeostatic function declines. Specifically, when the driver is in a normal state, when the line of sight is moved, the head also follows the movement of the line of sight in order to maintain a stable state of the line of sight, but the driver's homeostatic function is reduced. Even if the line of sight moved, it was found that it was difficult for the head to follow the line of sight. From this, the inventors of the present application have found that it is effective to detect an abnormality sign from the correlation between the head behavior and the eyeball behavior.

Therefore, in the technology according to the present disclosure, in addition to detection of an abnormality sign based only on the head behavior, an abnormality sign is also detected from the degree of matching between the eyeball behavior and the head behavior. This makes it possible to detect an abnormality sign as early as possible even in a driving scene where it is difficult to detect an abnormality sign only from the behavior of the head, such as constant-speed driving. As a result, it is possible to detect, as early as possible, a sign that the driver of the moving object will be unable to drive.

In one embodiment of the driver state estimating device, the first detection unit calculates a periodicity feature amount for the time-series data indicating the head behavior of the driver, and calculates the periodicity feature amount obtained by the calculation. , a time-series variation pattern is calculated, and the time-series variation pattern obtained by the calculation is compared with a predetermined first threshold value to detect the presence or absence of an anomaly sign of the driver.

That is, basically, if the driver is in a normal state, the behavior of the driver's head becomes irregular according to the acceleration input to the vehicle. However, as a result of intensive research by the present inventors, it was found that periodicity often appears in the head behavior even when the driver is in a normal state. Therefore, the time-series variation pattern of the periodicity feature amount is calculated, and the obtained time-series variation pattern is compared with a predetermined threshold to determine whether there is an anomaly symptom of the driver. As a result, it is possible to detect a sign of abnormality of the driver earlier.

In the one embodiment, the first detection unit further calculates the coherence between the driver's head behavior and the lateral acceleration acting on the driver, and for the periodic feature amount and the coherence obtained by the calculation, Calculate the time series fluctuation pattern.

That is, in a driving scene such as cornering in which a large lateral acceleration acts on the driver's head, periodicity is likely to appear in the behavior of the head even if the driver is in a normal state. Therefore, in the technique according to the present disclosure, the coherence between the driver's head behavior and the lateral acceleration acting on the driver is calculated to calculate the time-series variation pattern. As a result, it is possible to detect a sign of abnormality in consideration of the correlation between the behavior of the driver's head and the lateral acceleration, which is an external factor, so that the sign of abnormality of the driver can be detected earlier.

In the driver state estimation device, the second detection unit compares the displacement amount of the eyeballs detected by the eyeball behavior detection unit and the displacement amount of the head detected by the head behavior detection unit, The configuration may be such that the presence or absence of an abnormality sign of the driver is detected.

That is, when the eyeballs are slightly displaced, the head may not follow. If an abnormality sign is detected based on the correlation between the amount of displacement of the eyeballs and the amount of displacement of the head, the trend of the head behavior relative to the eyeball behavior, including the slight and relatively large movements of the eyeballs, can be detected. Abnormal conditions can be detected. As a result, it is possible to accurately detect an abnormality sign of the driver.

In the driver state estimating device that compares the amount of displacement of the eyeballs and the amount of displacement of the head, the second detection unit creates a two-dimensional map in which the amount of displacement of the head is plotted against the amount of displacement of the eyeballs detected at the same time. and calculating the occurrence probability distribution of the plots in the calculated two-dimensional map, and comparing the degree of coincidence between the calculated occurrence probability distribution and the previously calculated occurrence probability distribution at the time of abnormality, thereby detecting the abnormality of the driver. It may be configured to detect the presence or absence of a sign.

According to this configuration, it is possible to appropriately determine the difference in the degree of cooperation between the eyeball behavior and the head behavior between the normal state and the abnormal state. As a result, it is possible to detect an abnormality sign of the driver early and with high accuracy.

As described above, according to the technology disclosed herein, it is possible to detect, as early as possible, a sign that the driver of the moving object will be unable to drive.

FIG. 1 is a conceptual diagram explaining the positioning of the technology according to the present disclosure. FIG. 2 is an example of time-series data of the pitch angle and roll angle of the head. FIG. 3 is a graph showing the distribution of the autocorrelation indices of the pitch angle and roll angle of the head. FIG. 4 is a graph showing time series fluctuations of the autocorrelation index. FIG. 5 is a graph showing classification results of time-series variation patterns of autocorrelation indices. Fig. 6 shows the time-series data of the pitch angle of the head. (a) shows the normal state during straight running, (b) shows the abnormal simulated state, and (c) shows the normal state during cornering. show FIG. 7 shows data obtained from running tests, in which (a) shows the time when the vehicle enters a corner from a straight section, and (b) shows the time when the vehicle transitions from a normal state to an abnormal simulated state. FIG. 8 is a graph showing time-series variation of coherence of head behavior and lateral acceleration. FIG. 9 is a graph showing classification results of time-series variation patterns of coherence and autocorrelation indices. FIG. 10 shows examples of time-series data of head behavior and eyeball behavior, where (a) shows a normal state and (b) shows an abnormal state. FIG. 11 is a diagram showing the process of creating data for comparing the correlation between head behavior and eyeball behavior. FIG. 12 is a two-dimensional map plotting the amount of eyeball displacement and the amount of head displacement, in which (a) indicates a normal state and (b) indicates an abnormal state. 13A and 13B are distribution diagrams obtained by calculating the appearance probability distribution from the two-dimensional map of FIG. 12, where (a) shows a normal state and (b) shows an abnormal state. FIG. 14 is time-series data showing head shaking, head-eye coincidence, and head displacement amount and eyeball displacement amount, (a) showing normal state, (b) showing abnormal state indicates FIG. 15 is a configuration diagram of an in-vehicle system including a driver state estimation device. FIG. 16 is a flow chart showing processing for estimating the driver's state based only on head behavior. FIG. 17 is a flow chart showing processing for estimating the driver's state based on coordination between head behavior and eyeball behavior.

Exemplary embodiments are described in detail below with reference to the drawings.

FIG. 1 is a conceptual diagram showing the positioning of the technology according to the present disclosure. State changes that cause the driver to become unable to drive can be summarized into three patterns. Case A is a pattern of partial deterioration of perception, judgment, and movement, Case B is a pattern of gradual deterioration of general functions, and Case C is a pattern of sudden loss of consciousness. Of these cases, in cases A and B, as shown in FIG. 1, after the disease develops, the driver's driving ability level gradually declines, eventually leading to an inability to drive. Therefore, if it is possible to detect the deterioration state of the driving ability, it is possible to detect a sign of impossibility of driving of the driver. Once the sign of impossibility of driving can be detected, it becomes possible to take emergency measures such as confirming the intention of the driver and evacuating the vehicle to the shoulder of the road by automatic driving control. Also, even in case C, emergency measures can be taken if a sign before loss of consciousness can be detected.

The technology according to the present disclosure focuses on the function of maintaining homeostasis of people, and detects signs of impossibility of driving (hereinafter referred to as signs of abnormality) from the behavior of the driver's head and the behavior of the driver's eyeballs. Specifically, in the present embodiment, an abnormality sign detection based only on the driver's head behavior and an abnormality sign detection based on the coordinated movement of the driver's head behavior and eyeball behavior are performed.

Human beings have a function called homeostasis to maintain a constant state against disturbances. Consistency of head behavior refers to the property of trying to maintain the head posture during driving. In addition, when focusing on the coordinated movement of the head and eyeballs, it means moving the head in the direction in which the eyeballs move in order to maintain a stable state of the line of sight.

<Detection of signs of abnormality based only on head behavior>
First, the detection of the sign of abnormality based only on the behavior of the head will be described.

Various studies have been conducted on the homeostasis of humans as described above. While fluctuating, it is stated that the movement of the head becomes smaller and stabilizes during disease. In addition, Non-Patent Document 2 describes that the head periodically fluctuates in a critical deceleration state in which a normal state transitions to a diseased state.

The inventors of the present application paid attention to the periodicity of head behavior, and considered detection of an abnormality sign of the driver using the periodicity feature amount of head behavior.

FIG. 2 shows the results of an experiment conducted by the inventors of the present application on the behavior of the driver's head. In this experiment, a driver was made to drive a test course, and the head behavior of the driver during driving was measured. The driver was instructed to drive normally as usual (corresponding to a normal state), and while driving, an abnormality simulation task (difficult mental arithmetic) was given as a task along with a signal (corresponding to an abnormal predictive state). As data representing the behavior of the head, the pitch angle (angle in the front-rear direction) and the roll angle (angle in the left-right direction) of the head were measured from the video of the driver's camera. Then, an autocorrelation index (scaling index α) was obtained by DFA (Detrended Fluctuation Analysis) for the time-series data of the pitch angle and roll angle of the head. DFA is a technique that examines scaling by removing very slowly varying components, so-called trends. An autocorrelation index (scaling index α) is an example of a feature quantity indicating the periodicity of data.

FIG. 2 is an example of time-series data of the pitch angle and roll angle of the head. FIG. 3 is a graph showing the distribution of the autocorrelation indices of the pitch angle and roll angle of the head. In FIG. 3, the horizontal axis is the roll angle autocorrelation index, and the vertical axis is the pitch angle autocorrelation index. A smaller value of the autocorrelation index indicates a stronger autocorrelation, and a larger value indicates a weaker autocorrelation.

As shown in FIG. 3, in the abnormal portent state, the distribution of the autocorrelation index is skewed toward the stronger autocorrelation than in the normal state. This experimental result agrees with the above findings that the head fluctuates irregularly in the normal state, and that the head fluctuates periodically in the critical deceleration state. However, as can be seen from FIG. 3, the distribution of the normal state spreads over a relatively wide range, and a large range overlaps with the distribution of the abnormal portent state. In other words, it can be seen that it is not always easy to determine the state of predicting anomaly using only the autocorrelation index.

Therefore, the inventors of the present application focused on the time-series variation pattern of the autocorrelation index. As shown in FIG. 4, the time-series data of the autocorrelation index was cut out in chronological order, and pattern classification was performed using this as a time-series variation pattern. Here, a nonlinear dimension compression method (UMAP: Uniform Manifold Approximation and Projection) is used as a pattern classification method.

FIG. 5 shows the results of classification of time-series variation patterns of autocorrelation indices. In FIG. 5, the time-series variation pattern is reduced to two dimensions by UMAP and mapped on a two-dimensional map. In FIG. 5, compared with FIG. 3, the distribution of the normal state and the distribution of the abnormal portent state are more separated. The decision line LTH shown in FIG. 5 is obtained using a support vector machine for two-dimensional data. With this judgment line LTH, an erroneous judgment rate of 15% was achieved for the judgment of the abnormal portent state.

Based on the above, the periodic feature amount is calculated for the time-series data representing the behavior of the driver's head, and the time-series variation pattern is calculated for the periodic feature amount obtained by the calculation. By comparing the obtained time-series variation pattern with a predetermined threshold value, it is conceivable that a sign of an abnormality of the driver can be detected.

However, as a result of continued studies by the inventors of the present application, it has been found that there are cases in which the behavior of the head resembles that of an anomaly sign state even when the driver is in a normal state.

FIG. 6 shows time-series data of the pitch angle of the head obtained by experiments by the inventors of the present application. In the case of driving, (c) is the case in which the driver runs a corner in a normal state. Each graph shows the raw data and its moving average. As shown in FIG. 6(a), in the case of driving in a straight line under normal conditions, the head fluctuates irregularly, and as shown in FIG. fluctuates regularly. However, as shown in FIG. 6(c), in the case of running a corner in a normal state, the head regularly fluctuates.

That is, in the case of FIG. 6(c), although the driver is in a normal state, the autocorrelation index of the head pitch angle data is large. become difficult. It is presumed that the reason why the driver's head regularly fluctuates during cornering is the lateral acceleration acting on the head.

Figure 7 shows the data obtained from the driving test, where (a) is the data when the vehicle enters the corner from the straight part of the circuit, and (b) is the data when the driver changes from the normal state to the simulated abnormal state. is. In FIGS. 7A and 7B, the upper graph shows the time-series changes in the vehicle lateral acceleration and the head roll angle, and the lower graph shows the coherence (cross-correlation index) between the vehicle lateral acceleration and the head roll angle. represents the time-series change of The coherence is obtained by correlating the power spectrum obtained from the time-series variation of the vehicle lateral acceleration and the power spectrum obtained from the time-series variation of the head roll angle. In the coherence graph, the horizontal axis is time and the vertical axis is frequency. The lighter the color, the higher the correlation, and the darker the color, the lower the correlation.

Looking at the data on the upper side of FIG. 7A, it can be seen that the head moves in conjunction with changes in lateral acceleration after the vehicle enters the corner. That is, when the lateral acceleration acts and the head starts to shake, the driver corrects the head in the opposite direction of the lateral acceleration. Therefore, the correlation between the vehicle lateral acceleration and the head roll angle gradually begins to appear after corner entry. On the other hand, in the case of FIG. 7B, the head roll angle is not linked to changes in the vehicle lateral acceleration, and the coherence hardly changes.

From the experimental results described above, it was found that periodicity is likely to appear in the behavior of the driver's head when a large lateral acceleration acts on the driver's head, such as during cornering. The frequency at which correlation appears is mainly 2 Hz or less. This is thought to reflect the frequency response of the head to external forces.

Therefore, in order to further improve the accuracy of determination of signs of abnormality, the inventors of the present application used not only the autocorrelation index of head behavior, but also the cross-correlation index with an external factor (here, lateral acceleration) to obtain a feature amount. We calculated and considered pattern classification. That is, as shown in FIG. 8, the coherence time-series data was cut out in chronological order and combined with the time-series data of the autocorrelation index described above to perform pattern classification. Here, as the coherence time-series data, data with a frequency of 1 Hz, in which the correlation with the vehicle lateral acceleration is likely to appear, and data with a frequency of 4 Hz, in which the correlation with the vehicle lateral acceleration does not appear so much, are used. A nonlinear dimensionality compression method (UMAP) was used as a method of pattern classification.

FIG. 9 shows the results of classification of time-series variation patterns of coherence and autocorrelation indices. In FIG. 9, similarly to FIG. 5, the time-series variation pattern is reduced to two dimensions by UMAP and mapped on a two-dimensional map. The decision line LTH2 shown in FIG. 9 is obtained using a support vector machine for two-dimensional data. With this determination line LTH2, it was possible to achieve an erroneous determination rate of 1% in the determination of the abnormal portent state.

From the above, in the present embodiment, when detecting an abnormality sign of the driver based only on the driver's head behavior, the periodic feature amount is calculated for the time-series data representing the driver's head behavior. Furthermore, the coherence between the behavior of the driver's head and the lateral acceleration acting on the driver's head is calculated, and the time-series variation pattern is calculated for the periodic feature amount and coherence obtained by the calculation, and obtained by the calculation. By comparing the time-series variation pattern with a predetermined threshold value, an abnormality sign of the driver is detected. As a result, it is possible to make a determination that takes into account the correlation between the head behavior and the lateral acceleration, which is an external factor, and to detect an abnormality sign of the driver with higher accuracy.

<Detection of signs of abnormality based on coordinated head-eye movement>
Next, the detection of an abnormality sign based on the coordinated movement of the driver's head behavior and eyeball behavior will be described.

FIG. 10 shows simultaneous measurements of the driver's head behavior and eyeball behavior. Both the driver's head behavior and eyeball behavior are calculated from the images captured by the vehicle interior camera. In FIG. 10, (a) shows each behavior when the driver is in a normal state, and (b) shows each behavior when the driver is in an abnormal state. In each graph, the vertical axis represents the angle (hereinafter referred to as yaw angle) by which the head and eyeballs move left and right from the reference position.

As shown in FIG. 10A, when the yaw angle of the eyeballs is very small, the head hardly moves. It can be seen that it is moving to On the other hand, referring to FIG. 10(b), it can be seen that when the driver has an anomaly sign, the head hardly moves even if the eyeballs move greatly. This is because the driver's homeostatic function is degraded, and as a result, the brain cannot keep up with the movement of the head, and the driver tries to see the object only with his/her eyes.

The inventors of the present application examined detection of an abnormality sign of a driver by comparing the yaw angle of the head and the yaw angle of the eyeballs based on the knowledge obtained from such experiments. Then, the inventors of the present application made a two-dimensional map of the yaw angle of the driver's eyeballs and the yaw angle of the head when the yaw angle was measured, and compared the two-dimensional map with the yaw angle of the driver's abnormal state. The inventors have found that by calculating the degree of matching with a two-dimensional map, it is possible to detect a sign of an abnormality in the driver.

In this embodiment, in order to calculate the degree of matching between two-dimensional maps, a two-dimensional map of the yaw angle of the driver's eyeballs and the yaw angle of the head is subjected to probability density estimation (KDE). , the appearance probability distribution Q of the plots on the two-dimensional map is calculated.

Specifically, as shown in FIG. 11, first, from the data for a predetermined period of time in the time-series data of the head yaw angle and the eyeball yaw angle, the head angle corresponding to the driver's eyeball yaw angle detected at the same time is obtained. A two-dimensional map 1101 plotting the partial yaw angle is created. The predetermined time is, for example, about 5 to 10 seconds.

Next, KDE creates a probability distribution map 1102 showing the occurrence probability distribution Q for the created two-dimensional map 1101 . In calculating the probability distribution map 1102, the two-dimensional map 1101 is divided into meshes of a predetermined size, and the appearance probability of plots is calculated for each mesh. As shown in FIG. 11, in the probability distribution map 1102, the higher the density of plots, the higher the appearance probability. A known program can be used to calculate the probability distribution map 1102 from the two-dimensional map 1101 .

FIG. 12 shows a two-dimensional map 1101a ((a) in FIG. 12) when the driver is in a normal state and a two-dimensional map 1101b ((b) in FIG. 12) when the driver is in an abnormal state. As shown in FIG. 12, when the driver is in a normal state, there are many plots with large head yaw angles. In particular, when the driver is in a normal state, the head yaw angle tends to increase as the eyeball yaw angle increases. This indicates that the head moves in accordance with the movement of the eyeballs due to human constancy. On the other hand, it can be seen that when the driver is in an abnormal state, the head yaw angle is small even if the eyeball yaw angle is large. This is because the homeostatic function is degraded, making it difficult to move the head.

FIG. 13 shows a probability distribution map 1102a ((a) in FIG. 13) calculated for the two-dimensional map 1101a of FIG. A distribution map 1102b (FIG. 13(b)) is shown. As shown in FIG. 13, when the driver is in a normal state, the occurrence probabilities are distributed even in areas where the head yaw angle is large. It can be seen that it is distributed only in a small part.

In this embodiment, a probability distribution map 1102b showing the occurrence probability distribution P when the driver is in an abnormal state is stored in advance as data, as shown in FIG. A degree of coincidence with Q is obtained to detect an abnormality sign of the driver. Specifically, the Kullback-Leibler distance (KL distance: Kullback-Leibler distance) between the latest occurrence probability distribution Q and the occurrence probability distribution P serving as a model of the abnormal state is calculated. The KL distance is calculated by the following formula.

In this equation, i represents the coordinates of the mesh.

This KL distance becomes smaller as the degree of matching between the latest occurrence probability distribution Q and the occurrence probability distribution P at the time of abnormality becomes higher, and becomes 0 when they completely match. Therefore, when the KL distance is equal to or less than a predetermined value, it can be considered that there is a sign of abnormality in the driver, and when the KL distance is greater than the predetermined value, it can be considered that the driver is in a normal state.

FIG. 14 shows the result of detecting the head yaw angle and the eyeball yaw angle for the driver and calculating the KL distance between the occurrence probability distribution calculated from the detection results and the occurrence probability distribution in the abnormal state. In this experiment, the driver was caused to drive using a drive simulator, and the head behavior and eyeball behavior of the driver during driving were measured. FIG. 14(a) shows the results detected from the driver in the normal state, and FIG. 14(b) shows the results detected from the case C patient described above. In both FIGS. 14A and 14B, the upper diagram shows the KL distance, and the lower diagram shows the detected head yaw angle and eyeball yaw angle. In addition, in FIG. 14B, an abnormality occurs in the driver at the end of the graph.

As shown in FIG. 14(a), when the driver is normal, the head and eyeballs of the driver generally move in conjunction with each other, and the KL distance remains relatively high. On the other hand, as shown in FIG. 14(b), when a sign of abnormality appears in the driver, the KL distance abruptly decreases and changes to a value close to zero. This is considered to be due to a decline in the homeostatic function and a decline in the coordinated motor function between the head and the eyeballs as a sign of abnormality. Therefore, even in actual experiments, by calculating the KL distance between the latest appearance probability distribution Q and the appearance probability distribution P at the time of an abnormality, it has been proved by actual experiment results that it is possible to detect a sign of an abnormality of the driver. rice field. In FIG. 14(b), it is detected that there is an anomaly sign during the time indicated by the dashed line after the KL distance becomes equal to or less than the predetermined value. Thus, even based on coordinated head-eyeball movements, it is possible to detect signs of abnormality before an abnormality actually occurs. In particular, it was found to be effective also for the case C patient. An error prediction rate of 0.2% could be achieved by using this KL distance to determine the abnormal sign state.

<Configuration of driver state estimation device>
FIG. 15 shows a configuration example of an in-vehicle system including a driver state estimation device. In the in-vehicle system shown in FIG. 15, a camera 10, an acceleration sensor 11, a speaker 12, an information presenting section 13, a switch 14, and a microphone 15 are installed inside the vehicle. The information processing device 20 is configured by, for example, a single IC chip including a processor and memory, or a plurality of IC chips including processors and memories. The vehicle stop control unit 40 receives an instruction from the information processing device 20 and performs control to automatically evacuate the vehicle from the road shoulder and stop the vehicle.

The camera 10 is installed, for example, inside the windshield, and photographs the situation inside the vehicle including the driver. An image captured by the camera 10 is transmitted to the information processing device 20 via, for example, an in-vehicle network.

In the information processing device 20 , the head behavior detection unit 21 detects the behavior of the driver's head from the image captured by the camera 10 . For example, the driver's head is recognized from the image, and the tilt angles of the head, such as the pitch angle, roll angle, and yaw angle, are obtained. The processing in the head behavior detection unit 21 can be realized by existing image processing technology. Through the processing by the head behavior detection unit 21, time-series data of head behavior as shown in FIGS. 2, 10, and 14 can be obtained. The time-series data of the head behavior is sent to the first detection unit 110 and the second detection unit 120 that detect an abnormality sign of the driver.

In the information processing device 20 , the eyeball behavior detection unit 22 detects the eyeball behavior of the driver from the image captured by the camera 10 . For example, the eyeball of the driver is recognized from the image, and the tilt angle of the iris of the eyeball, for example, the yaw angle is obtained. The processing in the eyeball behavior detection unit 22 can be realized by existing image processing technology. Through the processing by the eyeball behavior detection unit 22, it is possible to obtain time series data of eyeball behaviors as shown in FIGS. 10 and 14 . The time-series data of the eyeball behavior is sent to the second detection unit 120 that detects an abnormality sign of the driver.

The first detection unit 110 detects an abnormality sign of the driver only from the behavior of the head. The first detection unit 110 includes a periodicity feature calculation unit 111 , a time-series variation pattern calculation unit 112 , an abnormality determination unit 113 , an abnormality determination threshold database 114 , and a coherence calculation unit 115 . The periodicity feature amount calculation unit 111 calculates a periodicity feature amount from the time-series data of the head behavior obtained by the head behavior detection unit 21 . Specifically, for example, by DFA, an autocorrelation index (scaling index α) is obtained as a periodicity feature amount. Time-series data of the periodic feature amount as shown in FIG. 4 can be obtained by the periodic feature amount calculation unit 111 .

The coherence calculation unit 115 uses the output of the acceleration sensor 11 and the time-series data of the head behavior obtained by the head behavior detection unit 21 to calculate the behavior of the driver's head and the lateral force acting on the driver's head. Calculate coherence with acceleration. For example, the coherence calculator 115 treats the lateral acceleration of the vehicle recognized from the output of the acceleration sensor 11 as lateral acceleration acting on the driver's head, and obtains the power spectrum from the time-series data of this lateral acceleration. Further, the coherence calculator 115 obtains the power spectrum from the time-series data of the head behavior, and obtains the coherence by taking the correlation between the power spectra. Then, the coherence calculator 115 obtains time-series data of, for example, 1 Hz as a frequency at which the correlation with the lateral acceleration is likely to appear, and time-series data at, for example, 4 Hz as the frequency at which the correlation with the lateral acceleration does not appear so much.

The time-series variation pattern calculation unit 112 cuts out the time-series data of the periodic feature quantity obtained by the periodic feature quantity calculation unit 111 and the time-series data of the coherence obtained by the coherence calculation unit 115 in chronological order, A time-series variation pattern is calculated from a combination of cut-out time-series data, and the pattern is classified. Specifically, for example, time-series data of combinations of periodicity feature amounts and coherence are subjected to dimensionality reduction using UMAP, which is one of nonlinear dimensionality reduction techniques, and converted into two-dimensional data.

The abnormality determination unit 113 compares the data obtained by the time-series variation pattern calculation unit 112 with threshold values stored in the abnormality determination threshold database 114, and determines whether or not an abnormality sign has occurred in the driver. For example, the determination line LTH2 in the two-dimensional map of FIG. 9 corresponds to the threshold stored in the abnormality determination threshold database 114. FIG. The abnormality determination unit 113 determines whether or not an abnormality sign has occurred in the driver depending on which side of the determination line LTH2 on the two-dimensional map the data obtained by the time-series variation pattern calculation unit 112 is located. do.

The second detection unit 120 detects an abnormality sign of the driver from the correlation between the head behavior and the eyeball behavior. The second detection unit 120 includes a head-eyeball displacement amount distribution calculation unit 121 , an abnormality degree calculation unit 122 , an abnormality determination unit 123 , an abnormality probability distribution database 124 , and an abnormality determination threshold database 125 . The head-eyeball displacement amount distribution calculation unit 121 calculates the same A two-dimensional map 1101 of the amount of displacement of the eyeballs and the amount of displacement of the head detected in time is calculated.

The degree-of-abnormality computation unit 122 divides the two-dimensional map 1101 into a plurality of meshes, computes the appearance probability distribution for each mesh by, for example, KDE, and creates a probability distribution map 1102 . As a result, a probability distribution map 1102 as shown in FIG. 13 can be obtained. In addition, the degree-of-abnormality calculation unit 122 calculates the degree of coincidence between the created probability distribution map 1102 and the probability distribution map in the abnormal state stored in the abnormality probability distribution database 124 . The degree-of-abnormality computing unit 122 computes the degree of coincidence between the probability distribution maps by, for example, computing the KL distance between the probability distribution maps.

The abnormality determination unit 123 compares the data obtained by the abnormality degree calculation unit 122 with the threshold values stored in the abnormality determination threshold database 125, and determines whether or not the driver has an abnormality sign. The threshold is set to 1, for example. When the KL distance obtained by the abnormality degree calculation unit 122 is 1 or less, the abnormality determination unit 123 determines that there is an abnormality sign in the driver. Then, it is determined that there is no sign of abnormality in the driver.

The first detection unit 110 and the second detection unit 120 output an inquiry request to the driver to the inquiry unit 23 when the abnormality judgment units 113 and 123 judge that the driver has an abnormality sign.

When receiving an inquiry request from at least one of the first detection unit 110 and the second detection unit 120, the inquiry unit 23 makes an inquiry to the driver. This question is for confirming the intention of the driver as to whether or not the vehicle should be evacuated automatically. The question is, for example, voiced through the speaker 12 or displayed through the information presentation unit 13 such as a monitor.

The response detection unit 24 detects the driver's response to the inquiry by the inquiry unit 23 . The driver's response is performed by operating the switch 14 or speaking through the microphone 15, for example. When the driver's intention is confirmed, or when there is no response from the driver, the information processing device 20 instructs the vehicle stop control unit 40 to automatically evacuate the vehicle from the road shoulder and stop the vehicle.

The driver state estimation device according to the present embodiment includes at least the head behavior detector 21, the eyeball behavior detector 22, the first detector 110, and the second detector 120 in the information processing device 20. . Also, the driver state estimation device according to the present disclosure may include camera 10 .

FIG. 16 is a flow chart showing the processing operation of driver state estimation based only on head behavior.

First, in step S11, the head behavior detection unit 21 performs image recognition of the driver's head from the image captured by the camera 10, and calculates the tilt angle, here the pitch angle and the roll angle, of the recognized head. do. This calculation is performed, for example, every 100 ms.

Next, in step S<b>12 , the first detection unit 110 determines whether or not a predetermined number or more of tilt angle data of the head have been accumulated. When the first detection unit 110 determines YES that the tilt angle data of the head is accumulated in a predetermined number or more, the first detection unit 110 proceeds to step S13. On the other hand, when the first detection unit 110 determines NO that the tilt angle data of the head is not accumulated in the predetermined number or more, the process returns to step S11.

In step S<b>13 , the first detection unit 110 calculates the periodicity feature amount for the time-series data of the inclination angle of the head. The first detection unit 110 obtains an autocorrelation index (scaling index α) as a periodicity feature amount by DFA, for example. The calculation of the periodicity feature amount is performed using, for example, 256 pieces of tilt angle data, for example, every 100 ms while shifting the time range of the target tilt angle data.

Next, in step S14, the first detection unit 110 detects the behavior of the driver's head and the lateral acceleration acting on the driver's head using the time-series data of the output of the vehicle acceleration sensor 50 and the tilt angle of the head. Compute the coherence of

Next, in step S<b>15 , the first detection unit 110 determines whether or not a predetermined number or more of periodicity feature amount and coherence data have been accumulated. The first detection unit 110 proceeds to step S16 when the determination is YES that data equal to or more than the prescribed number have been accumulated. On the other hand, the first detection unit 110 returns to step S<b>11 when the data is not accumulated to the specified number or more (NO).

In step S16, the first detection unit 110 calculates a time-series variation pattern for combined data of the periodicity feature amount and coherence. The first detection unit 110 uses, for example, UMAP, which is one of the nonlinear dimension compression techniques, to reduce the dimensions of the periodic feature amount and coherence time-series variation data, and convert the data into two-dimensional data. The calculation of the time-series variation pattern is performed using, for example, 256 periodic feature amounts and coherence data every 100 ms while shifting the time range of the target periodic feature amount and coherence data.

Next, in step S<b>17 , the first detection unit 110 acquires the threshold held in the abnormality determination threshold database 114 . Specifically, for example, the first detection unit 110 acquires information of a determination line in a two-dimensional map of classification patterns, such as the determination line LTH2 shown in FIG. 9, as a threshold.

Then, in step S18, the first detection unit 110 compares the two-dimensional data, which is the time-series variation pattern obtained in step S16, with the determination line acquired in step S17, and determines whether the two-dimensional data matches the determination line. It is determined whether or not it has exceeded. When the two-dimensional data exceeds the determination line and is on the abnormality portent side (YES), the first detection unit 110 determines that there is an abnormality portent in the driver, and proceeds to step S19. On the other hand, when the two-dimensional data does not exceed the determination line and is on the normal side and NO, the first detection unit 110 determines that there is no sign of abnormality in the driver, and returns to step S11.

At step S<b>19 , the first detection unit 110 outputs an inquiry request to the driver to the inquiry unit 23 . After step S19, the process ends.

FIG. 17 is a flowchart showing the processing operation of driver state estimation based on the correlation between head behavior and eyeball behavior.

First, in step S21, the head behavior detection unit 21 performs image recognition of the driver's head from the image captured by the camera 10, and calculates the displacement amount (here, yaw angle) of the recognized head. to calculate Further, the eyeball behavior detection unit 22 performs image recognition of the driver's eyeballs from the image captured by the camera 10, and calculates an eyeball displacement amount (here, yaw angle) for the recognized eyeballs. This calculation is performed, for example, every 100 ms.

Next, in step S<b>22 , the second detection unit 120 determines whether or not a specified number or more of the head displacement amount data and the eyeball displacement amount data are accumulated. When the second detection unit 120 determines YES that the displacement amount data of the head and the eyeballs is accumulated in a predetermined number or more, the second detection unit 120 proceeds to step S23. On the other hand, when the second detection unit 120 determines NO that the displacement amount data of either the head or the eyeballs is not accumulated to the specified number or more, the process returns to step S21.

In step S23, the second detection unit 120 calculates a two-dimensional map showing the head-eyeball displacement amount distribution for the time-series data of the head and eyeball displacement amounts.

Next, in step S24, the second detection unit 120 calculates a probability distribution map indicating the appearance probability distribution for the two-dimensional map calculated in step S23. For example, the second detection unit 120 divides the two-dimensional map into a plurality of meshes and calculates the probability distribution map by calculating the appearance probability distribution for each mesh using KDE.

Next, in step S25, the second detection unit 120 compares the occurrence probability distribution calculated in step S24 with the occurrence probability distribution at the time of abnormality stored in the probability distribution database at the time of abnormality. Specifically, the second detection unit 120 calculates the KL distance of the two appearance probability distributions.

Next, in step S26, the second detection unit 120 determines whether or not the KL distance calculated in step S25 is equal to or less than a predetermined value. When the KL distance is equal to or less than the predetermined value, that is, YES, the second detection unit 120 determines that there is an anomaly sign in the driver, and proceeds to step S27. On the other hand, when the KL distance is larger than the predetermined value (NO), the second detection unit 120 determines that there is no sign of abnormality in the driver, and returns to step S21.

In step S<b>27 , the second detection unit 120 outputs an inquiry request to the driver to the inquiry unit 23 . After step S27, the process ends.

In this way, in addition to detection of anomaly signs based only on head behavior, anomaly signs can be detected early and with high accuracy by detecting an anomaly sign based on the correlation between head behavior and eyeball behavior. . In other words, for example, in a constant-speed driving scene in which almost no acceleration is applied to the head and the periodicity feature of the driver's head behavior cannot be clearly calculated, the detection accuracy of an abnormality sign based only on the head behavior is low. descend. However, by detecting an abnormality sign based on the correlation between the head behavior and the eyeball behavior, it is possible to appropriately detect the driver's abnormality sign even in such a driving scene.

In addition, in the present embodiment, the first detection unit 110 that detects an abnormality sign of the driver calculates a periodicity feature amount with respect to the time-series data representing the behavior of the driver's head, and calculates the behavior of the driver's head. The coherence with the lateral acceleration acting on the head of the driver is calculated, the time-series variation pattern is calculated with respect to the periodicity feature amount and the coherence, the obtained time-series variation pattern is compared with a predetermined threshold value, and the driver Determines whether or not there is an anomaly sign. As a result, it is possible to make a determination that takes into account the correlation between the head behavior and the lateral acceleration, which is an external factor, so that it is possible to detect a sign of an abnormality in the driver earlier and with higher accuracy. In addition, the behavior of the driver's head can be detected from the image taken by the camera 10 installed in the vehicle, so this technology can quickly detect signs of illness that can lead to inability to drive, using existing on-vehicle sensors. .

Further, in the present embodiment, the second detection unit 120 that detects an abnormality sign of the driver calculates a two-dimensional map plotting the displacement amount of the head with respect to the displacement amount of the eyeball detected at the same time, and calculates the calculated two-dimensional map. The occurrence probability distribution Q of plots in the dimensional map is calculated, and the degree of coincidence between the calculated occurrence probability distribution Q and the pre-calculated occurrence probability distribution P at the time of abnormality is compared to detect the presence or absence of an abnormality sign of the driver. . That is, when the eyeballs are slightly displaced, the head may not follow. By calculating the appearance probability distribution of plots in a two-dimensional map in which the amount of displacement of the eyeball and the amount of displacement of the head are plotted, and detecting an abnormality sign on the basis of the appearance probability distribution, a minute movement of the eyeball can be detected. Abnormal conditions can be detected by the tendency of head behavior relative to eye movement, including relatively large movements. As a result, it is possible to detect an abnormality sign of the driver early and with high accuracy.

(Other embodiments)
The technology disclosed herein is not limited to the above-described embodiments, and substitutions are possible without departing from the scope of the claims.

For example, in the above-described embodiment, the first detection unit 110 obtains an autocorrelation index (scaling index α) as a periodicity feature amount by DFA for time-series data of head behavior. Not limited to this, as a method for obtaining the periodicity feature amount, for example, FFT (Fast Fourier Transform) may be used, or simple autocorrelation may be calculated.

Further, in the above-described embodiment, the second detection unit 120 uses the KL distance as the degree of matching between the probability distribution map at the time of abnormality and the latest probability distribution map in determining the sign of abnormality. Alternatively, for example, the degree of matching between the abnormal two-dimensional map and the latest two-dimensional map may be compared using, for example, the chi-square method.

Further, in the above-described embodiment, when there is an inquiry request from either one of the first detection unit 110 and the second detection unit 120, the inquiry unit 23 asks the driver. The configuration is not limited to this, and the inquiry unit 23 may ask the driver only when an inquiry request is received from both the first detection unit 110 and the second detection unit 120 . Alternatively, the information processing device 20 may make an inquiry request to the inquiry unit 23 only when both the first detection unit 110 and the second detection unit 120 detect an abnormality sign.

Further, in the present embodiment described above, the occurrence probability distribution P at the time of abnormality was stored in the abnormality probability distribution database 124 . The occurrence probability distribution P at the time of abnormality may be updated based on the accumulated occurrence probability distribution Q by accumulating the occurrence probability distribution Q including the time of an abnormality sign of the driver. Also, the thresholds stored in the abnormality determination threshold database 125 may be updated based on the accumulated occurrence probability distribution Q. FIG.

In addition, applications of the technology according to the present disclosure are not limited to automobiles. It is effective for detecting signs of abnormality of a driver in moving bodies other than automobiles, such as trains.

Also, the technology according to the present disclosure may be realized in a form other than a single information processing device. For example, the first detection unit 110 and the second detection unit 120 may be realized by an information processing device different from the head behavior detection unit 21 and the eyeball behavior detection unit 22 . Further, for example, an information processing device that is not mounted on a moving object, such as a smartphone or a tablet held by a driver, may realize the functions of the first detection unit 110 and the second detection unit 120 . Alternatively, the cloud may execute part or all of the arithmetic processing performed by the first detection unit 110 and the second detection unit 120 .

The above-described embodiments are merely examples, and should not be construed as limiting the scope of the present disclosure. The scope of the present disclosure is defined by the claims, and all modifications and changes within the equivalent range of the claims are within the scope of the present disclosure.

The technology disclosed herein is useful as a driver state estimation device that estimates the state of a driver driving a mobile object.

21 head behavior detector 22 eyeball behavior detector 110 first detector 120 second detector

Claims (5)

A driver state estimation device for estimating the state of a driver driving a mobile object,
a head behavior detection unit that detects the behavior of the driver's head;
an eyeball behavior detection unit that detects the behavior of the driver's eyeballs;
a first detection unit that detects an abnormality sign of the driver only from the head behavior detected by the head behavior detection unit;
a second detection unit that detects an abnormality sign of the driver from a correlation between the head behavior detected by the head behavior detection unit and the eyeball behavior detected by the eyeball behavior detection unit. Driver state estimator. In the driver state estimation device according to claim 1,
The first detection unit is
calculating a periodicity feature value for the time-series data indicating the head behavior of the driver;
Calculate the time-series variation pattern for the periodicity feature obtained by the calculation,
A driver state estimating device that compares a time-series variation pattern obtained by calculation with a predetermined first threshold value to detect the presence or absence of an anomaly sign of the driver. In the driver state estimation device according to claim 2,
The first detection unit is
further computing the coherence between the driver's head behavior and the lateral acceleration acting on the driver;
A driver state estimating device that calculates a time-series variation pattern with respect to periodic feature quantities and coherence obtained by calculation. In the driver state estimation device according to any one of claims 1 to 3,
The second detection unit compares the amount of displacement of the eyeballs detected by the eyeball behavior detection unit and the amount of displacement of the head detected by the head behavior detection unit, thereby determining whether there is an anomaly sign of the driver. A driver state estimation device characterized by detecting In the driver state estimation device according to claim 4,
The second detection unit is
Calculate a two-dimensional map plotting the displacement of the head against the displacement of the eyeball detected at the same time,
Calculate the appearance probability distribution of plots in the calculated two-dimensional map,
A driver state estimating device that detects the presence or absence of an abnormality sign of the driver by comparing the degree of coincidence between the calculated appearance probability distribution and a previously calculated appearance probability distribution at the time of an abnormality.

Images