JPH07101265A - Driving state detecting device - Google Patents

Abstract

PURPOSE:To extract the eye image of a driver positively to detect the driving state by processing the face image of the photographed driver. CONSTITUTION:An image from a camera 10 for photographing a vehicle driver is supplied to an image processing device 12. The image processing device 12 extracts a face area using a standard template stored in a memory 14. The image processing device 12 then extracts plural prospective eye areas from the extracted face area using the standard template and extracts a true eye area image from plural prospective eye areas using layout data on the layout of eyebrows and eyes. With the extracted eye area image as an object template, the correlative arithmetic result to the input image is outputted to an ECU 16 to monitor the state of eyes.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a driving state detecting device, and more particularly to a device for detecting a driving state of a driver by processing a face image of the driver obtained by photographing.

[0002]

2. Description of the Related Art Conventionally, various devices have been developed and installed in order to improve the safety of vehicle running. The driver's driving condition is monitored by an in-vehicle camera or the like to detect drowsiness or looking aside. The operating condition detection device that gives an alarm is also part 1
Is one.

For example, in Japanese Unexamined Patent Publication No. 63-174490, a video signal is A / D converted and stored in a video memory at regular intervals, and the contents of the video memory are compared with a reference video. It discloses the technology that there is a movement. That is, both the digital reference video signal and the digital video signal are input to the comparator, and the comparator compares (subtracts) both digital signals and outputs a motion detection signal when there is a difference of a certain value or more. Here, the reference image is stored in the reference image memory, but the content of the reference memory is appropriately updated at a predetermined timing according to the application to prevent erroneous detection of a change in brightness such as the shade of the sun.

[0004]

However, when such a motion detecting device is mounted on a vehicle, the brightness at the time of photographing changes variously according to the traveling environment, so the reference image must be updated frequently. However, there is a problem that the significance of the reference image is lost. In order to detect whether or not the driving condition of the driver is normal, it is necessary to store the driver image in a normal driving condition as a reference image in advance, but the reference image must be updated frequently like this. This is because if it is not possible to determine which is the image of the normal driving state.

Therefore, in order to detect the driving state of the driver even when the luminance changes due to various running environments as described above, the applicant of the present application has previously described the following in Japanese Patent Application No. 5-47597. A driving state detection device was proposed. That is, a target template peculiar to the driver, which is a target for detecting the driving state, is created by performing a correlation calculation between a predetermined standard template and the normalized captured image. Then, a correlation calculation is performed between the target template and the image in the corresponding range of the normalized image, and the state of the eyes and the direction of the face of the driver are detected from the change in the correlation value. By using the normalized image, it is possible to remove the influence of the change in the background brightness that changes depending on the driving environment, and by performing the correlation calculation, the change state of the driver can be reliably detected.

Here, in order to detect the state of the driver's eyes by the correlation calculation with the target template, it is premised that the target driver's eye region is detected reliably. The driver's eye area will be extracted from the predetermined brightness change, but there is a possibility that the area where the area brightness is close to the eye area (for example, the eyebrows or the rubbing part) is erroneously detected as the eye area. In this case, the target template itself is incorrect, which causes a problem that the driving state of the driver cannot be accurately detected.

The present invention has been made in view of the above conventional problems, and an object of the present invention is to provide a driving state detecting device capable of surely extracting the driver's eye region to determine the driving state of the driver. .

[0008]

In order to achieve the above object, a driving state detecting apparatus according to claim 1 detects a driving state of a driver by processing a face image of the vehicle driver. An apparatus, which is an image capturing unit that captures an image of a vehicle driver, a normalizing unit that normalizes the density of an obtained image, and a standard that specifies a face and an eye region in advance with respect to the normalized image. Correlation calculation is performed using the template to generate a target face template and a target eye template for the driver, and a correlation calculation is performed on the normalized image using the target template to obtain a correlation value. Detecting means for detecting the driving state of the driver based on the change of the driver, and the target template creating means uses the standard template for a normalized image of the driver. Face area extracting means for extracting a face area image, eye area candidate extracting means for extracting an eye area candidate of the driver using the standard template for the extracted face area, and a driver of the driver from the extracted eye area candidates. Eye area extracting means for extracting the eye area of the driver by using the information on the arrangement of the face elements.

Further, in order to achieve the above object, a second aspect is provided.
The driving state detection device described is a driving state detection device that detects the driving state of the driver by processing the face image of the vehicle driver, and an imaging means for photographing the vehicle driver and the obtained image. Correlation calculation is performed using a normalization means for performing grayscale normalization and a standard template in which the face area, the eye vicinity area and the eye area are specified in advance for the normalized image, and the target face template for the driver, the target Correlation calculation is performed by sequentially using the target face template, the target eye vicinity template, and the target eye template with respect to the normalized image, the target template creating means for creating the eye vicinity template and the target eye template, and the change of the correlation value. And a detection means for detecting the driving state of the driver based on the above.

[0010]

In the operating state detecting device according to claim 1,
When creating the target driver's eye template by the target template creating means, information about the arrangement of face elements such as eyebrows and eyes is used. In the target eye template, the face area is first extracted by the correlation calculation between the normalized image and the standard face template, and then the eye area is extracted by the correlation calculation between the face area and the standard eye template. Correlation calculation is also performed to extract eyebrows, a massaging part, and the like having a brightness change similar to that of the eye region. When the true eye region is extracted from these extracted eye region candidates, the face element arrangement information is used. Here, the placement information is, for example, that the eyebrows are in a horizontal direction in a substantially horizontal direction and are separated by 1
A pair of eyes, the eyes are approximately parallel to the lateral direction and are spaced apart 1
A pair of eyes, a pair of eyes are below a pair of eyebrows, and so on. When the true eye area and the eyebrow area are extracted as the eye area candidates, the eye area is separated and extracted from the eyebrow area by using the arrangement information that the eye area is below the eyebrow.

Further, in the driving state detecting apparatus according to the second aspect, when creating the target eye template, three types of standard templates, that is, standard face templates,
A standard eye vicinity template and a standard eye template are prepared and a correlation calculation is performed. A face image is extracted by first using a standard face template for the normalized image. Next, by using the standard eye vicinity template, the eye vicinity area including the eyebrows is extracted. Further, by using the standard eye template in the eye vicinity area including the eyebrows, only the eye area of the target driver is extracted.

In the above-mentioned Japanese Patent Application No. 5-47597, the eye area image is directly extracted from the face area image, but by sequentially performing the three-step extraction processing of face area-near eye area-eye area in this way, Only the eye area can be reliably detected.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT A preferred embodiment of the operating condition detecting apparatus of the present invention will be described below with reference to the drawings.

First Embodiment FIG. 1 shows a block diagram of the configuration of this embodiment. A camera 10 for photographing the driver is provided at a predetermined position of the vehicle and photographs a face image of the driver. An image processing device 12 is connected to the camera 10, and the obtained face image of the driver is supplied to the image processing device 12. The image processing device 12 includes an A / D converter, a normalization circuit, and a correlation calculation circuit, converts the input image signal into a digital signal, and further performs a grayscale normalization process. The image processing device 12 has a memory 14
Are connected. The memory 14 stores in advance standard templates (face template and eye template) and arrangement data of face elements such as eyebrows and eyes. The contents of the arrangement data are “two for each eyebrow and eye”,
The difference between the left and right gray value of the eyebrows and the eyes is small, the eyebrows and the eyes are parallel to the lateral direction (x direction), and the eyes are below the eyebrows. The image processing device 12 is further connected to the electronic control unit ECU 16, and the processing result is stored in the ECU.
Supply to 16. The ECU 16 is configured to determine the driving state of the driver from the processing result and output a control signal to an alarm device (not shown) to issue an alarm.

FIG. 2 is a processing block diagram showing the manner of processing in the configuration of this embodiment shown in FIG. The face image of the driver is captured by the camera 10 and output to the image processing device 12. The image processing apparatus 12 detects the target template (target face template, target eye template) of the target driver by the correlation calculation using the standard template stored in the memory 14. The face layout data stored in the memory 14 is used when creating the target eye template. The created target template is stored in the memory 14, and the target template is used to monitor the change of the driver's eye state from the driver's face image by the correlation calculation. The result of the correlation calculation is output to the ECU 16 and it is determined whether the driver is in an abnormal state.

The configuration of this embodiment is as described above, and its operation will be described in more detail below with reference to the flowcharts of FIGS.

FIG. 3 shows a detection flowchart for detecting an abnormal state such as a driver's drowsiness or looking aside using the apparatus of this embodiment. First, the face image of the driver is captured by the camera 10 (S101). Next, the obtained face image is gray-scale normalized by the image processing device 12 (S1).
02). The normalization processing is performed by the grayscale normalization in which the minimum brightness in the image is 1 and the maximum brightness is 256. After the normalization process is performed, the target template is detected using the standard template (S103). Details of the target template detection process are shown in the process flowchart of FIG. In FIG. 4, first, a target face area is searched from the obtained face image (S201). This search is performed by performing a correlation calculation between the normalized face image input from the camera 10 and the standard face template stored in the memory 14.
That is, the correlation calculation is performed on the face image while sequentially shifting the standard face template, the position where the correlation is highest (that is, the correlation value is minimum) is obtained, and the face area is extracted.

After the search for the target face area is completed, a small area is set in this target face area (S202). This small area divides the extracted face area into a plurality of strip-shaped areas, and is shown in FIG. In FIG. 5, reference numeral 100
The frame indicated by indicates the area of the face image, and the reference numeral 200
The frame indicated by is the face area extracted in S201 described above, and the frame indicated by reference numeral 300 is the small area. The width of the small area 300 is set such that the driver's eyebrows and eyes are not included at the same time. After a small area is set in the target face area, a target eye area candidate is searched in this small area (S203). That is, the correlation calculation is performed on the image in the small area using the standard eye template stored in advance in the memory 14, and the position with a high degree of correlation is searched for.
When the horizontal direction of the position where the correlation value is the minimum is represented by the x-coordinate and the vertical position is represented by the y-coordinate, the x-coordinate is separated by a predetermined threshold value xth or more and the y-coordinate is the predetermined minimum value yth or less. The correlation value area is stored in the memory 14 as the first and second area candidates. After the search in a certain small area is completed, as shown in FIG. 6A, the y-coordinate in the vertical direction for identifying the small area is updated, and the process moves to the next small area search (S20).
4). Hereinafter, similarly, the search for the first and second eye area candidates is performed for all the small areas 300 in the face area 200 (S205). After searching the eye area candidates in all the small areas, the true eye areas are detected from these eye area candidates (S206). The placement data stored in the memory 14 is used for detecting the true eye region. As described above, the arrangement data includes data such as “two eyebrows and two eyes are present” and “eyes are below the eyebrows”. Therefore, a pair of small area A and eye area 1 that have a pair of eyebrow areas as eye area candidates
When there is a small area B having a pair of eye areas as eye area candidates, the y coordinate of the small area B is lower than the y coordinate of the small area A, and thus the small “B” is in the arrangement data. The eye area publication of the area B is determined to be a true eye area. Further, even if there are two eye region candidates, the first and second eye region candidates, in a certain small region C, if the difference in shade between these first and second eye region candidates is large, "eyebrows, eye In both cases, the difference between the gray values on the left and right is small. " In this way, only the true eye region 500 is extracted from the plurality of eye region candidates as shown in FIG. 6 (b). The extracted face area and eye area are the target face template and the memory 1 as the target eye template, respectively.
Stored in 4.

The processing of S103 in FIG. 3 is performed as described above. After the target template is detected, the face image of the driver is captured again by the camera 10 and the grayscale normalization is performed (S104). Then, the eye region is tracked for this normalized face image (S105). That is, the face image area is extracted by the correlation calculation between the normalized face image and the target face template stored in the memory 14, and the correlation calculation is performed on this face image area using the target eye template stored in the memory 14. . The result of the correlation calculation is sequentially output to the ECU 16. The ECU 16 is sequentially input. A change in the driver's state is determined based on the correlation calculation result. This determination is made based on whether or not there is a change in the correlation value as in Japanese Patent Application No. 5-47597, and how long the change cycle is. When it is determined that the driver's state has changed (S107), it is determined that the driver is dozing or looking aside, and the alarm device is activated to call the driver's attention (S108).

As described above, in the present embodiment, a plurality of eye area candidates are extracted from the face image of the driver by performing the correlation calculation using the standard eye template, and a plurality of eye area candidates and eye arrangement information are used to obtain a plurality of eye area candidates. Since the target eye template is created by extracting the true eye area from the eye area candidates, the target eye template can be reliably created from the obtained face image, and the eye condition can be monitored.

Second Embodiment In the above-described first embodiment, the eye area is extracted from the face image extracted using the information on the arrangement of the face elements.
In this embodiment, the eye area is not directly extracted from the face image, but the eye vicinity area including eyebrows and eyes is first extracted from the face image, and the eye area is extracted from the eye vicinity area.

The overall configuration of this embodiment is the first shown in FIG.
Similar to the embodiment, the face image of the driver captured by the camera 10 is supplied to the image processing device 12. The image processing apparatus 12 creates the target template using the standard template stored in the memory 14 in advance. In this embodiment, the standard template stored in the memory 14 in advance is a standard face template, a standard eye vicinity template, and a standard eye template. Three templates of the template are stored,
A target face template, a target eye vicinity template, and a target eye template are sequentially created using these three templates. The created three target templates are stored in the memory 14. Correlation calculation is performed on the driver's face image sequentially input from the camera 10 using these target templates, and the correlation calculation result of the eye region is sequentially output to the ECU 16 to detect the abnormal state of the driver. It

FIG. 7 shows a processing flowchart in this embodiment. First, similar to the first embodiment described above, the target face area is searched from the driver's face image (S30).
1). This search is performed by a correlation calculation between the standard face template stored in advance in the memory 14 and the input face image. After the face area is extracted, a search for a target eye vicinity area is performed from this face area (S302). The search for the eye vicinity area is also performed by the correlation calculation between the extracted face area and the standard eye vicinity template stored in advance in the memory 14. The standard eye vicinity template is a template having a size including both the eyebrows and eyes of the driver, and an example thereof is shown in FIG. In FIG. 8, a face area 200 is set in the face image 100, and an eye vicinity area 400 is further set in the face area 200. This eye vicinity region is a range that includes both the eyebrows and eyes of the driver, and there is no region in the driver's face region that has the same brightness change as this eye vicinity region. Therefore, the eye vicinity region 400 is uniquely determined from the brightness change. Note that in FIG. 8, the eye area that is the final extraction target is indicated by reference numeral 500.

After the eye vicinity area is extracted, the target eye area is searched from the next extracted eye vicinity area (S30).
3). This search is performed by a correlation calculation between the extracted image in the eye vicinity region 400 and the standard eye template. However, similar to the first embodiment described above, the eyebrows and the eyes may have similar brightness changes. The result of the correlation calculation may be extracted as a pair of eyebrows and a pair of eye area candidates.
Therefore, also in this case, the data regarding the arrangement of the face elements stored in the memory 14 in advance is used, and only the eye region is extracted by the arrangement data such as "the eyes are below the eyebrows". FIG. 9 shows the relationship between the eye vicinity area 400 and the eye area 500, and the eye area 500 is extracted by performing the correlation calculation on the eye vicinity area 400 including the eyebrows and the eyes.

In this way, three target templates of the target driver's target face template, target eye neighborhood template, and target eye template are created, and stored in the memory 1
Stored in 4. Then, the correlation calculation with the input face image is performed using the target template created in this way, and the state to the eye region is monitored in the same manner as in the first embodiment described above, and it is determined that the state is abnormal. In this case, activate the alarm device to call attention to the driver.

As described above, in this embodiment, since the eye region is extracted by performing the processing in three stages of the face region, the eye vicinity region, and the eye region, as compared with the case where the eye region is directly extracted from the face region, It is possible to more reliably extract only the eye region.

[0027]

As described above, according to the driving condition detecting apparatus of the present invention, the driving condition of the driver can be grasped by surely extracting and monitoring the driver's eye region.

[Brief description of drawings]

FIG. 1 is a configuration block diagram of an embodiment of the present invention.

FIG. 2 is a processing block diagram in the first embodiment of the present invention.

FIG. 3 is a processing flowchart of the first embodiment of the present invention.

FIG. 4 is a processing flowchart of the first embodiment of the present invention.

FIG. 5 is an explanatory diagram of a small area according to the first embodiment of this invention.

FIG. 6 is an explanatory diagram of a small area and an eye area according to the first embodiment of this invention.

FIG. 7 is a processing flowchart of the second embodiment of the present invention.

FIG. 8 is an explanatory diagram of an eye vicinity area according to a second embodiment of the present invention.

FIG. 9 is an explanatory diagram of an eye vicinity region and an eye region according to the second embodiment of the present invention.

[Explanation of symbols]

 10 Camera 12 Image Processing Device 14 Memory 16 ECU

Claims (2)

[Claims] 1. A driving state detection device for detecting a driving state of a driver by processing a face image of the vehicle driver, the image pickup means for picking up an image of the vehicle driver, and the density of the obtained image. Normalizing means for performing normalization, and for the normalized image, a correlation calculation is performed using a standard template in which the face and eye regions are specified in advance, and a target face template and target eye template for the driver are created. A target template creation means, and a detection means for performing a correlation operation on the normalized image using the target template, and detecting a driving state of the driver based on a change in the correlation value, the target template The creating means is a face area extracting means for extracting a face area image of a driver using the standard template for the normalized image, and the standard area for the extracted face area. Region candidate extraction means for extracting a driver's eye region candidate using a template, and an eye region extraction for extracting the driver's eye region from the extracted eye region candidate using information on the arrangement of the driver's face elements An operating condition detection device comprising: 2. A driving state detection device for detecting a driving state of a driver by processing a face image of the vehicle driver, the image pickup means for photographing the vehicle driver, and the normalization of gradation of the obtained image. And a normalization means for performing a correlation operation using a standard template in which the face area, the eye vicinity area, and the eye area are specified in advance for the normalized image, and the target face template for the driver and the target eye vicinity template And a target template creating means for creating a target eye template, and the target image template, the target eye neighborhood template, and the target eye template are sequentially used for the normalized image to perform a correlation calculation, and the operation is performed based on a change in the correlation value. A driving state detection device comprising: a detection unit that detects a driving state of a person.