JP3405793B2 - Image type pedestrian detection device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image type pedestrian detection device for detecting a pedestrian crossing through the day and night by using image processing technology.

[0002]

2. Description of the Related Art Road traffic in Japan continues to increase and the number of traffic accident fatalities exceeds 10,000 per year. Many of these cases include fatal accidents of pedestrians, and in particular, there are many cases of fatal accidents of pedestrians due to carelessness of dryd, and countermeasures are desired.

As one of such measures, a vehicle manufacturer has been researching a system in which a front obstacle detection sensor is mounted on the vehicle itself and an emergency brake is applied when it is dangerous, and it is possible that it will be realized in the near future. high. Especially, it is important to detect a pedestrian crossing and warn the driver. However, the sensing area of such a vehicle-mounted sensor is limited to the front of the vehicle or the like, and is not sufficient for detecting a pedestrian crossing. Therefore, it is necessary to detect an object that cannot be sensed by the vehicle alone by a road sensor and warn the driver to prevent an accident.

For use in the above-mentioned applications, as a pedestrian detection means for remotely sensing a pedestrian crossing, a) ultrasonic type, b) laser beam type, c) visible light image type, and d) thermal image type. Can be considered. However, in the ultrasonic method of b), the aerial ultrasonic wave has a limit of about 10 m reflection, and a rectangular area such as a pedestrian crossing cannot be sensed by one device. The laser beam method of (b) is for detecting that the laser beam is blocked by a pedestrian crossing, but it is difficult to distinguish it from a car, and it is impractical because the transmitter and receiver must be installed on the sidewalk. .

Further, the visible light image type of (c) requires illumination at night and is difficult to apply by itself. The thermal image formula of d) is based on the wavelength of 3 to 5 μm in the mid-infrared region or 8 to 13 μm.
Its usefulness is well known because it obtains infrared images corresponding to infrared rays in the far infrared region, but it is expensive to replace a normal visible light camera and use it alone to detect a pedestrian crossing. Infrared device is -200 ℃ by adding equipment
It has to be cooled back and forth, and there is no hope of commercializing or popularizing it in terms of price.

[0006]

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned circumstances, and solves the problems of the above-described conventional detection devices, is used for day and night, is inexpensive, and is practical. It aims at proposing a high image quality pedestrian detection device.

[0007]

In the present invention for solving the above problems, a thermal image in which an image type pedestrian detection device is added with a visible light video camera and a Peltier electronic cooling element such as a three-stage Peltier electronic cooling element is added. A sensor camera and an image processing unit that processes the output of the visible light video camera in the daytime and processes the output of the thermal image sensor camera in the nighttime, and the image processing unit is the visible light video camera or the visible light video camera. The image signal from the thermal image sensor camera is processed and binarized into an image by the pattern correlation between the road luminance pattern and the pedestrian luminance pattern and sequentially stored, and the latest frame image binarized data and the previously stored binarized image are stored. By taking the difference of data, the imaging area is quaternized into four values of no object, object present, object retreat, and object approach for each pixel, and object present, object retreat, object advance The movement vector of the object is calculated from the distribution of the pixels corresponding to each input, and the pedestrian on the pedestrian crossing is identified based on the comparison between the direction of the movement vector and the crossing direction ,
Or one of the direction of movement vector and the direction of pedestrian crossing
Calculate the angle between the reference line that matches the direction and this angle
A person whose degree is 0 or 180 degrees is identified as a pedestrian .

[0008]

In the image type pedestrian detection apparatus of the present invention, in the daytime, the image processing unit, based on the video signal corresponding to the imaging range including the pedestrian crossing obtained from the visible light video camera,
Image is binarized by the pattern correlation between the road luminance pattern and the pedestrian luminance pattern, binary images are sequentially stored, and an image is obtained by taking the difference between the latest frame image binarized data and the previously stored image binarized data. Area without objects,
There is an object, the object retreats, and the object entrance is quaternized into four values for each pixel, and the movement vector of the object is calculated from the distribution of pixels corresponding to the object in the image, the object retreat, and the object entrance, and the movement vector Identify pedestrians on a pedestrian crossing based on comparison of direction and crossing direction , or cross direction and crossing movement vector
Angle formed by the reference line that matches one direction of the sidewalk
Calculate the degree and walk if this angle is 0 or 180 degrees.
Identified as ascetic.

Then, at dusk or at night, the same image processing unit, based on the video signal representing the temperature distribution corresponding to the imaging range including the pedestrian crossing obtained from the thermal image sensor camera, is exactly the same as in the daytime. The image is binarized according to the pattern correlation between the road luminance (temperature) pattern and the pedestrian luminance (temperature) pattern, the binary images are sequentially stored, and the latest frame image binarized data and the previously stored image binarized data are stored. By taking the difference, there is no object in the imaging area, there is an object,
Object retreat and object approach 4 values are converted into 4 values for each pixel, there is an object in the image, the object movement vector is calculated from the distribution of pixels corresponding to the object retreat and object entrance, and the direction of the movement vector and the crossing Identify pedestrians on a pedestrian crossing based on direction comparison , or movement vector direction and pedestrian crossing direction
Calculates the angle between the reference line that matches one direction
However, if the angle is 0 or 180 degrees, it is a pedestrian.
To identify Te. At the time of operation of the thermal image sensor camera, an additional inexpensive 3-stage Peltier thermoelectric cooler,
Cool the thermal image sensor to approximately -60 ° C, which is sufficient for device operation. Therefore, the image type pedestrian detection device of the present invention can be used day and night and is inexpensive.

[0010]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The basic idea of the present invention is to process a video signal corresponding to the brightness or temperature of a range including a pedestrian crossing to process a brightness pattern (temperature pattern) corresponding to a road and a brightness pattern (temperature pattern) corresponding to a pedestrian. Image binarization by pattern correlation with the temperature pattern), and by taking the difference between the latest frame image binarization data and the previously stored image binarization data, the imaging area is no object, there is an object, there is an object retreat, The four values of the object entry are converted into four values for each pixel, the object movement vector is calculated from the distribution of pixels corresponding to the object exit, the object retreat, and the object entry, and the pedestrian on the pedestrian crossing based on the direction of the movement vector. To identify.

In combination with the above-mentioned technique, a pedestrian crossing is detected by image processing of a video signal from a visible light video camera in the daytime, and a thermal image array (12) is used at dusk and at night.
8x128 element) and add Peltier electronic element-
The video signal from the thermal image sensor camera cooled to about 60 ° C. is used to detect a pedestrian crossing by substantially the same image processing as during the daytime, so that the camera operates regardless of day or night.

It should be noted that the thermal image sensor camera is equipped with a Peltier electronic element, paying attention to the fact that the road surface temperature is low at night and can be easily distinguished from the human body (36.5 ° C.) by the S / N ratio. It can be realized by Peltier 3-stage cooling using Peltier electronic elements, and by operating it by cooling it to around -60 ℃, it is possible to use it at night much more cheaply than before and put it into practical use. It is possible.

Hereinafter, the present invention will be described in detail based on embodiments with reference to the accompanying drawings. FIG. 1 is a block diagram showing an embodiment of the image type pedestrian detection device of the present application. The image type pedestrian detection apparatus of FIG. 1 includes a visible light video camera CA that captures an imaging range including a pedestrian crossing on a road, a thermal image sensor camera PCA that captures the same imaging range, and both cameras CA.
And an image processing unit 100 which receives signals from the PCA and selectively performs image processing on these video signals. Hereinafter, the configuration of each unit will be described in order.

The visible light camera CA is a video camera using a CCD element which has been generally used in this field. This video camera is, for example, 512 × 512
Of the detection area AR including the pedestrian crossing of the road from above the road (refer to the image example of FIG. 2A), and the NTSC video signal is converted into the luminance data conversion unit 20 of the image processing unit 100. Send to.

On the other hand, thermal image line sensor camera PCA
Is a lens 1 31 that is properly housed and fixed in the housing, and this lens 1
Three-stage Peltier thermoelectric cooler placed near the focal point of 31 1 33
Of the thermal image array element 1 32, a readout circuit 1 3 4 for reading out photoelectric conversion results of the thermal image array element 1 32, and an amplifier 1 35. Thermal image array element 1 32
In this embodiment, an electric signal corresponding to the amount of light is obtained in response to light rays in the infrared region. In the embodiment, 128 × 128 pixels are arranged in a matrix.

[0016] Thus the thermal imaging array element 1 32 is not attached is 3-stage Peltier thermoelectric cooler 1 33 to remove thermal noise, electronically cooled to near -60 ° C during operation of the embodiment device . According to the present invention, the cost of this portion is significantly reduced by performing the cooling by the three-stage Peltier electronic cooling element instead of the cooling device which has been generally used conventionally.

In the thermal image sensor camera PCA, a voltage corresponding to the amount of heat received by successive pixels (thus, the object temperature of the pixel-corresponding portion) is sequentially applied in accordance with the synchronization signal input from the outside to the readout circuit 1 3 4. Output. In the embodiment, this output (hereinafter referred to as a temperature conversion signal) is equivalent to a signal whose scanning line corresponds to a normal NTSC video signal. That is, as the read clock of the thermal image sensor camera PCA, 128 pixels in the horizontal direction are read in one scanning line time (63.5 μsec) of the NTSC signal. The vertical scanning is selected so as to be sequentially scanned.

Since the screen of the thermal image sensor camera PCA is compressed in the vertical direction as compared with the 512 × 512 CCD video camera, adjustments such as changing measurement points are made. After adjustment, the same image processing can be applied to both cameras.

The image processing unit 100 includes a brightness data conversion unit 20,
It is composed of an arithmetic processing unit 30 and a control unit 40. The brightness data conversion unit 20 is the NTSC from the visible light camera CA.
A video signal switching circuit 21 to which a video signal and a signal from the thermal image sensor camera PCA are input, a video amplifier 22 following the video signal switching circuit 21 for amplifying a transmitted video signal to an appropriate level, and a video signal It has a clamp level fixing circuit 23 for reproducing a DC level and a video A / D conversion circuit 24. The output of the video A / D conversion circuit 24 is an arithmetic processing unit.
Connected to 30 and input to dual port ram 33.

Further, the luminance data conversion section 20 has a sync signal extraction circuit 25 for obtaining a horizontal sync signal and a vertical sync signal from a video signal (output of the video amplifier 23), and a current sync line extraction circuit 25 based on both sync signals. A horizontal address counter 26A and a vertical address counter 26B for digitizing the scanning position to obtain the corresponding horizontal and vertical coordinates. These outputs and the microprocessor 41 sequentially specify the measurement address memory.
It has a digital comparator 27 which compares the coordinates output via 28 and generates a capture signal at the moment of coincidence, and the video A / D conversion circuit 25 responds to the generation of the capture signal by the video A / D conversion circuit 25. The brightness at the corresponding moment is digitized and output as brightness data.

The luminance data converter 20 further includes a thermal image camera read signal generation circuit 29, and generates a sync and read clock signal based on the horizontal sync signal and the vertical sync signal from the sync signal extraction circuit described above. The thermal image sensor camera P
The signal is output to the CA read circuit 1 34. Further, a signal is also supplied from the synchronizing signal extracting circuit 25 to the video signal switching circuit 21, and as described later, the thermal image sensor camera P
The signal from the CA is mixed with the video signal of the visible light camera CA and synthesized.

The arithmetic processing unit 30 is a circuit for successively receiving the luminance data from the video A / D conversion circuit 24, averaging a plurality of luminance data, and calculating the correlation, and requires high speed. For dual port ram 33,34
Also, a signal processor 31 capable of high-speed calculation is used, and a storage unit (ROM 32, RAM 3
It also has 5).

The control unit 40 controls the above-mentioned units and
A part that receives data from the arithmetic processing unit 30 and further performs arithmetic processing on the data, detects the presence of a pedestrian or a pedestrian or extracts the pedestrian and outputs the pedestrian to a subsequent device. It comprises a ROM 42 storing a program, a RAM 43 for storing data, and an I / O circuit 44 for outputting. The I / O circuit 44 has an input switch 47 and an output display 48.
Are connected. In addition, the image parameter setting tool 49 for adjustment when installing the two cameras described above is also I / O.
It is adapted to be connected to the O circuit 44.

In addition, the apparatus is further provided with a D / A conversion circuit 46 and the like which sends out a signal for increasing the sensitivity when the level of the video signal from the visible light camera CA is low and inappropriate for processing. Has been done.

Next, the operation of the above-described embodiment apparatus will be described. FIG. 2A schematically shows an image example of an image of an area including a pedestrian crossing. The measurement area of the pedestrian crossing is set at the starting point A of the measurement line La at the end of the pedestrian crossing in FIG.
s and the end point Ap, and the coordinates of the start point Bs and the end point Bp of the measurement line Lb at the beginning of the pedestrian crossing are set using the image parameter setting tool 49 connected to the image type pedestrian detection device and stored in the image processing unit 100. .

In dusk and at night, an image equivalent to that in the daytime can be obtained by the thermal image sensor camera PCA.
However, when a 128 × 128 thermal image array is used, the screen is 1 / vertical in the vertical direction as compared with a 512 × 512 CCD camera.
Since it is compressed to 2, it is necessary to set the measurement area separately from daytime, and it is necessary to switch and use it during dusk and night. 2B is a diagram showing the luminance value obtained along one of the measurement lines in FIG.

In the image processing unit 100, first, the video signal is processed to binarize the image in the measurement area for each pixel in the measurement area every m frames (for example, m = 2). That is, for all measurement lines, pixels with high brightness level (corresponding to temperature in the case of a thermal image sensor camera),
That is, the pixel (measurement point) where a vehicle or a person exists is 1,
The measurement points that do not exist are set to 0. Regarding this image binarization method, Japanese Patent Application No. 63-322940, Japanese Patent Application No. 2-
318992, Japanese Patent Application No. 3-043011 and the like.

FIG. 3 shows one of the measurement lines (for example, FIG. 2).
The state of image binarization along La of (a) is illustrated. Subsequently, a road luminance pattern (reference pattern) shown in FIG. 3B and a correlated luminance pattern (FIG. 3C) with the current luminance pattern are obtained and binarized by comparison with the detection level Ts, A binarization result as shown in (3) of FIG. 3 is obtained.

In the image binarized data, a defect of "0" actually occurs even at a measurement point where a vehicle or a person exists. Therefore, it is advisable to perform planar spatial smoothing to remove the "0" defect. FIG. 3E shows the result of binarizing one screen by binarizing each measurement line. The binarized data thus obtained is stored in the image processing unit.

If an image DSP (digital signal processor) is used, 400 measurement points are taken for 5 ms.
Since it is executed in ec, if the surface image is processed by 100 measurement lines at 1 measurement line and 100 measurement points, it can be executed at (100 × 100) / 400 = 25 msec. Image 2 at m = 2 frame (66 msec) intervals
It is quite possible to obtain the digitized data over the entire crosswalk.

Next, the image binarized data group thus obtained is used for quaternarization. FIG. 4 shows this state. Image binarized data of the current frame ((e) in Fig. 4)
And for each corresponding pixel (measuring point) of the image binarized data (for example, 2 frames before) of the image binarized data ((f) in FIG. 4).

In the case of (0-0), "0" (m
Frame last time and this time without vehicle or people), (1-1)
The case of "2" (there is a vehicle or a person in the previous m frames and this time), and the case of (1-0) is "-1" (there is a vehicle or a person before m frames, and the current frame is the vehicle Or if there is no person, that is, the vehicle retreats), in the case of (0-1), "+
The image is binarized by dividing it into four cases: 1 "(no vehicle or person before m frame, and vehicle or person at this frame. That is, vehicle approach).
FIG. 4 is a schematic diagram showing an example of this result (symbol 0 is not entered for measurement points with a value of “0”).

Next, the four-valued data is processed, and the data is
A portion that continues from "-1" to "2" and "+1" is extracted and recognized as a movement vector. From the coordinates (X1, Y1) of the start point (value: “−1”) and the coordinates (X2, Y2) of the end point (value: “+1”) of each movement vector, θ is set as a value indicating the direction of the movement vector. Ask for. θ = tan ■ -1 (Y2-Y1 / X2-X1) θ is the reference line that matches one direction of the pedestrian crossing in the screen (matched with the horizontal direction of the screen), and the movement vector Is the angle formed by.

After the direction θ of the movement vector is obtained in this way, a vehicle whose movement vector direction is orthogonal to the pedestrian crossing is removed as a vehicle. And, the direction θ of the movement vector is parallel to the crosswalk, that is, the angle θ = 0 degree or 180 degrees.
Degrees are detected as pedestrians. As a result, only crossing pedestrians can be accurately detected.

At night, a video signal of a level corresponding to the temperature of the measurement area is obtained from the thermal image sensor camera. In this case, the vertical and horizontal synchronizing signals of the NTSC video signal from the visible light video camera are used to switch only a part to the video signal of the thermal image sensor camera by the video signal switching circuit (video switch) 21.

In this way, when the 128 × 128 thermal image array is used, the screen is 512 × 5 as described above.
Since the image is compressed compared to the 12 CCD cameras, the setting of the measurement area is switched to an appropriate one that corresponds to twilight and night. As shown in FIG. 5A, the thermal image sensor has 128 × 128.
The timing of reading when the array is used is shown in the pattern of the compressed image in FIG.

Except for processing the video signal obtained as described above, the image processing itself for detection is substantially the same as the processing during the daytime described above, that is, image binarization → image quaternization →
Main image processing such as movement vector detection is performed in the same process as described above.

As described above, according to the present invention,
The thermal image sensor camera is used in combination with the visible light video camera, and the three-stage electronic Peltier cooling (-60
C)), it is possible to provide an inexpensive device capable of detecting a pedestrian crossing from day to night.

Since detecting a pedestrian crossing and appropriately controlling the relevant parts such as a traffic light and an alarm is life-threatening, the detection device part for this purpose has various environmental conditions (rain, fog, etc.). ), The detection probability must always be highly accurate. The device of the present invention is
These requirements are sufficiently satisfied. Also, in terms of cost, it has a structure that is sufficiently inexpensive to support the promotion of diffusion.

[0040]

As described above, the vehicle detection device of the first invention of the present application processes the output of the visible light video camera, the thermal image sensor camera to which the Peltier electronic cooling element is added, and the visible light video camera during the daytime. At night, it is configured to include an image processing unit that processes the output of the thermal image sensor camera,
The image processing unit processes a video signal from the visible light video camera or the thermal image sensor camera, binarizes the image based on a pattern correlation between a road luminance pattern and a pedestrian luminance pattern, and sequentially stores the binarized frame image binary. By taking the difference between the binarized data and the previously stored image binarized data, the imaging area is binarized into four values of no object, object present, object retreat, and object entrance for each image, and there is an object,
The moving vector of the object is calculated from the distribution of pixels corresponding to the object retreating and the object entering respectively, and the pedestrian on the pedestrian crossing is identified based on the direction of the moving vector. The image-type pedestrian detection device has a possible and inexpensive structure and is highly practical and can be expected to be widely used.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of an image type pedestrian detection device of the present invention.

FIG. 2 is a diagram (b ′) showing an image example (a) in which an area including a pedestrian crossing is imaged and a luminance value obtained along one of measurement lines according to the present invention.

FIG. 3 is a diagram ((b), (c), (d), (e ')) illustrating an example of binarization processing in the device of the present invention.

FIG. 4 is a diagram ((e), (e), (e)) illustrating an example of the four-valued processing in the device of the present invention.

FIG. 5 is a timing chart (A) showing a read timing of the thermal image sensor in the apparatus of the embodiment, and an example of a compressed image obtained by the thermal image sensor.

[Explanation of symbols]

CA: Visible light video camera, PCA ... Thermal image sensor camera, 33 ... Peltier thermoelectric cooler, 100 ... Image processing unit.

Claims (2)

(57) [Claims] 1. A thermal image sensor camera (PCA) having a visible light video camera (CA) and a Peltier thermoelectric cooler (133).
And an image processing unit (100) that processes the output of the visible light video camera (CA) during the daytime and the output of the thermal image sensor camera (PCA) at night, and the image processing unit (100 ) Is the visible light video camera (CA)
Alternatively, the video signal from the thermal image sensor camera (PCA) is processed and image binarized by the pattern correlation of the road luminance pattern and the pedestrian luminance pattern and sequentially stored, and the latest frame image binarized data and the previously stored By taking the difference of the image binarized data, the imaging area is quaternized into four values of no object, object present, object retreat, and object approach for each pixel, and there are object, object retreat, and object approach respectively. The movement vector of the object is calculated from the pixel distribution, and the angle between the movement vector and the reference line that matches one direction of the pedestrian crossing is calculated. This angle is 0 degree or 180 degrees.
An image-type pedestrian detection device characterized by identifying a degree of pedestrian as a pedestrian. 2. A thermal image sensor camera (PC) equipped with a visible light video camera (CA) and a three-stage Peltier thermoelectric cooler (133).
A) and an image processing unit (100) that processes the output of the visible light video camera (CA) in the daytime and the output of the thermal image sensor camera (PCA) in the nighttime. (100) is the visible light video camera (CA)
Alternatively, the video signal from the thermal image sensor camera (PCA) is processed and image binarized by the pattern correlation of the road luminance pattern and the pedestrian luminance pattern and sequentially stored, and the latest frame image binarized data and the previously stored By taking the difference of the image binarized data, the imaging area is quaternized into four values of no object, object present, object retreat, and object approach for each pixel, and there are object, object retreat, and object approach respectively. The movement vector of the object is calculated from the pixel distribution, and the angle between the movement vector and the reference line that matches one direction of the pedestrian crossing is calculated. This angle is 0 degree or 180 degrees.
An image-type pedestrian detection device characterized by identifying a degree of pedestrian as a pedestrian.