JP2009301494A - Image processing unit and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing unit and an image processing method, capable of effectively eliminating a shadow portion of a moving body imaged on an imaged image and correctly detecting the moving body type-by-type. <P>SOLUTION: Between an image data acquired from a first imaging unit imaging in a far infrared wavelength band and an image data substantially simultaneously acquired from a second imaging unit imaging in either a visible light wavelength band or a near infrared wavelength band, each difference is obtained from each background difference etc., so that a candidate of the moving body area having the moving body imaged thereon is extracted. An overlapped area of the difference obtained by composing each difference is decided to be a genuine moving body area. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus for detecting a moving body that appears in a captured image captured by an imaging apparatus. In particular, the present invention relates to an image processing apparatus and an image processing method that can effectively remove a shadow portion of a moving body that appears in a captured image and correctly detect the moving body by type.

  Conventionally, various traffic control systems for preventing traffic accidents at intersections such as T-shaped roads and crossroads with poor visibility have been proposed. In particular, a system has been proposed in which an imaging device that captures road conditions in the visible light wavelength band from above is installed on the roadside in the vicinity of the intersection, and the driver can be alerted and alerted by transmitting the captured image to the automobile. (For example, refer to Patent Document 1).

In this case, preprocessing for detecting a car from a captured image may be performed (see, for example, Patent Document 2). In this case, by using techniques such as “background difference”, “edge extraction”, “time difference (frame difference)”, and “optical flow” as preprocessing, an area corresponding to the vehicle in the captured image is determined. Yes.
JP 2003-109199 A JP 2008-46761 A

  In order to reduce various traffic accidents such as a collision between a car and a bicycle, a collision between a car and a person as well as a collision between cars, it is expected to realize a more advanced safe driving support system. Specifically, information on the presence and position of other bicycles, people, or fallen objects at the intersection is transmitted to the vehicle that is about to enter the intersection. And the control for avoiding an accident is automatically performed based on the information received by the vehicle side. This is expected to prevent accidents and minimize damage. Therefore, in this case, it is necessary to detect the moving body after identifying not only the presence / absence of the moving body at the intersection but also the type of the moving body such as a bicycle, a person, or a falling object.

  In order to distinguish and detect the type of moving object from the captured image, the feature amount of each moving object can be detected after extracting the area corresponding to the moving object in the captured image using the background difference or the time difference. It is effective to determine whether or not. For example, the shape and size of the detection target in the captured image are set as the feature amount, and the moving object depends on whether or not the shape and size of the area corresponding to the moving object matches the set shape and size. Can be identified.

  However, in an imaging device using a visible light wavelength band used in a conventional system, when an area corresponding to a moving object is to be identified from a captured image in a daytime sunshine situation, the moving object is projected onto the background. There is a possibility that the type of the moving object is erroneously identified by the shadowed portion. This is because when calculating the difference by image processing, the shadow portion may be extracted as a part of the moving body, and the size of the moving body may be detected largely.

  In addition, it is difficult for an imaging apparatus using a visible light wavelength band to detect a moving body other than an automobile at night. Since vehicles such as automobiles and bicycles have light-emitting parts such as headlights and taillights, they can be detected using these, but people usually do not have light-emitting parts and can capture images at night when the ambient illuminance is low. Have difficulty. In addition, even if the bicycle has a light emitting part, it may run without a light. Therefore, it is difficult to detect a bicycle or a person at night using an imaging device using a visible light wavelength band.

  The present invention has been made in view of such circumstances, and an image processing apparatus and image processing capable of correctly detecting, by type, a moving body that appears in a captured image based on a captured image regardless of nighttime or daytime. It aims to provide a method.

  An image processing apparatus according to a first aspect of the present invention is an image processing apparatus that performs image processing for detecting a moving object based on a captured image. The first imaging unit that captures an image in a far-infrared wavelength band, and the first imaging unit A moving body in which a moving body is shown from a second imaging means for imaging the imaging range of the first imaging area in the visible light wavelength band or near-infrared wavelength band, and captured images simultaneously captured by the first and second imaging means. Extracting means for extracting region candidates, specifying means for specifying overlapping portions of the candidates extracted by the extracting means as moving object regions, and moving objects based on feature quantities of moving object regions specified by the specifying means And means for detecting.

  In the image processing apparatus according to the second invention, the first and second imaging means are configured to image a range including a substantially flat background surface, and are associated with a plurality of different positions in the captured image. First storage means for storing a distance in real space from the predetermined position on the background surface to the object when there is an area where the object is captured at each position, and imaging of the moving body area specified by the specifying means It is characterized by comprising means for obtaining a position in the image and distance identifying means for identifying a distance in real space from the predetermined position to the moving body based on a distance associated with the obtained position.

  An image processing apparatus according to a third aspect of the present invention stores a correspondence relationship between a type of a moving body, a distance from the predetermined position, and a vertical and horizontal size of an area that the moving body area should occupy in the captured image. A storage unit; and an identification unit that identifies the type of the moving body based on the distance to the moving body specified by the distance specifying unit and the vertical and horizontal sizes of the moving body area specified by the specifying unit. Features.

  In an image processing apparatus according to a fourth aspect of the present invention, the first and second imaging means are installed so as to look down on the intersection, and the first storage means is a high-level device from the road surface of the first and second imaging means. Based on the height and the depression angle, the position of the object region in the captured image and the distance from the predetermined position in the intersection in the real space to the object are stored in association with each other, and the second storage means The vertical and horizontal sizes that the moving body region should occupy in the captured image are stored in association with the distance, and the identification means is a position stored by the first and second storage means. The type of the moving body is identified based on the correspondence between the distance and the distance and the vertical and horizontal sizes.

  An image processing apparatus according to a fifth aspect of the present invention is a third imaging unit installed so as to capture an imaging range of the first and second imaging units from an angle different from that of the first and second imaging units; Simultaneously with the first and second imaging means, an extraction means for extracting a candidate for a moving body region from a captured image captured by the third imaging means, a moving body area specified by the specifying means, and the extracting means Is based on the difference between the position specified by the means for associating the candidate extracted from the captured image of the third imaging means, the means for specifying the position of the associated moving body region and the candidate in each captured image, and And a means for calculating a distance in real space from any one of the first to third imaging means to the moving body.

  The image processing apparatus according to a sixth aspect of the invention is characterized in that the first to third imaging means are attached to a vehicle, and the distance from the vehicle to the moving body is calculated.

  An image processing method according to a seventh aspect of the invention is a first imaging means for imaging in the far infrared wavelength band, and a second imaging means for imaging the imaging range of the first imaging means in the visible light wavelength band or near infrared wavelength band. An image processing method for detecting a moving body from a picked-up image picked up by the first imaging means, wherein each of the moving body regions in which the moving body is captured from the picked-up images picked up simultaneously by the first and second image pickup means. A candidate is extracted, an overlapping portion of each extracted candidate is specified as a moving body region, and a moving body is detected based on a feature amount of the specified moving body region.

In the first invention and the seventh invention, images are picked up by the first image pickup means and the second image pickup means that pick up an image of substantially the same image pickup range.
The shadow of the moving body causes a color difference or a luminance difference with the surroundings, but does not cause a temperature change due to the presence or absence of solar radiation because it moves with the moving body in a short time. Therefore, the area of the shadow part of the moving body is the moving body area extracted using the background difference or the time difference in the captured image captured by the second imaging unit that captures an image in the visible light wavelength band or the near infrared wavelength band. Included in the candidate. However, the picked-up image picked up by the first image pick-up means picked up in the far-infrared wavelength band is not included in the moving object region candidate because there is no temperature difference, that is, no luminance difference from other parts. Thereby, the shadow part of the moving body located outside the moving body area in which the moving body itself is shown is removed from the overlapping part.
On the other hand, a portion within the moving body region where the moving body itself is shown but having no temperature difference from the surroundings is included in the moving body region candidates in the captured image by the first imaging means that captures images in the far infrared wavelength band. I can't. Accordingly, there is a possibility that a region corresponding to the moving body itself in the first captured image may be extracted separately. In this case, in the image picked up by the second image pickup means, the region corresponding to the moving body itself is extracted as a series of regions due to the difference in color or brightness from the peripheral part. Even if a plurality of regions separated from the captured image captured by the first imaging unit are extracted, a portion overlapping with the region extracted as a series of regions from the captured image captured by the second imaging unit is one moving body It can be specified as a region.

  In the second invention, the first and second imaging means are adapted to image a range including a background surface such as a substantially flat ground, floor or road surface, and a moving body region in the captured image. And a correspondence between a predetermined position on the background surface and a distance in the real space from the moving body is stored in advance. Thereby, the distance in the real space from a predetermined position to a moving body can be specified easily.

  In the third invention, depending on the type of the moving body, the vertical and horizontal sizes of the area that the moving body area should occupy can be determined according to the distance from the first and second imaging means of the moving body. When the area specified as the mobile object area corresponds to the vertical and horizontal size conditions stored in advance for each type of mobile object, the mobile object is identified as the type corresponding to the corresponding condition. . Since shadows can be accurately removed, moving object regions can be accurately extracted, and the size of the region can be used as a feature value to accurately identify the type of moving object. It is.

  In the fourth invention, features such as the vertical and horizontal sizes of the moving body are accurately removed by removing shadow portions on the road surface of the moving body such as automobiles, bicycles, people or falling objects moving on the road of the intersection. The amount can be specified. This makes it possible to identify the type of a moving body such as a vehicle such as an automobile or a bicycle that has entered the intersection, a person, or a fallen object, and specify the position of the real space of the moving body at the intersection.

  In the fifth invention, the position in the captured image of the moving body region that is extracted from the captured images by the first and second imaging means and is specified by removing the shadow portion, and the third imaging means Based on the difference between the position of the moving body region candidate extracted from the picked-up image in the picked-up image, that is, the difference in the incident angle to the two image pickup means, the distance to the moving object is determined using the triangulation method. It is possible to calculate with high accuracy.

  In the sixth invention, the first to third imaging means are installed in a vehicle that is itself a moving body. Since it is possible to remove the part corresponding to the shadow on the road surface of the moving body such as other vehicles, people, or falling objects that are moving bodies for the own vehicle, the method of triangulation method from the vehicle to the moving body It is possible to calculate the distance with high accuracy.

  In the case of the present invention, it is possible to avoid the shadow portion being erroneously detected as a part of the moving body by using the first imaging means for imaging in the far infrared wavelength band. Further, even when a part of the surface of the moving body does not cause a temperature difference from the surroundings and does not cause a luminance difference in the image captured by the first imaging means, the second imaging means causes the visible light wavelength band or the near infrared wavelength band to be generated. The moving body region is accurately extracted by taking into account the region discriminated from the captured image. As a result, it is possible to correctly identify the moving object by type with high accuracy.

  Hereinafter, the present invention will be specifically described with reference to the drawings showing embodiments thereof.

  In the embodiment shown below, the image processing apparatus according to the present invention makes it possible to accurately detect a moving object such as an automobile, a bicycle, a person, or a fallen object on an intersection, and an accident occurs in an automobile entering the intersection. A case where the present invention is applied to a safe driving support system that executes traveling control for prevention will be described as an example.

(Embodiment 1)
FIG. 1 is a perspective view from above showing an example of an intersection where the safe driving support system according to Embodiment 1 is installed. Reference numeral 1 in FIG. 1 denotes an image processing apparatus configured integrally with two imaging apparatuses that capture an intersection. The image processing apparatus 1 is attached to a column installed on the road side and extending to the road side at a predetermined height. The lenses of the two image pickup devices that are configured integrally with the image processing device 1 are installed so as to overlook the intersection and the vicinity thereof, and the image processing device 1 and the image pickup device are far away from the lens side of the image pickup device. It is housed in a casing made of a material that transmits infrared rays and visible rays or near infrared rays. Reference numeral 2 in FIG. 1 denotes a roadside machine, which is attached to a roadside column like the image processing apparatus 1. The image processing apparatus 1 and the roadside machine 2 are connected by a communication line.

  The image processing apparatus 1 can transmit and receive data via wireless communication via the roadside device 2. For example, two cars are shown in FIG. The vehicle-mounted device of each automobile is configured to be able to perform wireless communication with the roadside device 2 by, for example, DSRC (Dedicated Short Range Communication) technology.

  In the safe driving support system according to the first embodiment, the image processing apparatus 1 performs the operation described below, thereby realizing traveling control in each automobile. The image processing apparatus 1 detects a moving body such as an automobile, a bicycle, a pedestrian, or a fallen object at the intersection from image data obtained by an imaging apparatus that captures an image of the intersection. Then, the image processing apparatus 1 transmits information on the detected moving body to each automobile via the roadside device 2. Thereby, in each automobile, it is possible to realize traveling control such as automatically operating a brake when the presence of a pedestrian is detected in the traveling direction based on information received via the roadside device 2. In addition, even when a collision accident occurs, such as operating a shock absorber, it is possible to realize safety control for reducing the impact caused by the collision.

  In order to realize such a safe driving support system, image processing performed by the image processing apparatus 1 based on image data and various information obtained by the image processing will be described below.

  FIG. 2 is a block diagram showing an internal configuration of the image processing device 1, the roadside device 2, and the first and second imaging devices 3 and 4 constituting the safe driving support system in the first embodiment. The image processing apparatus 1 includes a control unit 10 that controls each component using a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc., an EEPROM (Electrically Erasable and Programmable Read Only Memory), a flash memory, and the like. A storage unit 11 using a nonvolatile memory, an image memory 12 using a memory such as SRAM (Static Random Access Memory) and DRAM (Dynamic Random Access Memory), and the first and second imaging devices 3 and 4 A first image acquisition unit 13 and a second image acquisition unit 14 that receive image signals, and a communication unit 15 that realizes communication with the roadside device 2 are provided.

  The control unit 10 reads out and executes a control program stored in a built-in ROM (Read Only Memory) or the storage unit 11 to control each component unit and perform image processing described later. is there. The storage unit 11 may be built in the control unit 10. Various data generated by the processing of the control unit 10 is temporarily stored in a non-volatile memory provided separately or in a memory built in the control unit 10.

  The image memory 12 stores the image data acquired by the first and second image acquisition units 13 and 14. When the image data is acquired by the first and second image acquisition units 13 and 14, the control unit 10 stores data in the storage areas in the image memory 12 corresponding to the first and second image acquisition units 13 and 14, respectively. Instruct to write. Each process described below is performed on the image data stored in the image memory 12 or a copy thereof.

  The image memory 12 may be a partial storage area of a RAM (Random Access Memory) built in the control unit 10. The storage unit 11 or the image memory 12 may be installed outside the image processing apparatus 1 and connected to the image processing apparatus 1 so that the control unit 10 of the image processing apparatus 1 can read and write.

  The first image acquisition unit 13 receives an image signal output from the first imaging device 3, and the second image acquisition unit 14 receives an image signal output from the second imaging device 4. Then, the first image acquisition unit 13 and the second image acquisition unit 14 respectively extract image data from the received image signal and store it in the image memory 12. More specifically, the first and second image acquisition units 13 and 14 specify the luminance value or color difference of each pixel constituting the image from the received image signal according to the instruction from the control unit 10 and use it as image data. Write to the image memory 12.

  The communication unit 15 realizes transmission / reception of information via a communication line connecting the roadside unit 2. The control unit 10 transmits information obtained by the image processing to the roadside device 2 through the communication unit 15, and receives information from the roadside unit 2 through the communication unit 15.

  The roadside machine 2 includes a control unit 20 that controls each component using a CPU or MPU, a communication unit 21 that realizes communication with the image processing apparatus 1, and communication with an on-vehicle device mounted on each vehicle. And a wireless communication unit 22 that realizes the above wirelessly.

  The communication unit 21 can transmit and receive information in correspondence with the communication unit 15 of the image processing apparatus 1. When the communication unit 21 receives information transmitted from the image processing apparatus 1, the control unit 20 receives a notification from the communication unit 21 and receives it.

  The wireless communication unit 22 can perform wireless communication using a DSRC technique. The control unit 20 transmits information received from the image processing apparatus 1 by the communication unit 22 to the vehicle-mounted device of each automobile by the wireless communication unit 22. The wireless communication unit 22 may be configured not only to communicate with the vehicle-mounted device by the DSRC method but also to transmit information to a communication device in a traffic control center at a remote location.

  The first imaging device 3 is a far-infrared camera that includes an infrared lens and a far-infrared imaging device and performs imaging in the far-infrared wavelength band. The infrared lens is manufactured using zinc sulfide, chalcogen glass, germanium, zinc selenium, or the like as a raw material. As the far-infrared imaging device, an uncooled type using vanadium oxide (VOx), amorphous silicon, SOI (Silicon on Insulator) diode, thermopile or the like is used. The wavelength band that can be detected by the far-infrared imaging device is, for example, 8 μm to 12 μm. A filter may be provided to block light of other wavelengths.

  The far-infrared imaging device of the first imaging device 3 converts the temperature difference into a luminance difference and outputs it. When a shadow part projected from the moving body onto the road surface is included in the angle of view, the shadow part also moves in a short time, so that a temperature change hardly occurs and a luminance difference does not occur. Therefore, no difference appears in the brightness value with respect to other road surfaces in the shadow portion of the moving object in the image data acquired from the image signal output by the far-infrared imaging device.

  The second imaging device 4 is a camera that includes a visible light lens and a visible light imaging device, and performs imaging in the visible light wavelength band. As the visible light imaging element, a complementary metal oxide semiconductor (CMOS) and a charge coupled device image sensor (CCD) are used. The sensitivity characteristic of the visible light imaging element is, for example, 450 nm to 700 nm. A filter may be further provided to block light of other wavelengths. In addition, the 2nd imaging device 4 is good also as a structure provided with an infrared lens and a near-infrared imaging element so that it may have a sensitivity characteristic in a near-infrared (for example, 900 nm-1.2 micrometers) wavelength band.

  The visible light imaging element of the second imaging device 4 converts the intensity of light into a luminance value and outputs it. When a shadow portion projected from the moving body onto the road surface is included in the angle of view, of course, the shadow portion has a light intensity different from that of the surrounding road surface. Therefore, a luminance difference is generated in the shadow portion of the moving body in the image data acquired from the image signal output from the visible light imaging device as compared with the portion other than the shadow.

  The number of pixels of the far-infrared imaging device of the first imaging device 3 and the visible light imaging device of the second imaging device 4 are both 320 × 240, for example, and an image signal of an image generated at a rate of about 30 frames per second Is output. Although it is desirable that the imaging timings in the first and second imaging devices 3 and 4 are strictly synchronized, exact synchronization is not always required. For example, the time difference between the imaging timings of the first and second imaging devices 3 and 4 affects image processing, which will be described later. If the time difference can be specified or estimated, an error due to the time difference can be absorbed.

  The angle of view of the first imaging device 3 and the angle of view of the second imaging device 4 are configured to substantially match. That is, in the first and second imaging devices 3 and 4, the optical arrangement of the built-in lens group is configured so that the focal length is the same. In order to easily realize imaging at the same angle of view, the first and second imaging devices 3 and 4 use imaging elements having the same number of pixels and are arranged in parallel in the same direction, that is, toward the intersection.

  As described above, it is desirable that the angles of view of the first and second imaging devices 3 and 4 match for the processing described later, but it is difficult to match them completely. However, it is necessary that at least the angles of view overlap. When it is not possible to arrange them so as to be substantially coincident, the overlapping range of the image data acquired by the first and second image acquisition units 13 and 14 may be stored in the storage unit 11. Then, a correspondence relationship is stored so that pixels corresponding to each other within the overlapping range can be specified. Accordingly, the control unit 10 detects an area corresponding to the same subject within the overlapping range of the two image data acquired by the first and second image acquisition units 13 and 14 stored in the image memory 12. be able to.

  Next, details of the image processing performed on the image data by the control unit 10 of the image processing apparatus 1 configured as described above will be described.

  The control unit 10 detects a moving body such as an automobile, a bicycle, a person, or a fallen object from the image data acquired by the first and second image acquisition units 13 and 14. Specifically, the control unit 10 extracts a moving body region from the image data. Conventionally, there are various techniques as a moving body detection method based on image data. In the first embodiment, the background difference method is used. However, a frame difference method that obtains a difference from one or more previous frames of image data may be used, or a method called an optical flow may be used. Good.

  FIG. 3 is an explanatory diagram illustrating an example of the contents of background image data and size information for each type of moving body, which are stored in advance in the storage unit 11 of the image processing apparatus 1 according to the first embodiment. FIG. 3A shows an example of the content of background image data, and FIG. 3B shows an example of the content of size information for each type of moving object. It should be noted that far-infrared background image data and visible light background image data as shown in FIG. 3A are stored in the storage unit 11, respectively.

  As shown in FIG. 3A, the background image data includes a road surface that is a fixed object. Movement is performed by calculating the background difference between the image data acquired from the first imaging device 3 and the background image data, and the background difference between the image data acquired from the second imaging device 4 and the background image data, respectively. Body region candidates are extracted.

  Note that the imaging range is determined according to the installation position and installation height of the image processing apparatus 1 and the depression angle and field angle of the first and second imaging apparatuses 3 and 4. In the case of the first embodiment, the first and second imaging devices 3 and 4 are installed so as to look down on the intersection, so that they correspond to positions in the image data, in particular, positions in the vertical direction and positions at the intersection in the real space. There is a relationship. Each horizontal line represented by a broken line in FIG. 3A indicates the distance from the image processing apparatus 1 in the real space corresponding to the vertical position in the image data. For example, a part of the road surface shown on the lowermost broken line in FIG. 3A is located at a distance of 20 m from the position where the image processing apparatus 1 is installed. A part of the road surface shown on the broken line at the center is located at a distance of 50 m from the position where the image processing apparatus 1 is installed. A part of the road surface shown on the uppermost broken line is located at a distance of 100 m from the position where the image processing apparatus 1 is installed.

  The size of the moving body is determined to some extent depending on the type of moving body. For example, when the moving body is a normal passenger car and a small passenger car, they generally fit within a volume of 5.5 m × 2 m × 1.8 m in length (vehicle length) × width (vehicle width) × height (vehicle height). It should be. In addition, a person should fit within a volume of approximately 80 cm × 80 cm × 2 m (height) in length × width × height. When the moving body is imaged by the first and second imaging devices 3 and 4, the size of the moving body region in the image data is smaller as it is farther from the image processing device 1 and is larger as it is closer. The size of the moving body region is determined according to the distance from the first and second imaging devices 3 and 4 to the moving body and the angle of view of the first and second imaging devices 3 and 4.

  Therefore, the vertical and horizontal sizes to be occupied in the image data of the moving object region located at a predetermined distance from the image processing apparatus 1 are obtained for each type of moving object. Therefore, as shown in FIG. 3B, the vertical and horizontal sizes of the moving body region located at a predetermined distance from the image processing apparatus 1 are stored in the storage unit 11 for each type of moving body. For example, passenger cars generally have a vehicle width of 2 m or less and a vehicle height of 1.8 m or less. Therefore, when a passenger car traveling at a position 50 m from the image processing device 1 in the direction corresponding to the vertical direction of the image data is imaged by the first and second imaging devices 3 and 4, The size is, for example, within 30 pixels × 48 pixels in the vertical × horizontal direction. On the other hand, when a passenger car that is traveling 50 m from the image processing device 1 in the direction corresponding to the horizontal direction of the image data is imaged by the first and second imaging devices 3 and 4, the region that the region of the passenger car should occupy For example, the vertical size and the horizontal size are within 30 pixels × 64 pixels.

  In the first embodiment, as shown in FIG. 3B, the vertical and horizontal sizes that should be occupied by the moving object region are stored in advance in the storage unit 11 for each type of moving object. Thereby, the control part 10 can identify the kind of moving body reversely based on the vertical and horizontal size of a moving body area | region. For example, when the moving body region is at a position corresponding to a distance of 50 m from the image processing apparatus 1, the control unit 10 can identify the type from the vertical and horizontal sizes of the region as follows. For example, when the vertical size of the moving object region is 30 pixels or less, the horizontal size is 64 pixels or less, and the horizontal size is larger than 48 pixels, the control unit 10 selects the moving object. The direction corresponding to the horizontal direction of the image data can be distinguished from the traveling car. Similarly, when the vertical size of the moving object region is within 30 pixels and the horizontal size is within 15 pixels, the control unit 10 can identify the moving object as a person.

  When the position in the image data of the moving body region is a position different from a predetermined position corresponding to the size information stored in the storage unit 11 (a position at a distance of 50 m in FIG. 3B), 10 calculates the vertical and horizontal sizes that should be occupied by the area corresponding to the moving object according to the difference. The control unit 10 can similarly identify the type of the moving object based on the vertical and horizontal sizes in the image data of the moving object region and the calculated size. For example, when the moving body region is located at a position corresponding to a distance of 100 m from the image processing apparatus 1, the vertical and horizontal sizes should be within half because the distance is double from the predetermined distance of 50 m.

  The aspect ratio of the area corresponding to the moving body should be constant depending on the type of the moving body. Therefore, the type of the moving body may be identified based on the aspect ratio obtained from the vertical and horizontal sizes shown in the explanatory diagram of FIG.

  Thus, by storing the position in the image data and the position in the real space in association with each other, the type of the moving object can be identified based on the position in the image data of the moving object region. In addition, the position of the moving body in the real space can be specified. When the position of the moving body is specified, information on the moving body is transmitted to each automobile, so that the traveling control device of each automobile can receive the information and recognize the position of the moving body, thereby performing highly accurate traveling control. be able to.

  The control unit 10 of the image processing apparatus 1 performs the following process using the information stored in the storage unit 11 as described above. FIG. 4 is a flowchart showing an example of an image processing procedure performed by the control unit 10 of the image processing apparatus 1 according to the first embodiment.

  The control unit 10 outputs an instruction signal simultaneously to the first image acquisition unit 13 and the second image acquisition unit 14, so that an image based on the image signals output from the first imaging device 3 and the second imaging device 4 is obtained. Data is acquired simultaneously (step S1). In addition, the acquisition timing of the image data in step S1 is arbitrary. For example, it may be configured to be performed at regular intervals such as 1 second, or may be configured to be performed when the vehicle detects entry into an intersection, such as when the roadside device 2 receives information from the vehicle-mounted device.

  The control unit 10 includes the image data acquired by the first image acquisition unit 13 in step S <b> 1 from the image signal output from the first imaging device 3, and far-infrared background image data stored in the storage unit 11. Is calculated (step S2). The calculated difference data is temporarily stored in the image memory 12 or the control unit 10 built-in memory. Thereby, the area | region where the mobile body which is not reflected in background image data exists is extracted from the image data obtained from the 1st imaging device 3 imaged in a far-infrared wavelength band.

  Further, the control unit 10 acquires the image data acquired by the second image acquisition unit 14 in step S1 from the image signal output from the second imaging device 4, and the background image data for visible light stored in the storage unit 11. (Background difference) is calculated (step S3). The calculated difference data is temporarily stored in the image memory 12 or the control unit 10 built-in memory. Thereby, the area | region where the moving body which is not reflected in background image data exists from the image data obtained from the 2nd imaging device 4 imaged in a visible light wavelength band is identified.

  The control unit 10 calculates the difference data as follows, for example. For each pixel, the control unit 10 calculates a difference in luminance value between the background image data and the acquired image data, and distinguishes between a pixel whose difference is greater than or equal to a predetermined value and a pixel whose difference is less than the predetermined value. The control unit 10 calculates, as difference data, a two-dimensional array of 1-bit information indicating whether the difference is greater than or equal to a predetermined value for each pixel constituting the image data.

  More specifically, the control unit 10 performs an operation based on Expression 1 shown below for each pixel of the acquired image data.

However, f (x, y) in Expression 1 is 1-bit information of each pixel specified by the position x in the horizontal direction and the position y in the vertical direction constituting the acquired image data. The position x in the horizontal direction is the number of pixels from the left end of the image data, and y in the vertical direction is the number of pixels from the upper end of the image data. I _{x, y} is a luminance value for each pixel specified by _{x, y} in the acquired image data. I ′ _{x, y} is a luminance value for each pixel specified by _{x and y} in the background image data.

  Note that when the background difference is calculated as shown in Equation 1, it may be calculated based on the luminance value after performing edge extraction processing on the acquired image data and background image data, respectively.

  The control unit 10 combines the difference data of the far-infrared image data and the difference data of the visible light image data calculated in steps S2 and S3, respectively (step S4). Specifically, the control unit 10 performs a logical product of the difference data. That is, the control unit 10 performs a true logical product of whether or not the luminance value difference is equal to or greater than a predetermined value between pixels located at the same place of the two difference data.

  More specifically, the control unit 10 performs an operation based on Equation 2 shown below for the calculated difference data.

  h (x, y) = f (x, y) * g (x, y) (2)

  However, f (x, y) in Expression 2 is difference data of far-infrared image data obtained by Expression 1, and g (x, y) is difference data of visible light image data obtained by Expression 1. Further, “*” in Equation 2 represents a theoretical product.

  By this processing, only the pixel that is true that the difference in luminance value is equal to or greater than a predetermined value in any difference data is true. In other words, the moving object region is extracted from the image data in the far-infrared wavelength band, and only the region extracted as the moving object region from the image data in the visible light wavelength band remains as the moving object region. It should be noted that the 1-bit information of the pixel may be set to false unless the true pixel is true, that is, if the true pixel is not continuous. This is to remove pixels whose difference is equal to or greater than a predetermined value due to noise.

  The control unit 10 identifies the moving body region from the result of combining the difference data of the far-infrared image data and the difference data of the visible light image data obtained in step S4 (step S5). At this time, of course, a plurality of mobile object regions may be specified. Specifically, as a result of the logical product of the difference data, the control unit 10 combines pixels that are true continuously in the horizontal direction and the vertical direction of the image data to form a moving body region. In this case, the specified moving body region has an arbitrary shape. The control unit 10 finally sets a rectangle circumscribing the moving body area as the moving body area.

  In addition, in the specific process of step S5, the control part 10 performs the following process about the area | region extracted as several different area | regions by the logical product of difference data in step S4. Even when the images are separated into a plurality of different regions, some of the differences in the visible light image data are included in one region. In this case, the control unit 10 specifies a rectangle that comprehensively circumscribes a plurality of regions as corresponding to one moving body. This is because even if separated into a plurality of different regions, each region corresponds to a part of the same moving body.

  Next, the control part 10 identifies a moving body based on the moving body area | region specified by step S5 (step S6). As shown in FIG. 3B, the vertical and horizontal sizes that the moving object region should occupy are set for each type of moving object with respect to a predetermined position in the image data of the moving object region. Based on the size of the circumscribed rectangle of the moving object area identified in step S5, the position of the area in the image data, and the size information set as shown in FIG. Identify the type of mobile. As a result, the moving body is detected after its type is identified. If the size does not match the vertical and horizontal sizes indicated by any of the size information, processing is performed assuming that no moving object has been detected.

  In step S6, the control unit 10 identifies the type based on the size information as shown in FIG. However, in the present invention, the method for identifying the type of the moving body is not limited. For example, a template matching method for identifying the type of moving object by pattern matching with the template for each type of moving object may be used for the moving object region (circumscribed rectangle). In addition, pattern recognition may be performed using a machine learning method such as support vector machine or boosting. Further, the control unit 10 cuts out visible light image data in the region specified in step S5, correlates it with color image data serving as a model for each type of moving object, and evaluates similarity based on the level of correlation. . And you may make it identify the kind of moving body based on the height of similarity.

  In the present invention, it is possible to accurately identify the moving body region from which the shadow portion has been removed in step S5. Therefore, regardless of which identification method is used in step S6, it is possible to correctly identify the type of the moving object.

  Next, the control part 10 transmits the information of a moving body to the roadside machine 2 by the communication part 15 (step S7), and complete | finishes a process. Specifically, the control unit 10 transmits the distance in the real space from the image processing apparatus 1 specified from the type of the moving body identified in step S6 and the position of the moving body area in the image data as the moving body information. . When a plurality of mobile bodies can be detected, the mobile body information transmitted to the roadside device 2 is transmitted to the vehicle-mounted device of each automobile by the wireless communication unit 22 of the roadside device 2.

  Since the control unit 10 of the image processing apparatus 1 transmits information indicating the type of the moving body together by the above-described processing, the control for preventing an accident can be performed with high accuracy on the traveling control apparatus side of the automobile.

  Next, processing by the control unit 10 of the image processing apparatus 1 will be described with a specific example. FIG. 5 is an explanatory diagram illustrating a specific example of an image processing result performed by the control unit 10 of the image processing apparatus 1 according to the first embodiment. FIG. 5A schematically shows an example of the content of the image data acquired by the second image acquisition unit 14, that is, visible light image data. In addition, the image data shown to Fig.5 (a) shows the outline in black and white, and has shown the shadow part imaged in a visible light wavelength band in a ray area.

  FIG. 5B schematically shows difference data between the visible light image data shown in FIG. 5A and the background image data for visible light. FIG. 5C schematically shows difference data between far-infrared image data and background image data for far-infrared rays. FIG. 5D schematically shows the image data after combining the difference data calculated from the visible light image data and the difference data calculated from the far-infrared image data. In FIGS. 5A, 5B, and 5C, a dashed line indicates a line corresponding to a road contour in order to clarify the position in the image data.

  As shown in FIG. 5A, the visible light image data in the specific example shows two cars traveling at the intersection and a person walking at the intersection. Since the image data is acquired from the second imaging device 4 that captures an image in the visible light wavelength band, the image data clearly shows a shadow portion that has a difference in luminance value from other road surface portions.

  Accordingly, the control unit 10 calculates difference data from the background image data for visible light, so that the areas of the two automobiles and the walking are shown as shown in the three ray portions in FIG. The area of the person inside is extracted. When the circumscribed rectangle of each area is obtained as indicated by the broken line as it is, the difference data includes the shadow portion, and therefore the vertical and horizontal sizes increase by the size of the shadow portion. Therefore, when trying to identify the type of the moving body based on the circumscribed figure shown in FIG. 5B, the size and aspect ratio are different from the actual ones, so there is a high possibility that they are erroneously identified.

  On the other hand, in the far-infrared image data, the shadow portion of the moving body does not produce a luminance difference from other road surface portions. Also in this case, the control unit 10 calculates the difference data from the background image data for far-infrared rays, thereby extracting the areas of the two cars and the area of the person who is walking. However, as shown in FIG. 5C, none of the shadow areas are included. In FIG. 5C, the area corresponding to the car traveling on the left side is separated into two.

  Thus, no shadow portion remains in the far-infrared image data. This is because the shadow portion of the moving body having no temperature difference from the surroundings does not cause a difference in brightness from other road surfaces. Therefore, by calculating the logical product of the difference data in the visible light image data and synthesizing the difference data, the shadow portion can be removed as shown in FIG.

  In the far-infrared image data, since there is no luminance difference in a portion where there is no temperature difference, as shown in FIG. 5C, the moving body region of the same moving body is extracted as two separate regions. About this, when specifying the moving body region based on the far infrared difference data and the visible light difference data, the control unit 10 specifies the moving body region of the same moving body. Even if the far-infrared image data is extracted into two separate areas, both are included in the continuous true area in the difference data of the visible light image data, and are areas corresponding to different parts of the same moving object. Because there is. As a result, as indicated by the broken line in FIG. 5D, the areas of the two cars and the person are accurately identified by removing the shadow portion.

  In this way, the first imaging device 3 that captures an image in the far-infrared wavelength band and the second imaging device 4 that captures an image in the visible light wavelength band or the near-infrared wavelength band, but a captured image captured by both of them. The overlapping part of the region extracted from each is specified as the true moving body region. By using the first imaging device 3, it is possible to eliminate shadows and avoid false detections, and also to use the second imaging device 4 to move in consideration of information that is insufficient only with far infrared rays. The region corresponding to the body itself can be accurately specified. Thereby, it becomes possible to detect the moving body with high accuracy after identifying the type of the moving body, and an excellent effect can be obtained.

  In addition, since it uses the fact that the shadow portion of the moving body is inevitably removed by using the first imaging device 3 in the far-infrared wavelength band, whether or not the shadow portion is included is determined by whether or not there is sunlight. There is no need to make a judgment based on this. As described above, the image processing by the control unit 10 of the image processing apparatus 1 has an excellent effect in that a shadow portion can be effectively removed with a simple configuration.

(Modification)
The image processing apparatus 1 according to the first embodiment described above is configured to store the vertical and horizontal sizes that should be occupied by the moving body region for each type of moving body in the storage unit 11 and identify the type of moving body. However, you may make it memorize | store the magnitude | size of the moving body in real space for every kind of moving body in the memory | storage part 11 of the image processing apparatus 1. FIG. In this case, the control unit 10 calculates the position and size of the detected moving object in the real space from the size of the specified moving object region, treats the calculated size as the feature value of the moving object, and the feature value. The type of the moving object is identified based on the above.

  FIG. 6 is an explanatory diagram schematically illustrating the principle by which the control unit 10 of the image processing apparatus 1 according to the modification of the first embodiment calculates the size of the moving object in real space.

  FIG. 6A shows the relationship between the position of the image processing apparatus 1 where the first and second imaging devices 3 and 4 are installed and the position of a person in an intersection to be detected by the image processing apparatus 1. Yes. FIG. 6B schematically illustrates an example of content of image data captured by the first and second imaging devices 3 and 4 when the image processing apparatus 1 and the person are in the positional relationship illustrated in FIG. Is shown.

As illustrated in FIG. 6A, when the angle of view θ _{v of} the first and second imaging devices 3 and 4 is set, it can be specified that the person is at a distance D from the installation position of the image processing device 1. At this time, the size T of the person in the real space can be obtained by a relational expression as shown in the following Expression 3.

However, H _c in Expression 3 is the number of pixels in the vertical direction of the image data as shown in FIG. In the first embodiment, the number of pixels of the imaging elements of the first and second imaging devices 3 and 4 is 320 × 240 in the horizontal direction × vertical direction. H _o is the number of pixels indicating the size in the vertical direction (vertical) of the area of the person to be detected in the image data as shown in FIG.

By storing the relational expression of Expression 3 in the storage unit 11 in advance, it corresponds to the distance D from the installation position of the image processing apparatus 1 in the real space of the moving object and the moving object to be detected extracted from the image data. based on the vertical size of H _o of the region, it is possible to calculate the size in the real space.

  And the control part 10 can identify the kind of moving body based on the magnitude | size in the real space calculated by Formula 3. FIG. For example, the control unit 10 moves when the calculated height in the real space is 1.7 m and the aspect ratio (horizontal / vertical) of the extracted area is about 0.30 (= 60 cm / 2.0 m). Identify the body as a person.

  It should be noted that the distance D of the person from the image processing apparatus 1 shown in FIG. 6A is obtained from information stored in the storage unit 11 in advance as described in the first embodiment. Specifically, the control unit 10 obtains the distance from the image processing apparatus 1 in the real space stored in association with the vertical position Y in the image data at the lower end of the region extracted as the region corresponding to the person. .

However, the relationship between the vertical position Y in the image data and the distance D from the image processing apparatus 1 including the first and second imaging devices 3 and 4 is shown in FIGS. 6 (a) and 6 (b). In addition to the vertical angle of view θ _v and the number of pixels H _c in the vertical direction of the image data, the image processing apparatus 1 including the first and second imaging devices 3 and 4 is installed and includes the height h and the depression angle φ _c as follows. The relational expression shown in Expression 4 is established.

  By storing Equation 4 in the storage unit 11, when the control unit 10 specifies the moving body region (circumscribed rectangle), the control unit 10 specifies the vertical coordinate Y in the image data at the lower end of the region. Based on this, the distance D from the image processing apparatus 1 to the moving body in the real space may be calculated.

  In the first embodiment, the correspondence between the coordinate Y calculated in advance based on Expression 4 and the distance D is stored in the storage unit 11, and the distance D corresponding to the specified coordinate Y is stored in the storage unit. The distance from a predetermined position in the real space was obtained by reading from 11. Thereby, the position (distance from the 1st and 2nd imaging devices 3 and 4) of the mobile body in real space can be easily specified, without calculating. Therefore, the position information in the real space in the intersection can be transmitted to the automobile as the moving body information, and high-precision traveling control can be realized for safe driving support.

(Embodiment 2)
FIG. 7 is a schematic diagram schematically showing an example of an automobile in which the safe driving support system according to the second embodiment is installed. Reference numeral 3 in FIG. 7 denotes a first imaging device 3 that performs imaging in the far-infrared wavelength band. Reference numeral 4 in FIG. 7 denotes a second imaging device 4 that performs imaging in the visible light wavelength band or the near-infrared wavelength band. Reference numeral 5 in FIG. 7 denotes a third imaging device that performs imaging in the far-infrared wavelength band. Reference numeral 6 in FIG. 7 denotes an image processing apparatus, which is connected to a travel control apparatus 8 that performs a travel control process of the automobile via the in-vehicle LAN 7. Further, the first to third imaging devices 3, 4, and 5 are connected to the image processing device 6 through the signal line 9.

  The first and second imaging devices 3 and 4 are juxtaposed forward on the left side of the top of the automobile. The third imaging device 5 is installed forward on the right side of the front portion of the automobile. As in the first embodiment, the first and second imaging devices 3 and 4 are installed so that substantially the same range is used as the imaging range. The third imaging device 5 also includes a far-infrared imaging device having the same number of pixels as the imaging device of the first imaging device 3, and the installation direction so that imaging with the same angle of view as the first and second imaging devices 3 and 4 is possible. Has been adjusted. Each of the first to third imaging devices 3, 4, 5 is configured to be able to output an image signal to the image processing device 6 via the communication line 9.

  When the image processing device 6 acquires image data from the image signals output from the first to third imaging devices 3, 4, and 5 and detects a moving body based on the acquired image data, The detected information on the moving body is transmitted to the travel control device 8 via the in-vehicle LAN 7. The information of the mobile body to be transmitted is the type of mobile body and the distance from the automobile to the mobile body. The traveling control device 8 can perform control such as instructing to perform braking automatically or starting a shock absorber inside and outside the vehicle based on the type of the moving body and the distance to the moving body.

  FIG. 8 is a block diagram showing the internal configuration of the first to third imaging devices 3, 4, 5 and the image processing device 6 constituting the safe driving support system in the second embodiment. Details of the configuration of the first to third imaging devices 3, 4, and 5 are the same as those in the first embodiment, and a description thereof will be omitted.

  The image processing device 6 includes a control unit 60, a storage unit 61, an image memory 62, a first image acquisition unit 63, and a second image acquisition unit 64, and further to the travel control device via the in-vehicle LAN 7. A communication unit 65 that realizes transmission of information and a third image acquisition unit 66 that receives an image signal from the third imaging device 5 are provided. Details of the control unit 60, the storage unit 61, the image memory 62, and the first and second image acquisition units 63 and 64 are the control unit 10, the storage unit 11, the image memory 12, and the image processing apparatus 1 according to the first embodiment. Further, since it is the same as the first and second image acquisition units 13 and 14, detailed description thereof is omitted.

  The communication unit 65 transmits and receives information based on standards for in-vehicle LANs such as CAN (Controller Area Network), LIN (Local Interconnect Network), and FlexRay (registered trademark). The control unit 60 transmits information on the moving body to the travel control device through the communication unit 65.

  The third image acquisition unit 66 extracts image data from the received image signal and stores it in the image memory 62. Specifically, the third image acquisition unit 66, like the first and second image acquisition units 63 and 64, follows the instruction from the control unit 60 and the luminance value or color difference for each pixel constituting the image from the received image signal. Are specified as image data and written to the image memory 62.

  The image processing apparatus 6 according to the second embodiment configured as described above processes the image data acquired from the first and second imaging apparatuses 3 and 4 in the same manner as the image processing procedure described in the first embodiment. To do. That is, the first and second image acquisition units 63 and 64 acquire image data at the same time to obtain difference data, and specify the moving object region by combining the difference data. However, in the first and second imaging devices 3 and 4 mounted on the automobile, the background changes at any time during traveling, so it is desirable to obtain the frame difference instead of the background difference.

  The control unit 60 of the image processing device 6 according to the second embodiment performs template matching on the type of the moving body from the moving body region specified by the above-described processing based on the image data acquired from the first and second imaging devices 3 and 4. You may identify using the method of.

  However, the control unit 60 of the image processing device 6 according to the second embodiment uses the image data acquired from the third imaging device 5 installed away from the first and second imaging devices 3 and 4 and uses the image data of the automobile. The distance from the head of the object to the moving body is calculated by the process described below.

  Since the 3rd imaging device 5 is installed so that it may become the same view angle as the 1st and 2nd imaging devices 3 and 4, the mobile body imaged with the 1st and 2nd imaging devices 3 and 4 is set to the 1st. The three image pickup devices 5 can also pick up images. Since the installation directions are different, the position of the moving body region in the image data obtained from the third imaging device 5 is obtained from the first and second imaging devices 3 and 4 even if the captured images are the same moving body. Different from the position of the moving object region in the image data to be obtained. This is because the incident angle of light from the moving body is different.

  The control unit 60 simultaneously acquires image data based on the image signals output from the first and second imaging devices 3 and 4 and further based on the image signal output from the third imaging device 5 simultaneously. Get image data. The control unit 60 obtains a frame difference for the image data acquired from the third imaging device 5 and extracts a candidate for the moving body region. The control unit 60 performs the first and second operations based on the similarity based on the shape or position of the moving object region candidate extracted from the image data acquired from the first and second imaging devices 3 and 4 by taking a frame difference. The moving body region specified from the image data acquired from the imaging devices 3 and 4 and the moving body region candidate extracted from the image data acquired from the third imaging device 5 are associated with each other. That is, the control unit 60 associates the specified moving body region corresponding to the same moving body with the moving body region candidate. The control unit 60 obtains a difference in position within the same angle of view between the identified moving body region and the candidate associated with the moving body region. The difference in position is obtained, for example, by the difference in position of specific coordinates such as the coordinates located at the upper right of the moving body region. Since the difference in position is the difference in incident angle, the distance to the moving body can be obtained based on the difference in incident angle and the distance between the first and second imaging devices 3 and 4 and the third imaging device 5. It is.

  As described above, in the first imaging device 3 for imaging in the far infrared wavelength band, the second imaging device 4 for imaging in the visible light wavelength band or the near infrared wavelength band, in the far infrared (visible light or near infrared) wavelength band. With the configuration in which the third imaging device 5 and the image processing device 6 for imaging are mounted on an automobile, the moving body can be detected with high accuracy after identifying the type of the moving body ahead in the automobile. Further, the third imaging device 5 is installed at a distance, and the distance to the moving body is calculated using the image data that can be acquired from the third imaging device 5, and the distance to the moving body is determined according to the type of the moving body and the distance to the moving body. Traveling control is possible.

  In the second embodiment, the third imaging device 5 is a far infrared camera that performs imaging in the far infrared wavelength band. However, the third imaging device 5 may be a camera that performs imaging in the visible light wavelength band or the near infrared wavelength band.

  In the first embodiment, similarly to the configuration in the second embodiment, a third imaging device that performs imaging from different angles may be separately provided, and the distance in the real space may be calculated by stereo vision.

  In the first and second embodiments described above, the control unit performs image acquisition and image processing by the first and second image acquisition units or the third image acquisition unit. However, the present invention is not limited to this, and the image acquisition processing and image processing by the first to third image acquisition units may be realized by an integrated circuit in which the image processing portions are separated.

  Embodiments 1 and 2 show the case where the present invention is applied to a safe driving support system. However, the detection of the moving object by the image processing apparatus according to the present invention is not limited to the safe driving support system, and can be applied to other fields. For example, when detecting a moving body based on a captured image, such as a security system or a monitoring system, it is possible to accurately detect the moving body and identify the type of the moving body.

  It should be understood that the embodiment disclosed above is illustrative in all respects and is not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the meanings described above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

It is a perspective view from the upper part which shows the example of the intersection in which the safe driving assistance system in Embodiment 1 is installed. 1 is a block diagram illustrating an internal configuration of an image processing device, a roadside machine, and first and second imaging devices that constitute a safe driving support system in Embodiment 1. FIG. FIG. 6 is an explanatory diagram illustrating an example of the contents of background image data and size information for each type of moving object stored in advance in the storage unit of the image processing apparatus according to the first embodiment. 3 is a flowchart illustrating an example of an image processing procedure performed by a control unit of the image processing apparatus according to the first embodiment. 6 is an explanatory diagram illustrating a specific example of an image processing result performed by a control unit of the image processing apparatus according to Embodiment 1. FIG. FIG. 10 is an explanatory diagram schematically illustrating the principle by which the control unit of the image processing apparatus according to the modification of the first embodiment calculates the size of the moving object in real space. It is a schematic diagram which shows typically the example of the motor vehicle in which the safe driving assistance system in Embodiment 2 is installed. 6 is a block diagram illustrating an internal configuration of first to third imaging devices and an image processing device that constitute a safe driving support system according to Embodiment 2. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1,6 Image processing apparatus 10,60 Control part 11,61 Memory | storage part 3 1st imaging device 4 2nd imaging device 5 3rd imaging device

Claims (7)

In an image processing apparatus that performs image processing for detecting a moving object based on a captured image,
First imaging means for imaging in the far-infrared wavelength band;
Second imaging means for imaging an imaging range of the first imaging means in a visible light wavelength band or a near infrared wavelength band;
Extraction means for extracting candidates for a moving body region in which a moving body is shown, respectively, from captured images simultaneously captured by the first and second imaging means;
A specifying means for specifying an overlapping portion of each candidate extracted by the extracting means as a moving object region;
An image processing apparatus comprising: means for detecting a moving object based on a feature amount of the moving object region specified by the specifying means. The first and second imaging means are configured to image a range including a substantially flat background surface;
First storage means for storing a distance in real space from a predetermined position on the background surface to the object when there is a region where the object is captured at each position in association with a plurality of different positions in the captured image When,
Means for obtaining a position in the captured image of the moving body region identified by the identifying means;
The image processing apparatus according to claim 1, further comprising: a distance specifying unit that specifies a distance in real space from the predetermined position to the moving body based on a distance associated with the obtained position. Second storage means for storing a correspondence relationship between the type of the moving body, the distance from the predetermined position, and the vertical and horizontal sizes of the area that the moving body area should occupy in the captured image;
An identification means for identifying the type of the moving body based on the distance to the moving body specified by the distance specifying means and the vertical and horizontal sizes of the moving body area specified by the specifying means. Item 3. The image processing apparatus according to Item 2. The first and second imaging means are installed so as to overlook the intersection,
The first storage means is based on the height and depression angle from the road surface of the first and second imaging means, and the position of the object region in the captured image and the predetermined position at the intersection in real space to the object. The distance is stored in association with it,
The second storage means is configured to store the vertical and horizontal sizes that the moving body region should occupy in the captured image in association with the distance.
The identification means is characterized in that the type of the moving body is identified based on the correspondence between the position and distance stored in the first and second storage means and the vertical and horizontal sizes. The image processing apparatus according to claim 3. A third imaging unit installed so as to capture an imaging range of the first and second imaging units from an angle different from that of the first and second imaging units;
Extraction means for extracting a candidate for a moving body region from a captured image captured by the third imaging means simultaneously with the first and second imaging means;
Means for associating the moving body region specified by the specifying means with the candidates extracted from the captured image of the third imaging means by the extracting means;
Means for specifying the position in the captured image of each of the associated moving body region and candidates;
2. The apparatus according to claim 1, further comprising: a unit that calculates a distance in real space from any one of the first to third imaging units to the moving body based on a difference between positions specified by each of the first to third imaging units. Image processing device. The image processing apparatus according to claim 5, wherein the first to third imaging units are attached to a vehicle and calculate a distance from the vehicle to the moving body. The first imaging unit that captures an image in the far-infrared wavelength band, and the moving body from the captured image captured by the second imaging unit that captures the imaging range of the first imaging unit in the visible light wavelength band or the near-infrared wavelength band. An image processing method for detecting,
Extracting moving body region candidates in which the moving body is shown from the captured images simultaneously captured by the first and second imaging means,
Identify each extracted candidate overlap as a mobile region,
An image processing method, comprising: detecting a moving object based on a feature amount of a specified moving object region.

Images