JP2008092459A - Periphery monitoring apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a periphery monitoring apparatus which accurately traces an object even in an area where imaging regions are not overlapped. <P>SOLUTION: A visual line direction of a three-dimensional object 22 and image templates are acquired in advance in a picked-up image of a front camera 11, and when the three-dimensional object 22 enters a single imaging region of a left-side camera 13, an image template is selected based on the visual line direction of the three-dimensional object 22. On the basis of the selected image template, the three-dimensional object 22 is traced. Thus, the three-dimensional object 22 is accurately traced. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a periphery monitoring device for monitoring the surroundings of a vehicle or the like.

2. Description of the Related Art Conventionally, as a periphery monitoring device for monitoring the periphery of a vehicle or the like, as described in, for example, Japanese Patent No. 3494434, the periphery of the vehicle is imaged, and information on a three-dimensional object around the vehicle is obtained based on the captured image. Devices that communicate to the driver are known. This device captures a plurality of different peripheral images by a single camera during the movement of the vehicle, detects a three-dimensional object by stereo vision based on these images, and detects the position data of the three-dimensional object on the image and the movement data of the vehicle Based on this, the distance from the vehicle to the three-dimensional object is calculated, and a peripheral image is generated based on the distance and transmitted to the driver.
Japanese Patent No. 3494434

  However, in such an apparatus, there is a problem that it is impossible to provide appropriate information about surrounding solid objects. For example, a stereoscopic object cannot be viewed in stereo in a range that does not overlap in a plurality of peripheral images, and the tracking of the stereoscopic object cannot be performed. For this reason, the three-dimensional information of the three-dimensional object cannot be acquired sufficiently. In this case, when an overhead view is created based on the captured image, the three-dimensional object is distorted and becomes unnatural.

  Therefore, the present invention has been made to solve such a technical problem, and an object of the present invention is to provide a periphery monitoring device capable of accurately tracking an object even in an area where imaging areas do not overlap. .

  That is, the periphery monitoring device according to the present invention includes a first imaging unit that images the first imaging region, a second imaging unit that images a second imaging region having a single imaging region that does not overlap the first imaging region, Feature information acquisition means for acquiring feature information of an object imaged by the first imaging means, and first line of sight for acquiring a first line-of-sight direction with respect to the object from the first image pickup means when the feature information of the object is acquired Direction acquisition means and a predetermined direction relative to the line-of-sight direction from the second imaging means to the object when the area where the object is imaged is relatively moved from the first imaging area to the single imaging area in the second imaging area; Object tracking means for tracking the object on the basis of the feature information of the object imaged in the first line-of-sight direction within the range of the direction difference and the captured image imaged by the second imaging means. It is.

  According to this invention, when the area where the object is imaged is relatively moved to the single imaging area of the second imaging area, the object imaged in the first line-of-sight direction close to the line-of-sight direction from the second imaging unit to the object is displayed. By tracking the object based on the feature information and the captured image of the object imaged by the second imaging unit, the position of the object in the captured image of the second imaging unit can be accurately tracked.

  Further, the periphery monitoring device according to the present invention is characterized in that the feature information of the object imaged in the first line-of-sight direction closest to the line-of-sight direction from the second imaging unit to the object and the image area in the captured image of the second imaging unit Preferably, the object tracking means tracks the object with the point having the highest correlation value as the position of the object.

  Further, the periphery monitoring device according to the present invention includes a plurality of pieces of feature information of the object imaged in the first line-of-sight direction within a predetermined direction difference range with respect to the line-of-sight direction from the second imaging unit to the object. Correlation value calculating means for calculating a correlation value with an image area in a captured image of the second imaging means, and the object tracking means tracks the object with the point having the highest correlation value as the position of the object. Is preferred.

  Further, in the periphery monitoring device according to the present invention, an object existing in an overlapping region where the first imaging region and the second imaging region overlap by stereo vision based on the captured images of the first imaging unit and the second imaging unit. A first object three-dimensional information acquisition means for acquiring three-dimensional information; a tracking result by the object tracking means; and a single image of the second imaging region based on the three-dimensional information acquired by the first object three-dimensional information acquisition means. It is preferable to include a second object 3D information acquisition unit that acquires 3D information of an object existing in the imaging region.

  According to this invention, the three-dimensional information of the object existing in the single imaging region of the second imaging region is acquired based on the tracking result by the object tracking unit and the three-dimensional information acquired by the first object three-dimensional information acquisition unit. Accordingly, even when the object exists in the single imaging region of the second imaging region where it is difficult to acquire the three-dimensional information of the object by stereo vision, the three-dimensional information of the object can be reliably acquired.

  Further, in the periphery monitoring device according to the present invention, when the first object three-dimensional information acquisition unit performs stereo vision, the first object imaging unit and the second imaging unit can capture images by being blocked by the object. A blind spot area for identifying a blind spot area where three-dimensional information cannot be obtained, and an area that can be imaged by the first imaging means or the second imaging means in the blind spot area, and the luminance is continuously changed. A blind spot interpolation means for interpolating the blind spot area based on the three-dimensional information of the peripheral area of the blind spot area, and a blind spot interpolation area recording means for recording the area that can be interpolated by the blind spot interpolation means and the area that cannot be interpolated. It is preferable to provide.

  According to the present invention, based on the three-dimensional information of the surrounding area of the blind spot area when the brightness continuously changes in the blind spot area that can be imaged by the first imaging means or the second imaging means. By interpolating the blind spot area, it is possible to reduce the blind spot area and interpolate three-dimensional information that cannot be obtained by stereo vision.

  Further, in the periphery monitoring device according to the present invention, a third imaging unit that images a third imaging region having an overlapping region that overlaps the second imaging region, and a region where the object is imaged is a single region of the second imaging region. When the relative movement from the imaging area to the overlapping area of the second imaging area and the third imaging area, the second imaging is performed with reference to the 3D information of the object acquired by the second object 3D information acquisition means And a first object three-dimensional information acquisition unit that acquires the three-dimensional information of the object by stereo vision based on the captured image of the third imaging unit.

  According to this invention, the three-dimensional information of the object acquired by the second object three-dimensional information acquisition unit when acquiring the three-dimensional information of the object by stereo vision based on the captured images of the second imaging unit and the third imaging unit. By referring to, it is possible to accurately derive the three-dimensional information of the object by stereo vision.

  Moreover, in the periphery monitoring apparatus according to the present invention, the first imaging unit and the second imaging unit are installed on a moving body to image the surroundings of the moving body, and the first object three-dimensional information acquisition unit and the second imaging unit The object three-dimensional information acquisition means preferably acquires the relative position coordinate value of the object with respect to the moving body as the three-dimensional information of the object.

  According to the present invention, it is possible to grasp the relative position of an object even when an object around the moving object moves out of the overlapping area of the imaging area as the moving object moves.

  Moreover, it is preferable that the periphery monitoring apparatus according to the present invention further includes an overhead view creation means for creating an overhead view around the moving body based on the relative position coordinate value of the object.

  According to the present invention, by creating an overhead view using the three-dimensional information of the object, it is possible to create an easy-to-see overhead view with reduced distortion of the object.

  Further, in the periphery monitoring device according to the present invention, when it is determined that the contact determination means determines whether or not the moving body contacts the object based on the relative position coordinate value of the object, and the contact determination means contacts It is preferable to include contact avoiding means for avoiding the contact.

  According to the present invention, even when an object exists in an area other than the overlapping area where the imaging areas of the imaging means overlap, it is possible to accurately avoid contact with the moving body based on the relative position of the object.

  According to the present invention, an object can be accurately tracked even in an area where imaging areas do not overlap.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a schematic configuration diagram of a periphery monitoring device according to an embodiment of the present invention, and FIG. 2 is a plan view showing the arrangement of cameras and the imaging region of the camera in the periphery monitoring device according to the present embodiment.

  The periphery monitoring device 1 according to the present embodiment is a device that monitors the periphery of the vehicle 3 and includes a plurality of cameras that image the periphery of the vehicle 3. For example, the front camera 11, the right side camera 12, and the left side A side camera 13 and a rear camera 14 are provided. The front camera 11, the right side camera 12, the left side camera 13, and the rear camera 14 are installed so as to pick up images in different directions, and are arranged so as to have non-overlapping regions where the image pickup regions do not overlap with adjacent cameras.

The front camera 11 is disposed toward the front of the vehicle and functions as an imaging unit that images the imaging area a _{F in} front of the vehicle. The right-side camera 12 is disposed toward the right side of the vehicle and functions as an imaging unit that captures an area a _R on the right side of the vehicle. The left-side camera 13 is disposed toward the left side of the vehicle and functions as an imaging unit that images a region a _L on the left side of the vehicle. The rear camera 14 is disposed toward the rear of the vehicle, and functions as an imaging unit that captures an area a _B behind the vehicle.

Front of an image area a _F has an overlapping area that overlaps with the imaging area a _R of the right side at one side, and has an overlapping region overlapping the imaging area a _L of the left side at the other end . Further, the rear imaging area a _F has an overlapping area that overlaps the right imaging area a _R on one end side, and an overlapping area that overlaps the left imaging area a _L on the other end side. ing.

The front camera 11, the right side camera 12, the left side camera 13, and the rear camera 14 are wide-angle cameras having a large viewing angle, and include, for example, a fish-eye lens with an angle of view of 180 ° for obtaining an image I _LO as shown in FIG. It is preferable to use the same.

  For example, when the guard rail 4 extends to the left side of the vehicle 3, the foremost rail portion 40 and the rearmost rail portion 46 are outside the field of view of the camera. The rail parts 41 and 42 are located in the field of view of the front camera 11, the rail parts 42 to 44 are located in the field of view of the left camera 13, and the rail parts 44 and 45 are located in the field of view of the rear camera 14, respectively. To do. That is, the rail portions 41, 43, and 45 are located in the field of view of only one of the front camera 11, the left side camera 13, and the rear camera 14, but the rail portion 42 is the front camera 11 and the left side camera. The rail portion 44 is positioned in a region where the fields of view of the left side camera 13 and the rear camera 14 overlap.

  Each camera 11 to 14 is connected to the image processing unit 2. Images captured by the cameras 11 to 14 are sent to the image processing unit 2 as digital signals. The digital conversion of the image signal may be performed in the cameras 11 to 14, but the conversion may be performed by providing an A / D converter between the cameras 11 to 14 and the image processing unit 2, or the image processing unit. 2 may include an A / D converter.

  The image processing unit 2 performs image processing on images captured by the cameras 11 to 14, and includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. ing. The image processing unit 2 reads the captured images of each camera and performs image processing. When there is a solid object in the imaging overlap area of the two cameras, the image processing unit 2 performs the three-dimensional information derivation process and elevation in the overlap area. Map creation processing, blind spot interpolation processing, tracking processing, solid object information recording update processing, and boundary flag recording processing are executed (see FIG. 5).

  Further, when the three-dimensional object moves from the imaging overlap area of the two cameras, the image processing unit 2 executes a tracking process, a three-dimensional information derivation process, an elevation map creation process, a cooperation process, and a record deletion process. (See FIG. 5). Details of the contents of these processes will be described later.

  The periphery monitoring device 1 is provided with a vehicle speed sensor 15. The vehicle speed sensor 15 is a sensor that detects the speed of the vehicle. A detection signal detected by the vehicle speed sensor 15 is input to the image processing unit 2.

  The periphery monitoring device 1 is provided with an output unit 30. The output unit 30 outputs information obtained by image processing of a captured image, and corresponds to a monitor, a speaker, a buzzer, a lamp, and the like installed in the vehicle.

  Next, the operation of the periphery monitoring device according to the present embodiment will be described.

  FIG. 4 is a flowchart showing the operation of the periphery monitoring apparatus according to the present embodiment. The control process of FIG. 4 is repeatedly executed at a predetermined timing after the vehicle power is turned on, for example. For example, the processing may be performed for each frame of images to be acquired or every several frames in synchronization with the image acquisition rates of the cameras 11 to 14.

  First, as shown in S10 of FIG. 4, a captured image reading process is performed. The captured image reading process is a process of reading images captured by the front camera 11, the right side camera 12, the left side camera 13, and the rear camera 14. And it transfers to S20 and it is judged whether the vehicle is moving forward. This determination process is a process for determining whether or not the vehicle is moving forward based on the detection signal of the vehicle speed sensor 15. In this determination process, it may be determined whether the vehicle is moving forward based on the detection signal of the vehicle speed sensor 15 and the operation state of the shift lever of the transmission.

  When it is determined in S20 that the vehicle is moving forward, image processing is performed on the front camera image and the left camera image (S30), and image processing is performed on the front camera image and the right camera image (S40). . Details of the image processing in the front camera image and the left-side camera image in S30 and the image processing in the front camera image and the right-side camera image in S40 will be described later.

  On the other hand, when it is determined in S20 that the vehicle is not moving forward, it is determined whether or not the vehicle is moving backward (S50). This determination process is a process for determining whether or not the vehicle is moving backward based on the detection signal of the vehicle speed sensor 15. In this determination process, it may be determined whether the vehicle is moving backward based on the detection signal of the vehicle speed sensor 15 and the operation state of the transmission shift lever.

  If it is determined in S50 that the vehicle has not moved backward, the control process is terminated because the vehicle has not moved forward or backward. In this case, since the vehicle is not running, the tracking process is not performed, but the three-dimensional information is acquired based on the captured images of the cameras 11 to 14, and the position information of the three-dimensional object, the elevation map, and the like. May be updated.

  On the other hand, if it is determined in S50 that the vehicle is moving backward, image processing is performed on the rear camera image and the left camera image (S60), and image processing is performed on the rear camera image and the right camera image. (S70). Details of the image processing for the rear camera image and the left side camera image in S60 and the image processing for the rear camera image and the right side camera image of S70 will be described later.

  Then, the process proceeds to S80, and an elevation map integration process is performed. This integration process is a process of creating one large elevation map by integrating the elevation maps created by the image processing of S30, S40, S60, and S70. The elevation map is a height map of a three-dimensional object in the monitoring area. For example, the elevation map is configured to correspond to an overhead view of the imaging area around the vehicle viewed from above, and is created so that the height information of a three-dimensional object can be recorded in units of 1 cm × 1 cm. The The three-dimensional object referred to here includes not only those that are higher than the road surface such as guardrails, columns, and pylons, but also those that are lower than the road surface such as side grooves.

  Then, the process proceeds to S90 and output processing is performed. The output process is a process of outputting information obtained by performing image processing on the captured image to the output unit 27 and performing warning processing and vehicle control processing based on the information.

  As the output process, for example, a process of displaying an overhead view of the imaging region on the monitor is performed. In this case, the driver of the vehicle can easily recognize a three-dimensional object around the vehicle through vision. At that time, the three-dimensional object may be displayed in 3D graphics using the three-dimensional information of the three-dimensional object.

  In addition, as an output process, if there is a three-dimensional object that obstructs driving or the vehicle 3 may come into contact with the three-dimensional object, the presence of an obstacle through hearing, such as sound, or through visual perception, such as lamp lighting, lamp blinking, and warning display Alternatively, the driver may be warned of the possibility of contact. In this case, the driver can travel the vehicle avoiding the obstacle by receiving this warning.

  Further, as an output process, when there is a three-dimensional object that becomes a travel obstacle, a signal may be output as route search information. In this case, a route search can be performed while avoiding obstacles, and an appropriate route search can be realized. As an output process, an automatic stop control signal may be output when there is a three-dimensional object that becomes a running obstacle. For example, a stop control signal is output to a braking control ECU (Electronic Control Unit) to automatically stop the vehicle. In this case, it is possible to avoid such a situation that the vehicle contacts an obstacle. When the output process of S90 is finished, the control process is finished.

  FIG. 5 is a flowchart showing image processing in the front camera image and the left camera image in S30 of FIG.

As shown in S300 of FIG. 5, first, it is determined whether or not the three-dimensional object has moved outside the overlapping area where the imaging areas overlap. That is, it is determined whether or not the three-dimensional object has moved from within the overlapping area a _LF between the imaging area a _F of the front camera 11 and the imaging area a _L of the left camera 13 to the outside. This determination is made based on, for example, the previously executed image processing result and vehicle travel information.

If there is a three-dimensional object in the overlap area a _LF as a result of the image processing executed earlier, but the vehicle is traveling more than a predetermined distance and it is estimated that the three-dimensional object is not in the overlap area a _LF , It is determined that the three-dimensional object has moved out of the overlapping region _aLF . On the other hand, when it is estimated that there is a three-dimensional object in the overlapping area a _LF and the vehicle has not traveled to a predetermined distance and the three-dimensional object is still in the overlapping area a _LF as a result of the image processing executed earlier. Is determined that the three-dimensional object has not moved out of the overlapping region _aLF . If there is no solid object in the overlapping area a _LF as a result of the image processing executed previously, it is determined that the three-dimensional object has not moved outside the overlapping area a _LF .

If it is determined in S300 that the three-dimensional object has not moved outside the overlapping area, a three-dimensional information derivation process in the overlapping area is performed (S302). The three-dimensional information deriving process is a process of deriving three-dimensional information of a three-dimensional object in the overlapping area a _LF of the imaging area a _F of the front camera 11 and the imaging area a _L of the left camera 13.

As this three-dimensional information derivation process, first, a processing range of an image acquired by the left side camera 13 is set. FIG. 6 shows the imaging area of each camera 11-14. Here, an area a _LF where the imaging area a _F of the front camera 11 and the imaging area a _L of the left-side camera 13 overlap is set as the processing area. At this time, for example, the viewing angle range γ of the left side camera 13 may be set as the processing range.

Then, a virtual viewpoint image I _{LF in the} direction of 45 degrees forward is generated from the captured image I _LO of the left side camera 13. For example, in the captured image I _LO , a virtual line-of-sight direction is set in the direction of 45 degrees forward, and a virtual viewpoint image I _LF is generated by performing line-of-sight conversion on the left front image portion with respect to a plane perpendicular to the line-of-sight direction. Is done. FIG. 7 is an explanatory diagram of generation of a virtual viewpoint image, and FIG. 8 is a diagram illustrating a virtual viewpoint image generated based on a captured image.

Unlike a normal camera lens, the fish-eye lens adopts an equisolid angle projection method or an equidistant projection method, not an orthographic projection method. For this reason, except for the straight line passing through the center of the screen, as shown in FIG. 3, the straight line shape in space (for example, the contour of the guardrail) is also displayed as a curve on the screen. A straight line can be displayed as a straight line as shown in FIG. 8 by converting the projection method from this image into an orthographic projection and converting the line-of-sight direction at that time. Here, _IL is a virtual viewpoint image in the leftward direction, and _ILB represents a virtual viewpoint image in the backward 45 degree direction.

When the image I _LF is generated, the processing range of the captured image acquired by the front camera 11 is set, and the virtual viewpoint image I _{FL in the} 45 ° left direction is generated from the captured image of the front camera 11. As shown in FIG. 6, the processing range in the captured image of the front camera 11 is set by processing an area a _LF that overlaps the imaging area a _L of the left camera 13 in the imaging area a _F of the front camera 11. This is done by setting as a range.

Generation of the virtual viewpoint image I _{FL in} the direction of 45 degrees to the left is performed, for example, in a captured image I _FO of the front camera 11 with a virtual line-of-sight direction set in the direction of 45 degrees to the left and a plane perpendicular to the line-of-sight direction. On the other hand, it is performed by changing the line of sight of the left front image portion. FIG. 9 is an explanatory diagram of generation of a virtual viewpoint image. Here, I _F is the forward direction of the virtual viewpoint image, I _FR represents a virtual viewpoint image of the right 45 ° direction.

Next, the corresponding points of the pixels of the images I _LF and I _FL are searched. Specifically, as shown in FIG. 10, a small region on the image I _FL corresponding to an arbitrary small region on the image I _LF exists on the epipolar polar line e _FL . Therefore, in the image _{I FL,} located on the epipolar polar _{e FL} corresponding to the region including the edge configuration points of the image _{I LF,} and edge configuration of the image _{I LF} for the area near the edge configuration points of the image _{I FL} A correlation value with an area including a point is calculated, and an area with the highest correlation value is obtained. The maximum correlation value and S _L. Then, by comparing the _{S L} and the threshold THs, if _{S L} threshold THs above, it is determined that the corresponding has been established between the images. The threshold value THs is a preset value. As a method for deriving the correlation value, for example, a pattern recognition technique such as a normalized correlation method or a sum of differences may be used.

The left front 45 degree image I _LF created from the captured image of the left side camera 13 and the left front 45 degree image I _FL created from the captured image of the front camera 11 have each camera installation distance (distance of the imaging position) as a base line length. It can be used as a stereo camera image. In this case, since the image I _LF and the image I _FL are images viewed from the same direction by viewpoint conversion, the corresponding points can be easily searched.

Then, based on the corresponding pixel position information in the image I _FL and the image I _LF , the spatial position of the part corresponding to the pixel of the subject is calculated. FIG. 11 is a diagram for explaining the principle of calculating the spatial position.

なお、ｙｌとｙｒとは理想的には一致する。 Here, it is assumed that the sensor coordinate system XsYsZs is taken, and the left and right cameras are arranged apart from the sensor origin by a base line length b in the X-axis direction. That is, the optical center of the left camera is located at the coordinate position (−b / 2, 0, 0), and the optical center of the right camera is located at the coordinate position (b / 2, 0, 0). The focal length of both cameras is f, the position of the object P on the screen of the left camera is Pl (xl, yl), and the position of the right camera on the screen is Pr (xr, yr) (X _L , The X _R axis is parallel to the Xs axis, and the Y _L and Y _R axes are parallel to the Zs axis.) And the spatial position of the object P (X _ps , Y _ps , Z) according to the principle of triangulation _ps ) is expressed by the following equation.

Note that yl and yr ideally match.

Further, conversion from the sensor coordinate system to the vehicle coordinate system is performed. FIG. 12 is a diagram illustrating coordinate conversion from the sensor coordinate system to the vehicle coordinate system. Consider a coordinate system XcYcZc in which the center of gravity of the vehicle is the origin and the sensor coordinate system and each axis are parallel. At this time, the axes Xc, Yc, and Zc are obtained by shifting the axes Xs, Ys, and Zs of the sensor coordinate system by ax, ay, and az, respectively. Similarly, consider a coordinate system XfYfZf where the center of gravity of the vehicle is the origin, the vertical axis is the Zf axis, and the axes Xc and Yc are projected on the horizontal plane are Xf and Yf. Here, assuming that the angle formed by Xf and Xc is the roll angle φ, and the angle formed by Yf and Yc is the pitch angle ρ, the spatial position (X _pc , Y _pc , Z _pc ) and XfYfZf coordinates of the object P in the XcYcZc coordinate system. The spatial position (X _pf , Y _pf , Z _pf ) of the object P in the system is represented by the following equation.

  Thereby, the position in space can be expressed. Note that ax, ay, and az are constants determined based on the camera arrangement position, and φ and ρ can be obtained from the attitude information of the vehicle 3. The spatial position thus obtained, that is, the three-dimensional position information is recorded in association with the pixel position on the screen.

  When the three-dimensional information derivation process in S302 of FIG. 5 is finished, the process proceeds to S304, and an elevation map creation process and a blind spot interpolation process are performed.

The elevation map creation process is a process for creating an elevation map related to height information in the overlapping area a _LF of the imaging area a _F of the front camera 11 and the imaging area a _L of the left camera 13 based on the three-dimensional position information. It is. For example, as shown in FIG. 13, the elevation map 21 is configured to correspond to an overhead view of the overlapping area a _LF as viewed from above, and the height information of the three-dimensional object 22 in units of 1 cm × 1 cm. Is created so that can be recorded. The three-dimensional object referred to here includes not only those that are higher than the road surface such as guardrails, columns, and pylons, but also those that are lower than the road surface such as side grooves.

In the elevation map 21, when there is a three-dimensional object 22 such as a pylon or a support in the overlapping area a _LF , the position information, height information, and luminance information (color information) of the three-dimensional object 22 are recorded in the elevation map 21. . For pixels in which the three-dimensional information of an object such as the three-dimensional object 22 is not recorded, if there are pixels in which the three-dimensional information is recorded in the surrounding pixels, it is preferable to interpolate based on the three-dimensional information.

  The blind spot interpolation process is a process of interpolating a blind spot area where three-dimensional information cannot be obtained due to a three-dimensional object or the like based on imaging information of another camera. This blind spot interpolation process is performed, for example, by sequentially executing a blind spot area derivation process, a blind spot area information recording process, and an interpolation process.

  The process of deriving the blind spot area by the three-dimensional object is performed based on the three-dimensional information of the three-dimensional object. FIG. 14 is an explanatory diagram of the blind spot area derivation process. FIG. 14A is a view of the monitoring area as viewed from the side, and FIG. 14B is a view of the monitoring area as viewed from above.

  The blind spot area derivation process is performed, for example, by calculating the length L and the width W of the blind spot area based on the three-dimensional information of the three-dimensional object 22. First, based on the three-dimensional information of the three-dimensional object 22, the height h of the three-dimensional object 22 and the distance d of the three-dimensional object 22 from the camera 11 are calculated. Then, the length L of the blind spot area is calculated by calculating h · d / (Hc−h). Here, Hc is the installation height of the camera 11. Further, the width W of the blind spot area can be calculated by performing a geometric calculation based on the width of the three-dimensional object 22 and the distance from the camera 11.

  Further, this blind spot area derivation process may be performed using not only the three-dimensional information of the three-dimensional object 22 but also the three-dimensional information of the road surface. In this case, when the road surface is inclined, the blind spot area can be accurately derived.

  FIG. 15 is an explanatory diagram of the blind spot area derivation process. FIG. 15A is a view of the monitoring area as viewed from the side, and FIG. 15B is a view of the monitoring area as viewed from above. FIG. 16 is a flowchart of the blind spot area derivation process.

As shown in S400 of FIG. 16, as the blind spot area derivation process, first, a calculation process of the distance Dv to the measurement point Pv is performed. This calculation process is a process for calculating the distance Dv from the camera to the measurement point Pv where the three-dimensional information is acquired in the overlap region a _FL . This distance Dv is calculated using the three-dimensional coordinate data of the measurement point Pv. Then, the process proceeds to S402, calculation processing of the distance _{D V-1} to the measurement point _{P V-1} is performed. The measurement point P _V-1 is a measurement point adjacent to the measurement point Pv on the near side of the measurement point Pv. This distance D _V-1 is calculated using the three-dimensional data of the measurement point P _V-1 .

Then, the process proceeds to S404, and a calculation process of the distance Sv between the measurement points is performed. This calculation process is a process of calculating the distance Sv between the measurement point Pv and the measurement point P _V-1, and is performed by calculating, for example, Pv-P _V-1 . Then, the process proceeds to S406, and a blind spot distance Lr calculation process is performed. This calculation process is a process of calculating the blind spot distance Lr by the three-dimensional object 22 when it is assumed that the road surface is horizontal and not inclined. This blind spot distance Lr is calculated by calculating h · d / (Hc−h).

Then, the process proceeds to S408, and it is determined whether or not the height of the measurement point P _V-1 is lower than the height of the measurement point Pv. Whether or not the height of the measurement point P _V-1 is lower than the height of the measurement point Pv can be determined by comparing the respective three-dimensional coordinate data. Measurement point height of _{P V-1} at S408 is when it is determined not lower than the height of the measurement point Pv is calculation processing of the blind spot determining distance threshold TH _D is performed (S412). Process for calculating the blind spot judgment distance threshold TH _{D is} the arithmetic expression of TH D = Lr + K1 · d , is a process of calculating the blind spot determining distance threshold TH _D. Here, d is a distance from the camera to the three-dimensional object 22, and K1 is a coefficient set in advance.

On the other hand, the height of the measurement point _{P V-1} is when it is judged to be lower than the height of the measurement point Pv is calculation processing of the blind spot determining distance threshold TH _D is performed in S408 (S410). Process for calculating the blind spot judgment distance threshold TH _{D is} the arithmetic expression of TH D = Lr-K2 · d , is a process of calculating the blind spot determining distance threshold TH _D. Here, K2 is a preset coefficient.

Then, the process proceeds to S414, blind spot distance Lr is whether a blind spot determining distance threshold TH _D is larger than or not. If the blind spot distance Lr is not greater than the blind spot determining distance threshold TH _D from the measurement point P _V-1 to the measurement point Pv is determined not to be blind area (S416). On the other hand, when the blind spot distance Lr is determined to be greater than the blind spot determining distance threshold TH _D from the measurement point P _V-1 to the measurement point Pv is determined as a blind area (S418).

  Then, the process proceeds to S420, and it is determined whether or not the blind spot area has been determined for all the measurement points. If the blind spot area has not been determined for all measurement points, the process returns to S400. On the other hand, when the blind spot area is determined for all the measurement points, the blind spot area derivation process ends.

  As described above, by performing the blind spot area derivation process of FIG. 16, it is possible to accurately derive the blind spot area due to the three-dimensional object even when the road surface is inclined.

  The blind spot area information derived by this blind spot area derivation process is recorded in the elevator map. In this blind spot area derivation process, a blind spot area for the front camera 11 is derived and a blind spot area for the left camera 13 is also derived. Each of these blind spot area information is recorded in the elevator map.

  FIG. 17 is a view showing a blind spot area D11 for the front camera 11, and FIG. 18 is a view showing a blind spot area D13 for the left camera 13. As shown in FIG. As shown in FIGS. 17 and 18, the blind spot area D <b> 11 for the front camera 11 and the blind spot area D <b> 13 for the left camera 13 are different because the installation positions of the front camera 11 and the left camera 13 are different.

  19 to 24 are explanatory diagrams of the blind spot area interpolation processing.

  As the blind spot area interpolation processing, first, the blind spot area D11 of the front camera 11 and the blind spot area D13 of the left side camera 13 are used. A certain region is extracted. As shown in FIG. 19, the area excluding the area Dx overlapping the blind spot area D13 in the blind spot area D11 of the front camera 11 is a blind spot area with respect to the front camera 11 and a non-dead spot area with respect to the left side camera 13. It becomes an area.

Then, with reference to the elevation map, the pixel position in the image of the left camera 13 corresponding to the blind spot area of the front camera 11 is obtained. For example, in the elevation map 21 shown in FIG. 20, based on which position the pixel p0 that is in the blind spot area D11 of the front camera 11 and has no three-dimensional information corresponds to the image of the left camera 13 of FIG. Then, the pixel position in the image of the left side camera 13 corresponding to the blind spot area of the front camera 11 is obtained. As an image of the left side camera 13, for example, a virtual viewpoint image _ILF is used.

  Then, as shown in FIG. 22, the blind spot area D <b> 11 of the front camera 11 is labeled in the image of the left camera 13. Further, the peripheral area S of the blind spot area D11 is extracted. Then, as shown in FIG. 23, a vertical line upper / lower pair of small areas R, R adjacent to the peripheral area S and not including the blind spot area D11 is set. As the small region R, for example, a region of 5 × 5 pixels is set. Further, the small region R may be a rectangular shape, or may be a shape that flexibly varies along the shape of the edge or the like of the peripheral region S.

  Then, the blind spot partial area Q sandwiched between the small areas R and R is approximated in a plane by a least square method, a Hough transform, or the like. The three-dimensional coordinate values obtained by the plane approximation are recorded in the elevation map 21 shown in FIG.

Prior to the plane approximation process of the blind spot partial area Q, it is preferable to perform the blind spot investigation process on the blind spot partial area Q. For example, in the image of FIG. 23, it is investigated whether or not there is a portion where the luminance is greatly changed in the blind spot partial area Q. If there is a large change in luminance, the blind spot partial area Q is not continuously connected and is approximated in a plane. It is determined that it should not be, and the area is recorded as a blind spot. This as a research method is, for example, the image I _LF derived luminance variation by Sobel filter or the like from, the luminance change in the blind spot region Q is examined density present is more portions predetermined threshold, the threshold If there is an area with a large change in luminance as described above, the blind spot flag is recorded without performing the interpolation process. Further, the luminance distribution of the peripheral region S and the peripheral portion thereof is examined, and when these regions have a distribution width equal to or larger than a predetermined threshold, a blind spot flag may be recorded.

  Then, the setting positions of the small areas R and R are moved in the direction in which the blind spot area D11 extends, the above-described planar approximation process is performed, and the process is repeated until the entire range of the blind spot area D11 is covered.

  Such an interpolation process is similarly performed on the blind spot area D13 of the left-side camera 13. According to this interpolation processing, the blind spot area in the monitoring area can be reduced.

FIG. 25 is a diagram illustrating a result of the interpolation process. FIG. 25A shows a blind spot area before interpolation processing, and FIG. 25B shows a blind spot area after interpolation processing. Before the interpolation process shown in FIG. 25A, a blind spot area D11 of the front camera 11 and a blind spot area D13 of the left camera 13 are large. On the other hand, after the interpolation processing shown in FIG. 25B, only the blind spot area Dx where the blind spot area D11 and the blind spot area D13 overlap exists, and the blind spot area of the monitoring area is reduced. Thereby, in the overlapping area a _LF between the imaging area a _F of the front camera 11 and the imaging area a _L of the left side camera 13, it is possible to more accurately acquire the three-dimensional information such as the three-dimensional coordinate value of the three-dimensional object. . Therefore, it is possible to more accurately create a plan view of a three-dimensional object used for creating a bird's eye view of a monitoring area.

When the elevation map creation process and the blind spot interpolation process of S304 in FIG. 5 are finished, the tracking process of the three-dimensional object is performed (S306). The tracking process is a process of tracking a solid object detected in an overlapping area a _LF between the imaging area a _F of the front camera 11 and the imaging area a _L of the left camera 13. For example, when the vehicle travels forward, the three-dimensional object moves rearward relative to the vehicle and moves to the outside of the overlapping region _aLF . At this time, the three-dimensional object can be accurately tracked by using the three-dimensional information of the three-dimensional object acquired when the three-dimensional object is within the overlapping region a _LF .

As the tracking process, first, using the elevation map created in S304, the three-dimensional object in the overlapping area a _LF is higher than the set threshold value and the hollow part below the set threshold value. Labeling is performed and a unique ID number is added to each. For example, as shown in FIG. 26, a unique ID number is added to the three-dimensional object 22 detected in the overlapping area a _{LF as} ID = i. Then, the same three-dimensional object 22 is tracked using the elevator map information derived from the image acquired at the next timing and the image information. As an image acquired at the next timing, for example, the next frame image is used. As a tracking method, for example, an image processing technique such as an optical flow may be used. In this tracking process, as shown in FIG. 27, when a new three-dimensional object 22a appears, a new unique ID number, for example, ID = i + 1, is added to be a tracking target.

Then, the process proceeds to S308 in FIG. 5 and the recording update process of the three-dimensional object information is performed. This recording update process is a process for recording and updating information of a three-dimensional object in the overlapping area a _LF . In this process, for example, information on the captured image of the front camera 11 that captured the three-dimensional object, position information of the three-dimensional object, and information on the direction of the line of sight from the camera with respect to the three-dimensional object are recorded and updated. The captured image information of the front camera 11 is recorded and updated as an image template. In this case, this image template is recorded as feature information of the three-dimensional object. The image template of the three-dimensional object may be an image of the entire three-dimensional object or an image obtained by partially dividing the three-dimensional object. Further, instead of the image template of the three-dimensional object, edge information, luminance pattern information, etc. of the three-dimensional object may be recorded and updated as feature information of the three-dimensional object. At this time, NULL is recorded when there is no feature information of the three-dimensional object or when feature information cannot be acquired.

  The position information of the three-dimensional object is recorded and updated based on the three-dimensional coordinate information of the three-dimensional object. The line-of-sight direction information of the three-dimensional object is recorded and updated as a line-of-sight direction vector based on, for example, relative position information of the three-dimensional object with respect to the camera.

Then, the process proceeds to S310, and a boundary flag recording process is performed. The boundary flag recording process is a process of recording a boundary flag as information of a three-dimensional object when the three-dimensional object moves to the boundary position of the outer edge portion of the overlapping region _aLF . For example, as shown in FIG. 28, when the vehicle moves forward, the three-dimensional object 22 moves relatively behind the monitoring area. When the three-dimensional object 22 moves to the boundary position of the outer edge portion of the overlapping area a _LF , a boundary flag is set as information on the three-dimensional object 22.

Thereby, it can be recognized that the three-dimensional object 22 comes out from the inside of the overlap region _aLF . Further, by setting this boundary flag, it is possible to prepare and trigger a tracking process and a motion-based stereo process based on an image captured by the left camera 13. In the boundary flag recording process of S310 in FIG. 5, if the three-dimensional object has not moved to the boundary position of the outer edge portion of the overlapping area _aLF , the control process is terminated without recording the boundary flag.

  By the way, when it is determined in S300 that the three-dimensional object has moved out of the overlapping area, tracking processing is performed (S312). This tracking process is a process of tracking a three-dimensional object based on the previously acquired feature information of the three-dimensional object and the captured image of the three-dimensional object captured by the left side camera 13.

For example, the tracking of the three-dimensional object 22 is performed using the captured image information of the front camera 11 when the three-dimensional object 22 is present in the overlapping area a _LF and the line-of-sight direction information from the front camera 11 at that time. As the captured image information of the front camera 11, the image template of the three-dimensional object 22 acquired in S308 is used. Further, as the line-of-sight direction information from the front camera 11, a line-of-sight direction vector calculated based on the position information of the three-dimensional object 22 recorded together with the image template is used.

Specifically, as shown in FIG. 29, when the tracking processing of the three-dimensional object 22 is performed at time Tn + 1, first, the previous information, that is, the relative of the three-dimensional object 22 from the left-side camera 13 at time Tn. A line-of-sight direction unit vector G _Ln is obtained from the position P _Ltn1 (X _Ltn1 , Y _Ltn1 , Z _Ltn1 ). The line-of-sight direction unit vector G _Ln may be obtained by calculating P _Ltn1 / | P _Ltn1 |.

Then, based on the relative position information from the front camera 11 of the three-dimensional object 22 having the same ID number at a past time Tl to Tn-1, required viewing direction unit vector _G LF1 _{~G LFn-1} by the similar method It is done. Then, the line _-of- sight direction vector closest to G _Ln is obtained from the line _-of- sight direction unit vectors G _{LF1 to} G _LFn−1 . The method of calculating the closest line-of-sight direction vector G _Ln, for example by the following equation, obtains a Ji, Ji may be the closest line-of-sight direction vector closest line-of-sight direction vector 1 to G _Ln.

Ji = G _Ln · G _Fi / | G _Ln | · | G _Fi |

Then, when prompted closest line-of-sight direction vector G _Ln, using an image template of the front camera 11 at the time of the set range from the time acquiring the positional information of the visual line direction vector, at time Tn + 1 of the left side camera 13 The point with the highest correlation value is searched for in the image.

  Then, it is determined that the position having the highest correlation value is the position in the image of the three-dimensional object 22, and the position information is recorded in the image processing unit 2.

  The in-image search range may be limited to the vicinity of the position predicted using the optical flow obtained from the images of the left side camera 13 at time Tn + 1 and time Tn. In this case, since the search range is narrowed, the search process can be performed quickly. Moreover, it is preferable to change the image template size of the front camera 1 based on the ratio of the distance from the front camera 11 to the three-dimensional object 22 and the distance from the left side camera 13 to the three-dimensional object 22. Thereby, the search process can be performed with high accuracy.

  At the next time Tn + 2, the line-of-sight direction information is calculated from the position information of the solid object with the same ID in the past by the same processing as at the time Tn + 1 described above, and the solid object is used by using the image template with the closest line-of-sight direction. Is searched for and the tracking of the three-dimensional object is performed.

  In the tracking process described above, the position of the three-dimensional object 22 is searched using the image template of the three-dimensional object 22 captured by the front camera 11 in the line-of-sight direction closest to the line-of-sight direction from the left-side camera 13 to the three-dimensional object 22. However, the position of the three-dimensional object 22 using a plurality of image templates of the three-dimensional object 22 imaged by the front camera 11 in a line-of-sight direction with a predetermined direction difference with respect to the line-of-sight direction from the left-side camera 13 to the three-dimensional object 22. May be searched. For example, a position with a high correlation value may be searched for each of a plurality of image templates with respect to the captured image of the left camera 13, and a position with the highest correlation value may be set as the position of the three-dimensional object 22.

  And it transfers to S314 of FIG. 5 and a three-dimensional information derivation process is performed. The three-dimensional information deriving process is a process of deriving three-dimensional information of a three-dimensional object by using a motion-based stereo method using an image captured by the left camera 13. For example, when the three-dimensional information of the three-dimensional object 22 is derived at time Tn + 1, the position in the captured image of the three-dimensional object 22 at time Tn + 1, the position in the captured image of the three-dimensional object 22 at time Tn, and the time difference between time Tn and time Tn + 1. The three-dimensional position of the three-dimensional object 22 is derived by the motion-based stereo technique using the captured images at times Tn + 1 and Tn based on the vehicle speed.

  At this time, it is assumed that the elevation map created when the three-dimensional object 22 exists in the overlapping area of the front camera 11 and the left-side camera 13 is on the extension line of the movement locus of the three-dimensional object 22 on the elevation map. Thus, the three-dimensional position of the three-dimensional object 22 at time Tn + 1 may be estimated from the intersection with the position of the three-dimensional object 22 in the captured image of the left camera 13 at time Tn + 1. By estimating the three-dimensional position of the three-dimensional object 22 by this method, the motion-based stereo process can be omitted.

  Further, the accuracy of deriving the three-dimensional position of the three-dimensional object 22 may be improved by using together the movement trajectory data of the three-dimensional object 22 in this elevation map and the position data derived by the motion-based stereo. For example, the reliability of one data of the three-dimensional object 22 can be improved by confirming the position data derived by one method with the position data derived by the other method. Further, the position data of the three-dimensional object 22 may be obtained by taking the average of the position data derived by both methods.

Then, the process proceeds to S316, and an elevation map creation process is performed. The elevation map creation process is a process for creating elevation map of the imaging area a _L excluding the overlapping region a _LF. For example, the position of the three-dimensional object 22 is set using the data acquired when the position of the three-dimensional object 22 is set based on the three-dimensional position derived in S314 and the three-dimensional object 22 exists in the overlapping area _aLF . Three-dimensional information and luminance information are set. Thus, elevation maps excluding the overlapping region a _LF imaging area a _L is created.

  It should be noted that it is preferable to interpolate a portion having no three-dimensional information in the three-dimensional object 22 or the like from a neighboring region having the three-dimensional information using a technique such as linear interpolation.

  In the tracking process in S <b> 312, the range in which the image template of the front camera 11 can be used is the time when the three-dimensional object 22 is in a position directly beside the left camera 13. For this reason, when the three-dimensional object 22 moves rearward from the position just beside the left-side camera 13, tracking processing and three-dimensional information derivation processing are performed using a motion-based stereo method based on optical flow using time-series images. What is necessary is just to create an elevation map.

And it transfers to S318 and a cooperation process is performed. In cooperative processing, the three-dimensional object takes cooperation when entering the overlap region a _LB from a single imaging area a _L of the left side camera 13 _(overlapping region a _LF and overlapping area imaging region excluding a _LB a _L) process is there. For example, the captured image of the left-side camera 13 when the three-dimensional object 22 is in the single imaging area a _L of the left-side camera 13 is recorded in advance as an image template, and the three-dimensional object 22 is recorded in the imaging area a _{B of the} rear camera 14. By using the image template of the left side camera 13 when entering, it is possible to increase the accuracy of the three-dimensional derivation result by the stereo view of the rear camera 14 and the left side camera 13. In addition, the tracking process can be performed using the captured image of the rear camera 14 and the image template of the left camera 13 even at a distance where the three-dimensional object 22 cannot be tracked from the left camera 13 due to being too far from the left camera 13.

  Then, the process proceeds to S320, and a deletion process is performed. The deletion process is a process of deleting an image template such as a three-dimensional object or position information. For example, when the three-dimensional object 22 moves away from the field of view of the front camera 11, unnecessary image templates and position information data are deleted.

  When the deletion process is finished, the image process for the front camera image and the left camera image shown in FIG. 5 is finished.

  The image processing for the front camera image and the right side camera image in S40 in FIG. 4 is performed in the same manner as the image processing for the front camera image and the left side camera image described above. That is, by performing processing by replacing the left-side camera image with the right-side camera image, it can be executed in the same manner as the image processing for the front camera image and the left-side camera image.

  Further, the image processing for the rear camera image and the left camera image in S60 of FIG. 4 is performed in the same manner as the image processing for the front camera image and the left camera image described above. That is, by performing processing by replacing the front camera image with the rear camera image, it can be executed in the same manner as the image processing for the front camera image and the left camera image.

  Further, the image processing for the rear camera image and the right side camera image in S70 of FIG. 4 is performed in the same manner as the image processing for the front camera image and the left side camera image described above. That is, by replacing the front camera image with the rear camera image and replacing the left side camera image with the right side camera image, processing can be performed in the same manner as the image processing for the front camera image and the left side camera image.

  As described above, according to the periphery monitoring device according to the present embodiment, the feature information of the object is acquired while acquiring the feature information such as the image template of the three-dimensional object from the captured image of the camera that first captured the three-dimensional object. By acquiring the first line-of-sight direction with respect to the object from the camera at the time when the three-dimensional object is imaged, when the area where the three-dimensional object is imaged is relatively moved from the overlapping area of the imaging area to the single imaging area of the other camera, The object is tracked based on the feature information of the object imaged in the first line-of-sight direction close to the object's line-of-sight direction and the captured image of the three-dimensional object imaged by another camera. Thereby, the position of the object in the captured image of the other camera can be accurately tracked.

  Further, according to the periphery monitoring device according to the present embodiment, the three-dimensional information of the three-dimensional object existing in the overlapping region is acquired by stereo vision based on the captured images of the two cameras having the overlapping imaging region, and tracking is performed by the tracking process. Single imaging area where it is difficult to obtain three-dimensional information of a three-dimensional object by stereo vision by acquiring three-dimensional information of an object existing in the single imaging area of the camera based on the result and three-dimensional information obtained in advance by stereo vision Even if an object exists in the three-dimensional object, the three-dimensional information of the three-dimensional object can be reliably acquired.

  In addition, according to the periphery monitoring device according to the present embodiment, a blind spot area in which one of the two cameras cannot be captured and the three-dimensional information of the three-dimensional object cannot be obtained is identified, and one of the blind spot areas is When it is an imageable area and the luminance changes continuously, the blind spot area is interpolated based on the three-dimensional information of the peripheral area of the blind spot area. Thereby, it is possible to interpolate three-dimensional information that cannot be obtained by stereo viewing by reducing the blind spot area.

  Further, according to the periphery monitoring device according to the present embodiment, when acquiring the three-dimensional information of the three-dimensional object by stereo vision based on the captured images of the two cameras, refer to the three-dimensional information of the three-dimensional object acquired previously. By doing so, it is possible to accurately derive the three-dimensional information of the object by stereo vision.

  In addition, according to the periphery monitoring apparatus according to the present embodiment, each camera is installed in the vehicle 3 that is a moving body, images the surroundings of the vehicle 3, and the relative position coordinate value of the three-dimensional object with respect to the vehicle 3 based on the captured image. As a result, the relative position of the three-dimensional object can be easily grasped even when the vehicle 3 moves and the three-dimensional object deviates from the overlapping area of the imaging regions.

  Also, by creating a bird's-eye view around the vehicle based on the relative position coordinate value and three-dimensional coordinate value of the three-dimensional object around the vehicle, the three-dimensional object can be displayed at an accurate position, and the bird's-eye view with less distortion of the three-dimensional object can be displayed. Can be created.

  Furthermore, according to the periphery monitoring device according to the present embodiment, when it is determined whether the vehicle 3 and the three-dimensional object are in contact based on the relative position coordinate value of the three-dimensional object, and it is determined that the vehicle 3 is in contact with the three-dimensional object. To avoid that contact. Thereby, even when a three-dimensional object exists in an area other than the overlapping area where the imaging areas of the cameras overlap, it is possible to accurately avoid contact with the vehicle 3 based on the relative position of the three-dimensional object.

  In addition, embodiment mentioned above shows an example of the periphery monitoring apparatus which concerns on this invention. The periphery monitoring device according to the present invention is not limited to the periphery monitoring device according to these embodiments, and the periphery monitoring device according to the embodiment is modified or changed without changing the gist described in each claim, or It may be applied to other things.

It is a block diagram which shows the structure outline | summary of the periphery monitoring apparatus which concerns on embodiment of this invention. It is a top view which shows the arrangement | positioning of the camera and the imaging area of a camera in the periphery monitoring apparatus of FIG. It is a figure which shows an example of the image obtained with the camera in the periphery monitoring apparatus of FIG. It is a flowchart which shows operation | movement of the periphery monitoring apparatus of FIG. It is a flowchart which shows the image process in the front camera image and left side camera image of the periphery monitoring apparatus of FIG. It is explanatory drawing about the setting of the image processing range of the periphery monitoring apparatus of FIG. It is explanatory drawing about the production | generation of the virtual viewpoint image in the periphery monitoring apparatus of FIG. It is the figure which showed the virtual viewpoint image produced | generated based on the captured image. It is explanatory drawing about the production | generation of the virtual viewpoint image in the periphery monitoring apparatus of FIG. It is a figure explaining an epipolar polar line. It is a figure explaining the spatial position calculation principle by stereo vision. It is a figure explaining conversion from a sensor coordinate system to a vehicle coordinate system. It is explanatory drawing of an elevation map. It is explanatory drawing of the derivation process of a blind spot area | region in the periphery monitoring apparatus of FIG. It is explanatory drawing of the derivation process of a blind spot area | region in the periphery monitoring apparatus of FIG. It is a flowchart which shows the blind spot derivation | leading-out process in the periphery monitoring apparatus of FIG. It is explanatory drawing of the blind spot area | region by a solid object. It is explanatory drawing of the blind spot area | region by a solid object. It is explanatory drawing of the interpolation process of the blind spot area | region in the periphery monitoring apparatus of FIG. It is explanatory drawing of the interpolation process of the blind spot area | region in the periphery monitoring apparatus of FIG. It is explanatory drawing of the interpolation process of the blind spot area | region in the periphery monitoring apparatus of FIG. It is explanatory drawing of the interpolation process of the blind spot area | region in the periphery monitoring apparatus of FIG. It is explanatory drawing of the interpolation process of the blind spot area | region in the periphery monitoring apparatus of FIG. It is explanatory drawing of the interpolation process of the blind spot area | region in the periphery monitoring apparatus of FIG. It is explanatory drawing of the interpolation process result of a blind spot area | region in the periphery monitoring apparatus of FIG. It is explanatory drawing of the tracking process in the periphery monitoring apparatus of FIG. It is explanatory drawing of the tracking process in the periphery monitoring apparatus of FIG. It is explanatory drawing of the boundary flag recording process in the periphery monitoring apparatus of FIG. It is explanatory drawing of the tracking process in the periphery monitoring apparatus of FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Perimeter monitoring apparatus, 2 ... Image processing part, 11 ... Front camera, 12 ... Right side camera, 13 ... Left side camera, 14 ... Back camera, 22 ... Three-dimensional object, 30 ... Output part.

Claims (9)

First imaging means for imaging the first imaging region;
Second imaging means for imaging a second imaging area having a single imaging area that does not overlap with the first imaging area;
Feature information acquisition means for acquiring feature information of an object imaged by the first imaging means;
First gaze direction acquisition means for acquiring a first gaze direction with respect to the object from the first imaging means when acquiring the feature information of the object;
A range of a predetermined direction difference with respect to the line-of-sight direction from the second imaging means to the object when the area where the object is imaged is relatively moved from the first imaging area to the single imaging area in the second imaging area Object tracking means for tracking the object based on feature information of the object imaged in the first line-of-sight direction and the captured image captured by the second imaging means,
Perimeter monitoring device with Correlation value calculation for calculating the correlation value between the feature information of the object imaged in the first line-of-sight direction closest to the line-of-sight direction from the second imaging unit to the object and the image area in the captured image of the second imaging unit With means,
The object tracking means is configured to track the object using the point having the highest correlation value as the position of the object;
The periphery monitoring device according to claim 1. A plurality of feature information of the object imaged in the first line-of-sight direction within a predetermined direction difference range with respect to the line-of-sight direction from the second imaging unit to the object, and an image area in the captured image of the second imaging unit Correlation value calculating means for calculating the correlation value with
The object tracking means is configured to track the object using the point having the highest correlation value as the position of the object;
The periphery monitoring device according to claim 1. A first object tertiary that obtains three-dimensional information of an object existing in an overlapping region where the first imaging region and the second imaging region overlap by stereo vision based on the captured images of the first imaging unit and the second imaging unit. Original information acquisition means;
A second object that acquires three-dimensional information of an object existing in a single imaging region of the second imaging region based on the tracking result by the object tracking unit and the three-dimensional information acquired by the first object three-dimensional information acquisition unit. Three-dimensional information acquisition means;
The periphery monitoring device according to any one of claims 1 to 3, further comprising: When the first object three-dimensional information acquisition means performs a stereo view, the blind spot area is blocked by the object and cannot be imaged by at least one of the first imaging means and the second imaging means and cannot acquire three-dimensional information. Blind spot identifying means for identifying
The blind spot area based on the three-dimensional information of the peripheral area of the blind spot area when the brightness is continuously changing in the blind spot area that is imageable by the first imaging means or the second imaging means. Blind spot interpolation means for interpolating
A blind spot interpolation area recording means for recording an area that can be interpolated by the blind spot interpolation means and a non-interpolable area;
The periphery monitoring device according to claim 4, further comprising: A third imaging means for imaging a third imaging area having an overlapping area overlapping the second imaging area;
Acquired by the second object three-dimensional information acquisition means when the area where the object is imaged is relatively moved from the single imaging area of the second imaging area to the overlapping area of the second imaging area and the third imaging area. First object three-dimensional information acquisition means for acquiring three-dimensional information of the object by stereo vision based on the captured images of the second imaging means and the third imaging means with reference to the three-dimensional information of the object,
The periphery monitoring device according to claim 4 or 5, comprising: The first imaging means and the second imaging means are installed on a moving body and images the periphery of the moving body,
The first object three-dimensional information acquisition means and the second object three-dimensional information acquisition means acquire the relative position coordinate value of the object with respect to the moving body as three-dimensional information of the object;
The perimeter monitoring device according to any one of claims 4 to 6.   The periphery monitoring device according to claim 7, further comprising an overhead view creation unit that creates an overhead view around the moving body based on a relative position coordinate value of the object. Contact determining means for determining whether or not the moving body and the object are in contact based on a relative position coordinate value of the object;
Contact avoiding means for avoiding contact when the contact determining means determines to contact;
The periphery monitoring device according to claim 7, comprising:

Images