JP2010178018A - Device for displaying periphery of vehicle by image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide image processing preventing the states distorting images of the peripheries and joints of sensed images and synthesizing the images without generating a discontinuity although the periphery of a vehicle is image-sensed for supporting a driver and the images are synthesized and displayed. <P>SOLUTION: A device loaded on the vehicle includes one image-sensing means which caries out image-sensing of the periphery of the vehicle and acquiring an original image. In the original image, the images inside regions positioned at the left and the right of a central region respectively are subject to a variable zoomed-magnification at a second magnification indicating an extension as the images in the central region in the horizontal direction are subject to the variable zoomed-magnification by a first magnification indicating a contraction while the images in joint regions among the central region and the side regions are subject to the variable zoomed-magnification by using a magnification gradually changing from the first magnification to the second magnification towards the side regions from the central region, and the variably zoomed-magnified images are displayed on a display. The discontinuity in the images is avoided while distortions in the side regions of the images are suppressed even when the images are sensed by a lens having a wide field angle by such a zoomed-magnified processing. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an apparatus for displaying an image of the periphery of a vehicle, and more specifically to an apparatus for processing and displaying a captured image so that an occupant can easily view the image.

  2. Description of the Related Art Conventionally, there has been proposed an apparatus that mounts a camera on a vehicle and processes and displays an image obtained by capturing the periphery of the vehicle so that it can be easily seen by a passenger.

  The following Patent Document 1 discloses an apparatus that displays an image of the rear side of a vehicle so that a sense of distance can be easily grasped. According to this device, a mark attached to the host vehicle is displayed below the monitor screen. In one example, the mark represents the vehicle width of the host vehicle.

  In Patent Document 2 below, a camera is mounted on at least one of the door mirrors, and the rear side so that the object in the image obtained by the camera is approximately the same size as the object reflected on the door mirror. An apparatus is disclosed in which image data in a region is enlarged and image data in another region is compressed and displayed.

Japanese Patent Laid-Open No. 11-5501 JP 2008-22125 A

  In order to assist the driver, a camera for imaging the periphery of the vehicle is attached. For example, a camera is attached to the rear of the vehicle in order to visually recognize the rear of the vehicle on the display device. By visually recognizing the rear image on the display device, the driver can perform the reverse operation more appropriately.

  Referring to FIG. 10, several scenes (situations) in which the vehicle V moves backward are shown. (A) shows a scene where the vehicle V comes out from the garage while moving backward, (b) shows a scene where the vehicle V is moved backward from a predetermined parking space, and (c) shows a predetermined scene provided on the side of the road. The scene which makes the vehicle V reverse from a parking space is shown. As shown in these scenes, exiting the parking space while retreating is a common practice in some countries. In any scene, the camera 101 is mounted in the center of the rear part of the vehicle V, and reference numeral 103 indicates an imaging region of the camera 101. Here, the horizontal angle of view (viewing angle) of the camera 101 is 130 degrees in this example.

  As shown in the figure, the person 111 and the other vehicle 113 are not displayed on the display device because they are moving toward the vehicle V and are present in an area that is not captured by the camera 101 (a blind spot area). . Therefore, it is difficult for the driver to visually recognize the object located in the blind spot area. Further, it is often difficult to visually recognize an object existing in such a blind spot region even visually.

  It is conceivable to mount a camera with a wider angle of view so that a wider area can be visually recognized. A camera having a wide-angle lens such as a so-called fish-eye lens can widen the horizontal angle of view to 180 degrees or more, for example.

  However, when a wide-angle lens such as a fish-eye lens is used, the imaged area is widened, but when an image of such a wide imaging area is displayed on the display device, the image of the peripheral (side) portion of the captured image is displayed. It becomes distorted. Since such distortion is difficult to visually recognize, various methods have been proposed to correct it. One of them is to use viewpoint conversion processing. That is, the captured image is converted into a plurality of images viewed from several viewpoints, and the converted plurality of images are connected (synthesized) to generate one image. Since distortion is removed from the generated image, it is easy to visually recognize the peripheral image.

  However, when such viewpoint conversion processing is performed, images generated from different viewpoints are joined together, so that discontinuities occur in the joint portions of the images. Even if the peripheral distortion is removed, such discontinuities cause the viewer to feel uncomfortable.

  Conventionally, a plurality of image pickup devices are provided, and a plurality of images picked up by these image pickup devices are connected to display a composite image. If a wide-angle lens is not used, one composite image can be generated without causing distortion. However, since this method requires a plurality of imaging devices, the cost becomes high.

  Accordingly, one object of the present invention is to use a single image pickup device and pick up an image without causing discontinuity even in the joint region between the center region of the picked-up image and its side region. It is to propose an image processing for processing the image.

  According to one aspect of the present invention, an apparatus mounted on a vehicle and displaying an image of the periphery of the vehicle has one imaging means for capturing an original image by imaging the periphery of the vehicle, and the original image , The image of the central area in the horizontal direction is scaled at a first magnification indicating reduction, and the images of the side areas respectively positioned on the left and right of the central area are scaled at a second magnification indicating enlargement. In addition, for the image of the joint area between the central area and the lateral area, gradually from the first magnification to the second magnification from the central area toward the lateral area. Image scaling means for changing the magnification using a changing third magnification, and means for displaying an image scaled by the image scaling means on a display device.

  According to the present invention, the image of the lateral region positioned on the left and right of the central region is obtained by scaling the image of the central region in the horizontal direction with the first magnification indicating reduction with respect to the captured original image. Is scaled at the second magnification indicating enlargement, the degree of image recognition in the side region can be improved. Further, for the image of the joint area between the central area and the lateral area, the image gradually changes from the first magnification to the second magnification from the central area toward the lateral area. Therefore, it is possible to prevent discontinuity from occurring in the central region to the lateral region. Therefore, the user can visually recognize the image on the display device without a sense of incongruity. In addition, since there is one imaging device and a wide imaging area can be displayed from a single captured image in a more natural form, the cost can be reduced compared to combining a plurality of images using a plurality of imaging devices. Can be reduced.

  In one embodiment, the third magnification is a magnification that changes in a quadratic curve. Therefore, the magnification can be gradually changed while maintaining continuity.

  According to an embodiment of the present invention, the first magnification, the second magnification, and the first magnification to the second magnification so that the average magnification of the central region, the side region, and the joint region is 1 ×. The third magnification that gradually changes to the magnification of is set.

  According to the present invention, since the average magnification of the entire captured image is 1, the entire image of the captured area can be displayed on the display device without exception. Therefore, even when an image is captured using a wide-angle lens, the captured image of a wide area can be viewed on the display device.

1 is a block diagram of an image display device according to one embodiment of the present invention. The figure which shows an example of the angle of view of an imaging device, the model of the image acquisition by a fisheye lens, and a captured image according to one Example of this invention. The figure which shows an example of the image by which the angle of view of the imaging device according to a prior art, the outline | summary of the viewpoint conversion process of the image acquired with the fish-eye lens, and the viewpoint conversion process. 3 is a flowchart of image processing according to one embodiment of the present invention. The graph showing the magnification used for the scaling process in the horizontal direction of the image according to one Example of this invention. The figure which shows the mode of the scaling process in the horizontal direction of the image according to one Example of this invention. The figure for demonstrating the method of determining the value of (beta) in FIG. 5, and the quadratic curve of a joint area | region in accordance with one Example of this invention. The graph showing the magnification used for the scaling process in the orthogonal | vertical direction of the image according to one Example of this invention. The figure which shows the comparison with the image by which the scaling process according to one Example of this invention was performed, and the image produced | generated by the conventional viewpoint conversion process. The figure for demonstrating the visual recognition of the periphery of a vehicle.

  Next, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram of an apparatus mounted on a vehicle and displaying an image around the vehicle according to an embodiment of the present invention.

  The imaging device 10 is a device for imaging the periphery of the vehicle. In this embodiment, the imaging device 10 is a camera attached to the center in the vehicle width direction at the rear of the vehicle so as to capture the rear of the vehicle. The imaging device 10 includes a wide-angle lens with a wide angle of view. In this embodiment, the imaging apparatus 10 is realized by a camera including a fisheye lens whose horizontal field angle is 180 degrees.

  The image processing device 12 can be realized in an electronic control unit (ECU) mounted on a vehicle. The ECU is a computer including a central processing unit (CPU) and a memory. Alternatively, instead of providing the image processing device 12 separately from the imaging device 10, the image processing function may be realized integrally in the imaging device 10.

  The display device 14 is provided in the vehicle so that the driver can visually recognize it. In this embodiment, a display device that displays map information and the like of the navigation system is used. The image picked up by the image pickup device 10 is processed by the image processing device 12, and the resulting image is displayed on the display device 14. Of course, the image itself (referred to as an original image) captured by the imaging device 10 can be displayed on the display device 14.

  With reference to FIG. 2, the image imaged with the camera (imaging device) 10 provided with the fisheye lens is demonstrated easily. As shown to (a), the imaging area 20 of the camera 10 attached to the center of the rear part of the vehicle V1 has spread at an angle of 180 degrees in the horizontal direction.

  (B) is the figure which modeled the image acquisition by a fish-eye lens simply in the XYZ coordinate system using the virtual spherical surface 23. FIG. The Z axis represents the central axis of the fisheye lens. The light from the imaging object enters the phantom spherical surface 23 perpendicularly and forms an image where it falls vertically onto a circular plane 25 having a radius R corresponding to the imaging surface of the fisheye lens.

  (C) is an example of an original image captured by the camera 10. The vehicle V2 is imaged at the periphery (side area) of the image. An arrow 31 indicates a locus on the captured image when the vehicle V2 travels. As shown in (b), when a picture is taken using a fisheye lens, the object is captured in a spherical surface, so that an object with a wider field of view can be imaged. However, since the object is captured by the spherical surface, the peripheral image of the captured image is distorted as compared to the central image and is captured small.

  In order to remove such distortion, the viewpoint conversion process has been conventionally used as described above. This will be briefly described with reference to FIG. 3. FIG. 3 (a) shows the same view as FIG. 2 (a). The area A1 to A3 is divided into three, and three viewpoints, that is, a viewpoint for the area A1, a viewpoint for the area A2, and a viewpoint for the area A3 are set based on the position of the camera 10. Three viewpoint images are generated from the captured original image.

  (B) represents a simple modeling of the fisheye lens image acquisition as described above. Here, N1 to N3 indicate vectors representing the three viewpoints, respectively. Here, since it is divided into three in the horizontal direction, the vector N2 is along the Z-axis. The vectors N1 and N3 are along the XZ plane, and the angle θz with respect to the Z axis is 60 degrees. By setting planes P1 to P3 perpendicular to the vectors N1 to N3 at points where the vectors N1 to N3 intersect the phantom spherical surface 23, and obtaining images projected on these planes P1 to P3, a viewpoint is obtained. Each image is generated. The coordinates on the planes P1 to P3 are associated with pixels on the screen on the display device, and one image can be generated on the display device by connecting the planes P1 to P3.

  In this way, viewpoint conversion is performed on the original image in FIG. 2C to generate three images, and an image obtained by connecting these images is shown in FIG. Lines 41 and 43 indicate the joint portion of the image. Compared with the original image in FIG. 2C, the distortion in the peripheral portion is corrected, but discontinuities occur in the joints 41 and 43. The vehicle V <b> 2 is imaged to the right of the joint 43. As shown in FIG. 2 (c), the vehicle V2 travels with an essentially smooth trajectory, but in FIG. 3 (c), the vehicle V2 is discontinuous at the joint 43 as shown by the arrow 45. It has become a movement. This is due to the stitching of images generated from different viewpoints.

  As described above, when the synthesized image is generated using the viewpoint conversion process, the distortion in the peripheral portion is corrected, but discontinuity occurs in the joint portion of the image. This causes a sense of incongruity for the occupant who visually recognizes. In particular, when the object is moving across the joint, the trajectory of the movement is not smooth at the joint, and an unnatural feeling is felt.

  Therefore, the present invention corrects the difficulty in recognizing an object due to distortion in the peripheral part without performing viewpoint conversion, and corrects the inconvenience in the area (joint area) between the central part and the peripheral (side) part. We propose an image generation method that eliminates continuity.

  FIG. 4 shows a flow of processing executed by the image processing apparatus 12. In step S <b> 1, the image processing device 12 receives an image (original image) captured by the camera 10. In step S3, a scaling process in the horizontal direction is executed on the original image. In step S5, a scaling process in the vertical direction is executed. In step S7, the image scaled in steps S3 and S5 is displayed on the display device 14. Note that steps S3 and S5 may be performed in the reverse order. Therefore, the scaling process in the horizontal direction may be performed after the scaling process in the vertical direction.

  Details of the horizontal scaling processing executed in step S3 will be described. FIG. 5 shows a graph of a magnification curve 51 (represented by a thick solid line) used for horizontal scaling processing. The horizontal axis represents the angle of view in the horizontal direction. As described above, since the angle is 180 degrees in this embodiment, the angles of view of 90 degrees and −90 degrees are scaled with respect to the origin O (0 degree). The vertical axis represents the magnification. The magnification value “1” represents the same magnification, α is larger than the value 1 and therefore represents a magnification for enlargement, and β is smaller than the value 1 and represents a magnification for reduction.

  Here, the original image is divided into a central area, a side area, and a joint area according to the angle of view in the horizontal direction. In the figure, the angle of view range of -D1 to D1 is set in the central area, the angle of view range of -D2 to -90 and D2 to 90 is set to the side area, and the image angles of -D1 to -D2 and D1 to D2 are set. The corner range is set to a region between the central region and the side region, that is, a joint region.

  The magnification curve 51 is constant at the value β when the angle of view is zero to ± D1, and is constant at the value α when the angle of view is between ± D2 and ± 90. While the angle of view is between ± D1 and ± D2, it is set to gradually change from the value β to the value α. In this embodiment, the magnification curve 51 is set to change in a quadratic curve between ± D1 and ± D2.

  Thus, in the central area of the captured image, the image is reduced by being multiplied by β, and in the lateral area, it is enlarged by being multiplied by α. In the joint region between the central region and the side region, the magnification gradually changes from β times to α times. Therefore, in a lateral region where distortion occurs, by enlarging, an image captured small due to distortion can be converted into an enlarged image, and thus the recognition degree of the object can be improved.

  However, simply enlarging increases the area of the resulting image and does not fit on the display screen on the display device 14. Therefore, by reducing in the central area, the enlarged portion and the reduced portion are offset. As a result, the entire imaged area can be displayed on the display device 14. In order to display the entire imaged region on the display device 14 without fail, the magnification curve 51 is preferably set so that the average magnification of the entire image is 1. As a result, the area of the image does not change before and after the scaling process, so that the captured area can be displayed on the display device 14 without exception.

  Furthermore, since the magnification gradually changes in the joint region, it is possible to avoid discontinuity of the image between the central region and the side region. Therefore, even if the object moves between the center area and the side area, the locus of the object becomes smooth and does not cause discontinuity.

  The above values D1 and D2 are predetermined values. It is preferable to examine a region where a phenomenon in which an object is imaged small by distortion is generated by simulation or the like, and set values D1 and D2 based on this. For example, an angle of view specified as starting to generate distortion from the central area toward the side area is set as a value D1, and an angle of view defining a side area specified as having a high degree of distortion is set as a value D2. Of course, the values D1 and D2 are set so that the simulation of the scaling process is performed using the determined magnification curve 51 (this determination method will be described later), and the image displayed as a result is easier to visually recognize. Can be adjusted. In this example, D1 = 50 degrees and D2 = 70 degrees. When D1 = 50 degrees and D2 = 70 degrees, the angle of view D3 when the magnification is 1 in the magnification curve 51 is about 65 degrees.

  FIG. 6 shows how the pixels of the original image are expanded in the horizontal direction in the scaling process in the horizontal direction. In the original image, the area of each pixel is a square, and the horizontal and vertical lengths are the same. By the scaling process in the horizontal direction, the horizontal length of each pixel area in the central area of the image is reduced by a factor of β, and as a result, the horizontal length of each pixel area is smaller than the vertical length. It has become. The horizontal length of each pixel region in the lateral region of the image is enlarged by α, and as a result, the horizontal length of each pixel region is larger than the vertical length. In the joint region from the central region to the side region, the magnification gradually changes from β times to α times, and accordingly, the horizontal length of each pixel region gradually increases.

  The value of α is determined in advance based on simulations and experiments in consideration of the degree to which the distortion of the object in the lateral region is corrected. In this embodiment, the value of α is a value between 1.0 and 1.5. Set to take. Preferably, when an object to be detected such as a vehicle or a pedestrian is imaged in the side region and the display screen of the display device 14 is viewed from a point 1 meter away, the object is at least 5 in the horizontal direction. It is determined so that it can be seen in a millimeter (mm) size. This is based on the knowledge that an object smaller than 5 mm square is generally difficult to see when a human visually recognizes the screen.

  As described above, the value of β is preferably set so that the average magnification of the entire image is 1. In order to obtain a magnification of 1 times on average, the value of β and the magnification curve 51 in the joint area need to be set so that the amount of reduction and the amount of enlargement are equal in the area of the image region. There is. Here, this setting method will be described.

  FIG. 7 is an enlarged view of a region having a positive angle of view in the graph shown in FIG. In this embodiment, D1 and D2 are 50 degrees and 70 degrees, respectively.

The magnification curve 51 in the joint region at the angles of view D1 to D2 is a quadratic curve that passes through the points A and B. The coordinates of point A are (50, β), and the coordinates of point B are (70, α). Therefore, the magnification y in the joint area is expressed as in Expression (1). Here, θ represents the angle of view.

The magnification curve 51 has a value β between the angles of view 0 to 50 degrees, a value y between the angles of view 50 to 70, and a value α between the angles of view 70 to 90. This magnification curve 51 (represented by f (θ)) is integrated over an angle of view of 0 to 90 degrees as shown in equation (2).

An average magnification of 1 indicates that a value obtained by integrating the magnification curve 51 from 90 degrees to 90 degrees is 90 degrees. Therefore, equation (3) needs to hold.

  As described above, the value of α is determined in advance based on simulation or the like. Therefore, by substituting the value of α into the equation (3), the value of β is calculated and the quadratic curve (y) in the joint region is calculated.

For example, when α = 1.1, the value of β and the quadratic curve (y) are as follows.

When α = 1.5, the value of β and the quadratic curve (y) are as follows.

The quadratic curve in the joint area of the negative field angle can be calculated from the quadratic curve of the positive area. In this way, by predetermining the values of α, D1, and D2, β and a quadratic curve are calculated, so that a magnification curve 51 as shown in FIG. 5 can be determined. A map defining such a magnification curve 51 can be stored in the memory of the image processing device 12 in advance. For each pixel of the received image, the image processing device 12 obtains a magnification corresponding to the angle of view of the pixel from the map, and scales the magnification by applying the magnification in the horizontal direction of the pixel. Thus, an image obtained by scaling the original image in the horizontal direction is generated.

  Next, the vertical scaling process executed in step S5 will be described. In one embodiment, scaling is performed in the vertical direction at the same magnification as in the horizontal direction. In this case, for example, if a certain pixel is multiplied by β in the horizontal direction, it is similarly multiplied by β in the vertical direction. Thus, the pixels are scaled in the horizontal and vertical directions.

  In an alternative embodiment, for the vertical direction, a map as shown in FIG. 8 may be used and scaled at a constant magnification α as shown in the magnification curve 53 (in this case, the magnification curve 53 is actually A straight line). As described above, α is larger than the value 1 and is a magnification for enlarging. The horizontal axis of this map is the angle of view in the horizontal direction, and the vertical axis represents the magnification in the vertical direction of the pixels corresponding to the angle of view in the horizontal direction. Thereby, in the side region, not only the distortion in the horizontal direction but also the distortion in the vertical direction can be suppressed, and thus the object can be more easily seen.

  In this case, the enlargement process is also performed on the central region of the image. As a result, the area of the image becomes larger after the scaling process than before the scaling process in the vertical direction. However, the upper part of the captured image is mainly a part that does not hinder driving assistance even if it is cut, such as the sky or the upper part of a high-rise building. Therefore, for example, a portion that protrudes from the display screen of the display device 14 may be cut and an image after the cut may be displayed on the display device 14.

  In this way, the image scaled in the horizontal and vertical directions is displayed on the display device 14. By determining the value of α in the horizontal and vertical scaling processes so that the object can be seen with a size of at least 5 mm when viewing the display screen from a point 1 meter away as described above, the user can In the lateral region, an object having a size of at least 5 mm can be seen in both the horizontal and vertical directions, thus ensuring good visual recognition.

  Referring to FIG. 9, (a) is an example of an image that is scaled according to the present invention and displayed on the display device 14. Here, in the vertical direction, the scaling process is the same as the horizontal direction. Has been done. FIG. 3B is an example of an image that is synthesized according to the conventional viewpoint conversion process and displayed on the display device, as in FIG. In both images, the vehicle V3 is captured in the left side region. In both images, since the distortion in the side region is corrected, the vehicle V3 is displayed so as to be visible.

  The vehicle V3 actually travels so as to draw a smooth locus behind the host vehicle on which the camera is mounted. Referring to (a) with respect to this point, the traveling locus of the vehicle V3 is smooth as shown by the arrow 61, so that an occupant who visually recognizes the image does not cause a sense of incongruity in the behavior of the vehicle V3. On the other hand, in (b), as described above, discontinuity occurs in the joint portion 65 of the original image to be synthesized. Accordingly, the behavior of the vehicle V3 becomes discontinuous at the joint 65 as shown by the arrow 67, and is visually recognized as turning around the corner, thereby causing the passenger to feel uncomfortable.

  Thus, according to the present invention, in the resulting image, distortion in the lateral region is corrected and discontinuity in the image can be eliminated. While being able to visually recognize favorably, it can prevent that the behavior of a subject becomes discontinuous. Therefore, an imaging region with a wide angle of view can be displayed on the display device without causing the passenger to feel uncomfortable. Further, only one imaging device is required, and a more natural image of a wide imaging area as described above can be displayed based on an image obtained from one imaging device, so a plurality of imaging devices are used. There is no need, and the cost can be reduced.

  In the above embodiment, a fisheye lens having a horizontal angle of view of 180 degrees is used as the lens of the camera. However, the present invention is not limited to this, and the case where the angle of view is smaller than 180 degrees or larger than 180 degrees. However, the present invention is applicable. In other words, when a wide-angle lens is used, a phenomenon occurs in which the object is imaged smaller due to distortion in the peripheral part of the captured image than in the central part. The present invention can be preferably applied. The values of α, D1 and D2 in the map as shown in FIG. 5 are preferably determined in accordance with the size of the angle of view of the camera in the horizontal direction. As described above, the value of α is displayed on the display screen from a point away from, for example, 1 meter so that an object to be detected (for example, another vehicle or a pedestrian) existing in the side area can be satisfactorily recognized. Is determined through simulation or the like so that it can be seen in a size of at least 5 mm square when the image is visually recognized, and the values of D1 and D2 are determined in accordance with the angle of view range that is obstructive to be visually recognized due to distortion. It is better.

  As described above, specific embodiments of the present invention have been described. However, the present invention is not limited to these embodiments.

DESCRIPTION OF SYMBOLS 10 Imaging device 12 Image processing apparatus 14 Display apparatus

Claims (3)

An apparatus mounted on a vehicle for displaying an image around the vehicle,
One imaging means for capturing the original image by imaging the periphery of the vehicle;
In the original image, the image of the central area in the horizontal direction is scaled at the first magnification indicating reduction, and the images of the lateral areas respectively positioned on the left and right of the central area are expanded at the second magnification. For the image of the joint area between the central area and the lateral area, the first magnification to the second magnification are applied from the central area toward the lateral area. Image scaling means for scaling using a third magnification that gradually changes to
Means for displaying an image scaled by the image scaling means on a display device;
An apparatus comprising: The third magnification that gradually changes from the first magnification to the second magnification is a magnification that changes in a quadratic curve.
The apparatus of claim 1. Gradually from the first magnification, the second magnification, and the first magnification to the second magnification such that the average magnification of the central region, the side region, and the joint region is 1 ×. The third magnification to change is set,
The apparatus according to claim 1 or 2.

Images