JP2010252015A - Image composition device, image composition method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image composition device that prevents adverse effect of a shift of applying a correction value without need for a frame memory. <P>SOLUTION: An image composition device for composing a plurality of images captured by a plurality of cameras 2, includes: an image memory 11 for storing a plurality of camera images input from cameras 2; an image extracting section 12 for reading an image of at least part of overlapping area of the plurality of camera images from the image memory 11; a correction value calculating section 19 for calculating a correction value for correcting the plurality of camera images in such a manner that image levels of the plurality of camera images in the overlapping area are equalized, on the basis of their image levels; an image correcting section 14 for correcting the read camera images using the correction value; and an image composition section 15 for composing the camera images obtained after the correction. The image composition device has a construction such that calculation of the correction value by the correction value calculating section 19 is performed in a vertical blanking period just prior to video effective period during which the image is read out by the image extracting section 12. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an on-vehicle camera image composition device and an image composition method for compositing and displaying images captured by a plurality of cameras mounted on a vehicle, and more particularly to a technique for compositing images around a vehicle in an easy-to-view manner.

  In recent years, images using viewpoint conversion technology that transforms and synthesizes images from multiple cameras attached to a vehicle into an image viewed from a predetermined virtual viewpoint such as a viewpoint looking down from above the vehicle, and displays it on the screen in real time A synthesizer has been developed. The driver can check the situation around the vehicle on the screen, can avoid an accident, and can easily perform a driving operation in a parking lot or the like.

  FIG. 5A and FIG. 5B are diagrams showing the camera mounting positions of such an image composition device. FIG. 5A is a top view of an automobile on which the cameras 21 to 25 are mounted, and FIG. 5B is a left side view thereof. In the automobile shown in FIGS. 5A and 5B, five cameras 21 to 25 are mounted. The camera 21 captures an image directly behind the vehicle, the camera 22 attached to the front and rear of the left side of the vehicle, and the camera 23 images the oblique rear of the left side of the vehicle and the camera 24 attached to the front and rear of the right side of the vehicle. 25 images the rear right side of the vehicle.

  FIG. 6 is a diagram illustrating captured images of the five cameras 21 to 25 illustrated in FIGS. 5A and 5B and their combined images. In this example, an image when the vehicle goes back to the parking lot is shown. It is difficult for the driver to monitor the captured images of the five cameras 21 to 25 in order on the display screen. As shown in FIG. 6, the captured images of the five cameras 21 to 25 are converted into an image looked down from a virtual viewpoint at an upper position of the vehicle, synthesized, and displayed on the screen, so that the periphery of the vehicle can be seen at once. It can be monitored and the situation can be easily grasped. In the composite image shown in FIG. 6, the image representing the vehicle is not a composite of the camera image (because it is not reflected in the camera) but an image of illustration data.

  However, when the captured images of the plurality of cameras 21 to 25 are combined and displayed on the screen of the display device, the brightness, color balance, and the like are independently controlled for each camera even if the plurality of cameras have the same standard. For this reason, there is a problem in that differences in luminance and color balance between the cameras at the boundaries of the imaging ranges of the cameras 21 to 25 occur, resulting in an image that is very difficult to see.

  In order to solve this problem, the following methods have been proposed. The plurality of cameras are arranged such that the respective imaging ranges overlap at the boundary. When combining the captured images of the cameras, correction is performed with reference to the image level of the overlapping portion of the images so that the luminance and color balance between the cameras match.

  Patent Document 1 discloses an example of a correction method. The average value of the brightness of the image of the first camera, the average value of the brightness of the image of the second camera arranged adjacent to the first camera, and the average value of the brightness of the overlapping portion of these two images measure. Then, a correction value for each camera image is calculated so that the luminance values of the overlapping portions are equal, and the average value of the luminance of the entire screen becomes a preset target value, and the combined image is corrected. In Patent Document 1, only the luminance is corrected.

  In order to obtain the correction value of the video of a certain frame by the method described in Patent Document 1, it is necessary to scan the entire frame and measure the image level. In an image synthesizing apparatus that operates in real time, it is not possible to spend time for scanning the entire frame in order to calculate a correction value. Accordingly, the correction value applied to the frame at time N + 1 is a correction value calculated from the value calculated at the frame at time N. Originally, the correction value obtained from the frame at time N should be applied to the frame at time N. However, since it is actually applied to the frame at time N + 1, the frame for obtaining the correction value (here, time N frames) and the frame to be corrected (here, the frame at time N + 1) do not match (hereinafter referred to as “correction value application deviation”). The adverse effect of this correction value application shift is mainly that when the brightness of the subject fluctuates, the screen flickers and the difference in brightness between the cameras changes.

  With respect to the problem of correction value application deviation, the image comparison device described in Patent Document 2 has a frame memory for one frame of video, and applies a correction value by delaying the video side to which the correction value is applied by one frame. Misalignment is lost.

JP 2006-173988 A Japanese Patent Laid-Open No. 2001-169911

  However, the need for a frame memory is disadvantageous in terms of cost, and the input image is further delayed by one frame in addition to the system delay. Therefore, it is not desirable for in-vehicle applications that require real-time performance.

  In view of the above background, an object of the present invention is to provide an image composition device that can prevent the adverse effect of correction value application deviation without requiring a frame memory.

  An image composition apparatus according to the present invention is an image composition apparatus that combines a plurality of images captured by a plurality of cameras arranged so that a part of each imaging range overlaps, and a plurality of images input from the cameras. An image memory that stores a camera image, a first image extraction unit that reads out at least some of the overlapping portions of a plurality of camera images from the image memory, and an image level that measures an image level of the read image A correction value calculating unit that calculates a correction value for correcting a plurality of camera images so that the image levels of the plurality of camera images in the overlapping portion are the same based on the measurement unit, and the image level; A second image extraction unit that reads the camera image from an image memory; an image correction unit that corrects the read camera image using the correction value; and the corrected camera image. And an image synthesizing unit for synthesizing, has a configuration in which the calculation of the correction value by the correction value calculation unit performs the vertical blanking interval immediately before the effective image section for reading an image by the second image extraction unit.

  With this configuration, the correction value is obtained only from the overlapping portion of a plurality of camera images, so the correction value can be obtained in the vertical blanking interval, and correction is performed on the camera image read in the video effective interval following the vertical blanking interval. The value can be applied. Thereby, the image frame for obtaining the correction value can be matched with the image frame to be corrected.

  The image composition device of the present invention includes an overlapping part extraction table storing coordinates of the camera image corresponding to the overlapping part, and the first image extraction unit stores the coordinates stored in the overlapping part extraction table. The image data of the overlapping portion is read from the image data.

  With this configuration, it is possible to read the image of the overlapping portion without analyzing the image and determining the overlapping portion.

  The image composition apparatus of the present invention includes a composite image extraction table that stores a correspondence relationship between the coordinates of the camera image and the coordinates of the composite image, and the second image extraction unit stores the composite image in the composite image extraction table. The image data is read out from the coordinates of the camera image thus read out, and the image composition unit is configured to map the read image data to the coordinates of the corresponding composite image or by combining them.

  With this configuration, a plurality of camera images can be easily combined.

  The image composition method of the present invention is an image composition method for compositing a plurality of images captured by a plurality of cameras arranged so that a part of the imaging range of each other overlaps, and a plurality of images input from the cameras. Storing a camera image in an image memory; reading out at least a part of an overlapping portion of a plurality of camera images from the image memory in a vertical blanking interval; and measuring an image level of the read image And a step of calculating a correction value for correcting a plurality of camera images based on the image level so that the image levels of the plurality of camera images in the overlapping portion are the same, and the vertical blanking. A step of reading the camera image from the image memory in a video valid section following the section, and correcting the read camera image It has a step of correcting, the configuration that includes a step of combining the camera image after the correction using the.

  With this configuration, the image frame for obtaining the correction value and the image frame to be corrected can be matched with each other as in the above-described image composition device. Further, various configurations of the above-described image composition apparatus can be applied to the image composition method of the present invention.

  The program of the present invention is a program for synthesizing a plurality of images captured by a plurality of cameras arranged so that a part of each imaging range overlaps, and a plurality of images input to the computer from the cameras Storing in the image memory, reading out at least some of the overlapping portions of the plurality of camera images from the image memory in the vertical blanking interval, and determining the image level of the read image A step of measuring, a step of calculating a correction value for correcting a plurality of camera images so that the image levels of the plurality of camera images in the overlapped portion are the same based on the image level, and the vertical block A step of reading the camera image from the image memory in a video valid section following the ranking section; And correcting using the La image the correction value, and a step of synthesizing the camera image after the correction.

  With this configuration, the image frame for obtaining the correction value and the image frame to be corrected can be matched with each other as in the above-described image composition device. Various configurations of the above-described image composition apparatus can be applied to the program of the present invention.

  According to the present invention, the correction value is obtained in the vertical blanking interval, and the correction value is applied to the camera image read out in the video effective interval following the vertical blanking interval. It has an excellent effect that the target image frame can be matched.

Block diagram of an image composition device in an embodiment (A) The figure which shows an example of the synthesized image in this Embodiment, (b) The explanatory view of the synthetic image extraction table, (c) The explanatory drawing of the overlapping part extraction table A flowchart showing a procedure of image composition correction processing of an image composition device in an embodiment The figure which shows the frame timing of the image composition correction process of the image composition apparatus in embodiment (A) Top view showing an example of an automobile equipped with five in-vehicle cameras, (b) Side view showing an example of an automobile equipped with five in-vehicle cameras The figure which shows the example of each camera image of the vehicle carrying 5 in-vehicle cameras, and its composite image

  Hereinafter, an image composition device according to an embodiment of the present invention will be described with reference to the drawings. The image composition device according to the present embodiment is a device that converts the captured images of a plurality of in-vehicle cameras as shown in FIG.

  FIG. 1 is a diagram showing a configuration of an image composition device 1 according to an embodiment of the present invention. The image composition apparatus 1 is connected with n in-vehicle cameras 2 and a display device 3. The n in-vehicle cameras 2 are arranged so that a part of the imaging range overlaps with the in-vehicle cameras 2 adjacent to each other.

  The image composition apparatus 1 includes an image memory 11 that stores a camera image input from the in-vehicle camera 2, an overlapping part extraction table 16 that stores the coordinates of the camera image corresponding to the overlapping part, the coordinates of the camera image, and the composite image From the composite image extraction table 17 storing the correspondence relationship of the coordinates of the two, from the image memory 11 based on the two tables 16, 17, from the image extraction unit 12 that extracts the overlapping portion image or the composite image, and from the image memory 11 A correction signal generator 13 that calculates a correction value based on the extracted data of the overlapping portion, an image correction unit 14 that corrects image data based on the image signal correction value calculated by the correction signal generator 13, and a correction An image composition unit 15 that composes the later image data and displays the synthesized image on the display device 3 is provided. The image extraction unit 12 also has the functions of the “first image extraction unit” and the “second image extraction unit” recited in the claims. The correction signal generation unit 13 includes an image level measurement unit 18 and a correction value calculation unit 19. The image extraction unit 12 sends a coordinate value specified in an overlapping part extraction table 16 and a composite image extraction table 17 (to be described later) and a control signal indicating reading to the image memory 11. Read the coordinate image data.

  FIG. 2A to FIG. 2C are diagrams for explaining the details of the composite image extraction table 17 and the overlapped part extraction table 16. FIG. 2A is a diagram illustrating an example of a composite image in the present embodiment. The captured images captured by the five cameras as illustrated in FIGS. 5A and 5B are deformed and combined. This is an example of a composite image for one frame obtained in this way.

  FIG. 2B is a diagram illustrating an example of the composite image extraction table. The composite image extraction table 17 has a one-to-one correspondence with the pixel coordinates of the composite image. The composite image extraction table 17 describes the correspondence between which coordinate of which camera image and which coordinate of which camera image are combined in each coordinate of the output image. As a table of each coordinate, a camera number of a camera image for generating a composite image pixel corresponding to the coordinate and a two-dimensional array indicating the pixel coordinate are expressed. In the case where the coordinate is an overlapping portion with an adjacent camera, a two-dimensional array indicating each camera number and pixel coordinates is expressed. Information about whether or not illustrations are superimposed is also expressed. In the example shown in FIG. 2B, information of “illustration: NO, duplication: NO, camera number: 2, coordinates: (Xa, Yb)” is stored in the coordinates (x1, y1) of the composite image extraction table. ing. This indicates that the pixel at the coordinate (x1, y1) of the composite image is generated by the pixel at the coordinate (Xa, Yb) of the image of the camera 2. Further, the coordinates (x2, y1) of the composite image extraction table include “illustration: NO, duplication: YES, camera number: 2, coordinates: (Xc, Yd), camera number: 1, coordinates: (Xe, Yf). , Weighting: α ”is stored. This is because the pixel of the coordinate (x2, y1) of the composite image is an overlapping portion of the coordinate (Xc, Yd) of the image of the camera 2 and the coordinate (Xe, Yf) of the image of the camera 1, and each pixel value Is generated by blending with a weight of α: (1-α).

  FIG. 2C is a diagram illustrating an example of the overlapping part extraction table 16. In this overlapping part extraction table 16, correspondence between which coordinates of which camera image and which coordinates of which camera image overlap is described. The overlapping part extraction table 16 stores a two-dimensional array indicating the camera number and pixel coordinates of each camera in all or part of the overlapping parts of adjacent camera images in the composite image. . In the example shown in FIG. 2C, the Mth data stores information of “camera number: 1, coordinates: (Xg, Yh), camera number: 2, coordinates: (Xi, Yj)”. . This indicates that the Mth pixel in the overlapping portion image is an overlapping portion between the coordinates (Xg, Yh) of the captured image of the camera 1 and the coordinates (Xi, Yj) of the captured image of the camera 2. ing.

  The image composition apparatus 1 described above is configured by a computer including a CPU, a ROM, a RAM, and a data bus that connects these. The above functions are realized by the CPU executing arithmetic processing according to the program written in the ROM. The overlapping part extraction table 16 and the composite image extraction table 17 may be stored in a ROM or an external storage medium.

  FIG. 3 is a flowchart showing a procedure of a correction process for an image for synthesis by the image synthesizing apparatus 1. The image synthesizing apparatus 1 captures, for example, 30 camera images per second. The image composition device 1 performs the correction process shown in FIG. 3 every time a camera image is captured.

  First, when entering the blanking section, the image extraction unit 12 reads out the image data of the overlapping portion from the image memory 11 based on the overlapping portion extraction table 16 (S10). The image level measurement unit 18 integrates the pixel levels of the image data in the overlapping portions of the images of the adjacent cameras necessary for correction for each color of R (red), G (green), and B (blue), and at the same time The number is measured (S11). At this time, if the values of R, G, and B are outside the predetermined threshold range, the pixel level is not integrated and the number of pixels is not measured. Thereby, the abnormal value of the pixel level can be excluded from the calculation of the correction value, and an appropriate correction value can be obtained. Subsequently, the integrated value of the R, G, and B pixel levels of the image data of each camera in each overlapping portion is divided by the number of pixels, and the average value of the signal level of each camera image in the overlapping portion is calculated (S12).

  Next, the correction value calculation unit 19 uses the average value of the signal level of each camera in each overlapping portion to correct the average value of the signal level of the overlapping portion between adjacent camera images. Is calculated (S13). When the video valid section starts, the image extraction unit 12 reads the composition image data from the image memory 11 based on the composition image extraction table 17 (S14). Based on the data described in the composite image extraction table 17, the image composition unit 15 generates a composite image by mapping or combining the corresponding pixel data of the camera image (S15).

  FIG. 4 is a diagram illustrating the timing for calculating the correction value. The vertical synchronizing signal shown in FIG. 4 is a synchronizing signal indicating a section of one frame, and when the vertical synchronizing signal is LOW, it is a video valid section, and when it is HIGH, it is a vertical blanking section. When the vertical synchronization signal changes from LOW to HIGH, one frame of video ends. When the vertical synchronization signal changes from HIGH to LOW again, the video of the next frame starts. An image of n frames will be described. When entering the vertical blanking interval, the image of the overlapping portion in the n frames is read from the image memory 11 based on the overlapping portion extraction table 16. The image level measurement unit 18 measures the pixel level of the image data in the overlapping portion of the images of each adjacent camera. When the reading of the overlapping image is completed, the correction value calculation unit 19 calculates a correction value and determines an n-frame correction value. The overlapping part extraction table 16 does not need to correspond one-to-one with the pixel coordinates of the composite image, and is data of all or only a part of the overlapping part. Since the number of pixels read out from the image memory 11 is sufficiently smaller than that of the composite image, it is possible to sufficiently perform processing from reading out the image of the overlapping portion in the vertical blanking interval to the correction calculation. When the video valid period starts, the image extraction unit 12 reads out the n-frame composition image data from the image memory 11 based on the composite image extraction table 17, and the image correction unit 14 applies the n-frame composition image to the composition image. Then, correction is performed using the n-frame correction value.

  The image synthesizing apparatus 1 according to the embodiment of the present invention has been described above. The image synthesizing apparatus of the present invention refers to the image level of the overlapping portion (overlapping portion) of the image and performs correction so that the luminance and color balance between the cameras match, so that the correction value is set in the vertical blanking interval. Can be sought. As a result, the correction value can be applied to the camera image read in the video valid section following the vertical blanking section, and the image frame for obtaining the correction value can be matched with the image frame to be corrected. . As can be seen in FIG. 4, the correction value obtained from n frames can be applied to an image of n frames.

  In the present embodiment, the overlapping part extraction table 16 and the composite image extraction table 17 are described separately, but the present invention can be implemented even if both data are held in one data holding device. . The present invention can also be implemented by using the image memory 11 as well.

  In the present embodiment, five cameras are described, and the front of the vehicle is described as an unimaged area. However, the present invention can also be implemented in a system of other cameras or a system that captures and displays the entire periphery of the vehicle.

  As described above, the present invention has an excellent effect that an image frame for obtaining a correction value can be matched with an image frame to be corrected, and an image obtained by combining images captured by a plurality of cameras. It is useful as a synthesizer.

DESCRIPTION OF SYMBOLS 1 Image composition apparatus 2 Car-mounted camera 3 Display apparatus 11 Image memory 12 Image extraction part 13 Correction signal generation part 14 Image correction part 15 Image composition part 16 Overlapping part extraction table 17 Composite image extraction table 18 Image level measurement part 19 Correction value Calculation unit

Claims (5)

An image composition device that synthesizes a plurality of images captured by a plurality of cameras arranged so that a part of each imaging range overlaps,
An image memory for storing a plurality of camera images input from the camera;
A first image extraction unit that reads out at least some of the overlapping portions of the plurality of camera images from the image memory;
An image level measurement unit for measuring the image level of the read image;
A correction value calculation unit that calculates a correction value for correcting a plurality of camera images based on the image level so that the image levels of the plurality of camera images in the overlapping portion are the same;
A second image extraction unit for reading the camera image from the image memory;
An image correction unit for correcting the read camera image using the correction value;
An image composition unit for compositing the corrected camera image;
With
An image synthesizing apparatus that performs correction value calculation by the correction value calculation unit in a vertical blanking interval immediately before a video valid interval in which an image is read by the second image extraction unit. An overlapping part extraction table storing coordinates of the camera image corresponding to the overlapping part;
The image synthesizing apparatus according to claim 1, wherein the first image extraction unit reads data of an image of an overlapping portion from coordinates stored in the overlapping portion extraction table. A composite image extraction table storing the correspondence between the coordinates of the camera image and the coordinates of the composite image;
The second image extraction unit reads out image data from the coordinates of the camera image stored in the composite image extraction table;
The image synthesizing apparatus according to claim 1, wherein the image synthesizing unit performs mapping by mapping or synthesizing the read image data to the coordinates of the corresponding synthesized image. A method of combining a plurality of images captured by a plurality of cameras arranged so that a part of each other's imaging range overlaps,
Storing a plurality of camera images input from the camera in an image memory;
Reading at least some of the overlapping portions of the plurality of camera images from the image memory in a vertical blanking interval;
Measuring the image level of the read image;
Calculating a correction value for correcting a plurality of camera images based on the image level so that the image levels of the plurality of camera images in the overlapping portion are the same;
Reading the camera image from the image memory in a video valid section following the vertical blanking section;
Correcting the read camera image using the correction value;
Synthesizing the corrected camera image;
An image composition method comprising: A program for synthesizing a plurality of images captured by a plurality of cameras arranged so that a part of each imaging range overlaps,
Storing a plurality of camera images input from the camera in an image memory;
Reading at least some of the overlapping portions of the plurality of camera images from the image memory in a vertical blanking interval;
Measuring the image level of the read image;
Calculating a correction value for correcting a plurality of camera images based on the image level so that the image levels of the plurality of camera images in the overlapping portion are the same;
Reading the camera image from the image memory in a video valid section following the vertical blanking section;
Correcting the read camera image using the correction value;
Synthesizing the corrected camera image;
A program that executes