JP2009123131A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of magnifying a part of a received image detected by an image sensor of a fisheye camera and displaying it to a display picture with little distortion and to correct it when the fisheye camera is fitted at a tilt. <P>SOLUTION: An image of a part of a section 30a in a received image 20 taken by the fisheye camera 11 is magnified and displayed to a display picture with a magnification rate differed in every part inside the section 30a. In the image processing, a variable function differed in magnification rate of an angle of view in every place is used, converted into central projection to generate the display picture, and a zenith angle and azimuth angle of an image point are obtained. By adding an angle correction value to the azimuth angle to generate a display image, an installation angle error of the fisheye camera 11 can be corrected. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention detects light collected by a lens such as a fish-eye lens, a wide-angle lens, or a telephoto lens with an image sensor, enlarges a part of an image signal detected by the image sensor, corrects distortion, and displays a display screen. The present invention relates to an imaging device that can be displayed on the screen.

  2. Description of the Related Art An imaging device that captures an external scene using a camera equipped with a fish-eye lens, a wide-angle lens, and the like and an imaging device and displays the image on a display device is used for crime prevention or on-vehicle use. In particular, in a vehicle, a camera having a fisheye lens or a wide-angle lens is attached to the vehicle body, and a wide range of scenery outside the vehicle body is photographed by the camera. However, the received image detected by the image sensor of a camera using a fish-eye lens or the like is distorted as the angle of view increases from the optical axis of the lens.

  Therefore, Patent Document 1 below discloses an invention in which distortion of an image is corrected by calculating distortion aberration for an image received by a camera. However, Patent Document 1 does not describe how to calculate distortion aberration. Since an image taken with a camera equipped with a fisheye lens has a three-dimensional depth, a three-dimensional geometric process is required when calculating distortion, and the amount of calculation is enormous. Become a thing. Therefore, the burden on the CPU is too large to perform the calculation for the received image captured by the camera for a long time.

  The display system described in Patent Document 2 sets a conversion table for correcting distortion of a curved image, and converts the curved image detected by the camera into display data using the conversion table. It is.

  However, in this method, since the conversion table corresponds only to a section having a predetermined size at a predetermined position in the received image captured by the camera, among the received images detected by the camera. When a plurality of sections are taken out and displayed on the display device, it is necessary to prepare a conversion table for each section. In other words, since the conversion table corresponds to a certain space of the area photographed by the camera, it is not versatile, for example, a section where an actual image is planned in the conversion table because the camera mounting angle is shifted. When the image is different from the first image, normal correction cannot be applied to the received image.

  The invention described in Patent Document 3 below also prepares a matching table corresponding to the display area in advance, and displays the pixels of the received image detected by a plurality of cameras in correspondence with the predetermined coordinate positions of the mapping table. That's it. Similarly to the invention described in Patent Document 2, the invention described in Patent Document 3 can only apply distortion correction to an image in a predetermined section.

  In addition, since the distortion-corrected image follows the principle of plane development, it cannot be avoided that the entire angle of view becomes narrow.

  Further, since the inventions described in Patent Documents 2 and 3 are systems in which image signals (luminance signals) at each detection point detected by the image sensor are applied to a predetermined conversion table, for example, an automobile When the camera is attached to the vehicle body or the like at an angle, an angle error occurs in the image rotation direction between the received image detected by the image sensor and the conversion table. As a result, a received signal at a different position is given to the position to be defined on the display screen, and the original display image cannot be obtained.

For this reason, it is necessary to position and attach a camera equipped with a fisheye lens or the like with high accuracy to the body of an automobile, and the installation work becomes extremely complicated.
JP 2000-324386 A Japanese Patent Laid-Open No. 2004-40411 Japanese Patent No. 3300334

  The present invention solves the above-described conventional problems and provides an imaging apparatus capable of displaying a received image detected by a camera having a lens such as a fisheye lens, a wide-angle lens, or a telephoto lens as a natural image with a wide field of view. The purpose is that.

  It is another object of the present invention to provide an imaging apparatus that can obtain a display image with an accurate angle even when the camera mounting angle accuracy is rough.

The present invention provides a lens, an image sensor that detects light collected by the lens, and image data for displaying an image on a display screen having a predetermined area based on an image reception signal detected by the image sensor. In an imaging device provided with an image processing unit to generate,
The image processing unit performs image processing for enlarging and displaying an image of a part of the section within the image receiving angle of view detected by the image sensor on the display screen. In the image processing, in the rotation direction of the received image, The correction is possible.

  The present invention can also correct the rotation direction by giving a correction value to the azimuth angle of the spatial point by polar coordinate conversion.

  The present invention can be configured such that in the image processing, the following calculation is performed on each pixel Pi of the display screen, and processing is performed for associating the pixel Pi with the received signal at the detection point of the image sensor. .

(1) The position rx of the image point on the display screen from the field angle αx in the X direction corresponding to the pixel Pi on the display screen, the field angle αy in the Y direction, and the focal length f corresponding to the magnification of the display image. And ry are calculated.
(2) The zenith angle θi and the azimuth angle Φi of the spatial point to be displayed on the pixel Pi are obtained from the rx and ry.
(3) The rotation angle correction value ΔΦ is added to the azimuth angle Φi.
(4) The pixel Pi is made to correspond to an imaging signal detected at any detection point of the imaging device from the zenith angle θi, the azimuth angle after correction (Φi + ΔΦ), and the projection polynomial of the lens.
(5) The processes (1) to (4) are performed on all the pixels on the display screen.

  For example, in the process (1), the coordinate position in the X-axis direction of the pixel Pi on the display screen is substituted into the change function, and the angle of view αx in the X direction corresponding to the coordinate position is calculated. It is preferable to obtain the angle of view αy in the Y direction corresponding to the coordinate position by substituting the coordinate position in the Y axis direction of the pixel Pi at the change function.

  The change function gives an enlargement ratio with respect to the angle of view of the position information of the pixel Pi, but the enlargement ratio in this specification means not only literal expansion that extends the angle of view. It is a concept that also means a rate of reducing the angle of view.

  The process (1) is performed by the calculation of rx = f · tan (αx) and ry = f · tan (αy).

  Furthermore, in the process (4), the coordinates mx, my of the detection point corresponding to the pixel Pi on the image sensor are calculated by the following calculation.

r (θi) = k ₁ θi + k ₂ θi ³ + k ₃ θi ⁵ + k ₄ θi ⁷ +.
mx = r (θi) · cos (Φi + ΔΦ)
my = r (θi) · sin (Φi + ΔΦ)
(However, the r (θi) is calculated by a polynomial obtained by rounding down the terms of an arbitrary order.)

  In the present invention, the rotation direction of the received image can be corrected by inputting a correction value to a storage unit provided in the imaging apparatus.

  The image pickup apparatus of the present invention takes out a part of the received image detected by an image pickup device provided in a camera equipped with a lens such as a fish-eye lens, a wide-angle lens, or a telephoto lens, and sets the magnification of the angle of view in the location. Therefore, a part of the received image can be enlarged or reduced and displayed on the display screen. As a result, an image with a wide field of view can be displayed.

  Furthermore, since the rotation direction of the received image can be corrected, the azimuth angle of the display image displayed on the display screen can be set freely. Therefore, for example, even when the camera is attached to the body of an automobile or the like, a display image with the rotation angle corrected can be displayed.

FIG. 1 is a block diagram of an imaging apparatus according to an embodiment of the present invention.
The imaging device 10 has a fisheye camera 11. The fisheye camera 11 is attached to the vehicle body of the automobile 1, for example. In the embodiment of FIG. 1, the fisheye camera 11 is attached to the fender mirror 2 of the automobile 1, and its optical axis is located between the front wheel 3 and the rear wheel 4. The optical axis is directed toward the front side of the paper surface of FIG. 1 and is set downward at an angle of approximately 45 degrees with the surface of the ground.

  As shown in FIG. 2A, the fisheye camera 11 has a fisheye lens 12 and an image sensor 13 having a plurality of detection points for detecting light collected by the fisheye lens 12. The image sensor 13 is a CCD or a CMOS.

  The received image signal (luminance signal) detected at each detection point of the image sensor 13 is converted into a digital value by the image detection unit 14 and given to the image processing unit 15. The image processing unit 15 is provided with a memory 16 as a storage unit, and the image processing unit 15 and the memory 16 are controlled by the control unit 17. The image processing unit 15 can perform arithmetic processing on a digital image signal sent from the image detection unit 14. The control unit 17 is mainly composed of a CPU, and can control the image processing unit 15 and the display driver 18. Image data obtained by processing by the image processing unit 15 is sent to the display driver 18 and an image is displayed on a display panel 19 such as a liquid crystal display panel provided in the vehicle.

  Furthermore, the imaging apparatus 10 is provided with an input unit, and an angle correction value (ΔΦ) can be input to the control unit 17 from this input unit. The input angle correction value (ΔΦ) is stored in the memory 16. This input unit may be operated using any of the operation buttons provided on the instrument panel of the automobile when the imaging device 10 is attached to the automobile. Alternatively, when a terminal is provided in any of the circuits of the imaging device 10 and a computer is connected to the terminal and the imaging device 10 is mounted on the automobile 1, the angle from the computer to the control unit 17 is part of the work. A correction value (ΔΦ) may be given.

  FIG. 4 shows a plane H including the X axis and Y axis, which are orthogonal axes, and a Z axis extending perpendicularly to the plane H from the intersection of the X axis and the Y axis. When the plane H is the imaging plane of the imaging device, and A1 positioned in front of the plane is a spatial point to be projected on the imaging device, the zenith angle of the spatial point A1 with respect to the Z axis is θi, and the X axis is the reference. In this case, the azimuth angle of the space point A1 is Φi.

  FIG. 2B is a plan view of the image sensor 13 provided in the fisheye camera 11. A general model of a fisheye lens is equidistant projection, where ra = f · θi, where ra is the image height of the image point A2 and f is the focal length. As another model, the equal solid angle projection is ra = 2f · sin (θi / 2), and the orthographic projection is ra = f · sin θi.

When these various types of fisheye lenses are expressed by one model equation, the image height ra can be expressed as r (θi) as follows.
r (θi) = k ₁ θi + k ₂ θi ³ + k ₃ θi ⁵ + k ₄ θi ⁷ +.

In this embodiment, a polynomial which is a model expression of a fisheye lens in which an arbitrary high-order term is omitted is stored in the memory 16. For example, the following polynomial is stored in the memory 16. However, the degree of the polynomial can be arbitrarily set.
r (θi) = k ₁ θi + k ₂ θi ³

FIG. 3 shows a central projection which is a projection model using a theoretical convex lens having no pinhole or aberration. In the central projection, the relationship between the image height r and the focal length f is as follows.
r = f · tan θ

  In the central projection model shown in FIG. 3, an image without distortion can be captured, but the angle of view that can be projected onto an image sensor with a limited area is narrow. On the other hand, in the fisheye camera 11 using the fisheye lens 12 shown in FIGS. 2A and 2B, a wide-angle image can be formed on the image sensor 13, and the angle of view is approximately 180 degrees. However, the received image formed on the image sensor 13 of the fisheye camera 11 is distorted as the angle of view increases.

  FIG. 5 shows an example of the received image 20 that is imaged on the image sensor 13 when imaged by the fisheye camera 11. In FIG. 5, the position of the optical axis of the fisheye lens 12 is shown as point O1. 6 shows a display image 30 displayed on the screen 19a of the display panel 19, and FIG. 8 shows a display image 30A displayed on the screen 19a. The display image 30 shown in FIG. 6 has not been corrected by the angle correction value ΔΦ, and the display image 30A shown in FIG. 8 has been corrected by the angle correction value ΔΦ.

  The display image 30 shown in FIG. 6 and the display image 30A shown in FIG. 8 show a partial section 30a of the image receiving image 20 shown in FIG. Since the received image 20 shown in FIG. 5 is formed by the fisheye lens 12, the amount of distortion increases as the distance from the optical axis O1 increases. Therefore, in the image processing unit 15, the angle of view in the section 30a of the received image 20 shown in FIG. 5 is enlarged so that the enlargement ratio differs depending on the location, and is reduced depending on the location to correct the distortion and displayed on the display screen 19a. The image data for generating is generated.

  At this time, the magnification of the angle of view is changed according to the position away from the optical axis O1. For example, the field angle is enlarged in a region close to the optical axis O1, and the enlargement ratio is reduced or gradually reduced as the distance from the optical axis O1 increases. Or conversely, the enlargement ratio may be gradually increased as the distance from the optical axis O1 increases. By performing such image processing, an image to be displayed as the display image 30 can be displayed in a state that is easy to see with a wide viewing angle.

  In the example shown in FIG. 5, the change function T (x) in the X-axis direction shown in FIG. 7 is used in the image processing for generating the display image 30 or the display image 30A by enlarging a part of the section 30a of the received image 20. Is done. This change function T (x) is predetermined for correction of image processing and is held in the memory 16.

  6, the coordinate position in the X-axis direction is indicated by 0, 0.1, 0.2, 0.3,... On the display screen 30 from the right end to the left end. The horizontal axis of FIG. 7 shows the coordinate position of the display screen 30 in the X-axis direction, and the vertical axis shows the angle of view of the image point at each position in the X-axis direction shown on the horizontal axis. The amount of enlargement and image processing to be performed is shown numerically.

  The change function T (x) shown in FIG. 7 is, for example, information to be displayed at the coordinate position “0.1” in the X-axis direction on the display screen 19a in FIG. For example, information to be displayed at a coordinate position of “0.9” in the X-axis direction in FIG. 6 is displayed as “1.0” in the X-axis direction. It shows that the angle of view is enlarged to the position for processing.

  Similarly, the memory 16 also stores a view angle change function T (y) in the Y-axis direction. The angle-of-view change function T (y) in the Y-axis direction is a curvature function different from the angle-of-view change function T (x) in the X-axis direction.

  By performing image processing using the change functions T (x) and T (y), not only the magnification of the angle of view differs depending on the place, but also the direction in which the angle of view is enlarged gradually differs depending on the place. Images can be generated.

  The change function T (x) shown in FIG. 7 varies the enlargement ratio depending on the location when a part of the section 30a of the received image 20 of the fisheye camera 11 is enlarged and displayed as the display screen 19a shown in FIG. By using this change function T (x), which is a correction function, it is possible to enlarge the region of interest and generate an easy-to-view display image 19a with no distortion as a whole. This change function T (x) is generated according to the position of the section 30a from which the image is cut out in the received image 20 of the fisheye camera 11, the area of the section 30a, the magnification of the image of the section 30a, and the like.

  The change function T (x) determines a plurality of enlargement ratios according to the position, area, and enlargement ratio of the section 30a to be cut out from the received image 20, and connects the plurality of enlargement ratios with a parametric curve such as a B-spline curve. Is generated. That is, by providing the memory 16 with position information of a plurality of magnifications and a parametric curve function, the change function T (x) shown in FIG.

  The image processing method is performed according to the following procedure. In this image processing, an image signal (luminance signal) detected at which detection point of the image sensor 13 is displayed in correspondence with an arbitrary pixel Pi on the display screen 19a of the display panel 19 shown in FIG. 6 or FIG. Decide.

  First, the angle of view αx to be normalized is obtained by substituting the position of the pixel Pi in the X-axis direction into the horizontal axis of the change function T (x) shown in FIG. For example, when the position of the pixel Pi in the X-axis direction is “0.1”, the position in the X-axis direction is “0.18” when this “0.1” is substituted into the change function of FIG. The image processing unit 15 determines how much the angle of view in the X-axis direction corresponds to the position of “0.18” in the X-axis direction. The obtained value is the normalized image receiving angle αx.

  In the example shown in FIG. 5, the “0” point on the X axis of the section 30a coincides with the X direction position of the optical axis O1. That is, the origin of the X-axis coordinate in FIG. 6 is the same X-coordinate position as the optical axis O1. Therefore, in this example, the vertical axis shown in FIG. 7 means the magnification of the angle of view. For example, when the pixel Pi is at the position “0.1” on the X-axis coordinates, the image angle at the position “0.1” on the display screen 16a in FIG. The angle of view αx can be obtained. Similarly, the normalized image receiving field angle αy of the pixel Pi in the Y-axis direction can be obtained using the change function T (y) in the Y-axis direction. In the change function T (x) in FIG. 7, the magnification of the angle of view gradually decreases as the distance from the optical axis O1 increases.

  Next, image point positions rx and ry in the display image are calculated from the focal length f corresponding to the magnification of the display image, the angle of view αx, and the angle of view αy.

  For example, using the normalized image receiving angle αx and αy, the image forming position on the image receiving surface Ha in the central projection without distortion as shown in FIG. 3 is obtained. In this calculation, the value of the focal length f obtained from the magnification corresponding to the overall enlargement ratio for displaying the section 30a of the received image 20 shown in FIG. 5 as the display image 30 shown in FIG. 6 is used. That is, the normalized coordinate positions rx and ry corresponding to the central projection of the pixel Pi can be obtained from the following expression relating to the central projection.

rx = f · tan (αx)
ry = f · tan (αy)

  By knowing the coordinate positions rx and ry normalized as described above, the zenith angle θi and the azimuth angle Φi shown in FIG. 4 can be known. The zenith angle θi and the azimuth angle Φi are the normalized zenith angle and azimuth angle corrected by the change function T (x) shown in FIG. 7, and are the zenith angle and azimuth angle of the spatial point to be displayed on the pixel Pi. is there.

  A display image 30 shown in FIG. 6 shows the result of the following calculation using the zenith angle θi and the azimuth angle Φi. On the other hand, the display image 30A shown in FIG. 8 is a result of performing the following calculation using (Φi + ΔΦ) obtained by adding the angle correction value ΔΦ to the azimuth angle Φi as the azimuth angle. A display image 30 shown in FIG. 6 is an image when the fisheye camera 11 is attached to the body of the automobile 1 with a slight angle. The display screen 30A shown in FIG. 8 uses (Φi + ΔΦ) obtained by adding an angle correction value ΔΦ for correcting the inclination of the mounting angle as an azimuth to correct the mounting angle error.

  In the image processing method, r (θi), which is the image height, is obtained by substituting the zenith angle θi into the model formula of the fisheye lens 12 used in the fisheye camera 11.

r (θi) = k ₁ · θi + k ₂ · θi ³ ,
It should be noted that r (θi) is any polynomial in which r (θi) = k ₁ θi + k ₂ θi ³ + k ₃ θi ⁵ + k ₄ θi ⁷ +. Even a formula for the number of terms can be used.

  Next, from the image height r (θi) and the azimuth angle Φi or (Φi + ΔΦ), an image reception signal (luminance signal) at any position in the image reception image 20 shown in FIG. 5 is used as a pixel Pi on the display screen 19a. Whether to use as image data.

The display image 30 shown in FIG. 6 is obtained by performing the following calculation.
mx = r (θi) · cos (Φi)
my = r (θi) · sin (Φi)

The display image 30A shown in FIG. 8 is obtained by performing the following calculation.
mx = r (θi) · cos (Φi + ΔΦ)
my = r (θi) · sin (Φi + ΔΦ)
The process is performed on all the pixels Pi in the display screen 19a.

  In the image processing method described above, the image formed on the image sensor 13 of the fisheye camera 11 is converted into a central projection image, whereby the enlarged display image 30 shown in FIG. 6 and the display image 30A shown in FIG. Can correct distortion. However, when the wide-angle image data obtained by the image sensor 13 is simply converted into a central projection image, the image displayed on the display screen is extremely reduced in the area close to the optical axis O1 and separated from the optical axis O1. The image area is extremely enlarged, resulting in an image in which visibility is significantly impaired.

  On the other hand, in the image processing, using the change function T (x) determined in advance as shown in FIG. 7, the normalized angle of view αx and αy of the image position corresponding to the pixel Pi is obtained. Using the angle of view αx and αy, the zenith angle θi and the azimuth angle Φi are obtained from the central projection formula. Then, the position (mx, my) of the detection point on the received image 20 to be given to the pixel Pi of the display screen 19a by substituting the zenith angle θi and the azimuth angle Φi into the model equation of the fisheye lens 12 being used. Is calculated.

  By using the change function, it is possible to correct the occurrence of extreme enlargement and reduction areas, such as when a fisheye camera image is directly converted into a central projection image, and a display image 30 with good visibility. Can be obtained.

  Further, in the display image 30A shown in FIG. 8, since the angle correction value ΔΦ is given to the azimuth angle Φi, the rotation angle of the display image can be corrected. Therefore, when the fisheye or the meta 11 is attached to the body of the automobile 1 or the like, the display unit 30 is inclined as shown in FIG. 6 by using the input unit or the like and giving the angle correction value ΔΦ. Instead, a display image 30A with corrected inclination can be obtained as shown in FIG.

  As a method for correcting the rotation direction, the rotation direction can also be corrected by giving a correction value to the azimuth angle of the spatial point by polar coordinate conversion.

  Further, the present invention is not limited to the one using a camera equipped with a fisheye lens as in the above-described embodiment, and can be applied to one using a wide-angle lens or a telephoto lens, for example. The wide-angle lens has an angle of view of approximately 60 degrees or more, and produces a distortion that can be recognized in the peripheral portion. In the present specification, the wide-angle lens is a concept including a fish-eye lens.

1 is a block diagram showing an imaging apparatus according to an embodiment of the present invention; (A) is a side view of the fisheye lens and the image sensor, (B) is a front view of the image sensor, Illustration of the central projection model, Explanatory diagram of spatial point, zenith angle and azimuth, An explanatory view showing an example of an image received by a fisheye camera, An explanatory diagram of a display image that is not corrected with the angle correction value ΔΦ, Explanatory drawing of the change function T (x) in the X-axis direction, Explanatory drawing of the display image corrected with the angle correction value ΔΦ,

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Fisheye camera 12 Fisheye lens 13 Image pick-up element 15 Image processing part 19 Display panel 19a Display screen 20 Received image 30 Display image 30A before correction Display image 30a after correction

Claims (7)

Image processing for generating image data for displaying an image on a display screen having a predetermined area based on a lens, an image sensor that detects light collected by the lens, and an image reception signal detected by the image sensor An imaging device provided with
The image processing unit performs image processing for enlarging and displaying an image of a part of the section within the image receiving angle of view detected by the image sensor on the display screen. In the image processing, in the rotation direction of the received image, An imaging apparatus characterized by being capable of correction.   The imaging apparatus according to claim 1, wherein the rotational direction is corrected by giving a correction value to the azimuth angle of the spatial point by polar coordinate conversion. 3. The imaging according to claim 1, wherein in the image processing, the following calculation is performed on each pixel Pi of the display screen, and processing is performed for associating the pixel Pi with an image receiving signal at a detection point of the imaging element. apparatus.
(1) The position rx of the image point on the display screen from the field angle αx in the X direction corresponding to the pixel Pi on the display screen, the field angle αy in the Y direction, and the focal length f corresponding to the magnification of the display image. And ry are calculated.
(2) The zenith angle θi and azimuth angle Φi of the spatial point to be projected on the pixel Pi are obtained from the rx and ry.
(3) The rotation angle correction value ΔΦ is added to the azimuth angle Φi.
(4) The pixel Pi is made to correspond to an imaging signal detected at any detection point of the imaging device from the zenith angle θi, the azimuth angle after correction (Φi + ΔΦ), and the projection polynomial of the lens.
(5) The processes (1) to (4) are performed on all the pixels on the display screen.   In the process (1), the coordinate position in the X-axis direction of the pixel Pi on the display screen is substituted into the change function, and the angle of view αx in the X direction corresponding to the coordinate position is calculated. The imaging apparatus according to claim 3, wherein the Y-direction angle of view αy corresponding to the coordinate position is obtained by substituting the coordinate position of the pixel Pi in the Y-axis direction into a change function.   The imaging apparatus according to claim 3 or 4, wherein the process (1) is performed by a calculation of rx = f · tan (αx) and ry = f · tan (αy). 6. The imaging apparatus according to claim 3, wherein in the processing (4), coordinates mx and my of the detection point corresponding to the pixel Pi on the imaging device are calculated by the following calculation.
r (θi) = k ₁ θi + k ₂ θi ³ + k ₃ θi ⁵ + k ₄ θi ⁷ +.
mx = r (θi) · cos (Φi + ΔΦ)
my = r (θi) · sin (Φi + ΔΦ)
(However, the r (θi) is calculated by a polynomial obtained by rounding down the terms of an arbitrary order.)   The imaging apparatus according to claim 1, wherein the rotation direction of the received image is corrected by inputting a correction value to a storage unit provided in the imaging apparatus.

Images