JP4919836B2 - Image processing apparatus, image pickup apparatus, and image distortion correction method for correcting image distortion - Google Patents

Description

  The present invention relates to an image processing apparatus that electronically corrects distortion of an image captured by an image sensor and an imaging apparatus using the same.

  Generally, in a photographing apparatus that photographs optical information captured through a lens, distortion occurs in a photographed image due to aberration of the lens. The types of distortion include tall distortion and pincushion distortion, and the degree of distortion varies depending on the magnification of the lens and the specifications of the lens. For example, in a zoom lens, distortion tends to occur on the wide side (side with a short focal length).

  In the field of electronic cameras, a technique has been devised to electronically correct distortion caused by such lens characteristics (hereinafter referred to as “distortion correction”) (see, for example, Patent Document 1).

  For example, when the subject shown in FIG. 18A is picked up by an electronic camera, an image distorted into a tall shape is formed on an image pickup device such as a CCD in the electronic camera due to lens characteristics as shown in FIG. The FIG. 18C is an image obtained by electronically correcting the distorted image. In FIG. 18, the points s1, a1, and b1 correspond to each other, and the points sm, am, and bm correspond to each other. It can be seen that the vicinity of the point a1 is greatly distorted compared to the vicinity of the point am. As shown in FIG. 18C, the points a1 and am are corrected to their original positions by the distortion correction processing.

  Image distortion correction processing is performed by mapping conversion as follows. Considering the coordinate system on the captured image (hereinafter referred to as “input coordinate system”) and the coordinate system on the corrected image (hereinafter referred to as “output coordinate system”), input to the pixel positions of the captured image and the corrected image. The positions of the grid points in the coordinate system and the output coordinate system are made to correspond. In the distortion correction process, in such a coordinate system, image data of each pixel on the input coordinate system is mapped to the output coordinate system using a predetermined function f. Then, the image information (luminance information, color difference information) of the grid points on the output coordinate system (hereinafter referred to as “output coordinate points”) is interpolated (synthesized) with the image information of the mapped pixels in the vicinity thereof. Ask for. In this way, distortion correction is performed by obtaining image information for each grid point (pixel) on the output coordinate system.

  This will be described more specifically with reference to FIG. A point a1 in the input coordinate system (coordinate system on the captured image) is mapped to a point b1 on the output coordinate system (coordinate system on the corrected image) by the mapping function f. Another point am in the input coordinate system is mapped to a point bm in the output coordinate system (corrected image coordinate system) by the mapping function f. The mapping function f is obtained based on lens characteristics and zoom magnification.

  Necessary information on the corrected image is image information on a grid point (output coordinate point) of the output coordinate system, and generally the points b1 and bm on the output coordinate system mapped are not on the output coordinate point. The image data of the output coordinate point is obtained by interpolating (synthesizing) image data of a plurality of pixels mapped in the vicinity thereof. For example, as shown in FIG. 19, the image data of the output coordinate point Q1 is obtained by interpolating (synthesizing) the image data of the pixels b1, b2, bk, bk + 1 mapped in the vicinity thereof. Similarly, the image data of the output coordinate point Qm is obtained by interpolating the image data of the pixels bm, bm + 1, bn, bn + 1 mapped in the vicinity thereof. Thus, in the output coordinate system, the image data of the output coordinate point, that is, the corrected data is obtained by interpolation (synthesis).

  While scanning each input coordinate point of the captured image in order, the mapping point on the output coordinate system is obtained and synthesized as described above, thereby correcting the distortion of the entire image.

Japanese Patent No. 3035992

  In FIG. 18C, the vicinity of the point b1 is obtained by correcting an area having a relatively large distortion in the captured image, and the vicinity of the point bm is obtained by correcting an area having a relatively small distortion. For correction of a region with large distortion such as the region near the point b1, information necessary for correction cannot be obtained unless images of a plurality of lines are read out in the captured image. On the other hand, for correction of a region with a small distortion such as a region near the point bm, information necessary for correction can be obtained by reading out an image of a relatively small number of lines in the captured image.

  As described above, in order to obtain one line of the corrected image by mapping, data corresponding to many lines of the input image may be required. In other words, in order to output the corrected image data to the buffer memory line by line, a large number of line memories are required to hold data of a large number of lines in the input image while waiting for processing on unprocessed pixels. is there. However, the provision of a large number of line memories in the imaging apparatus has a problem in that the number of components increases by the amount of the line memory due to the configuration.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide an imaging apparatus having an easy configuration in an imaging apparatus that enables correction of image distortion caused by lens characteristics. It is in.

  An image processing apparatus according to the present invention is an image processing apparatus that electronically corrects distortion of an input image to generate an output image. The image processing apparatus sequentially reads input image data for each pixel in the scanning order, and performs correction processing for each pixel of the output image using the read pixel data, thereby obtaining correction data for each pixel of the output image. A part. The correction processing unit outputs pixel data for which correction processing has been completed in an order different from the scanning order of the output image.

The correction processing unit includes a line memory that temporarily stores input pixel data for a plurality of lines, which is a part of input image data, and a plurality of input pixel data necessary to generate data for a certain output pixel. A control unit that determines whether or not the data is stored in the line memory, and when the control unit determines that the data of the plurality of input pixels necessary for generating the output pixel data is stored in the line memory. And a combining processing unit that performs correction processing by combining the data of the plurality of input pixels . The predetermined number of lines is a number equal to the number of lines in which each of the plurality of input pixels used for generating data of one output pixel is located.

  The correction processing unit may further include a coordinate calculation unit that obtains a reverse mapping point on the input image corresponding to the pixel of the output image. In this case, the control unit determines whether data of the plurality of input pixels is stored in the line memory based on the inverse mapping point obtained by the coordinate calculation unit.

  The correction processing unit may further include information that associates the pixel of the input image with the pixel of the output image that is synthesized using the pixel data of the input image.

  The correction processing unit may further include a cache memory that temporarily stores data of pixels for which correction processing has been completed and outputs the data in units of pages.

  An imaging apparatus according to the present invention includes a lens, an imaging element that converts optical information input through the lens into electrical image information, and electronically corrects distortion of image data captured by the imaging element. Image processing apparatus.

The distortion correction method according to the present invention is a method of generating an output image by electronically correcting distortion of an input image. The distortion correction method includes a step of sequentially reading input image data for each pixel in the scanning order, and performing correction processing for each pixel of the output image using the read pixel data in order to obtain data of each pixel of the output image. And a step of outputting pixel data for which correction processing has been completed in an order different from the order of the pixels at the time of reading. The step of performing the correction process is to temporarily store input pixel data for a predetermined number of lines, which is a part of the input image data, in a line memory, and to generate a plurality of input pixels necessary for generating data of a certain output pixel. Are determined to be stored in the line memory, and when it is determined that data of a plurality of input pixels necessary for generating output pixel data is stored in the line memory, The correction processing is performed by combining the data of the input pixels. The predetermined number of lines is a number equal to the number of lines in which each of the plurality of input pixels used for generating data of one output pixel is located .

  According to the present invention, when performing interpolation in distortion correction processing, it is sufficient to prepare as many input images as the number of lines necessary for interpolation processing in the line memory, and it is necessary to prepare input images for many lines. Absent. Therefore, according to the imaging apparatus of the present embodiment, the line memory capacity for storing input image data can be reduced, and the configuration of the imaging apparatus can be simplified.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.
In the following description, data relating to pixel luminance information and color difference information is referred to as “pixel data”. Further, image data before distortion correction processing obtained on the image sensor is referred to as “original data”, and image data after distortion correction processing is referred to as “output data”. Also, the coordinate system on the original data is called the “input coordinate system”, the coordinate system on the output data is called the “output coordinate system”, and the input coordinate system and the output coordinate corresponding to the position of each pixel of the original data and the output data. A grid point on the system is provided. The grid points on the input coordinate system and the output coordinate system are referred to as “input coordinate point” and “output coordinate point”, respectively.

Embodiment 1
(Configuration of imaging device)
FIG. 1 is a diagram showing a configuration of an imaging apparatus according to the present invention. The imaging apparatus includes a lens 3, a CCD 11 that is an imaging element that converts optical information into an electrical signal, an AD converter 13 that converts an analog signal into a digital signal, and an image processing circuit 15 that controls the operation of the entire imaging apparatus. A buffer memory 17 that temporarily stores image data before and after distortion correction, and a card slot 21 in which a memory card 19 that stores image data is inserted are provided. The imaging apparatus further includes a motor 9 for moving the lens 3 in the focus operation and the zoom operation.

  The image processing circuit 15 includes a memory controller 23 that controls access between each processing unit in the image processing circuit 15, the buffer memory 17, and the memory card 19, and a preprocessing unit that performs preprocessing such as gamma correction on the image data. 25, a YC processing unit 27 that performs YC processing, a synthesis processing unit 29 that synthesizes (interpolates) pixel data for a plurality of input coordinate points to obtain corrected pixel data, and performs compression and expansion processing on the image And a compression / decompression processing unit 31 to perform. The YC process is a process for separating the Y signal (luminance component) and the C signal (color component) of the image data in order to facilitate the subsequent compression process. The image processing circuit 15 further includes a coordinate calculation unit 33, a memory 35, a magnification acquisition unit 37, and a FIFO memory 41. In the present embodiment, the memory controller 23, the composition processing unit 29, the coordinate calculation unit 33, the memory 35, and the FIFO memory 41 constitute a correction processing unit.

The coordinate calculation unit 33 has a function of calculating an inverse map f ⁻¹ (details will be described later) for distortion correction. The magnification acquisition unit 37 acquires information regarding the zoom magnification from the motor 9. The memory 35 stores information necessary for calculating the inverse map f ⁻¹ (information on the function of the inverse map f ⁻¹ for various conditions).

Here, the inverse map f ⁻¹ will be described. The mapping f is a mapping that associates a captured image, that is, a point on the coordinate system (input coordinate system) of the original data with a point on the output coordinate system of the image after distortion correction. The inverse map f ⁻¹ is a reverse map of the map f, and is a function that associates each point of the output coordinate system on the image after distortion correction with each point of the input coordinate system on the image sensor before distortion correction. . The mapping f and the inverse mapping f ⁻¹ are determined according to the lens characteristics and the zoom magnification.

  The FIFO memory 41 stores information indicating which output coordinate point (pixel of the output image) in the output data is used for interpolation of each input coordinate point (pixel) of the original data (hereinafter, “corresponding coordinate point list”). Is stored. Details of the corresponding coordinate point list will be described later.

(Overall operation of the imaging device)
The overall operation of the imaging apparatus configured as described above will be described.
In the imaging apparatus, optical information input through the lens 3 is converted into an analog electrical signal by the CCD 11. The AD converter 13 converts the analog electric signal from the CCD 11 into image information of a digital signal. The image processing circuit 15 receives image information of a digital signal from the AD converter 13, performs predetermined image processing, and finally records it on the memory card 19.

  More specifically, in the image processing circuit 15, preprocessing such as gamma correction is performed on the image signal from the AD converter 13 by the preprocessing unit 25, and YC processing is performed by the YC processing unit 27. . The image data subjected to these processes is stored in the buffer memory 17.

  Next, distortion correction is performed on the image data stored in the buffer memory 17 by the synthesis processing unit 29, the coordinate calculation unit 33, and the like. The image data after distortion correction is stored in the buffer memory 17 again. In this way, a distortion-corrected image is obtained on the buffer memory 17.

(Corresponding coordinate point list generation)
Generation of the corresponding coordinate point list stored in the FIFO memory 41 will be described. The corresponding coordinate point list is generated by the coordinate calculation unit 33 as follows.

The coordinate calculation unit 33 obtains an inverse mapping point on the original data from the inverse mapping f ⁻¹ for each output coordinate point. The input coordinate point of the original data existing in the area within the predetermined range Δ centered on the obtained inverse mapping point is set as an input coordinate point necessary for interpolation for obtaining pixel data of the output coordinate point. The coordinate calculation unit 33 associates the input coordinate point obtained by such a method with the output coordinate point, and outputs it to the FIFO memory 41 as a corresponding coordinate point list.

  FIG. 2 is a diagram illustrating a correspondence relationship between the input coordinate points in the original data and the output coordinate points in the output data when the lens 3 has a tall distortion characteristic. FIG. 2 further shows an example of a corresponding coordinate point list stored in the FIFO memory 41.

  In FIG. 2, for example, the inverse mapping point of the output coordinate point A1 of the output data is located in the vicinity of the input coordinate point Z1. Similarly, the inverse mapping point of the output coordinate point B1 is also located in the vicinity of the input coordinate point Z1. Both inverse mapping points of the output coordinate points A1 and B1 are in a region within the predetermined range Δ of the input coordinate point Z1. Therefore, in the example shown in FIG. 2, the input coordinate point Z1 of the original data is a point necessary for interpolation for obtaining pixel data of the output coordinate points A1 and B1. Therefore, in the corresponding coordinate point list of the FIFO memory 41, the input coordinate point Z1 and the output coordinate points A1 and B1 are recorded in association with each other. From the corresponding coordinate point list shown in FIG. 2, it is understood that the input coordinate points X1 and X2 are not used for interpolation of any output coordinate point, and the input coordinate point Y2 is necessary for interpolation of the output coordinate point A2. . Note that the corresponding coordinate point list stored in the FIFO memory 41 does not need to include information on all the input coordinate points at the same time, and information on the input coordinate points of the number of lines necessary in the processing in which the corresponding coordinate point list is referred to. Should be included. For example, the corresponding coordinate point list may be generated for every 10 lines of the original data according to the input coordinate points of the original data to be processed.

(Distortion correction processing)
With reference to FIG. 3, an outline of the distortion correction processing of the imaging apparatus of the present embodiment will be described.
In the present embodiment, the pixel data of the input coordinate points on the original data are sequentially read from the buffer memory 17 and the corresponding coordinate point list stored in the FIFO memory 41 is referred to, and the input coordinate point is one of the output coordinate points. It is determined whether or not it is used for interpolation (synthesis) of the pixel data. For example, in the corresponding coordinate point list shown in FIG. 2, it is determined that the input coordinate point X1 is not used for interpolation of any output coordinate point (pixel of the output image), and the input coordinate point Y3 is the output coordinate point of the output data. It is determined to be used for interpolation of A3 and B3.

  When it is determined that the input coordinate point is used for interpolation of any output coordinate point, the pixel data of the input coordinate point is sent to the synthesis processing unit 29 and is output to the output coordinate point (pixel of the output image). It is associated and stored in a predetermined storage area. For each output coordinate point (pixel of the output image), the number of input coordinate points that require interpolation is determined in advance. FIG. 3 shows an example in which four input coordinate points are used for interpolation of one output coordinate point. The composition processing unit 29 determines whether or not input coordinate points necessary for interpolation are aligned for each output coordinate point, performs interpolation processing for the aligned output coordinate points, and outputs the result to the buffer memory 17. In FIG. 3, since input coordinate points necessary for interpolation are aligned at the output coordinate point corresponding to the pixel D, interpolation processing is performed and output to the buffer memory 17. Since the output coordinate points corresponding to the pixels A to C are not yet required, the interpolation process is not performed and the output coordinate points are not output to the buffer memory 17.

  As described above, in the distortion correction processing of this embodiment, it is determined whether or not each input coordinate point of the original data is used for interpolation of the output coordinate point, and is used for interpolation of the output coordinate point. Only the pixel data of the determined input coordinate point is stored in association with the output coordinate point (pixel of the output image). Then, the output coordinate points are interpolated in the order in which the necessary input coordinate point information is satisfied and output. For this reason, the scanning order of the output image does not match the output order of the corrected data. Here, the scanning order of the output image is the order in which the output image once stored in the buffer memory 17 is read again by the memory controller 23, and moves in the horizontal direction from the upper left of the screen. Is the order of repeating vertically downward.

FIG. 4 is a flowchart showing distortion correction processing by the imaging apparatus of the present embodiment.
The image data picked up by the CCD 11 is subjected to predetermined processing by the preprocessing unit 25 and the YC processing unit 27 and then stored as original data in the buffer memory 17 (S31). Next, the coordinate calculation unit 33 starts generating a corresponding coordinate point list indicating which output coordinate point of the output data is used for interpolation of the input coordinate point of the original data (S32). The generated corresponding coordinate point list is stored in the FIFO memory 41.

  The memory controller 23 reads the pixel data of one input coordinate point x1 from the buffer memory 17 (S33). When the input coordinate point x1 can be read (NO in S34), the memory controller 23 sequentially reads the data of the corresponding coordinate point list from the FIFO memory 41, and determines whether or not the input coordinate point x1 exists in the corresponding coordinate point list. (S35). That is, with reference to the corresponding coordinate point list, it is determined whether or not the input coordinate point x1 is used for interpolation of any output coordinate point.

  When there is no input coordinate point x1 in the corresponding coordinate point list, that is, when it is determined that the input coordinate point x1 is not used for interpolation of any output coordinate point, the input coordinate point is not particularly processed. The process proceeds to the next input coordinate point (S39), and the same processing as described above is performed. Note that the memory controller 23 reads and processes pixel data from the pixel at the upper left corner of the image sequentially from the top to the bottom for each line and from the left to the right during the same line.

  When the memory controller 23 has the input coordinate point x1 in the corresponding coordinate point list, that is, when it is determined that the input coordinate point x1 is used for interpolation of any output coordinate point, the memory controller 23 sets the input coordinate point x1. The pixel data is sent to the synthesis processing unit 29, and interpolation processing is performed by the synthesis processing unit 29 (S36). Specifically, the composition processing unit 29 holds the pixel data of the input coordinate point x1 in a storage area corresponding to the output coordinate point associated with the input coordinate point by the corresponding coordinate point list. After that, the composition processing unit 29 determines whether or not the number of input coordinate points necessary for interpolation has been prepared for each output coordinate point, and performs interpolation processing using the pixel data of the input coordinate points stored when the output coordinate points are aligned ( The pixel data of the output coordinate point, that is, the corrected pixel data is obtained.

  After completion of the interpolation process, the memory controller 23 determines whether there is an output coordinate point for which interpolation has been completed (S37). If there is an output coordinate point for which interpolation has been completed, the pixel data of the output coordinate point is stored in the buffer memory 17. Recording is performed (S38). When it is determined that there is no output coordinate point for which interpolation has been completed, the process proceeds to the next input coordinate point (S39), and the same processing as described above is repeated.

  On the other hand, in step S34, when reading of all the input coordinate points x1 of the original data from the buffer memory 17 is finished (YES in S34), the process is finished.

(Output status of distortion-corrected data)
With reference to FIGS. 5 and 6, the output state of the distortion-corrected data according to this embodiment will be described. FIG. 5A shows the original data, and FIGS. 5B to 5D show the progress of the distortion correction process in the output data. FIG. 6 is a diagram showing an output state of image data after distortion correction to the buffer memory 17.

  When the first line of the original data is scanned, the data after the input coordinate point A ′ (on the right side of A ′) is read, but the input coordinate point B ′ necessary for interpolation of the output coordinate points B and C, The data of C ′ has not been read yet. Therefore, at this time, as shown in FIG. 5B, the interpolation processing is completed only for a part of the output coordinate point group including the output coordinate point A, and is output to the buffer memory 17. Therefore, the output state of the corrected data to the buffer memory 17 is as shown in FIG.

  Next, when the second line of the original data is scanned, the data after the input coordinate point B ′ is read, but the data of the pixel C ′ necessary for interpolation of the output coordinate point C is not yet read. . Therefore, at this time, as shown in FIG. 5C, the data between the output coordinate point B and the output coordinate point A is newly interpolated in the first line of the output data, and is output to the buffer memory 17. . Further, the output coordinate point at the center is also interpolated in the second line of the output data. Therefore, the output state of the corrected data to the buffer memory 17 is as shown in FIG.

  Further, when the third line of the original data is scanned, data after the input coordinate point C ′ is read. Therefore, at this time, as shown in FIG. 5D, the data between the output coordinate point C and the output coordinate point B is newly interpolated and output to the buffer memory 17 in the first line of the output data. Further, the output coordinate point at the center is also interpolated in the second and third lines of the output data. Therefore, the output state of the corrected data to the buffer memory 17 is as shown in FIG.

  As described above, in this embodiment, even when the original data is scanned from the upper left end of the image, the image data after distortion correction is not output to the buffer memory 17 from the upper left end of the image, and the interpolation is completed. From the selected pixel, that is, from the center of the line as shown in FIG. Therefore, the output order of the distortion-corrected pixels to the buffer memory 17 is different from the order in which the corrected output image stored in the buffer memory 17 is read again from the buffer memory 17 (pixel order of the output image).

(Summary)
As described above, according to the imaging apparatus of the present embodiment, when performing interpolation in the distortion correction process, it is only necessary to have a line memory having a capacity capable of storing input image data for the number of lines necessary for the interpolation process. There is no need to store input image data for many lines. Therefore, according to the imaging apparatus of the present embodiment, the line memory capacity can be reduced, and the configuration of the image apparatus can be simplified.

  Further, according to the imaging apparatus of the present embodiment, the inverse mapping point in the original data corresponding to the output coordinate point of the output data is obtained, and the pixel data of the output coordinate point is obtained using the pixel data near the inverse mapping point. ing. Since the pixel data obtained by such a method is always on the grid points of the output coordinate system and is interpolated from the input coordinate system without distortion, a high-quality image containing no error can be obtained. On the other hand, in the conventional technique in which the input image is mapped once and then interpolated, the mapping point is not necessarily on the lattice point of the output image, and it approximates to a square lattice although the mapped coordinates are distorted. Since interpolation has been performed, there has been a problem that the output data includes an error. However, according to the imaging apparatus of the present embodiment, the pixel data is always on the grid points of the output coordinate system for the above reason. Can solve the conventional problem.

Embodiment 2
FIG. 7 is a diagram illustrating a configuration of an imaging apparatus according to another embodiment of the present invention.
The imaging apparatus according to the present embodiment further includes a cache memory 43 between the synthesis processing unit 29 and the memory controller 23 in addition to the configuration shown in FIG. The cache memory 43 stores the pixels for which interpolation (combination) has been completed in the composition processing unit 29 in units of pages. The basic operation of the imaging apparatus of the present embodiment is the same as that of the first embodiment, but the contents of the distortion correction processing relating to the cache memory 43 are different. In the present embodiment, the memory controller 23, the composition processing unit 29, the coordinate calculation unit 33, the memory 35, the FIFO memory 41, and the cache memory 43 constitute a correction processing unit.

  FIG. 8 is a diagram for explaining the distortion correction processing of this embodiment. The pixel data of the input coordinate point of the original data is sequentially read out from the buffer memory 17, and the corresponding coordinate point list stored in the FIFO memory 41 is referred to, and the input coordinate point is used for interpolation of any output coordinate point. Determine whether or not. If it is determined that it is used for interpolation of any output coordinate point, the information of the input coordinate point is sent to the composition processing unit 29. In the synthesis processing unit 29, pixel data of the input coordinate point is stored in association with the output coordinate point (pixel of the output image). The composition processing unit 29 determines whether or not input coordinate points necessary for interpolation are aligned for each output coordinate point, performs interpolation processing for the aligned output coordinate points, and outputs the result to the cache memory 43.

  The cache memory 43 manages output coordinate points (pixels of the output image) in units of pages, and outputs a page data to the buffer memory 17 when a page in which the output coordinate points are satisfied is generated. In the example of FIG. 8, page 2 data is output to the buffer memory 17. In the cache memory 43, the data area output to the buffer memory 17 is released.

  Thus, in this embodiment, pixel data is output to the buffer memory 17 in units of pages. In the example of FIG. 8, one page is composed of two pixels, but one page may be composed of more pixels.

  FIG. 9 is a flowchart illustrating distortion correction processing by the imaging apparatus according to the present embodiment. In addition to the steps of the flowchart shown in FIG. 4, the flowchart of FIG. 9 further includes steps S37a and S37b between step S37 and step S38, and steps S40 and S41 between steps S34 and S38.

  In this embodiment, the pixel data of the output coordinate point (pixel of the output image) for which interpolation has been completed is temporarily output from the synthesis processing unit 29 to the cache memory 43 (S37a), and the memory controller 23 stores the pixel data in the cache memory 43. The data of the page is output to the buffer memory 17 from the page satisfying (S37b, S38). In addition, when reading of all input coordinate points x1 of the original data from the buffer memory 17 is completed (YES in S34), data of a page that includes only some pixel data and is not completely satisfied is cached. The data is output from the memory 43 to the buffer memory 17 (S40, S41).

  In this embodiment, since the interpolated data is output to the buffer memory 17 in units of pages, the number of accesses to the buffer memory 17 can be further reduced and the processing speed can be further improved as compared with the case of the first embodiment. be able to.

Embodiment 3
FIG. 10 is a diagram illustrating a configuration of an imaging apparatus according to still another embodiment of the present invention.
The imaging apparatus includes a lens 3, a CCD 11 that is an imaging element that converts optical information into an electrical signal, an AD converter 13 that converts an analog signal into a digital signal, and an image processing circuit 15 that controls the overall operation of the imaging apparatus. A buffer memory 17 that temporarily stores image data before and after distortion correction, and a card slot 21 in which a memory card 19 is mounted. The imaging apparatus further includes a motor 9 for moving the lens 3 in the focus operation and the zoom operation.

  The image processing circuit 15 includes a memory controller 23 that controls the operation of each processing unit in the image processing circuit 15, a preprocessing unit 25 that performs preprocessing such as gamma correction on image information, and a YC processing unit that performs YC processing. 27, a composition processing unit 29 that performs interpolation for distortion correction processing, and a compression / decompression processing unit 31 that performs compression / decompression processing on an image. The image processing circuit 15 further includes a coordinate calculation unit 33, a memory 35, and a magnification acquisition unit 37. In the present embodiment, the memory controller 23, the composition processing unit 29, the coordinate calculation unit 33, and the memory 35 constitute a correction processing unit.

The coordinate calculation unit 33 has a function of calculating an inverse map f ⁻¹ for distortion correction. The magnification acquisition unit 37 acquires information regarding the zoom magnification from the motor 9. The memory 35 stores information necessary for calculating the inverse map f ⁻¹ (information on the function of the inverse map f ⁻¹ for various conditions).

The operation of the imaging apparatus configured as described above will be described.
In the imaging apparatus, optical information input through the lens 3 is converted into an analog electrical signal by the CCD 11. The AD converter 13 converts the analog electric signal from the CCD 11 into image information of a digital signal. The image processing circuit 15 receives image information of a digital signal from the AD converter 13, performs predetermined image processing, and then records it on the memory card 19.

  More specifically, in the image processing circuit 15, preprocessing such as gamma correction is performed on the image data from the AD converter 13 by the preprocessing unit 25, and YC processing is performed by the YC processing unit 27. . The image data subjected to these processes is stored in the buffer memory 17.

  Next, distortion correction is performed on the image data stored in the buffer memory 17 by the synthesis processing unit 29, the coordinate calculation unit 33, and the like. The image data after distortion correction is stored in the buffer memory 17 again.

Next, details of distortion correction processing by the imaging apparatus of the present embodiment will be described.
First, an outline of distortion correction processing by the imaging apparatus of the present embodiment will be described with reference to FIG. In the distortion correction processing of the present embodiment, the following processing is performed for each pixel while sequentially scanning all the pixels constituting the output data.

For one output coordinate point xo on the output data, a corresponding point P on the input coordinate system of the original data is obtained by inverse mapping f ⁻¹ . Next, the value of the pixel data of the corresponding point P is obtained by interpolating using the pixel data of the input coordinate point in the vicinity of the corresponding point P on the original data. Then, the pixel data value of the corresponding point P obtained in this way is set as the pixel data of the output coordinate point xo.

For example, in FIG. 11, the pixel data of the output coordinate point x01 is obtained as follows. The inverse map f ⁻¹ of the output coordinate point x01 is calculated, and the corresponding point P1 (= f ⁻¹ (x01)) on the original data is obtained. Then, by interpolating (combining) the pixel data of the input coordinate points xa, xa + 1, xb, xb + 1 around the corresponding point P1, the value of the pixel data of the corresponding point P1 is obtained. Similarly, the corresponding point Pm is obtained for another output coordinate point x0m, and interpolation is performed using the input coordinate points xm, xm + 1, xk, and xk + 1 around the corresponding point Pm. Obtain pixel data.

  Since the output coordinate points x01 and x0m for which the pixel data is obtained by the above method are always on the grid points of the output coordinate system, the phase does not become discontinuous in the output data, and the image quality caused by the discontinuity It will not cause any deterioration.

FIG. 12 is a flowchart illustrating distortion correction processing by the imaging apparatus according to the present embodiment.
The image data picked up by the CCD 11 is subjected to predetermined processing by the preprocessing unit 25 and the YC processing unit 27 and then stored as original data in the buffer memory 17 (S11). The coordinate calculation unit 33 calculates the inverse mapping point f ⁻¹ (x0) for one output coordinate point x0 (S12). At this time, the coordinate calculation unit 33 acquires an appropriate inverse mapping function f ⁻¹ from the conversion information stored in the memory 35 based on the magnification information from the magnification acquisition unit 37.

Memory controller 23 refers to the calculated inverse mapping point f ^-1 (x0), the pixels of the input coordinate points to be used for interpolation (synthesis) for obtaining the pixel data of the inverse mapping point f ^-1 (x0) Data is acquired from the buffer memory 17 (S13). For example, when the inverse mapping point f ⁻¹ (x0) is the inverse mapping point P1 in FIG. 11, the pixel data of the input coordinate points xa, xa + 1, xb, and xb + 1 around it are acquired.

The memory controller 23 interpolates (synthesizes) the obtained pixel data to obtain the pixel data of the inverse mapping point f ⁻¹ (x0) (S14). The pixel data obtained by interpolation in this way is recorded in the buffer memory 17 as data after distortion correction for the output coordinate point x0 (S15).

  The memory controller 23 performs the same processing for the next output coordinate point of the output data (S16, S17). The above processing is repeated until the above correction processing is performed for all output coordinate points of the output data. In the reverse mapping, the output data is scanned line by line from the output coordinate point at the upper left corner of the image and sequentially from the left in the same line. As described above, image data after distortion correction is obtained.

  As described above, in the imaging apparatus of the present embodiment, the inverse mapping point in the original data corresponding to the output coordinate point of the output data is obtained, and the pixel data of the output coordinate point is obtained using the pixel data in the vicinity of the inverse mapping point. Ask. Since the pixel data obtained by such a method is always on the grid points of the output coordinate system, the output data does not have a discontinuous phase, and a high-quality image without deterioration due to distortion correction can be obtained. .

Embodiment 4
FIG. 13 is a diagram illustrating a configuration of an imaging apparatus according to still another embodiment of the present invention.
The imaging apparatus according to the present embodiment further includes a line memory 45 in addition to the configuration shown in FIG. The line memory 45 stores data of input images (distortion correction target images) for a predetermined number of lines. In the present embodiment, the line memory 45 stores input image data for two lines. The basic operation of the imaging apparatus of this embodiment is the same as that of the above-described embodiment, but the content of the distortion correction process is different. In the present embodiment, the memory controller 23, the composition processing unit 29, the coordinate calculation unit 33, the memory 35, and the line memory 45 constitute a correction processing unit.

With reference to FIG. 14, an outline of distortion correction processing of the imaging apparatus of the present embodiment will be described.
The line memory 45 stores data of two lines of original data read from the buffer memory 17 (hereinafter referred to as “input line data”). The following processing is performed for each (output coordinate point) while sequentially scanning all output coordinate points (pixels of the output image) constituting the output data.

An inverse mapping point f ⁻¹ (x0) is obtained for one output coordinate point xo on the output data. Whether or not the inverse mapping point f ⁻¹ (x0) obtained in this way exists in the area of the image indicated by the input line data stored in the line memory 45 (hereinafter referred to as “input line area”). Judge whether or not. When the inverse mapping point f ⁻¹ (x0) exists in the input line area, the pixel data value of the input coordinate point stored in the line memory 45 in the vicinity of the inverse mapping point f ⁻¹ (x0) is calculated. The value of the pixel data at the inverse mapping point f ⁻¹ (x0) is obtained by interpolation (combination). The value obtained in this way is used as the pixel data of the original output coordinate point x0.

If the inverse mapping point f ⁻¹ (x0) is outside the input line area, the process proceeds to the next output coordinate point. Here, the next output coordinate point is determined in order along a path determined so as to avoid the interpolated (synthesized) area in the output data. For example, in the case shown in FIG. 15, since the center region R of the first and second lines has already been interpolated, a route along the periphery of the region R as shown by the broken line T is determined and output along that route. The coordinate points are advanced sequentially. In this way, by sequentially determining the coordinate points to be corrected along the path along the interpolated area, it is possible to avoid double processing for already processed coordinate points, thus reducing unnecessary processing. Processing efficiency can be improved.

  Interpolation is performed while advancing the output coordinate point in the above manner, and when the processing of the last output coordinate point on the line (the right end of the image) is completed, the input line data in the line memory 45 is changed. Advance one line. Note that to advance the input line data by one line is to update the data in the m + 1st line and the m + 2th line when the line memory 45 stores the data in the mth line and the m + 1st line.

  Details of the distortion correction processing by the imaging apparatus of the present embodiment will be described with reference to the flowchart of FIG.

  The image data picked up by the CCD 11 is subjected to predetermined processing by the preprocessing unit 25 and the YC processing unit 27 and then stored as original data in the buffer memory 17 (S51). Next, the memory controller 23 stores the data for the first two lines of the original data in the line memory 45 as input line data (S52).

The coordinate calculation unit 33 calculates the inverse mapping point f ⁻¹ (x0) for one output coordinate point x0 (S53). At this time, the coordinate calculation unit 33 acquires an appropriate inverse mapping function f ⁻¹ from the conversion information stored in the memory 35 based on the magnification information from the magnification acquisition unit 37.

The memory controller 23 determines whether or not the calculated vertical coordinate of the inverse mapping point f ⁻¹ (x0) matches the vertical coordinate of the input line data stored in the line memory 45 (S54). That is, the memory controller 23 determines whether or not the inverse mapping point f ⁻¹ (x0) exists in the input line area.

When the vertical coordinates of the two do not match, that is, when the inverse mapping point f ⁻¹ (x0) is outside the pixel area stored in the line memory 45, the process proceeds to the next output coordinate point (S59). . At this time, the next output coordinate point is determined along the interpolated (synthesized) area (see FIG. 15). For example, a region that has been interpolated (synthesized) among output coordinate points may be stored, and it may be determined whether or not interpolation (synthesized) has been completed when proceeding to a new output coordinate point. In addition, information indicating the boundary between the non-interpolated area and the already-interpolated area may be stored, and a new output coordinate point may be advanced based on the information. For example, when an uninterpolated area is scanned, information “DOWN” is stored in association with a pixel whose uninterpolated area changes to the lower line, and the uninterpolated area changes to the upper line. Information “UP” is stored in association with the pixel. Then, when scanning a certain line, the scanning line may be changed up and down to advance to a new output coordinate point in accordance with the information of “DOWN” and “UP” in the previous line.

When the output coordinate point advanced in this way exceeds the right end of the output image, the line of the image data stored in the line memory 45 is advanced by one line (S61). That is, the input line data in the line memory 45 is updated from the data of the mth line and the (m + 1) th line to the data of the (m + 1) th line and the (m + 2) th line. Thereafter, for the next input line data, it is determined whether or not the inverse mapping point f ⁻¹ (x0) exists in the input line area by the same method as described above (S53 to S54).

When the inverse mapping point f ^-1 (x0) exists in the input line area (YES in S54), the memory controller 23 performs pixel data of pixels in the vicinity of the inverse mapping point f ^-1 (x0) in the input line data. Is acquired (S55). The composition processing unit 29 interpolates (synthesizes) using the acquired pixel data, and obtains pixel data of the inverse mapping point f ⁻¹ (x0) (S56).

  The memory controller 23 records the pixel data thus obtained by interpolation in the buffer memory 17 as data after distortion correction for the output coordinate point x0 (S57). Then, the output coordinate point x0 is stored in the memory 35 as information on the output (combined) output coordinate point (S58). By referring to the stored information, the memory controller 23 can determine the next output coordinate point while avoiding the processed output coordinate point in step S59.

  Thereafter, the process proceeds to the next output coordinate point (S59), and the above-described processing is executed for each output coordinate point until the processing for the data for one screen is completed (S62).

  With reference to FIG. 17, the change of the input line data in the line memory 45 based on the above flow and the progress of the output coordinate point in the output data will be described with a specific example. Now, consider image areas A, B, C, and D in the output data as shown in FIG. The output coordinate points (pixels of the output image) in the areas A, C, and D are subjected to interpolation processing using the data of the third and fourth lines of the original data, and the output coordinate points in the area B are the original data It is assumed that interpolation processing is performed using data on the first and second lines. In addition, the arrow in the output data of FIG. 17 shows the advancing direction of the output coordinate point used as a process target. In this case, the distortion correction process of the present embodiment proceeds as follows.

  First, as shown in FIG. 17B, the line memory 45 stores the first and second line data of the original data as input line data. In the output data, the inverse mapping point is obtained (that is, scanned) sequentially from the output coordinate point (pixel of the output image) at the left end of the area A, and it is determined whether the inverse mapping point is in the input line area. To do. Since the area A is interpolated using the data of the third and fourth lines, at the time of the first scan for the output image, the inverse mapping point of the output coordinate point of the area A is the input line area. It is out of range and therefore no interpolation processing is performed. Since the area B is interpolated using the first and second line data of the original data, the area B is within the input line area, and the output coordinate point of the area B is interpolated. As with the area A, the output coordinate point of the area C is not interpolated.

  When the first scan is completed, that is, when the above processing is completed for the output coordinate point at the right end of the first line of the output data, the input line data is updated by one line as shown in FIG. The That is, the data in the line memory 45 is updated to the data of the second and third lines of the original data. Then, an inverse mapping point is obtained for the output coordinate point that has not been interpolated, and interpolation processing is performed while referring to the newly updated line memory 45. The scanning direction of the output coordinate point at this time is determined so as to be along the already interpolated image region (region B). That is, in the example of FIG. 17C, the output coordinate point to be processed is advanced to regions A, D, and C in the second scan. In the second scan, the line memory 45 stores the data of the second and third lines of the original data. Even at this time, the interpolation processing is performed using the data of the third and fourth lines of the original data. The output coordinate points of the regions A, C, and D in which are performed are not interpolated.

  When the second scan is completed, as shown in FIG. 17D, the input line data in the line memory 45 is updated to the data of the third and fourth lines of the original data. Then, a third scan is performed. In the third scan, since the inverse mapping points of the output coordinate points of the areas A, D, and C are within the image range of the data stored in the line memory 45, at this time, the areas A, D, and C Each output coordinate point is interpolated.

  When the third scan is completed, that is, when the interpolation process of the output coordinate point of the rightmost region C is completed, the data in the line memory 45 is the fourth and fifth lines of the original data as shown in FIG. The same processing is performed for output coordinate points that are updated to eye data and are not interpolated thereafter.

As described above, in this embodiment, the inverse mapping point f ⁻¹ (x0) of each output coordinate point x0 in the output data is compared with the input line data read to the line memory 45, and the output coordinate point is based on the comparison result. Determine whether x0 correction processing is possible.

  As described above, according to the imaging apparatus of the present embodiment, when performing interpolation in the distortion correction process, it is only necessary to have a line memory having a capacity capable of storing input image data for the number of lines necessary for the interpolation process. There is no need to store input image data for many lines. Therefore, according to the imaging apparatus of the present embodiment, the line memory capacity can be reduced, and the configuration of the image apparatus can be simplified.

  Also, as in the prior art, the method of obtaining the mapping at the output coordinate of the input pixel with reference to the coordinate point of the input data and obtaining the pixel data of the output coordinate point by linearly interpolating these is the input mapped to the output coordinate Since the coordinates are distorted nonlinearly, only an approximate value can be obtained, and a discontinuity occurs in the output image where the polarity of the error between the approximate value and the true value is reversed between pixels of the output image. However, according to the imaging apparatus of this embodiment, the inverse mapping point in the original data corresponding to the output coordinate point of the output data is obtained, and the pixel data of the output coordinate point is obtained using the pixel data in the vicinity of the inverse mapping point. Therefore, since the pixel data obtained by this method is a pixel on the grid point of the true output coordinate system, the phase does not become discontinuous in the output data, and a high-quality image without deterioration due to distortion correction Is obtained.

In step S54 of the present embodiment, an example in which output pixels are generated from input pixels for two lines has been described. That is, it is determined whether or not the reverse mapping point f ⁻¹ (x0) exists in the input line area by determining whether or not the reverse mapping point of the output pixel exists between the two lines of the input pixel. . However, the present invention is not limited to this. For example, output pixels may be generated from input pixels on three or more lines. Further, it may be determined whether or not a reverse mapping point exists in the input line area by determining whether or not a reverse mapping point exists in the intermediate line ± 0.5 line of the input pixel.

  In the present embodiment, a cache memory 43 may be provided as in the second embodiment.

  The present invention is an image pickup apparatus that picks up an image using an image pickup element, and is particularly applicable to an image pickup apparatus that electronically corrects image distortion caused by lens characteristics.

The figure which shows the structure of the imaging device of Embodiment 1 of this invention. The figure which showed the correspondence between the pixels of original data and output data, and the structural example of the data stored in FIFO memory The figure for demonstrating the distortion correction process in Embodiment 1. FIG. Flowchart of distortion correction processing by the imaging apparatus of Embodiment 1 The figure explaining the progress situation of distortion amendment processing by Embodiment 1. The figure explaining the production | generation condition of the image data after the distortion correction by the distortion correction process of Embodiment 1. The figure which shows the structure of the imaging device of Embodiment 2 of this invention. The figure for demonstrating the distortion correction process in Embodiment 2. FIG. Flowchart of distortion correction processing by imaging device of embodiment 2 The figure which shows the structure of the imaging device of Embodiment 3 of this invention. The figure for demonstrating the concept of the distortion correction process in Embodiment 3. FIG. Flowchart of distortion correction processing by imaging device of embodiment 3 The figure which shows the structure of the imaging device of Embodiment 4 of this invention. The figure for demonstrating the distortion correction process in Embodiment 4. FIG. 10 is a diagram for explaining a scan direction of pixels on an output image according to a fourth embodiment. Flowchart of distortion correction processing by imaging device of embodiment 4 The figure for demonstrating the distortion correction process in Embodiment 4 concretely. The figure explaining the image distorted due to lens characteristics and the image after distortion correction The figure for demonstrating the conventional distortion correction process

Explanation of symbols

3 Lens 9 Motor 11 CCD
13 AD converter 15 Controller 17 Buffer memory 19 Memory card 23 Memory controller 25 Preprocessing unit 27 YC processing unit 29 Compositing processing unit 31 Compression / decompression unit 33 Coordinate calculation unit 35 Memory 37 Magnification acquisition unit 41 FIFO memory 43 Cache memory 45 Line memory S Subject

Claims (9)

An image processing apparatus that electronically corrects distortion of an input image and generates an output image,
A correction processing unit that sequentially reads input image data for each pixel in the scanning order, and performs correction processing for each pixel of the output image using the read pixel data,
The correction processing unit
A line memory that temporarily stores data of input pixels for a predetermined number of lines that are a part of the input image data;
A control unit that determines whether data of a plurality of input pixels necessary for generating data of a certain output pixel is stored in the line memory;
When the control unit determines that data of a plurality of input pixels necessary for generating the data of the output pixel is stored in the line memory, the data of the plurality of input pixels is synthesized to Including a synthesis processing unit that performs correction processing,
The predetermined number of lines is a number equal to the number of lines where each of a plurality of input pixels used to generate data of one output pixel is located,
The correction processing unit outputs pixel data for which the correction processing has been completed in an order different from the scanning order of the output image;
An image processing apparatus. The correction processing unit further includes a coordinate calculation unit for obtaining a reverse mapping point on the input image corresponding to the pixel of the output image,
The control unit determines whether data of the plurality of input pixels is stored in the line memory based on the inverse mapping point obtained by the coordinate calculation unit.
The image processing apparatus according to claim 1 . The correction processing unit includes a pixel of the input image, the using data of the pixel of the input image are synthesized further comprises information associated with a pixel of the output image, the image processing according to claim 1, wherein the apparatus.   The image processing apparatus according to claim 1, wherein the correction processing unit further includes a cache memory that temporarily stores data of pixels for which the correction processing has been completed and outputs the data in units of pages. A lens,
An image sensor that converts optical information input through the lens into electrical image information;
Said an image processing apparatus according to distortion of the image data imaged in any one of claims 1 to 4 electronically corrected by the imaging device, an imaging apparatus, characterized in that. A distortion correction method for generating an output image by electronically correcting distortion of an input image,
Sequentially reading input image data for each pixel in scanning order;
Performing correction processing for each pixel of the output image using the read pixel data in order to obtain data of each pixel of the output image;
Outputting the pixel data for which the correction processing has been completed in an order different from the order of the pixels at the time of reading;
Including
The step of performing the correction process includes:
Data of input pixels for a predetermined number of lines, which is a part of input image data, are temporarily stored in a line memory,
Determining whether data of a plurality of input pixels necessary for generating data of a certain output pixel is stored in the line memory;
When it is determined that the data of a plurality of input pixels necessary for generating the data of the output pixels is stored in the line memory, the correction process is performed by combining the data of the plurality of input pixels. ,
The predetermined number of lines is equal to the number of lines in which each of a plurality of input pixels used to generate data of one output pixel is located.
A distortion correction method characterized by the above. The step of performing the correction process includes:
Find the inverse mapping point on the input image corresponding to the pixel of the output image,
Determining whether data of the plurality of input pixels is stored in the line memory based on the obtained inverse mapping point;
The distortion correction method according to claim 6 . In the step of performing the correction process, with reference to information relating the pixels of the input image and the pixels of the output image synthesized using the pixels of the input image, the information of the read pixels is the pixel of the output image. The distortion correction method according to claim 6 , wherein it is determined whether or not it is used for correction. The distortion correction method according to claim 6 , further comprising a step of temporarily storing data of pixels for which the correction processing has been completed and outputting the data in units of pages.

Images