JP2016095667A - Image processing device and electronics apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of suppressing increase in an access band to a memory and reducing degradation of image quality.SOLUTION: An image processing device 1 includes a free deformation part 12 which deforms an inputted image and outputs an output image. The free deformation part 12 divides the output image into a plurality of regions, and performs enlargement processing or reduction processing on each divided region based on a reference region respectively referred to within the inputted image for every divided regions.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus and an electronic apparatus.

  As an electronic device, for example, a digital camera has been rapidly reduced in size, and the built-in photographing lens is also required to be downsized. As for the downsizing of the photographic lens, the conventional one that uses a combination of a large number of lenses is often replaced with one or a small number of small lenses.

  In some cases, the lens diameter is replaced with a smaller one for miniaturization, or the lens material is replaced with an inexpensive one for cost reduction. However, in such a compact and low-priced photographing lens, it is difficult to sufficiently suppress image quality degradation caused by a lens such as so-called chromatic aberration and distortion. The downsizing and widening of the imaging device are factors that increase distortion at the periphery of the captured image.

  Therefore, an imaging apparatus described in Patent Document 1 has been proposed. The imaging device described in Patent Document 1 is a video for each color of R (red), G (green), and B (blue) from an imaging device such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. Retrieve the signal. Next, it is once converted into digital data and temporarily stored in individual field memories. Based on the driving state of the photographing lens, each field memory is individually vector-moved to enlarge or reduce each image. After that, it is described that R, G, and B are combined again to correct a color shift and distortion occurring in the photographing lens.

  In addition, a digital zoom technique that enlarges the magnification of an image by processing an image obtained from an image sensor after distortion correction in an imaging device such as a digital camera is already known.

  However, in the imaging device described in Patent Document 1, data read from the imaging element is temporarily stored in a memory such as SDRAM (Synchronous Dynamic Random Access Memory) and then read for distortion correction. After the distortion correction is performed, an operation of saving in the memory again is performed. Similarly, when digital zooming is performed, an operation is performed in which image data stored in the memory is read out, an enlarged image is generated by processing such as interpolation, and then stored again in the memory.

  For this reason, reading data from the memory and writing data to the memory are frequently performed, resulting in a problem that a bandwidth for accessing the memory is insufficient. In particular, since distortion correction and digital zoom processing are performed on image data before compression, the access bandwidth to the memory is greatly insufficient. For this reason, it is necessary to take measures such as increasing the speed of the system clock, resulting in problems that the power consumption increases and the system configuration becomes complicated.

  In addition, by increasing the number of pixels in an image sensor such as a CCD sensor or CMOS sensor, and widening the angle of the photographic lens, the range of magnification and reduction ratios when enlarging and reducing pixels according to the location of the image during distortion correction is increased. . In this case, depending on the pixel position, a reduction process of 1/2 or less may be performed. For this reason, when digital zoom is performed on an image after distortion correction by image processing, a position for performing enlargement processing appears after the reduction processing, resulting in a problem that image quality deteriorates.

  The present invention aims to solve such problems. That is, an object of the present invention is to provide, for example, an image processing apparatus capable of suppressing an increase in access bandwidth to a memory and reducing deterioration in image quality.

  In order to solve the problems described above, the present invention is an image processing apparatus having a deformation unit that performs a predetermined deformation process on an input image and outputs an output image, wherein the deformation unit includes: The output image is divided into a plurality of regions, and each of the divided regions is subjected to enlargement processing or reduction processing together with the deformation processing based on a reference region that is referred to in the input image. To do.

  According to the present invention, since the enlargement process or the reduction process is performed together with the deformation process for each divided region, it is not necessary to access the memory for each of the deformation process and the enlargement process or the reduction process, and an increase in the memory access bandwidth can be suppressed. it can. In addition, since it is determined whether the enlargement process or the reduction process is performed for each divided region, it is possible to reduce deterioration in image quality.

1 is a block configuration diagram of an image processing apparatus according to a first embodiment of the present invention. It is explanatory drawing of the coordinate data table shown by FIG. It is explanatory drawing of the example of an input image It is explanatory drawing which shows 4 points | pieces of the four corners of 1 area. FIG. 5 is an example of correspondence between coordinate data of an area shown in FIG. 4 and an input image. It is explanatory drawing of the relationship between the pixel number of the reference area (reference area) in the case of only a deformation | transformation process, and the pixel number of an output area. It is explanatory drawing of the expansion operation | movement in the free deformation part 12. FIG. It is explanatory drawing of the relationship between the pixel number of a reference area and the pixel number of an output area in the case of the deformation | transformation process accompanied by an expansion process or a reduction process. It is the flowchart which showed operation | movement of the free deformation | transformation part shown by FIG. It is explanatory drawing of the relationship between the number of pixels of a reference area and the number of pixels of an output area in the case of the deformation | transformation process accompanying the expansion process or the reduction process of the image processing apparatus concerning the 2nd Embodiment of this invention. It is the flowchart which showed operation | movement of the free deformation part concerning the 2nd Embodiment of this invention. It is a block block diagram of the image processing apparatus concerning the 3rd Embodiment of this invention. It is explanatory drawing which showed the example of a blending image. It is explanatory drawing which decomposed | disassembled the image shown in FIG. 13 into two images. It is explanatory drawing of the example of two images after a process was performed by the free deformation | transformation part shown in FIG. It is a timing chart showing operation of an image processing device concerning a 4th embodiment of the present invention. It is a block block diagram of the electronic device concerning one Embodiment of this invention.

(First embodiment)
An image processing apparatus according to a first embodiment of the present invention will be described with reference to FIGS. As illustrated in FIG. 1, the image processing apparatus 1 according to the present embodiment includes a coordinate data table 11, a free deformation unit 12, a memory controller 13, a memory 14, and an enlargement / reduction unit 15. ing.

  A coordinate data table 11 serving as a coordinate storage unit is a table in which an area to be read by the free deformation unit 12 to deform an input image is set. The coordinate data table 11 is stored in a non-volatile memory such as a ROM (Read Only Memory). Alternatively, it may be composed of a volatile memory such as a RAM and loaded from the outside when the power is turned on. The coordinate data table 11 is provided with a plurality of tables for each deformation process such as distortion correction and rotation.

  Based on the coordinate data set in the coordinate data table 11, the free deformation unit 12 as a deformation unit causes the memory controller 13 to read the input image stored in the memory 14. The free deformation unit 12 includes an internal memory such as a RAM, and the read input image is stored in the internal memory. Then, the free deformation unit 12 performs a predetermined deformation process associated with a distortion process, a rotation process, and the like, and an enlargement process or a reduction process described later, based on the input image stored in the internal memory. The image data after the above processing is performed is stored in the memory 14 via the memory controller 13.

  For example, the memory controller 13 stores an input image acquired by an external imaging unit or the like and input to the image processing apparatus 1 in the memory 14. Further, the memory controller 13 reads the input image, the image data after being processed by the free deformation unit 12, and the like from the memory 14 in accordance with instructions from the free deformation unit 12 and the enlargement / reduction unit 15.

  The memory 14 is composed of, for example, an SDRAM or the like, and stores an input image and image data after being processed by the free deformation unit 12 under the control of the memory controller 13.

  The enlargement / reduction unit 15 as the enlargement / reduction means performs an enlargement process or a reduction process on the input image stored in the memory 14 or the image data processed by the free deformation unit 12 and outputs the output image as an output image. Output.

  In the image processing apparatus 1 configured as described above, the memory 14 may be provided outside. Further, the image processing apparatus 1 may be configured as a semiconductor integrated circuit, or may be configured with a printed board, a module, or the like.

  Alternatively, the configuration of FIG. 1 can be configured as a computer program (image processing program) that causes a CPU (Central Processing Unit) or memory of a computer to function. For example, the CPU can function as the free deformation unit 12, the memory controller 13, and the enlargement / reduction unit 15, and the memory can function as the coordinate data table 11 and the memory 14.

  Next, the coordinate data table 11 will be described with reference to FIGS. FIG. 2 is an example of the coordinate data table 11. The coordinate data table 11 includes X coordinate data and Y coordinate data. Here, the X coordinate indicates the main scanning direction of the image (for example, the horizontal direction of the image), and the Y coordinate indicates the sub scanning direction of the image (for example, the vertical direction of the image).

  The coordinate data table 11 indicates pixel positions of a region to be referred to in the input image when processing is performed by the free deformation unit 12 as a set of X coordinate data and Y coordinate data. A specific example will be described with reference to FIG. FIG. 3A shows an input image. FIG. 3B is an output image corresponding to the shaded portion in FIG.

  In the case of FIG. 3, the X coordinate data and the Y coordinate data are set as 4 areas × 4 pixels as one area (region). For example, the pixel referred to in correspondence with the pixel a in FIG. 3B is set to the X coordinate data / Y coordinate data-0 in FIG. Similarly, the pixel referred to corresponding to the pixel b is set to X coordinate data / Y coordinate data-1 and the pixel referred to corresponding to the pixel c is set to X coordinate data / Y coordinate data-2. The pixel referred to in correspondence with d is set to X coordinate data / Y coordinate data-3, and the pixel d is set to X coordinate data / Y coordinate data-15.

  In the X coordinate data / Y coordinate data-16 to X coordinate data / Y coordinate data-31 of FIG. 2, the coordinate data of the pixel corresponding to the area on the right side of the area described in FIG. 3B is set. .

  The memory controller 13 calculates an address of the memory 14 based on the above-described X coordinate data and Y coordinate data, and reads an input image (pixel) of the coordinates indicated by the X coordinate data and the Y coordinate data.

For example, when the data format of the input image is YUV422, the address calculation of the memory 14 by the memory controller 13 is expressed by the following equation (1).
Memory address = memory offset address + X coordinate data × 2 + (Y coordinate data × (number of X pixels of input image × 2)) (1)

  Here, the memory offset address is an offset value of a predetermined address, and the number of X pixels is the number of pixels in the X direction (main scanning direction).

  In the present embodiment, both X coordinate data and Y coordinate data can be set up to 5 decimal places. Therefore, when the decimal point precision (sub-pixel precision) is designated, the free deformation unit 12 performs an interpolation calculation by a bicubic method that is well-known as a pixel interpolation method. In the bicubic method, adjacent peripheral pixels are required, and in this embodiment, 9 pixels are required. Of course, other pixel numbers such as 16 pixels may be used. That is, when the X coordinate data and the Y coordinate data are specified with decimal point precision, it is necessary to read out 9 pixel data for one coordinate set in the X coordinate data and the Y coordinate data. If there are no digits after the decimal point in the X coordinate data and the Y coordinate data, interpolation by the bicubic method is not performed, and reference (reading) of pixels necessary only for the interpolation by the bicubic method is performed. Absent.

  Thus, since nine pixels stored in the memory 14 are referred to one pixel of the output pixel data in the interpolation calculation by the bicubic method, for example, the X coordinate data, the Y coordinate data-0, the X coordinate data, There is a high possibility that the referenced data overlaps with the Y coordinate data-1. Therefore, in the present embodiment, the free deformation unit 12 divides the output image into 4 × 4 pixel area units and handles image data in area units. This 4 × 4 pixel output image area is called an output area or output area image. Then, 16 pieces of reference pixel data (X coordinate data / Y coordinate data 0 to 15) from the coordinate data table 11 are extracted from the memory 14 and stored in the internal memory of the free deformation unit 12, thereby accessing the memory 14. Improves efficiency.

  This output area (output area image) is set in advance, but is not limited to 4 × 4 pixels. The output area (output area image) is a reference for the enlargement process or the reduction process in the free deformation unit 12. When the free deformation unit 12 performs equal magnification processing, the size of the output area is equal to the size of the output result of the free deformation unit 12, but when enlargement processing or reduction processing is performed, the size of the output area The size of the output result of the free deformation unit 12 may be different. Details will be described later.

  In the example shown in FIG. 2, 16 coordinate data per area are set in the coordinate data table 11, but as shown in FIG. 4, only the four points that are the four corners of one area are stored in the coordinate data table. The other 12 points may be calculated by calculation. FIG. 4 is a diagram showing one output area image.

  In FIG. 4, the pixel f is referred to the input image of the coordinates indicated by the X coordinate data / Y coordinate data-0 of FIG. Similarly, the pixel g refers to the input image of the coordinates indicated by the X coordinate data / Y coordinate data-1, and the pixel h refers to the input image of the coordinates indicated by the X coordinate data / Y coordinate data-2. The pixel i is referred to the input image of the coordinates indicated by the X coordinate data / Y coordinate data-3.

  The remaining 12 pixels calculate coordinate data by linear interpolation, for example. An interpolation method other than linear interpolation may be used. Moreover, you may have a parameter which changes an interpolation method for every area.

  FIG. 5 shows a specific example of linear interpolation. FIG. 5A shows specific coordinate data shown in FIG. 4 (the upper row is X coordinate data and the lower row is Y coordinate data). In FIG. 5A, the coordinate data at the positions corresponding to the pixels f, g, h, i in FIG. 4 are values set in the coordinate data table 11 in advance. The coordinate data at other positions is a value calculated by linear interpolation. In the case of FIG. 5A, an area having a shape as shown in FIG. 5B is referenced with respect to the input image (note that FIG. 5B includes an interpolation by the bicubic method). Absent).

  FIG. 6 shows the relationship between the number of pixels in the reference area (reference region) and the number of pixels in the output area in the case of only deformation processing. Here, the output area refers to one divided area when the output image is divided into a predetermined arbitrary size as described above. In the example of FIG. 6, the number of pixels is 4 × 4 pixels (rectangular shape). Let it be a pixel. Further, the upper part of FIG. 6 shows the reference area, and the lower part shows the result of processing with reference to the reference area by the free deformation section 12.

  FIG. 6A shows a case where the number of pixels in the output area <the number of pixels in the reference area. FIG. 6B shows the case where the number of pixels in the output area> the number of pixels in the reference area, FIG. 6C shows the case where the number of pixels in the output area = the number of pixels in the reference area, and FIG. This is the case where the number <the number of pixels in the reference area.

  Note that, when the reference area is designated with the above-described sub-pixel accuracy, interpolation by the bicubic method is required, and therefore the reference area may not necessarily be a rectangle as shown in FIG. Further, depending on the content of the deformation process, an area that is not 4 × 4 pixels may be referred to. Therefore, the number of pixels in the reference area may be calculated based on the coordinate data table 11 using, for example, a well-known coordinate method.

  In FIG. 6, since the enlargement process and the reduction process are not performed, the processing results in the lower stage are all fixed at 4 × 4 pixels. That is, only the deformation (for example, FIG. 5) based on the reference area corresponding to the coordinates shown in the coordinate data table 11 is performed.

  Next, the expansion operation in the free deformation portion 12 will be described with reference to FIG. For example, when enlarging from 4 × 4 pixels to 5 × 5 pixels, the enlarging process can be performed using the coordinate data of the four corners set in the coordinate data table 11 as they are.

  As shown in FIG. 5, when only the four corners of one area are set in the coordinate data table 11 (when the coordinate positions of the input image corresponding to the four vertices of the rectangle are stored), other than the four corners The enlargement process is performed by increasing the number of pixels. For example, as shown in FIG. 7, in the case of 4 × 4 pixels, the coordinate data of 2 pixels between the vertices was calculated by linear interpolation, but when expanding to 5 × 5 pixels, between the vertices The coordinate data may be calculated with 3 pixels. In this way, an enlargement process of 1.25 times can be performed.

  In the case of FIG. 7, the coordinate data concerning the pixels f, g, h, i can use the same data for both 4 × 4 pixels and 5 × 5 pixels. Similarly, if it is handled as 6 × 6 pixels, it can be enlarged 1.5 times, and if it is handled as 7 × 7 pixels, it can be enlarged 1.75 times. That is, since the coordinate data is calculated by linear interpolation from four points at the four corners, the enlargement operation can be performed by changing the size of the output image in unit of area output from the free deformation unit 12. Instead of linear interpolation, other interpolation methods may be used.

  The above is an example of enlargement processing, but reduction processing is also possible with the same concept. That is, interpolation processing may be performed so as to reduce the number of pixels between the four corners. That is, the deforming means performs an enlargement process or a reduction process on the divided area based on the coordinate positions of the four vertices of the rectangle stored in the coordinate storage means.

  FIG. 8 shows the relationship between the number of pixels in the reference area and the number of pixels in the output area in the case of deformation processing involving enlargement processing or reduction processing in the free deformation section 12. FIG. 8 shows an example of a case where the image processing apparatus 1 as a whole performs a 1.25-times enlargement process by, for example, digital zoom. Further, the upper part of FIG. 8 shows the result of processing by referring to the reference area, and the lower part of FIG.

  FIG. 8A shows a case where the number of pixels in the output area <the number of pixels in the reference area. 8B shows the case where the number of pixels in the output area> the number of pixels in the reference area, FIG. 8C shows the case where the number of pixels in the output area = the number of pixels in the reference area, and FIG. This is the case where the number <the number of pixels in the reference area. Here, the number of pixels in the output area is 4 × 4 pixels = 16 pixels as in FIG.

In the present embodiment, the criteria for the enlargement process or the reduction process of the free deformation section 12 are determined by the following three.
(1) When the condition of the number of pixels in the output area <the number of pixels in the reference area is satisfied, the free deformation unit 12 performs an enlargement process. That is, when the number of pixels in the reference area is larger than the number of pixels in the divided area, enlargement processing is performed on the divided area.
(2) When the condition of the number of pixels in the output area> the number of pixels in the reference area is satisfied, the free deformation unit 12 performs a reduction process. That is, when the number of pixels in the reference area is smaller than the number of pixels in the divided area, a reduction process is performed on the divided area.
(3) When the condition of the number of pixels in the output area = the number of pixels in the reference area is satisfied, the free deformation unit 12 performs neither enlargement processing nor reduction processing (same size).

  When the above conditions are applied to FIG. 8, in the case of FIG. 8A, since the number of pixels in the output area (16 pixels) <the number of pixels in the reference area (27 pixels), enlargement processing of 1.25 times is performed. Is called. In the case of FIG. 8B, reduction processing is performed because the number of pixels in the output area (16 pixels)> the number of pixels in the reference area (9 pixels). In the case of the reduction process, the size is reduced to the size of the reference area. In the case of FIG. 8C, since the number of pixels in the output area (16 pixels) = the number of pixels in the reference area (16 pixels), neither enlargement processing nor reduction processing is performed. In the case of FIG. 8D, since the number of pixels in the output area (16 pixels) <the number of pixels in the reference area (21 pixels), enlargement processing of 1.25 times is performed.

  When the number of pixels in the output area in FIGS. 8A and 8D is smaller than the number of pixels in the reference area, the image data output by the free deformation unit 12 is subjected to a reduction process on the data in the reference area. Can be regarded as equivalent. For example, a case where the reference area image is 25 × 25 pixels and the output area image is 4 × 4 pixels and the enlargement process is performed twice while performing a predetermined deformation such as distortion correction will be described.

  In this case, only the deformation process is performed in the free deformation unit 12 and doubled by the enlargement / reduction unit 15 in the subsequent stage, the (4/25) × (4/25) pixels are performed in the deformation process in the free deformation unit 12. A double enlargement process is performed after the reduction process. On the other hand, when the free deformation unit 12 performs enlargement together with the deformation process, the free deformation unit 12 performs a process of (8/25) × (8/25) pixels. Since the enlargement / reduction unit 15 does not perform any processing at the same magnification, it can be determined that when the image is enlarged by the free deformation unit 12, the image quality is improved as compared with the enlargement / reduction unit 15.

  When the number of pixels in the output area in FIG. 8B> the number of pixels in the reference area, the reference area image is 3 × 3 pixels. Therefore, even when the size of the reference area image is 3 × 3 pixels, the free deformation unit 12 Then, the processing is substantially equal. Therefore, it can be determined that the image quality does not deteriorate even if the output area image size is reduced. The area of the output image output by the free deformation unit 12 is separately enlarged by the enlargement / reduction unit 15.

  In the case where the number of pixels in the output area = the number of pixels in the reference area in FIG. 8C, the free deformation unit 12 performs the same magnification processing. Therefore, it can be determined that the image quality does not improve even if the free deformation unit 12 enlarges. . Accordingly, the enlargement / reduction unit 15 separately performs enlargement processing on the area of the output image output by the free deformation unit 12.

  The operation (image processing method) of the above-described free deformation unit 12 is summarized in the flowchart of FIG. In the flowchart of FIG. 9, first, in step S <b> 101, the free deformation unit 12 reads the reference area of the area to be processed from the memory 14 via the memory controller 13 based on the coordinate data table 11. Next, in step S102, the free deformation unit 12 determines whether or not there is an enlargement process such as digital zoom. If yes (if YES), the number of pixels in the output area and the number of pixels in the reference area are determined in step S103. Compare On the other hand, when there is no enlargement process (in the case of NO), only the deformation process is performed in step S106, and the process proceeds to step S107.

  In step S103, the free deformation unit 12 performs enlargement processing and deformation processing in step S104 when the number of pixels in the output area <the number of pixels in the reference area. On the other hand, if the number of pixels in the output area> the number of pixels in the reference area, reduction processing and deformation processing are performed in step S105. Here, enlarging and reducing refers to processing compared with the number of pixels in the output area described above.

  When the number of pixels in the output area = the number of pixels in the reference area, the free deformation unit 12 performs only deformation processing in step S106. In step S107, the free deformation unit 12 determines whether or not there is a next area. If there is, the process returns to step S101 and the above-described processing is repeated for the next area.

  As is clear from the above description, steps S101 to S107 correspond to a deformation process. Further, the free deformation unit 12 divides the output image into a plurality of output areas, and enlarges or reduces the output area together with the deformation process based on the reference area that is referred to in the input image for each output area. We are processing.

  According to the present embodiment, in the image processing apparatus 1, the free deformation unit 12 divides the output image into a plurality of areas, and determines the divided areas based on the reference areas that are referred to in the input image for each of the divided areas. Zoom in or out. In this way, since the enlargement process or the reduction process is performed together with the deformation process for each divided area, it is not necessary to access the memory 14 respectively in the deformation process and the enlargement process or the reduction process, and an increase in the memory access bandwidth is suppressed. be able to. In addition, since it is determined whether to enlarge or reduce for each divided region, it is possible to reduce deterioration in image quality.

  The free deformation unit 12 performs a reduction process and a deformation process when the number of pixels in the output area> the number of pixels in the reference area, and performs an expansion process and a deformation process when the number of pixels in the output area <the number of pixels in the reference area. Do. In this way, when the number of pixels in the output area> the number of pixels in the reference area, the free deformation unit 12 uses the same size as the size of the reference area. Does not deteriorate. Further, when the number of pixels in the output area <the number of pixels in the reference area, the enlargement process is performed only by the free deformation unit 12, so that the image quality is improved as compared with the enlargement / reduction unit 15 separately.

  Moreover, the free deformation | transformation part 12 performs only a deformation | transformation process, when the number of pixels of an output area> the number of pixels of a reference area. By doing so, the free deformation unit 12 performs the same magnification processing, and therefore the image quality is not improved even if the free deformation unit 12 enlarges the image. Therefore, the enlargement / reduction unit 15 may perform separate processing.

(Second Embodiment)
Next, an image processing apparatus according to a second embodiment of the present invention will be described with reference to FIGS. Note that the same parts as those in the first embodiment described above are denoted by the same reference numerals and description thereof is omitted.

  The image processing apparatus 1 according to the present embodiment has the same block configuration as that in FIG. 1, but the processing in the free deformation unit 12 is different.

  FIG. 10 shows the relationship between the number of pixels in the reference area and the number of pixels in the output area in the case of the deformation process involving the enlargement process or the reduction process according to the present embodiment. FIG. 10 shows an example in which enlargement processing of 1.25 times is performed on the entire image processing apparatus 1 by digital zoom or the like as in FIG. 10 shows the result of processing with reference to the reference area, and the lower part of FIG. Also, the number of pixels in the output area is 4 × 4 pixels = 16 pixels, as in the first embodiment.

In the present embodiment, the criteria for the enlargement process or the reduction process of the free deformation section 12 are determined according to the following six. In this embodiment, the main scanning direction (X coordinate) is described as one direction, and the sub-scanning direction (Y coordinate) is described as another direction orthogonal to the one direction.
(1) When the condition of the X size of the image in the output area (number of pixels in the main scanning direction) <the X size of the image in the reference area is satisfied, the free deformation unit 12 increases the X size (enlargement process in the X direction). That is, when the number of pixels in one direction of the reference area is larger than the number of pixels in one direction of the divided area, enlargement processing is performed in the one direction of the divided area.
(2) If the condition X size of output area image> X size of reference area image is satisfied, the free deformation unit 12 reduces the X size (reduction processing in the X direction). That is, when the number of pixels in one direction of the reference area is smaller than the number of pixels in one direction of the divided area, the reduction process is performed in one direction of the divided area.
(3) When the condition of the X size of the image in the output area = the X size of the image in the reference area is satisfied, the free deformation unit 12 does not change the X size.

(4) If the condition of the Y size of the output area image (number of pixels in the sub-scanning direction) <the Y size of the reference area image is satisfied, the free deformation unit 12 increases the Y size (enlargement process in the Y direction). That is, when the number of pixels in the other direction of the reference area is larger than the number of pixels in the other direction of the divided area, the enlargement process is performed in the other direction of the divided area.
(5) When the condition of the Y size of the image in the output area> the Y size of the image in the reference area is satisfied, the free deformation unit 12 reduces the Y size (reduction processing in the Y direction). That is, when the number of pixels in the other direction of the reference area is smaller than the number of pixels in the other direction orthogonal to one direction of the divided area, the reduction process is performed in the other direction of the divided area.
(6) When the condition of the Y size of the image in the output area = the Y size of the image in the reference area is satisfied, the free deformation unit 12 does not change the Y size.

  When the above conditions are applied to FIG. 10, in the case of FIG. 10A, the X size of the output area (4 pixels) <the X size of the reference area (6 pixels), so the X size is changed from 4 pixels to 5 pixels. Increase (enlarge 1.25 times). Further, since the Y size of the output area (4 pixels)> the Y size of the reference area (3 pixels), the Y size is reduced from 4 pixels to 3 pixels (reduced to match the size of the reference area).

  In the case of FIG. 10B, since the output area X size (4 pixels)> reference area X size (3 pixels), the X size is reduced from 4 pixels to 3 pixels (to match the size of the reference area). to shrink). Further, since the Y size of the output area (4 pixels)> the Y size of the reference area (3 pixels), the Y size is reduced from 4 pixels to 3 pixels (reduced to match the size of the reference area).

  In the case of FIG. 10C, since the X size of the output area (4 pixels) = the X size of the reference area (4 pixels), the X size is not changed. Further, since the Y size of the output area (4 pixels) = the Y size of the reference area (4 pixels), the Y size is not changed.

  In the case of FIG. 10B, since the output area X size (4 pixels)> reference area X size (3 pixels), the X size is reduced from 4 pixels to 3 pixels (to match the size of the reference area). to shrink). Further, since the Y size of the output area (4 pixels) <the Y size of the reference area (7 pixels), the Y size is increased from 4 pixels to 5 pixels (enlarged by 1.25 times).

  In the case of the present embodiment, since the first embodiment is determined separately for each of the X size and the Y size, each determination of enlargement, reduction, and equal magnification can be considered in the same manner as in the first embodiment. . That is, when the output area X size <the reference area X size and the output area Y size <the reference area Y size, the enlargement / reduction unit 15 does not set the X size or the Y size to the same size. Processing is not performed. Therefore, it can be determined that when the image is enlarged by the free deformation unit 12, the image quality is improved as compared with the enlargement by the enlargement / reduction unit 15.

  In the case where the output area X size> the reference area X size and the output area Y size> the reference area Y size, the free deformation unit 12 performs substantially the same size processing on the X size or the Y size. Therefore, it can be determined that the image quality does not deteriorate even if the output area image size is reduced. The output image output by the free deformation unit 12 is separately enlarged by the enlargement / reduction unit 15.

  When the X size of the output area = the X size of the reference area and the Y size of the output area = the Y size of the reference area, the free deformation unit 12 performs the same magnification processing. It can be determined that the image quality is not improved. Therefore, the output image output by the free deformation unit 12 is separately enlarged by the enlargement / reduction unit 15.

  For example, in the case of FIG. 10A, the enlargement / reduction unit 15 performs enlargement processing with the X size being the same size and the Y size being 1.66 times. In the case of FIG. 10B, enlargement processing of 1.66 times is performed for both the X size and the Y size. In the case of FIG. 10C, enlargement processing of 1.25 times is performed for both the X size and the Y size. In the case of FIG. 10D, an enlargement process of 1.66 times is performed for the X size and an equal magnification process is performed for the Y size. In this way, the enlargement process of 1.25 times is performed for the entire image processing apparatus 1A.

  The operation (image processing method) of the free deformation unit 12 according to the present embodiment is summarized in the flowchart of FIG. In the flowchart of FIG. 11, first, the free deformation unit 12 reads a reference area of an area to be processed based on the coordinate data table 11 in step S <b> 201. Next, in step S202, the free deformation unit 12 determines whether or not there is an enlargement process such as digital zoom. If yes (if YES), the output area X size and the reference area X size are determined in step S203. Compare On the other hand, when there is no enlargement process (in the case of NO), only the deformation process is performed in step S212, and the process proceeds to step S213.

  In step S <b> 203, the free deformation unit 12 increases the X size in step S <b> 204 when the X size of the output area <the X size of the reference area. If the X size of the output area> the X size of the reference area, the X size is reduced in step S205. If the X size of the output area = the X size of the reference area, the X size is not changed in step S206.

  In step S207 advanced from steps S204 to S206, the free deformation unit 12 compares the Y size of the output area with the Y size of the reference area. In step S207, if the Y size of the output area <the Y size of the reference area, the Y size is increased in step S208. If Y size of the output area> Y size of the reference area, the Y size is reduced in step S209. If Y size of the output area = Y size of the reference area, the Y size is not changed in step S210.

  In step S211, the free deformation unit 12 proceeds from steps 208 to S210. The enlargement process or the reduction process based on the X size and the Y size determined in steps S204 to 206 and steps S208 to S210 and the deformation process based on the coordinate data table 11 are performed. I do.

  In step S213, the free deformation unit 12 determines whether or not there is a next area. If there is, the process returns to step S201 to repeat the above-described processing for the next area, and if not, the process ends.

  As is clear from the above description, steps S201 to S213 correspond to the deformation process. In the flowchart shown in FIG. 11, the X size is determined first, but the Y size may be determined first. Further, although the X size and the Y size are determined independently in the flowchart shown in FIG. 11, they may be determined simultaneously.

  According to the present embodiment, when comparing the output area and the reference area, the X size and the Y size are individually compared instead of the number of pixels, and the enlargement process or the reduction process is performed based on the respective comparison results. Is going. In this way, even when the reference area is not rectangular, it is possible to reduce image quality degradation. Therefore, image quality deterioration can be reduced in more various deformation processes.

(Third embodiment)
Next, an image processing apparatus according to a third embodiment of the present invention will be described with reference to FIGS. The same parts as those in the first and second embodiments described above are denoted by the same reference numerals and description thereof is omitted.

  FIG. 12 shows a block configuration diagram of the image processing apparatus 1A according to the present embodiment. In the image processing apparatus 1A shown in FIG. 12, a blending unit 16 is added to the image processing apparatus 1A shown in FIG. 1, and the memory controller 13 is changed to a memory controller 13a. The memory controller 13a is configured to be able to input a plurality of input images (input images A and B). Note that a plurality of input images may be sequentially input with one input as in the memory controller 13 shown in FIG.

A blending unit 16 as a mixing unit performs a blending process on a plurality of input images. The blending process is a process in which a plurality of input images are superimposed to form one output image in which a plurality of input images are mixed. For example, in the case of blending two images, a blending coefficient is set, the following equation (2) is calculated, and pixel data (image data) as a calculation result is output to the memory 14 via the memory controller 13.
(Pixel data of one image × blending coefficient) + (pixel data of the other image × (1−blending coefficient)) (2)

  FIG. 13 is an example of a blended image. In FIG. 13, the end portions of the two images of the A image and the B image are blended. That is, the right end portion of the A image and the left end portion of the B image are superimposed. This superposed area is hereinafter referred to as a blended image. In the present embodiment, the image shown in FIG. 13 and the like is not an image before the process of the free deformation unit 12 but an image after the process (after the deformation).

  That is, as shown in FIG. 14, the A image is composed of 15 areas of areas 1 to 5, areas 11 to 15, and areas 21 to 25. In addition, the B image is composed of 15 areas of area 6 to 10, area 16 to 20, and area 26 to 30. In this case, area 5 and area 6, area 15 and area 16, and area 25 and area 26 overlap each other.

  When the blending process is performed in the blending unit 16, if the enlargement process or the reduction process in the free deformation unit 12 described in the first and second embodiments is performed, for example, in an area included in the blended image, enlargement is performed between two images. The magnification of may be different. If the magnification such as enlargement is different, either the enlargement process or the reduction process must be performed during the blending process, and the processing efficiency as the image processing apparatus 1A is reduced.

  Therefore, in this embodiment, in the free deformation part 12, the area included in the blended image is adjusted to any magnification. Here, the magnification is expressed as a value larger than 1 in the case of enlargement processing, a value less than 1 in the case of reduction processing, and 1 as the same magnification.

  As a specific example, in the case of FIG. 13, when the magnification of the area of the A image> the magnification of the area of the B image, the enlargement process or the reduction process of the area is performed with the magnification of the area of the A image. If the magnification of the area of the A image is smaller than the magnification of the area of the B image, the enlargement process or the reduction process of the area is performed at the magnification of the area of the B image. That is, in the divided areas mixed by the mixing means, the deforming means performs an enlargement process or a reduction process in accordance with the maximum magnification among the enlargement or reduction magnifications calculated for the divided areas of the respective output images. .

  As processing of the free deformation unit 12 in FIG. 14, first, processing of areas A1 to A4 of the A image is performed. Next, area 5 is processed later, area 6 of the B image is subsequently processed, and the two areas are output to the blending unit 16 by combining the magnifications of area 5 and area 6. Then, processing of areas 7 to 10 of the B image is performed. That is, the blending unit 16 processes the area 5 of the A image and the area 6 of the B image included in the blended image. The processing of areas A1 to 4 of the A image and the processing of areas 7 to 10 of the B image are the same as in the first or second embodiment, and the processing by the blending unit 16 is not performed. As described above, in this embodiment, areas that are overlapped with each other in the blended image are processed continuously.

  FIG. 15 shows an example in which the processing as described above is performed. Area5 and area6, area15 and area16, and area25 and area26 included in the blended image have the same magnification.

  According to the present embodiment, when performing the blending process, the free deformation unit 12 performs enlargement or reduction in accordance with a higher magnification among the calculated magnifications in the area included in the blended image. By doing so, it is not necessary to perform enlargement processing or reduction processing in the blending unit 16, and the processing efficiency of the image processing apparatus 1A can be improved without complicating the processing of the blending unit 16.

  In the above-described example, the blending process of two images has been described. However, even in the case of three or more blending processes, the processing efficiency can be improved by similarly adjusting the magnification.

(Fourth embodiment)
Next, an image processing apparatus according to a fourth embodiment of the present invention will be described with reference to FIG. In addition, the same code | symbol is attached | subjected to the same part as the 1st-3rd embodiment mentioned above, and description is abbreviate | omitted.

  In this embodiment, the block configuration may be any of FIG. 1 and FIG. 12, and the operation of the free deformation portion 12 is different.

  If the operation | movement of the free deformation | transformation part 12 as described in the 1st-3rd embodiment is performed, the access band to the memory 14 may run short. For example, in the case of a digital camera, when the input image input from the imaging unit is a moving image, 30 or more images are input per second. Therefore, the image processing apparatus 1 (1A) needs to complete the processing within 33 milliseconds at the latest. However, depending on the processing contents of the free deformation unit 12, the amount of access to the memory 14 increases, and the processing may not be completed within 33 milliseconds.

  In the present embodiment, when memory access is insufficient in the immediately preceding input image, the free deformation unit 12 does not perform processing of the next input image, and further reduces the enlargement processing from the next input image. Only process or same-size processing is allowed.

  A specific example is shown in FIG. FIG. 16 is a timing chart showing the frame start signal and the processing state of the free deformation unit 12. The frame start signal is a signal indicating the start of a frame, and is input from, for example, an imaging unit or the like at the front stage of the image processing apparatus 1 (1A).

  In FIG. 16, the processing of the free deformation unit 12 is not completed within the maximum processing time (one frame period) for the first input image. Accordingly, the second input image has not been processed in the free deformation unit 12 since one frame period has passed. Then, the free deformation unit 12 restarts the process from the third input image, but does not perform the enlargement process. When enlargement processing is necessary, the enlargement / reduction unit 15 separately performs it. That is, the deformation means limits the enlargement process for subsequent input images when the processing of one input image has elapsed for a predetermined time or longer.

  Note that the predetermined period for determining that the enlargement process is not performed is not limited to one frame period, but the image processing apparatus 1 (1A) such as the overhead of processing of the read / write of the memory 14 and the enlargement / reduction unit 15 and the frame rate. It may be changed as appropriate according to the performance required.

  According to the present embodiment, the free deformation unit 12 restricts the enlarging process to be prohibited for subsequent input images when processing of one input image has passed for one frame period or longer. By doing so, the amount of data written by the free deformation unit 12 to the memory 14 and the amount of data read by the enlargement / reduction unit 15 are reduced, and the memory access bandwidth shortage can be solved.

  In addition, since the free deformation unit 12 does not perform processing of an input image immediately after an image for which processing of the input image has passed for one frame period or more, in the case of a moving image or the like in which the input image is input in real time, image degradation Can be minimized.

  Finally, an embodiment of an electronic apparatus having any one of the image processing apparatuses 1 (1A) according to the above-described four embodiments will be described with reference to FIG. FIG. 17 shows an example of the image processing apparatus 1 shown in FIG. 1, but it goes without saying that the image processing apparatus 1A shown in FIG. 12 may be used.

  In the electronic apparatus 100 illustrated in FIG. 17, an image acquisition unit 101 as an image acquisition unit and an image output unit 102 as an image output unit are added to the image processing apparatus 1. For example, when the electronic device 100 is a digital camera, the image acquisition unit 101 corresponds to an imaging element (imaging unit), a conversion unit to a YUV422 format, and the image output unit 102 includes a display unit such as a liquid crystal display, a memory This corresponds to a storage medium such as a card.

  In the electronic device 100 illustrated in FIG. 17, the image acquisition unit 101 converts an image captured by the image sensor into a YUV422 format and inputs the image to the image processing apparatus 1. In the image processing apparatus 1, the distortion correction process and the digital zoom process are performed as in the above-described embodiment, and the image subjected to the distortion correction process and the digital zoom process is displayed on the display unit by the image output unit 102.

  Note that the image acquisition unit 101 is not limited to an image sensor or the like. For example, if the image is stored in a storage medium such as a memory card or a hard disk drive, the storage medium may be used. If the image is acquired via a network or the like, a network interface may be used. May be.

  Similarly, the image output unit 102 is not limited to the display unit. For example, if the image is stored in a storage medium such as a memory card or a hard disk drive, the storage medium may be used. If the image is output via a network or the like, a network interface may be used. Also good.

  The electronic device 100 is not limited to a photographing device such as a digital camera, and may be a video device, an information device, a communication device, an image forming device, or the like having a deformation and enlargement / reduction function such as image distortion correction and rotation. .

  The present invention is not limited to the above embodiment. That is, those skilled in the art can implement various modifications in accordance with conventionally known knowledge without departing from the scope of the present invention. Of course, such modifications are included in the scope of the present invention as long as the configuration of the image processing apparatus of the present invention is provided.

1, 1A Image processing apparatus 11 Coordinate data table (coordinate storage means)
12 Free deformation part (deformation means)
13 Memory controller 14 Memory 15 Enlarging / reducing unit (enlarging / reducing means)
16 Blending part (mixing means)
DESCRIPTION OF SYMBOLS 100 Electronic device 101 Image acquisition part (image acquisition means)
102 Image output unit (image output means)
S101-S107 deformation process

Japanese Patent No. 5154361

Claims (10)

An image processing apparatus having deformation means for performing a predetermined deformation process on an input image and outputting an output image,
The deformation unit divides the output image into a plurality of regions, and enlarges or reduces the divided regions together with the deformation processing based on reference regions that are referred to in the input image for each of the divided regions. An image processing apparatus characterized in that The deformation means includes
When the number of pixels in the reference area is smaller than the number of pixels in the divided area, the reduction process is performed on the divided area, and when the number of pixels in the reference area is large, the enlargement process is performed on the divided area. Apply,
The image processing apparatus according to claim 1. The deformation means includes
When the number of pixels in the one direction of the reference area is smaller than the number of pixels in one direction of the divided area, the reduction process is performed in the one direction of the divided area, and the reference area When the number of pixels in one direction is large, the enlargement process is performed in the one direction of the divided area,
When the number of pixels in the other direction of the reference area is smaller than the number of pixels in the other direction orthogonal to the one direction of the divided area, the reduction process is performed in the other direction of the divided area. When the number of pixels in the other direction of the reference area is large, the enlargement process is performed in the other direction of the divided area.
The image processing apparatus according to claim 1. The divided area is rectangular, and further has coordinate storage means in which coordinate positions of the input image corresponding to at least four vertices of the rectangle are stored,
The said deformation | transformation means performs the said expansion process or the said reduction process to the said division area based on the said coordinate position stored in the said coordinate storage means, The Claim 1 thru | or 3 characterized by the above-mentioned. The image processing apparatus described. It further has a mixing means for mixing a plurality of images,
In the divided region mixed by the mixing unit, the deforming unit may perform the enlargement process or the scale-up according to the maximum magnification among the enlargement or reduction magnifications calculated for the divided regions of the respective output images. The image processing apparatus according to claim 1, wherein a reduction process is performed.   6. The method according to claim 1, wherein when the processing of one input image has passed for a predetermined period or longer, the deformation unit restricts the enlargement processing for subsequent input images. The image processing apparatus according to item.   The image processing apparatus according to claim 1, further comprising an enlarging / reducing unit that performs an enlarging process or a reducing process on an image that has been processed by the deforming unit.   An electronic apparatus comprising the image processing apparatus according to claim 1.   An image acquisition unit that acquires an image, the image processing apparatus according to claim 1, and an image output unit that outputs an image processed by the image processing apparatus. Electronic equipment characterized by An image processing method including a deformation step of performing a predetermined deformation process on an input image and outputting an output image,
The deformation step divides the output image into a plurality of regions, and enlarges or reduces the divided regions together with the deformation processing based on reference regions that are referred to in the input image for each of the divided regions. An image processing method characterized by:

Images