JP2015177244A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make image corrections through which sufficient correction effect on random offset noise is obtained for each row even when the random noise is large.SOLUTION: An image processing apparatus makes pixel corrections on random offset noise for each row by: selecting pixel outputs of a flat part of a subject image among pixel outputs of effective pixels of an imaging element included in image data obtained by the imaging element; calculating first linear data based upon the selected pixel outputs; calculating second linear data by extracting a component of random offset noise for each row from the calculated first linear data; and subtracting the calculated second linear data from the image data for each row.

Description

  The present invention relates to an image processing apparatus and an image processing method.

  When a pixel output is multiplied by a large gain when reading an image from an image sensor having two-dimensionally arranged pixels, random offset noise is generated in the image for each row due to disturbances or the influence of the readout circuit. It was sometimes visually recognized as pattern noise. For such horizontal stripe pattern noise, a correction technique using an output of a light-shielding pixel arranged in an image sensor has been proposed (for example, see Patent Document 1). In addition, a technique for reducing horizontal stripe-like fixed pattern noise by performing filter processing on an effective pixel output or an image arranged in an image sensor has been proposed (see, for example, Patent Document 2).

JP 7-67038 A JP 2011-76467 A

  However, in the technique described in Patent Document 1, when the pixel output is multiplied by a large gain, it is difficult to accurately extract pattern noise components from random noise because the number of light-shielding pixels is small. The effect may not be obtained. In the technique described in Patent Document 2, even with a noise component having a characteristic for each row such as a random offset noise for each row, a filtering process is performed only with information on neighboring pixels. For this reason, in the technique described in Patent Document 2, there is a possibility that a sense of resolution is sacrificed at the edge portion of the subject image or sufficient correction cannot be performed.

  An object of the present invention is to provide an image processing apparatus and an image processing method capable of performing image correction that can provide a sufficient correction effect for random offset noise for each row even when the random noise is large. is there.

  An image processing apparatus according to the present invention uses, as an input image, image data obtained by an image sensor having a plurality of pixels arranged in a two-dimensional shape, and outputs a pixel output of effective pixels in the image sensor included in the image data. Among them, a pixel selection unit that selects a pixel output of a flat portion of the subject image, a first calculation unit that calculates first one-dimensional data based on the pixel output selected by the pixel selection unit, and the first A second calculation unit for calculating second one-dimensional data by extracting a random offset noise component for each row from the first one-dimensional data calculated by the calculation unit; and the second calculation unit A subtracting unit that subtracts the second one-dimensional data calculated by the above-mentioned input image for each row.

  According to the present invention, even when random noise is large, a sufficient correction effect can be obtained in image correction for random offset noise for each row.

It is a figure which shows the structural example of the imaging device which concerns on embodiment of this invention. It is a figure showing the outline of the example of composition of the image sensor concerning this embodiment. It is a figure which shows the structural example of the unit pixel and readout circuit of the image pick-up element which concerns on this embodiment. 6 is a timing chart illustrating an example of driving of an image sensor according to the present embodiment. It is a figure which shows the example of the pixel layout of the image pick-up element which concerns on this embodiment. It is a figure which shows the structural example of the image process part which concerns on 1st Embodiment. It is a figure for demonstrating the flat part of the to-be-photographed image of image data. It is a figure for demonstrating the example of distribution of the pixel output value for every line in an image. It is a figure which shows the mode value for every line | wire when the long and thin subject is reflected in the horizontal direction. It is a figure for demonstrating the characteristic of 1st one-dimensional data. It is a figure which shows the structural example of the 2nd calculation part which an image process part has. It is a figure for demonstrating wavelet transformation. It is a figure which shows the to-be-photographed object edge removal with respect to a wavelet coefficient. It is a figure which shows the structural example of the image process part which concerns on 2nd Embodiment. It is a figure which shows the computer function which can implement | achieve the image processing apparatus which concerns on this embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
A first embodiment of the present invention will be described.
FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus according to an embodiment of the present invention. The imaging apparatus according to the present embodiment includes an imaging element 10, an analog-digital (AD) conversion unit 20, a timing generator (TG) 30, a system control unit 40, an image processing unit 50, a memory 60, a display 70, and a recording medium 80. Have

  The image sensor 10 converts an incident optical signal into an electrical analog signal. The analog signal output from the image sensor 10 is converted into a digital signal by the analog / digital converter 20 and output. The image sensor 10 and the analog-digital converter 20 operate in synchronization with a timing signal from the timing generator 30.

  The image processing unit 50 performs digital signal processing on the digital signal output from the analog-digital conversion unit 20 in accordance with control by the system control unit 40. This digital signal processing includes noise reduction, gain adjustment, defective pixel interpolation, image correction for random offset noise for each row, and the like. In the memory 60, the processing result of the image processing unit 50 is recorded. The memory 60 can be used as a temporary work area for signal processing.

  The system control unit 40 sends a control signal to the timing generator 30 to perform imaging control related to acquisition of image data, and transfers data necessary for signal processing to the image processing unit 50. Further, the system control unit 40 displays the image data recorded in the memory 60 on the display device 70, and records the image data on a recording medium 80 such as a compact flash (registered trademark) or an SD card.

  FIG. 2 is a diagram schematically illustrating a configuration example of the image sensor 10 according to the present embodiment. The image sensor 10 includes a pixel unit 100 in which a plurality of unit pixels 101 are arranged in a two-dimensional manner (row direction and column direction), a vertical scanning circuit 102 for reading out an arbitrary pixel output, a reading circuit 103, and horizontal scanning. Circuit 104.

  FIG. 3A is a diagram illustrating a configuration example of the unit pixel 101. In FIG. 3A, one unit pixel 101 arranged in the j-th row among the plurality of unit pixels 101 included in the pixel unit 100 is shown as an example. The unit pixel 101 includes a photodiode (photoelectric conversion element) 120, a transfer switch 121, a floating diffusion 122, a reset switch 123, a row selection switch 124, and a source follower amplifier 125. The source follower amplifier 125 is connected to the vertical signal line 126 of the arranged column.

  The transfer switch 121 is on / off controlled by a control signal φTX, the reset switch 127 is on / off controlled by a control signal φR, and the row selection switch 124 is on / off controlled by a control signal φSEL. Control signals φTX, φR, and φSEL are supplied from the outside of the pixel (vertical scanning circuit 102).

  FIG. 3B is a diagram illustrating a configuration example of the reading circuit 103. The read circuit 103 includes a column circuit 130 and an output amplifier 135 provided for each column. The column circuit 130 includes a noise signal (N) reading switch 131a, an imaging signal (S) reading switch 131b, an N reading capacity 132a, an S reading capacity 132b, an N reading transfer switch 133a, and an S reading transfer switch 133b. Each column circuit 130 is connected to the vertical signal line 126 of the corresponding column. The N read transfer switch 133a and the S read transfer switch 133b of each column circuit 130 are connected to the output amplifier 135 by horizontal output lines 134a and 134b, respectively.

  The N reading switch 131a is ON / OFF controlled by a control signal φTN, and the S reading switch 131b is ON / OFF controlled by a control signal φTS. The N reading transfer switch 133a and the S reading transfer switch 133b are ON / OFF controlled by a control signal φPH from the outside of the pixel (horizontal scanning circuit 104).

  FIG. 4 is a timing chart showing an example of a pixel signal readout operation in the present embodiment. In FIG. 4, readout from the pixels arranged in the i-th column and the j-th row of the image sensor 10 is shown as an example. Signals common to the pixels in the j-th row are represented as φSEL (j), φR (j), and φTX (j).

  First, in the period from time T1 to time T2, the reset signal φR (j) is asserted. Further, the signal charge of the photodiode 120 is reset by asserting the control signal φTX (j) during the period from the time T1 to the time T2. This operation is performed in all rows, and all the pixels in the pixel unit 100 are reset at the same time. After the reset is completed, the control signals φTX (j) and φR (j) are negated.

  Thereafter, at time T3, a mechanical shutter (not shown) that guides the subject image is opened, and all pixels start to accumulate signal charges simultaneously. The mechanical shutter is closed at time T4. A period from time T3 to time T4 is a signal charge accumulation period of the photodiode 120. At time T5, the row selection signal φSEL (j) is asserted, the row selection switch 124 of the pixel in the jth row is turned on, and the source follower amplifier 125 is in an operating state. Here, in the period from time T6 to time T7, the potential of the gate of the source follower amplifier 125 is reset by asserting the reset signal φR (j).

  At time T8, the control signal φTN is asserted, so that the dark level signal immediately after the reset becomes N in the column circuit 130 in the i-th column in the readout circuit 103 via the vertical output line 126 and the N readout switch 131a. It is held in the reading capacity 132a. At time T9 when the transfer of the dark level signal to the N reading capacitor 132a is completed, the control signal φTN is negated. The dark level signal held in the N reading capacity 132a is referred to herein as an N reading signal.

  At time T10, the control signal φTX (j) is asserted, and the signal charge accumulated in the photodiode 120 is transferred to the floating diffusion 122 via the transfer switch 121. The transferred charge is converted into a voltage by the capacitance of the floating diffusion 122, and the signal level is determined. The control signal φTX (j) is negated at time T11 when the signal charge is sufficiently transferred, and the control signal φTS is asserted at time T12. As a result, the signal level is held in the S reading capacitor 132b of the i-th column circuit 130 in the reading circuit 130 via the vertical output line 126 and the S reading switch 131b. The control signal φTS is negated at time T13 when the transfer of the signal level to the S reading capacitor 132b is completed. The signal level held in the S reading capacity 132b is referred to herein as an S reading signal.

  At time T14, the row selection signal φSEL (j) is negated, and reading of the pixel signal from the pixel in the jth row is completed. The N reading signal from the S reading signal held in this way is output to the output amplifier 135 via the horizontal output lines 134a and 134b in order from the column circuit of the i-th column by the control signal φPH from the horizontal scanning circuit 104. . The output amplifier 135 outputs a pixel signal having a good S / N ratio by subtracting the N reading signal from the S reading signal. The CMOS image sensor can obtain a good S / N ratio by the function of the readout circuit 103.

  However, when the voltage level of the power supply or the ground (GND) varies during reading of the selected row, the voltage variation appears as horizontal pattern noise in the entire selected row. This is because there is a time difference between the readout of the noise signal (N reading) and the readout of the imaging signal (S reading), and voltage fluctuations generated during that time appear as random offset noise for each row in the signal after subtraction. Is considered as the cause.

  FIG. 5 is a diagram illustrating an example of a pixel layout of the image sensor 10 according to the present embodiment. The imaging device 10 includes a vertical optical black (VOB) unit 140 in which light-shielded pixels are arranged, a horizontal optical black (HOB) unit 141, and an effective pixel unit 142 in which non-light-shielded pixels are arranged. The light shielding pixel is shielded from light by the wiring layer on the front surface of the photodiode in a semiconductor process for manufacturing an image sensor, for example.

  The random offset noise for each row described above appears not only in the pixel output of the effective pixel unit 142 but also in the pixel output of the HOB unit 141 that is shielded from light. Therefore, it is possible for the image processing unit 50 to digitally correct the pixel signal of the effective pixel unit 142 using the pixel signal of the HOB unit 141. This image correction process will be described below.

  By obtaining the horizontal average value of the pixel output of the HOB unit 141 in a certain row, it is possible to suppress random noise and estimate the offset noise amount in that row. The offset noise in the pixel output of the effective pixel unit 142 is corrected by subtracting the estimated offset noise amount. Alternatively, an average value may be obtained from the pixel outputs of the HOB units 141 for a plurality of rows including the row, and the offset noise amount of the row may be estimated. In the latter case, since the offset noise of different rows by the increased number of rows is added to the calculation, it is impossible to estimate the exact noise amount of the row, but the effect of suppressing random noise is large by increasing the average parameter. It is considered unlikely to be an error. It is desirable that the number of rows used when estimating the amount of offset noise is appropriately adjusted in consideration of the number of pixels for one row of the HOB unit 141. This process is referred to as HOB horizontal stripe correction.

  In order to obtain a sufficient correction effect by the above-described HOB horizontal stripe correction, a sufficient number of pixels for one row of the HOB portion 141 are necessary. In particular, at the time of setting a high photographing sensitivity (high ISO sensitivity) in which the influence of random noise is large, it is difficult to estimate the offset noise amount from the pixel output of the HOB unit 141.

  Therefore, in the present embodiment, even when the random noise is large, random offset noise is reduced for each row so that image correction with sufficient correction effect can be performed. FIG. 6 is a block diagram illustrating a configuration example of the image processing unit 50 according to the first embodiment. FIG. 6 illustrates only a portion of the configuration of the image processing unit 50 according to the first embodiment that corrects random offset noise for each row generated due to the above-described reading of the image sensor. .

  As illustrated in FIG. 6, the image processing unit 50 includes a pixel selection unit 502, a first calculation unit 503, a second calculation unit 504, and a row offset subtraction unit 505. The pixel selection unit 502 selects, for the image data (input image) input to the image processing unit 50, the pixel output of the flat portion of the subject image having a similar pixel value from the pixel outputs of the effective pixel unit.

  Here, the flat portion of the subject image will be described with reference to FIG. FIG. 7 is a diagram for explaining a flat portion of a subject image of image data. A hatched area 510 in FIG. 7B with respect to the image data shown in FIG. 7A indicates an area where a sky or a flat wall that is not a main subject such as a person is photographed. Particularly in night photography, the background portion of the main subject is easily exposed as a flat dark portion due to insufficient exposure. The pixel selection unit 502 selects the pixel output of the flat part of the subject image as shown in the area 510 of FIG.

  The first calculation unit 503 obtains an average value of the pixel output in the horizontal direction for each row from the pixel output selected by the pixel selection unit 502, and the first one-dimensional data DTA that is one-dimensional data in the vertical direction of the image. Is calculated. That is, the first calculation unit 503 extracts the low-frequency component in the horizontal direction of the image in the row by calculating the average value in the horizontal direction of the pixel output selected as the flat portion for each row. From the first one-dimensional data DTA, it is possible to know the change of the low frequency component in the horizontal direction of the image with respect to the vertical direction of the image.

  The first one-dimensional data DTA calculated by the first calculation unit 503 includes a random offset noise component and a subject image component for each row. The second calculation unit 504 performs the filtering process on the first one-dimensional data DTA, thereby calculating the second one-dimensional data DTB in which only random offset noise components are extracted for each row. By the processing by the pixel selection unit 502 described above, the first one-dimensional data DTA has a subject image component on the low frequency side and a lot of random offset noise components for each row on the high frequency side. High frequency components are removed. Therefore, only a random offset noise component is extracted for each row without being affected by the high-frequency component of the subject image only by performing a filtering process for reducing the low-frequency component on the first one-dimensional data DTA. The second one-dimensional data DTB can be extracted.

  A row offset subtraction unit 505 subtracts the second one-dimensional data DTB calculated by the second calculation unit 504 from the image data (input image) input to the image processing unit 50 for each row. As a result, it is possible to obtain image data with less random offset noise for each row than just the HOB horizontal stripe correction described above. In the present embodiment, by extracting only the pixel output of the flat part of the subject image by the pixel selection unit 502, the frequency band that can be extracted as a noise component at the time of filter processing can be widened, and the noise component that overlaps the frequency band of the edge part of the subject is also included. It becomes possible to extract.

  Details of the processing in the image processing unit 50 according to the first embodiment will be described below. First, specific processing contents of the pixel selection unit 502 will be described. FIG. 8 is a diagram for explaining a distribution example of pixel output values for each row in an image. As shown in FIGS. 8A and 8B, when a histogram of image output values for each row is created from the rows CA and CB in the image, the focused main subject has many high-frequency components and the background. Since there are many low frequency components, the mode value of the pixel output is highly likely to indicate the background portion. When the background part is the sky or a flat wall, the tendency becomes strong.

  Therefore, the pixel selection unit 502 creates a histogram of image output values for each row, and sets a pixel indicating a pixel output within the range between the first threshold value and the second threshold value as a flat portion pixel in the row. An example of a method for determining the first threshold value and the second threshold value will be described with reference to FIG. With the mode value κ, values λ1 and λ2 having a relationship of (κ−λ1) ≦ κ ≦ (κ + λ2) are used, and the first threshold value is (κ−λ1) and the second threshold value is (κ + λ2). As a method of determining the values λ1 and λ2, the standard deviation σ of the pixel value of the dark image obtained with the same shooting sensitivity (ISO sensitivity) setting as that of the captured image may be used as a reference. As a result of experiments conducted by the present inventors, ranges of (κ−2σ) ≦ (κ−λ1) ≦ (κ−σ) and (κ + σ) ≦ (κ + λ2) ≦ (κ + 2σ), respectively. Good results were obtained when determined in. Here, the first threshold value may be (κ′−λ1) and the second threshold value may be (κ ′ + λ2) depending on the mode value κ ′ in the upper row.

  In the processing described above, when a long and thin subject such as an electric wire is captured in the horizontal direction, the mode value of the pixel output for each row may change greatly in a short cycle. FIG. 9 is a diagram illustrating an example of the mode value of the pixel output for each row when a long and thin subject is captured in the horizontal direction. As shown in FIG. 9A, if the mode value of the pixel output for each row includes a high frequency component, the first one-dimensional data calculated based on this mode also includes an unnecessary high frequency component. Therefore, it becomes difficult to extract only random offset noise for each row by the filter processing. FIG. 9B shows data obtained by applying a moving average filter to the data shown in FIG. In this way, by performing low-pass filter processing such as a moving average filter on the mode value κ in each row, it is possible to make it difficult to make an erroneous determination in selecting a pixel in the subject flat portion.

  The method for determining the first threshold value and the second threshold value described above is an example, and the present invention is not limited to this. It is also possible to use a plurality of pixel outputs for determination of pixel selection in the flat portion. For example, the pixel selection unit 502 calculates a moving average value in the horizontal direction of the image around the pixel of interest, and when the average value is within the range between the first threshold value and the second threshold value, The pixel of interest is determined as a flat pixel. In this case, since the random noise can be suppressed by obtaining the moving average in the horizontal direction, it is possible to correctly capture the flat portion rather than judging only with a single pixel. In addition, regarding the use of a plurality of pixels, the pixel is not limited to a plurality of pixels in the horizontal direction, and pixels that are two-dimensionally adjacent may be used, or a median filter other than a moving average may be used. Also good.

  Next, specific processing of the first calculation unit 503 will be described. The first calculation unit 503 integrates the pixel output values of the pixels determined by the pixel selection unit 502 as the pixels of the flat portion of the subject image over one line, and counts the number of pixels. The first calculation unit 503 divides the integral value by the number of pixels when the processing is completed up to the pixel at the end of the row. When processing is completed for all rows, first one-dimensional data DTA is obtained.

  FIG. 10 is a diagram for explaining the characteristics of the first one-dimensional data DTA. As shown in FIGS. 10A and 10B, the first one-dimensional data is the horizontal average value of the pixels selected by the pixel selection unit 502 with the horizontal axis as the row of the pixel unit 100. This is the data. FIG. 10A shows the first one-dimensional data obtained by the first calculation unit 503 when random noise for each row exists in the input image. FIG. 10B shows the first one-dimensional data obtained by performing the same processing when there is no random offset noise for each row described above. When there is no random offset noise for each row, the first one-dimensional data is less likely to change greatly in the vertical direction of the image, and therefore data with a dominant low frequency component as shown in FIG. Can be expected. Therefore, it can be determined that the high-frequency component seen in FIG. 10A is mainly affected by random offset noise for each row.

  Next, specific processing of the second calculation unit 504 will be described. The second calculation unit 504 extracts random offset noise for each row by performing a process of extracting a specific frequency band on the first one-dimensional data calculated by the first calculation unit 503. To do. As a method of decomposing for each frequency band, for example, there is multi-resolution decomposition by discrete wavelet transform.

  FIG. 11 is a block diagram illustrating a configuration example of the second calculation unit 504. The second calculation unit 504 includes a frequency conversion unit 520, a band extraction unit 521, a subject edge removal unit 522, and an inverse conversion unit 523. The frequency conversion unit 520 applies discrete wavelet transform to the first one-dimensional data DTA and decomposes it into wavelet coefficients for each frequency band. A fast algorithm is known for discrete wavelet transform. An outline of the wavelet transform will be described with reference to FIG.

  FIG. 12A is a diagram showing an outline of wavelet transform. The discrete wavelet transform is decomposed into a plurality of frequency bands using a pair of low-pass filters and high-pass filters having symmetrical gain characteristics. The signal obtained as the output of the low-pass filter is called a scaling coefficient, and the signal obtained as the output of the high-pass filter is called a wavelet coefficient. Since the data length must be a power of 2 in order to use the high-speed algorithm, the numbers are matched with dummy data or the like.

  In FIG. 12A, scale 0 is an original signal and is represented as a scaling coefficient S0 for convenience. For the scaling coefficient S0, a scaling coefficient S1 that has passed through a low-pass filter and a wavelet coefficient T1 that has passed through a high-pass filter are obtained. Here, the data lengths of the scaling coefficient S1 and the wavelet coefficient T1 are each half of the scaling coefficient S0. Similarly, a scaling coefficient S2 that has passed through a low-pass filter and a wavelet coefficient T2 that has passed through a high-pass filter are obtained for the scaling coefficient S1. The data lengths of the scaling coefficient S2 and the wavelet coefficient T2 are half of the scaling coefficient S1. Further, no filter processing is performed on the wavelet coefficient T1. By repeatedly performing a series of filtering processes only on the scaling coefficients in this way, wavelet coefficients having different frequency bands can be obtained by the amount of repetition as shown in FIG.

  FIG. 12B is a diagram showing the frequency bands of the wavelet coefficients at each scale. The band extracting unit 521 extracts the offset noise for each row by multiplying the wavelet coefficient obtained by the frequency converting unit 520 as described above by a gain of 1.0 or less in addition to the specific frequency band component. . In the present embodiment, the first one-dimensional data DTA input to the second calculation unit 504 includes a subject image component as a low-frequency component, so that the wavelet coefficient below a preset scale has zero. Multiply In other words, the band extracting unit 521 cuts components that are equal to or lower than a predetermined frequency band. The frequency band to be cut is determined by what frequency band random offset noise appears in the vertical direction of the image sensor for each row.

  Next, the subject edge removing unit 522 performs gain adjustment on the wavelet coefficients not cut by the band extracting unit 521 according to the magnitude of the wavelet coefficients. FIG. 13 is a diagram showing subject edge removal for wavelet coefficients of scale m. The subject edge removal unit 522 multiplies the coefficient of the scale m wavelet coefficient by a gain of 1.0 or less to the coefficient of the negative threshold a1 or less and the coefficient of the positive threshold a2 or more. This processing uses the feature that random offset noise for each row has a smaller variation in pixel value than the edge portion of the subject image, and the edge portion of the subject image mixed in by the pixel selection unit 502 is used. This is done to reduce the information. Thereby, it is possible to suppress the loss of the edge information of the subject in the image after the image correction.

  The inverse transform unit 523 performs inverse discrete wavelet transform on the wavelet coefficient data processed by the subject edge unit 522 to obtain second one-dimensional data DTB. The second one-dimensional data DTB obtained here is obtained by extracting random offset noise for each row.

  Although an example using the discrete wavelet transform has been described here, a Fourier transform, other multi-resolution decomposition means, or a digital filter may be used as long as the components can be separated in the frequency band.

  Next, specific processing of the row offset subtraction unit 505 will be described. The row offset subtraction unit 505 multiplies the numerical value corresponding to the row of the original image in the second one-dimensional data DTB from the original image data input to the image processing unit 50 by a reliability coefficient described later. Subtract a uniform value. The reliability coefficient is multiplied by how accurately the second one-dimensional data DTB can extract random offset noise for each row. For example, the gain of 1.0 or less is multiplied as the reliability coefficient according to the count number of the pixel selected by the pixel selection unit 502. For example, the gain is increased when the count number of the pixel selected by the pixel selection unit 502 is large, and the gain is decreased when the count number is small.

  According to the first embodiment, by the processing in the image processing unit 50 described above, it is possible to remove random offset noise components for each row included in the image from the pixel output of the effective pixels. Therefore, image correction for random offset noise can be appropriately performed on the pixel output of effective pixels for each row included in the image, and a sufficient correction effect can be obtained even when the random noise is large.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In the second embodiment, the image correction in the first embodiment and the HOB horizontal stripe correction described above are used in combination. Hereinafter, only differences from the first embodiment will be described. FIG. 14 is a block diagram illustrating a configuration example of the image processing unit 50 according to the second embodiment. FIG. 14 also shows only a portion that corrects random offset noise for each row generated due to reading of the above-described imaging element in the configuration of the image processing unit 50 according to the second embodiment. ing. 14, components having the same functions as those shown in FIG. 6 are given the same reference numerals, and redundant descriptions are omitted.

  As illustrated in FIG. 14, the image processing unit 50 includes a HOB horizontal stripe correction unit 500, an ISO sensitivity determination unit 501, a pixel selection unit 502, a first calculation unit 503, a second calculation unit 504, and a row offset subtraction unit 505. Have Image data (input image) input to the image processing unit 50 is first input to the HOB horizontal stripe correction unit 500. The HOB horizontal stripe correction unit 500 performs the HOB horizontal stripe correction process described in the first embodiment. The number of rows of HOB pixels used for the HOB horizontal stripe correction process is determined in advance according to the setting of photographing sensitivity (ISO sensitivity). Here, the higher the setting of the shooting sensitivity (ISO sensitivity), the larger the random noise. Therefore, the higher the shooting sensitivity (ISO sensitivity) during shooting, the larger the number of rows of HOB pixels used for correction. Thereby, the 1st horizontal stripe correction image from which the low frequency component of the random offset noise was removed for every line is obtained.

  As a method for determining the number of rows of HOB pixels used for the HOB horizontal stripe correction process, for example, the pixel value of the effective pixel portion of an image (hereinafter referred to as a dark image) captured while being shielded so that light does not enter the image sensor 10 is used. There is a method of determining from the variation. When the pixel output value of the effective pixel portion has a distribution of average μ and standard deviation σ, the average value of n pixels is included in the distribution of average μ and standard deviation σ / √n based on the central limit theorem. The value n is determined so that the standard deviation σ / √n of the average value becomes sufficiently small. In addition, when the value n is set to a power of 2, division can be realized by a shift operation, which is preferable in terms of implementation.

  The ISO sensitivity determination unit 501 includes a switch corresponding to the imaging sensitivity (ISO sensitivity) of image data. The ISO sensitivity determination unit 501 determines whether to output the first horizontal stripe correction image to the pixel selection unit 502 based on the imaging sensitivity (ISO sensitivity) at the time of imaging of the image data input to the image processing device 50. To do. When the shooting sensitivity (ISO sensitivity) at the time of shooting of the image data input to the image processing device 50 is a preset shooting sensitivity, the first horizontal stripe correction image is output to the pixel selection unit 502, and the first implementation is performed. Processing similar to that of the form is performed. Thus, according to the second embodiment, by using the image correction in the first embodiment and the HOB horizontal stripe correction processing by the HOB horizontal stripe correction unit 500 in combination, High frequency components can be reduced together.

(Other embodiments of the present invention)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  For example, the image processing apparatus shown in the first and second embodiments has a computer function 1000 as shown in FIG. 15, and the CPU 1001 performs the operations in the first and second embodiments. As shown in FIG. 15, the computer function 1000 includes a CPU 1001, a ROM 1002, and a RAM 1003. Further, a controller (CONSC) 1005 of the operation unit (CONS) 1009 and a display controller (DISPC) 1006 of a display (DISP) 1010 as a display unit such as a CRT or LCD are provided. Furthermore, a hard disk (HD) 1011, a controller (DCONT) 1007 of a storage device (STD) 1012 such as a flexible disk, and a network interface card (NIC) 1008 are provided. These functional units 1001, 1002, 1003, 1005, 1006, 1007, and 1008 are configured to be communicably connected to each other via a system bus 1004.

  The CPU 1001 performs overall control of each component connected to the system bus 1004 by executing software stored in the ROM 1002 or the HD 1011 or software supplied from the STD 1012. That is, the CPU 1001 reads out the processing program for performing the operation as described above from the ROM 1002, the HD 1011 or the STD 1012 and executes it, thereby performing control for realizing the operation in the first and second embodiments. Do. The RAM 1003 functions as a main memory or work area for the CPU 1001. The CONSC 1005 controls an instruction input from the CONS 1009. The DISPC 1006 controls the display of the DISP 1010. The DCONT 1007 controls access to the HD 1011 and the STD 1012 that store a boot program, various applications, user files, a network management program, the processing program in the first and second embodiments, and the like. The NIC 1008 exchanges data with other devices on the network 1013 in both directions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

DESCRIPTION OF SYMBOLS 10: Image pick-up element 50: Image processing part 500: HOB horizontal stripe correction | amendment part 501: ISO sensitivity determination part 502: Pixel selection part 503: 1st calculation part 504: 2nd calculation part 505: Row offset subtraction part

Claims (11)

Image data obtained by an image sensor having a plurality of pixels arranged two-dimensionally is used as an input image,
A pixel selection unit that selects a pixel output of a flat portion of a subject image among pixel outputs of effective pixels in the image sensor included in the image data;
A first calculation unit that calculates first one-dimensional data based on a pixel output selected by the pixel selection unit;
A second calculation unit that extracts random offset noise components for each row from the first one-dimensional data calculated by the first calculation unit to calculate second one-dimensional data;
An image processing apparatus comprising: a subtraction unit that subtracts the second one-dimensional data calculated by the second calculation unit for each row from the input image.   The pixel selection unit calculates a first threshold value and a second threshold value based on a mode value of a pixel output for each row of the input image, and is within a range between the first threshold value and the second threshold value. The image processing apparatus according to claim 1, wherein a pixel indicating the pixel output is selected as a flat portion of the subject image.   The pixel selection unit performs low pass filter processing on the mode value of pixel output for each row of the input image, and calculates the first threshold value and the second threshold value from the low pass filter processed data. The image processing apparatus according to claim 2, wherein:   The first calculation unit calculates an average value of pixel outputs for each row based on the pixel output selected by the pixel selection unit, and uses the average value calculated for each row as the first one-dimensional data. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The second calculation unit performs a filtering process for suppressing low frequency components on the first one-dimensional data calculated by the first calculation unit, and calculates the second one-dimensional data. The image processing apparatus according to claim 1, wherein: The second calculation unit includes:
A frequency converter that converts the first one-dimensional data calculated by the first calculator into data of a plurality of frequency bands;
A band extractor for extracting specific frequency band components from data of a plurality of frequency bands obtained by the frequency converter;
A subject edge removing unit that removes subject edge information from the frequency band data extracted by the band extracting unit;
The image processing apparatus according to claim 1, further comprising: an inverse conversion unit that performs inverse conversion on an output from the subject edge removal unit and calculates the second one-dimensional data. .   7. The subtraction unit according to claim 1, wherein the subtraction unit multiplies each numerical value of the second one-dimensional data by a gain of 1.0 or less and then subtracts the input image for each row. The image processing apparatus according to claim 1.   8. The image according to claim 7, wherein the subtraction unit determines a gain to be multiplied by the second one-dimensional data based on the number of pixels selected as the flat portion of the subject image by the pixel selection unit. Processing equipment. Based on the pixel output of the light-shielded pixel included in the image data, a correction unit that obtains a first horizontal stripe correction image in which random offset noise is corrected for each row from the pixel output of the effective pixel;
The determination unit according to claim 1, further comprising: a determination unit configured to determine whether to output the first horizontal stripe correction image to the pixel selection unit based on a photographing sensitivity at the time of photographing the image data. The image processing apparatus according to any one of the above. Image data obtained by an image sensor having a plurality of pixels arranged two-dimensionally is used as an input image,
A pixel selection step of selecting a pixel output of a flat portion of a subject image among pixel outputs of effective pixels in the image sensor included in the image data;
A first calculation step of calculating first one-dimensional data based on the pixel output selected in the pixel selection step;
A second calculation step of calculating a second one-dimensional data by extracting a random offset noise component for each row from the calculated first one-dimensional data;
And a subtracting step of subtracting the calculated second one-dimensional data for each row from the input image. Image data obtained by an image sensor having a plurality of pixels arranged two-dimensionally is used as an input image,
A pixel selection step of selecting a pixel output of a flat portion of the subject image among pixel outputs of effective pixels in the image sensor included in the image data;
A first calculation step of calculating first one-dimensional data based on the pixel output selected in the pixel selection step;
A second calculation step of calculating a second one-dimensional data by extracting a random offset noise component for each row from the calculated first one-dimensional data;
A program for causing a computer to execute a subtraction step of subtracting the calculated second one-dimensional data from the input image for each row.

Images