JP2008172566A - Imaging apparatus and imaging method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus and imaging method capable of properly improving a dynamic range of an input image with a simple configuration. <P>SOLUTION: The imaging apparatus comprises: a gain correcting means (2) for multiplexing a first imaging signal (Xa) by a correction gain (G<SB>k</SB>) and outputting a second imaging signal (Xb) corrected for each pixel; an exposure control means (10) for determining a first exposure value (E1) for making an average value (APL) of the first imaging signal (Xa) coincide with a prescribed target value, further correcting the first exposure value (E1) by using a gain correction value (Kd) on the basis of the maximum value (Gmax) of a correction gain, calculating a second exposure value (E2) and controlling exposure with the second exposure value; and a correction gain determining means (15) for determining a correction gain (Gk) for each pixel on the basis of a result of filtering brightness components of the first imaging signal (Xa) and the maximum value of the correction gain. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an imaging method that appropriately improve the dynamic range of an input image.

  Conventionally, as a method for improving the gradation characteristics of an image, gradation conversion is performed so that the cumulative frequency (histogram) obtained by accumulating the number of pixels of the same luminance from an input image on one screen is evenly distributed. Histogram equalization is proposed (see, for example, Patent Document 1).

  Also, by calculating a weighted average of spatial luminance changes from the input image and calculating an improved luminance signal from the logarithmic conversion value of the calculated weighted average and the logarithmic conversion value of the input image, the dynamic image of the image is obtained. A method called RETINEX for improving the range has been proposed (see, for example, Non-Patent Document 1 and Patent Document 2).

JP 2002-27285 A (paragraphs 0029-0041, FIG. 1) Japanese Patent Laying-Open No. 2005-38119 (paragraphs 0028-0031, FIG. 1) Z. Rahman et al. "A Multiscale Retinex For Color Rendition and Dynamic Range Compression", XIX Proc. SPIE Vol. 2847, pp. 183-191, Nov. 1996

  However, in the above-described histogram equalization, histogram data of an image for at least one screen is recorded, histogram bias is analyzed from the recorded histogram data, and gradation characteristics are determined based on the analysis result. For this reason, a timing shift of one screen or more occurs between the image used for the analysis and the image reflecting the analysis result, so that the dynamic range of the input image may not be improved appropriately. For example, when there is a timing shift of one or more screens as described above in a moving image, the optimal level for the image reflecting the analysis result is different depending on the difference between the image used for the analysis and the image reflecting the analysis result. There was a problem that the tonal characteristics could not be determined.

  Further, in the method using RETINEX, calculation processing such as convolution integration for using weighted average and logarithm calculation of weighted average and input signal is complicated. Therefore, hardware (for example, ASIC (Application Specific Integrated Circuit) or FPGA) is used. (Field Programmable Gate Array)) or RETINEX is executed by an embedded microcomputer, there is a problem that processing time becomes long and mounting capacity (gate scale, memory capacity) becomes large.

  Therefore, the present invention has been made to solve the above-described problems of the prior art, and an object of the present invention is to provide an imaging apparatus and an imaging method that can appropriately improve the dynamic range of an input image with a simple configuration. It is to provide.

The present invention
Imaging signal generating means for receiving light from a subject and outputting a first imaging signal corresponding to the light;
Gain correcting means for correcting the first imaging signal for each pixel and outputting a corrected second imaging signal;
A first exposure value of the imaging signal generating means for making the average value of the first imaging signal coincide with a predetermined target value is determined, and the first exposure value is further determined based on the maximum value of the correction gain. An exposure control unit that corrects to obtain a second exposure value, and controls the exposure of the imaging signal generation unit by the second exposure value;
Filtering is performed on the value of each pixel of the luminance component of the first imaging signal and the value of the surrounding pixels, and a correction gain is determined for each pixel based on the filtering output and the maximum value of the correction gain. Correction gain determining means,
The gain correction unit provides the imaging apparatus, wherein the correction is performed by an operation including multiplication of the correction gain determined by the correction gain determination unit and the first imaging signal.

  According to the present invention, filtering is performed on the value of each pixel of the luminance component of the imaging signal and the surrounding pixels, a correction gain is obtained for each pixel based on the filtering output, and the pixel for the imaging signal is obtained using this correction gain. Since the correction is performed every time, the gradation characteristics of the imaging signal can be improved appropriately.

  In addition, according to the present invention, the exposure value is corrected according to the maximum value of the correction gain, and the exposure is controlled with the corrected exposure, thereby suppressing whitening of the bright portion of the imaging signal and a decrease in contrast. In addition, the contrast of the dark part can be improved, and thereby the visibility of the dark part can be achieved.

  Further, according to the present invention, since it is not necessary to perform complicated calculations and calculation and processing can be simplified, it is possible to simplify the configuration and reduce the costs associated therewith.

Embodiment 1 FIG.
FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus according to Embodiment 1 of the present invention (an apparatus that performs the imaging method according to Embodiment 1).
As shown in FIG. 1, the imaging apparatus according to the first embodiment includes an imaging device 5, an analog signal processing means 6, an A / D conversion means 7, a camera signal processing means 8, a timing generation means 9, The integrating means 11, the exposure correction means 10, the gain correction means 2, the luminance filtering means 3, the correction gain calculation means 4, and the maximum gain determination means 13 are included. The camera signal processing means 8, the integrating means 11, the exposure correcting means 10, the gain correcting means 2, the luminance filtering means 3, the correction gain calculating means 4, and the maximum gain determining means 13 are determined by hardware such as an electric circuit or a program. You may comprise either the software which operate | moves or the combination of hardware and software.

  For example, the imaging device 5 includes a photodiode array that receives a light signal from a subject and performs photoelectric conversion, and a CCD (including a vertical transfer CCD and a horizontal transfer CCD, and means for taking out the signal of the photodiode array to the outside. Charge Coupled Device) sensor.

  The image pickup device 5 can read out the charge accumulated in the photodiode array every frame period to the outside through the vertical transfer CCD and the horizontal transfer CCD by applying a charge read pulse from the timing generation unit 9. .

  Further, the charge accumulated in the photodiode array can be swept out to the substrate potential of the image sensor 5 by the charge sweeping pulse from the timing generation means 9. The charge accumulation time S is the time from when the application of the charge sweep pulse is stopped until the accumulated charge is read out to the vertical transfer CCD by the charge readout pulse. The reciprocal of the charge accumulation time S corresponds to the shutter speed. The charge accumulation time S is controlled by a control signal from the exposure control means 10.

  The analog signal processing means 6 inputs an image signal from the image sensor 5, performs a CDS (correlated double sampling) process, amplifies with a gain G, and outputs an analog signal. The amplification gain G is controlled by a control signal from the exposure control means 10.

  The A / D conversion means 7 receives the analog signal from the analog signal processing means 6, converts it into a digital signal, and outputs it.

  The timing generation unit 9 generates a drive timing pulse for the image sensor 5. For example, when the image pickup device 5 is a CCD sensor, the drive timing pulse is a horizontal transfer pulse for transferring the charge of the horizontal CCD, a vertical transfer pulse for transferring the charge of the vertical CCD, and sweeping the accumulated charges to the CCD substrate potential. These are pulses such as a charge sweep pulse for performing an electronic shutter operation and a reset pulse for resetting the horizontally transferred charge of the horizontal CCD for each pixel.

  The timing generation unit 9 also generates a sampling pulse of the analog signal processing unit 6 and an A / D clock of the A / D conversion unit 7.

  Note that the image sensor 5 is not limited to a CCD sensor, and may be a CMOS sensor, for example. A sensor that resets pixel charges, such as a CMOS sensor, or a driving method that reads out charges can be used as long as exposure can be controlled even in different sensors.

  In the above example, the sensor has an electronic shutter function. However, the present invention can also be applied to an imaging apparatus capable of adjusting the light amount by the diaphragm and adjusting the exposure time by the mechanical shutter.

  The camera signal processing means 8 receives the digital signal output from the A / D conversion means 7 and performs processing such as white balance processing, generation of RGB signals by interpolation processing, YCbCr conversion, color matrix conversion, gradation conversion and the like. The imaging signal Xa is output.

  The imaging element 5, the analog signal processing means 6, the A / D conversion means 7, and the camera signal processing means 8 receive light from the subject, and an imaging signal corresponding to the light (first imaging signal). An imaging signal generation means 14 for outputting Xa is configured.

  The imaging signal Xa is, for example, an imaging signal of three colors of red (R), green (G), and blue (B) that is two-dimensionally arranged in horizontal 640 × vertical 480 pixels with 8-bit gradation (hereinafter referred to as “RGB signal”). "" The R signal level of the imaging signal Xa is expressed as R (M, N), the G signal level of the imaging signal Xa is expressed as G (M, N), and the B signal level of the imaging signal Xa is expressed as B (M, N). Here, M represents a horizontal pixel position, and N represents a vertical pixel position.

The imaging signal Xa is not limited to an RGB signal, and may be a YCbCr signal, an L * a * b * signal, or an HSV (Hue, Saturation, Value) signal. When a YCbCr signal, an L * a * b * signal, or an HSV signal is used as the imaging signal Xa, a color conversion unit (not shown) that performs color conversion processing of each color space signal to an RGB signal is performed by the luminance filtering unit 3. Provided internally (in the input stage).
Further, the number of gradations of the imaging signal Xa is not limited to the above 8 bits, and may be other gradations such as 10 bits or 12 bits used in the still image file. Further, the number of pixels of the imaging signal Xa is not limited to the above value, and may be other number of pixels such as horizontal 1024 × vertical 960 pixels.

  The integrating means 11 obtains an integrated value As for each of a plurality of photometric windows that form part of the screen of the digital signal output of the A / D converting means 7. For example, the size of each photometric window is 10 pixels × 10 pixels, and the effective pixel area of the screen is divided. When a photometric window size of 10 pixels × 10 pixels is used, a number of photometric windows of 64 × 48 can be obtained when the number of effective pixels is 640 × 480. Note that the size and number of photometric windows, and further their positions can be arbitrarily determined.

The exposure control means 10 uses the integrated value As for each photometric window of the integrating means 11 to determine the exposure condition of the imaging apparatus and controls the exposure.
In the configuration of FIG. 1, the exposure is controlled by controlling the charge accumulation time (exposure time) S by the timing generation means 9 and the amplification gain G in the analog signal processing means 6. That is, the “exposure value” that is proportional to the product of the charge accumulation time S and the amplification gain G is controlled.

  FIG. 2 shows an example of the exposure control means 10. The illustrated exposure control means 10 includes a first exposure value generation means 16 and a second exposure value generation means 17.

The first exposure value generating means 16 selects an effective photometric window from the photometric window integrated value As according to an empirically or statistically obtained algorithm, and is obtained from the selected photometric window integrated value As. An exposure value (first exposure value) E1 for generating an APL (Average Picture Level) at a predetermined target value (for example, 50% of the maximum value) Tgt is generated. The target value Tgt is supplied from control means (not shown), for example.
The second exposure value generation means 17 receives the maximum value Gmax of the correction gain, and the first exposure value E1 will be described later with reference to FIGS. 12, 13, 14, 18, and 19. In this way, an exposure value (hereinafter, “second exposure value”) E2 corrected using the exposure correction value Kd obtained based on the maximum value Gmax of the correction gain is generated.
The first exposure value E1 defines, for example, the charge accumulation time S and the gain G,
E1 = Kf × S × G
(Where Kf is a constant), and the second exposure value E2 also defines, for example, the charge accumulation time S and the gain G,
E2 = Kf × S × G
(However, the values of S and G are different from the case of E1.)
Have the relationship.

  FIG. 3 is a log-log graph showing an example of the control characteristics of exposure control. The horizontal axis indicates the illuminance L of the subject, and the vertical axis indicates the reciprocal 1 / S of the charge accumulation time S and the gain G. Under a bright environment equal to or higher than the first predetermined illuminance Lb, the gain G is set to a constant value (G = 2), and the charge accumulation time S is shortened (shutter speed S is increased) as the illuminance increases. . On the other hand, below the first predetermined value Lb, the charge accumulation time S is fixed, and the gain G is increased as the illuminance decreases. Further, the gain G is also fixed at the maximum value Xg below the second predetermined value La (La <Lb). Further, the gain G and the charge accumulation time S are fixed even at the third predetermined value Lc or more. The characteristic shown in FIG. 3 is for APL to be a predetermined value, for example, 50% of the maximum value, in the range where the illuminance L is not less than the second predetermined value La and less than the third predetermined value Lc.

When determining the charge accumulation time S and the gain G using the control characteristics of FIG. 3, the charge accumulation time S and the gain G are set to certain values, for example, predetermined initial values S0 and G0, and from the integrated value As at that time, The illuminance L of the subject
L = Ka × As / (S0 × G0)
(Ka is a predetermined constant) calculated, and based on the illuminance L of the subject, the charge accumulation time S and gain G thus obtained are determined using the characteristics shown in FIG. And a predetermined constant Kf is equal to the first exposure value E1. After the first exposure value E1 is determined, the correction coefficient Kd is given, and when the second exposure value E2 is calculated by the correction coefficient Kd, the charge accumulation time S, the gain G, and the constant Kf described above One or both of the charge accumulation time S and the gain G is changed so that the product of the two coincides with the second exposure value E2. For example, in FIG. 3, in the range where the charge accumulation time S is fixed (the range less than the first predetermined value Lb), the gain G is changed, and the range where the gain G is fixed (the first predetermined value Lb or more). In the range), the charge accumulation time S is changed. In the range where both the charge accumulation time S and the gain G are fixed (greater than or equal to the third predetermined value Lc), neither the charge accumulation time S nor the gain G is changed.

Note that the above is only an example of exposure control, and other exposure control may be used. For example, in the above description, the first exposure value E1 is determined based on the integrated value As obtained from one frame, but the first exposure obtained from one frame (first frame). The next frame (second frame) is imaged with the charge accumulation time S and gain G corresponding to the value E1, and the first exposure value E1 is obtained again based on the integrated value obtained from the second frame. By repeating the process described above, the second exposure value E2 may be generated after the first exposure value E1 is converged to a more appropriate one.
In this case, the charge accumulation time S and the gain G (indicating control signal) corresponding to the first exposure value E1 are replaced with the charge accumulation time S and the gain G (indicating control signal) corresponding to the second exposure value E2. In addition, a switching unit may be provided in the exposure control unit 10 to supply the image sensor 5 and the analog signal processing unit 6.

The luminance filtering unit 3 includes each pixel of the luminance component (the luminance signal included in the imaging signal Xa or the luminance signal generated from the imaging signal Xa) of the imaging signal Xa output from the camera signal processing unit 8 and the surrounding pixels. Filter on values.
Correction gain calculating means 4, and filtering the output from the luminance filtering means 3 calculates a correction gain G _k for each pixel on the basis of the maximum value Gmax of the correction gain.
The luminance filtering unit 3 and the correction gain calculation unit 4 perform filtering on the value of each pixel of the luminance component of the imaging signal Xa and the surrounding pixels, and each pixel based on the filtering output and the maximum value Gmax of the correction gain. and it is configured to correct the gain determining unit 15 for determining the correction gain G _k in.

  FIG. 4 is a block diagram showing an example of the luminance filtering means 3. The illustrated luminance filtering unit 3 includes a luminance detection unit 31 and a filter unit 32. The luminance filtering unit 3 generates a luminance signal Y from the imaging signal Xa output from the camera signal processing unit 8, and a plurality of continuous luminance signals are generated. A filtering output over pixels, for example, an average luminance signal Yavg is obtained and output.

The luminance detection means 31 calculates and outputs a luminance signal component from the imaging signal Xa. ITU-R BT. In the case of 709, the luminance signal Y can be obtained from the RGB signal by the following equation (1).
Y = 0.299 × R (M, N) + 0.587 × G (M, N) + 0.114 × B (M, N)
... (1)
The conversion formula for obtaining the luminance signal Y from the RGB signal is not limited to the above formula (1), but is defined by the color space standard adopted by the system that performs image processing. If the luminance signal Y is included in the imaging signal Xa, the luminance detection unit 31 outputs the luminance signal Y of the imaging signal Xa to the filter unit 32 without performing calculation for obtaining the luminance signal Y.

The filter unit 32 is a one-dimensional n-tap acyclic digital filter, and includes a delay unit 33, a coefficient unit 34, and an addition unit 35. The delay means 33 delays the luminance signal of the imaging signal Xa, delay element DL (−1), delay element DL (0) delays the output of delay element DL (−1), and output of delay element DL (0). Has a delay element DL (1). The coefficient means 34 is a multiplier 34a that multiplies the output of the delay element DL (-1) by a coefficient a _k (-1), and a multiplier 34b that multiplies the output of the delay element DL (0) by a coefficient a _k (0). , And a multiplier 34c for multiplying the output of the delay element DL (1) by a coefficient a _k (1). The tap number n satisfies n = 2 × k + 1 (k is a positive integer).
Alternatively, the delay element DL (-1) may not be provided, and the output of the luminance calculation unit 31 may be directly input to the delay element DL (0) and the multiplier 34a.

The filter unit 32 performs a filter process on the luminance signal Y output from the luminance detection unit 31 and outputs a filter signal after the filter process. FIG. 4 shows a case where the number of taps n is 3 taps. The filter signal output from the filter unit 32 is, for example, the average luminance Yavg, and can be obtained by the following equation (2).

In equation (2), Y (−1), Y (0), and Y (1) are the luminance signal of the pixel one pixel after the pixel to be corrected, the luminance signal of the pixel to be corrected, and the correction, respectively. The luminance signal of the pixel immediately before the pixel to be performed is shown. Here, when the coefficient a _k (−1) = a _k (0) = a _k (1) = 1, the denominator of the equation (2) is as follows, and the equation (2) is a simple average: It represents the operation to be sought.

  Therefore, the average luminance of the pixel to be corrected and the peripheral pixels of the pixel to be corrected can be obtained by Expression (2). Note that “peripheral pixels of a pixel to be corrected” refers to a pixel from a pixel before i pixel of a pixel to be corrected to a pixel one pixel before the pixel to be corrected when i is a predetermined integer. From the pixel one pixel after the pixel to be performed to the pixel after i pixel of the pixel to be corrected. When the predetermined integer i is 1, the “peripheral pixels of the pixel to be corrected” are a pixel one pixel before the pixel to be corrected and a pixel one pixel after the pixel to be corrected.

  Thus, by using the configuration of the one-dimensional acyclic digital filter, the filter output in the one-dimensional direction of the luminance signal Y of the luminance detecting means 31 can be obtained. The obtained filter output is configured to obtain the luminance signal Y and the average value of the peripheral pixels of the pixel to be corrected, whereby the change in the brightness distribution in the one-dimensional direction can be obtained. Therefore, a correction gain corresponding to a change in brightness distribution can be detected, and a signal contrast can be corrected in consideration of a change in brightness distribution. In addition, the digital signal processing circuit has a general configuration, which can simplify the circuit scale and reduce the gate scale and reduce the cost.

  The number of taps n is not limited to 3 taps, and can be any number of taps. By increasing the number of taps, the characteristics of the cutoff frequency can be set finely, and a gradual change in luminance over a wide range can be detected. In this way, by switching the number of taps n, it is possible to configure the optimum filter means 32 according to changes in the luminance distribution due to different illumination conditions in the input image.

  In the above description, the case where the filter unit 32 is a one-dimensional acyclic digital filter has been described. However, the filter unit 32 may be a two-dimensional acyclic digital filter. By using a two-dimensional acyclic digital filter, it is possible to detect a regional change in luminance of the input image.

  The filter unit 32 is not limited to a configuration that performs the process of calculating the average luminance Yavg based on the above formula (2), and may be any configuration that can determine a change in brightness distribution, and outputs a weighted average. Other configurations, such as a configuration using a low-pass filter, a configuration using a band-pass filter, or the like.

Then, the correction gain calculation means 4, based on the average luminance signal Yavg output from the luminance filtering means 3 calculates and outputs the correction gain G _k.
For example, the correction gain Gk is _obtained by the following equation (3).

  In Expression (3), Yavg represents the average luminance output from the filter unit 32 and input to the correction gain calculation unit 4, Gmax represents the maximum value (maximum gain) of the correction gain, and Ymax represents the filter unit 32. Is the maximum output (maximum value within a range of values that the imaging signal Xa can take).

  The maximum luminance Ymax is uniquely determined by the digital resolution (the number of gradations) of the imaging signal. For example, in the case of 8-bit gradation, the maximum luminance Ymax is 255, and in the case of 10-bit gradation, the maximum luminance Ymax is 1023.

If for calculating a correction gain G _k in equation (3), can be determined according to the maximum value Gmax the quality of the image, fine, enables optimal image quality improvement. For example, it is possible to correct the luminance component of the illumination light using a luminance change in a relatively wide area (luminance change with a low spatial frequency) such as a luminance distribution due to illumination light on the screen to improve contrast and visual characteristics. it can.

For example, the corrected image pickup signal Xb output from the gain correction unit 2 is image-analyzed by the maximum correction gain calculation unit 13, and the maximum correction gain Gmax is obtained by detecting the regional distribution of the dark signal amount in the shooting screen. It can be calculated or determined. For example, the area of a region where dark pixels (for example, pixels with a luminance signal of 10% or less of Ymax) in the imaging screen are continuous and the ratio of the area to the entire screen is calculated and exists over a specified value (for example, 5%) If so, Gmax (which should be used instead of a predetermined value) is obtained. Gmax is, for example, Gmax = Ytg / Ydk
It can ask for. In the above formula, Ytg is a prescribed signal level, and is set to a value of 50% of Ymax, for example. Ydk is an average luminance in a dark portion, that is, a region where the luminance signal is 10% or less of Ymax.
When the area of a region where dark pixels (for example, pixels having a luminance signal of 10% or less of Ymax) continue in the imaging screen is less than the specified value (for example, 5%) for the entire screen, The Gmax of a predetermined value is used without performing the calculation according to the above formula.

  The numerical values used here (“10%” defining the dark region, “5%” representing the specified value, “50%” being the specified signal level Tg) are experimental, empirical, It is a numerical value derived statistically and can be changed according to the characteristics of the display device, the configuration of the image processing circuit, and the like.

  Instead of determining Gmax by calculation as described above, Gmax may be provided at the subsequent stage of the imaging apparatus, displaying the imaging result on the display means, confirming the displayed image, and determining Gmax by sensory judgment.

  Furthermore, based on the corrected imaging signal Xb, advanced information analysis such as histogram analysis of luminance distribution, regional luminance distribution analysis of images, color information analysis of sky and human skin, pattern analysis of shape, etc. By performing this, it is possible to determine Gmax with higher accuracy.

  As the maximum gain Gmax, it is possible to use the one calculated by the image pickup device by performing image analysis in the image pickup device as described above and held in the image pickup device. Therefore, an interface for controlling the maximum gain Gmax may be configured, and Gmax may be switched from a higher system.

  As an example of the above-mentioned “higher system”, an image analysis device that analyzes an image of the imaging device based on the imaging signal Xb, and a feature point detection that extracts and detects a feature point of an object or a person from the image of the imaging device There are devices. In this case, for example, the image analysis device analyzes the outdoor scenery facing the window, the bright part signal such as the sky, the indoor scenery, the dark part signal such as the shadow of the mountain, etc. Gmax is set so that the image quality of the captured image is improved.

  Further, in a feature point detection device or the like, Gmax is set so that the feature point of the subject can be detected optimally.

  By using the host system in this way, it is possible to configure a system with high accuracy and optimal image quality improvement for the host system.

Figure 6 is a graph showing a correction gain G _k output from the correction gain calculation means 4, FIG. 7 is a table showing a correction gain G _k output from the correction gain calculation means 4. 6, the horizontal axis represents the average brightness YAVG / Ymax standardized by the maximum brightness, the vertical axis represents the correction gain _{G k.}

As shown in FIGS. 6 and 7, when the maximum gain Gmax is larger than 1, as the average luminance Yavg / Ymax normalized by the maximum luminance increases (that is, as the average luminance Yavg increases), correction gain _{G k} decreases from a maximum gain Gmax, the average luminance YAVG is equal to the maximum luminance Ymax (i.e., at the YAVG / Ymax = 1), the correction gain _{G k} becomes 1 times. When the maximum gain Gmax is 1, the correction gain _Gk is 1 time.

Incidentally, the correction gain calculation means 4 has described an arrangement for obtaining a correction gain G _k by performing the operation according to equation (3), the correction gain G _k corresponding to advance the average luminance Yavg as a look-up table (LUT) You can keep it. When such an LUT is used, it is not necessary to perform division processing, so that the calculation processing in the correction gain calculation means 4 can be simplified.

FIG. 5 shows a block diagram when the LUT is used. A plurality of LUTs 41a, 41b, 41c corresponding to different values of Gmax, a selection unit 42 for selecting an LUT suitable for Gmax among the plurality of LUTs 41a, 41b, 41c, and an LUT selected by the selection unit 42 The LUT 43 that holds the contents of the written LUT is mounted. LUT43, depending on the average luminance Yavg inputted, outputs a correction gain _{G k.}

Thus, the correction gain calculation means 4 of FIG. 5, LUT41a, 41b, 41c has a configuration that holds the correction gain _{G k} with respect to the average luminance Yavg for different values of Gmax. Further, LUT 43 has a configuration that inputs the average luminance YAVG, outputs the corresponding correction gain _{G k.}

  By configuring the correction gain calculation means 4 with an LUT, the multiplication means and the division means become unnecessary, and there are effects such as a reduction in gate scale and a reduction in processing time.

An example of the gain correction means 2 of FIG. 1 is shown in FIG. Gain correcting means 2 shown in FIG. 8 is made from the multiplication means 21 is supplied with the correction gain G _k output from the correction gain calculation means 4, the correction gain G _k in the imaging signal Xa from the camera signal processing unit 8 Multiply and output the imaging signal Xb. That is, the imaging signal Xb is obtained by the following equation.
Xb = G _k × Xa

Correction gain G _k, the imaging signal Xb is Yes set to be a monotonically increasing function of the image signals Xa, without magnitude relationship between the signal value is inverted between adjacent pixels, the gradation reversal does not occur, the image quality Is preventing deterioration.

FIG. 9 is a graph showing a value obtained by multiplying the correction gain used in the first embodiment by the average luminance normalized by the maximum luminance. In FIG. 9, the horizontal axis indicates the average luminance Yavg / Ymax normalized by the maximum luminance, and the vertical axis indicates the value G _{k obtained} by multiplying the average luminance Yavg / Ymax normalized by the maximum luminance by the correction gain G _k. X Indicates Yavg / Ymax.

Here, the correction gain G _k is a value obtained so that the correction luminance G _k × Yavg / Ymax is a monotonically increasing function. FIG. 9 shows the corrected luminance G _k × Yavg / Ymax when the maximum gain Gmax is 1, 3 or 5 times. As the correction luminance, a correction luminance based on a calculation other than G _k × Yavg / Ymax may be used. As can be seen from FIG. 9, when the maximum gain Gmax is 1, the imaging signal Xa is output as it is. As can be seen from FIG. 9, as the maximum gain Gmax increases, the slope on the low brightness side increases and the slope on the high brightness side decreases. By increasing the inclination on the low luminance side, it is possible to amplify and output a low-frequency signal component that tends to be blacked out, and to improve the contrast of the low luminance portion. Further, the luminance signal and contrast signal on the high luminance side are maintained by reducing the inclination on the high luminance side to about 1.0 times that on the low luminance side. Accordingly, it is possible to prevent the problem that the high luminance side is crushed white, and it is possible to extract a high contrast signal even in a high luminance or low luminance signal, thereby improving visibility.

  Hereinafter, the processing procedure will be described with reference to FIGS. 10 and 11. FIG. 10 shows an overall processing procedure of the imaging apparatus. FIG. 11 shows details of the gain correction process (ST5) of FIG.

As shown in FIG. 10, the imaging apparatus according to the first embodiment includes a Gmax determination process (ST1), a Gmax change process (ST2), an exposure process (ST3), an imaging process (ST4), a gain correction process (ST5), An end determination process (ST6) is performed.
In step ST1, it is determined whether it is necessary to change Gmax. When the power is turned on, a predetermined initial value is output on the assumption that no change is required. If a request for changing Gmax occurs after power-on, Gmax is changed. The Gmax change request includes a Gmax change request based on an image analysis result, a Gmax change request from a host system, and the like.

  If it is determined in step ST1 that the Gmax needs to be changed, the Gmax changing process in step ST2 is performed. The Gmax changing process replaces the initial value or the currently held Gmax value with a new Gmax value. If there is no Gmax change request in step ST1, the process proceeds to step ST3 without performing the process of step ST2.

  Note that the example shown in FIG. 10 includes the Gmax determination process (ST1) and the Gmax change process (ST2). However, these processes (steps ST1 and ST2) may be performed when still shooting or when Gmax may remain at the initial value. It is also possible to adopt a configuration that does not perform the above process.

In the exposure process of step ST3, the exposure state of the subject is detected from the captured image, the first exposure value E1 is obtained, and further, the second exposure value E2 is obtained from Gmax and the first exposure value E1.
Further, the charge accumulation time S and the gain G of the image sensor 5 are adjusted based on the first exposure value E1 or the second exposure value E2.

Here, switching between the first exposure value E1 and the second exposure value E2 is controlled by a control means (not shown) in the imaging apparatus or a control means for controlling a system in an external system. The control means is constituted by a microcomputer, for example.
As an example, when setting optimal exposure correction in still image shooting or the like, the first exposure value E1 is set, and a correction gain maximum value Gmax changing process (ST2) or a gain correction process (ST5 described later) is performed. ), The maximum value Gmax of the correction gain and the exposure correction value Kd are obtained from the obtained imaging result, and the second exposure value E2 is obtained by using these values to obtain an image in a subsequent frame. It may be reflected.

  In the imaging process of step ST4, exposure control is performed based on the exposure value (first exposure value E1 or second exposure value E2) obtained in the exposure process (ST3), imaging is performed, and an imaging signal is output. To do.

The gain correction process (ST5), obtains the correction gain G _k of the image signal obtained by imaging processing (ST4) for each pixel, which is corrected by multiplying a correction gain G _k in the imaging signal Xa (signal of each pixel) Processing for obtaining the imaging signal Xb is performed.
In the end determination process in step ST6, it is determined whether or not the imaging is to be ended, and if it is continued, the process returns to step ST1.

An example of the details of the gain correction process (ST5) of FIG. 10 is shown in FIG. Of the steps shown in FIG. 11, step ST10, step ST14, step ST15, step ST16, and step ST17 are processes for performing processing for one frame in time series, and “steps ST10, ST14, ST15” “i” indicates the number in each line of the pixel to be processed, and “j” in steps ST10, ST16, and ST17 indicates the number of the line to which the pixel to be processed belongs. XPixel in step ST15 represents the number of pixels in one line, and YLine in step ST17 represents the number of lines in the screen.
When image data is stored in the memory, these processes need not be performed in time series, and only the processes from step ST11 to step ST13 are sufficient.

In step ST10, the pixel number i and the line number j are set to initial values (0).
The luminance filtering step ST11 is a step for performing the same processing as the luminance filtering means 3, and obtains the average luminance Yavg of the pixel and the surrounding pixels.

Correction gain calculating step ST12 is a step of performing the same processing as correction gain calculation means 4 calculates a correction gain G _k based on the average luminance Yavg determined by the luminance filtering step ST11.

Gain correction calculation step ST13, the correction gain G _k determined by the correction gain calculating step ST12, by multiplying the Xa (data for each pixel) image signal, (data of each pixel) corrected image signal seeking Xb.

In step ST14, the pixel number i is incremented by one. If i is smaller than XPixl in step ST15, the process returns to step ST11. If i is not smaller than XPixl in step ST15, the process proceeds to step ST16.
In step ST16, the line number j is incremented by one. If j is smaller than YLine in step ST17, the process returns to step ST11. If j is not smaller than YLine in step ST17, the process is terminated.

FIG. 12 shows an example of the relationship between the exposure correction value Kd used for obtaining the second exposure value E2 and Gmax.
In the illustrated example, Kd = 0 when Gmax is less than a predetermined threshold Gta (Gta = 2 in the illustrated example), and Kd = Kd1 (predetermined positive real number) when Gmax is equal to or greater than the threshold Gta (= 2).
The second exposure value E2 is obtained by the following equation (4) from the first exposure value E1 and the exposure correction value Kd.
E2 = Ke * E1 * (1/2) ^ Kd (4)
(In the above equation, Ke is a correction coefficient and is a positive real number. “^” Means that the sign “Kd” that follows it represents a power exponent. The same applies hereinafter. )
The second exposure value E2 obtained by the above formula becomes a smaller value as Kd is larger.

  Here, the correction coefficient Ke is usually 1, but a constant obtained experimentally is used depending on the configuration of the imaging apparatus. Alternatively, image analysis of the imaging state may be performed, and the value of the correction coefficient Ke may be switched in the state of the luminance level of the image.

In FIG. 12, as described above, when the maximum gain Gmax is equal to or greater than the predetermined threshold Gta (= 2), Kd = Kd1, and when Gmax is smaller than the threshold Gta (= 2), Kd = 0. .
By setting Kd = 0, the first exposure value and the second exposure value are proportional functions.
When Kd = 2 or more,
E2 = Ke * E1 * (1/2) ^ Kd1
Is used as the second exposure value E2. As a result, the exposure is reduced and the exposure state becomes dark.
Here, the threshold value Gta for switching the exposure correction value Kd is set to “2”, but it may be a number equal to or greater than “1”. When Gmax is less than the threshold value Gta, Kd = 0. Set it. When Gmax is greater than or equal to the threshold Gta, the exposure is controlled in the direction of narrowing by setting Kd to a positive real number. In this way, by controlling the exposure correction value Kd according to Gmax, it is possible to improve the darkness visibility while improving the whitening of the bright part and the contrast reduction of the bright part.

Instead of obtaining the exposure correction value Kd from the relationship shown in FIG. 12, the exposure correction value Kd may be obtained from the relationship shown in FIG. 13 or FIG.
In the example shown in FIG. 13, the exposure correction value Kd increases stepwise as Gmax increases in a range where Gmax is equal to or greater than the threshold value Gta.
In the example shown in FIG. 14, the exposure correction value Kd continuously increases as Gmax increases in a range where Gmax is equal to or greater than the threshold value Gta.

  Furthermore, a configuration in which different relationships such as the relationship shown in FIG. 12, the relationship shown in FIG. 13, and the relationship shown in FIG. By adopting a configuration in which the exposure correction value changes as shown in FIGS. 13 and 14, or adopting a configuration in which different relationships can be dynamically switched as in FIGS. 12, 13, and 14, The optimum exposure correction value Kd for the captured image can be determined, the quality of the corrected image can be improved, and the visibility can also be improved.

  In the relationship between Gmax and Kd shown in FIG. 12, the value of Kd in the range equal to or greater than Gta is fixed to Kd1, but the value of Kd may be changed based on the histogram of the signal level of the imaging signal Xa. good.

  The exposure control means 10 used in this case is shown in FIG. The illustrated exposure control means 10 is generally the same as the exposure control means 10 of FIG. 2, but includes a histogram generation analysis means 19. The histogram generation / analysis means 19 receives the imaging signal Xa, generates and analyzes a histogram, and outputs an analysis result Ha. The second exposure value generation means 17 generates the second exposure value E2 based on the first exposure value E1, the maximum correction gain value Gmax, and the histogram analysis result Ha in the histogram generation analysis means 19. .

  16 and 17 show two different examples of histograms. In the example shown in FIG. 16, the image pickup signal Xa is distributed in the vicinity of the center. In this case, the exposure correction value Kd is set to a low value so that, for example, the second exposure value E2 given by Expression (4) becomes a larger value. An example in which the exposure correction value Kd is set low is shown in FIG. In the illustrated example, a lower Kd4 (solid line) is used for the same Kd1 (shown by a broken line) as shown in FIG. Thus, by using a higher exposure value by reducing the exposure correction value Kd, the high-intensity signal is somewhat compressed, but the halftone with a large amount of information and the luminance correction of the dark part are performed. Can do.

  As shown in FIG. 17, when there are many high-luminance signals, the exposure correction value Kd is set higher, for example, so that the second exposure value E2 given by the equation (4) becomes a smaller value. To do. An example in which the exposure correction value Kd is set higher is shown in FIG. In the illustrated example, a higher Kd5 (solid line) is used for the same Kd1 (shown by a broken line) as shown in FIG. Thus, by using a lower exposure value by increasing the exposure correction value Kd, it is possible to avoid a reduction in the amount of information in the high luminance part due to compression of the high luminance part.

  As described above, by switching the exposure correction value Kd based on the histogram analysis result Ha, the accuracy of the exposure value setting can be improved, and the visibility can be improved. The exposure correction value may be switched using an image analysis result other than the histogram analysis.

  Further, not only the exposure correction value but also the correction gain may be controlled based on the histogram or other image analysis results.

  FIG. 20 is a diagram illustrating an example of an image before correction processing by the imaging device according to Embodiment 1, and FIG. 21 is a diagram illustrating an example of an image after correction processing by the imaging device according to Embodiment 1. is there.

  In FIG. 20, the bright area, that is, the scenery outside the window (the part indicated by the symbol DS1) is clearly reproduced, but the dark area, that is, the indoor person HD1 and the like is in a state close to black crushing. It shows that.

  22A and 22B relate to processing of a bright area of a captured image (that is, an area DS1 where the outside of the room can be seen through a window in FIG. 20) in the imaging apparatus according to Embodiment 1. FIG. ) Is a normalized luminance signal Xin / Ymax of the image signal Xa before the gain correction normalized from pixel position p0 to pixel position p6 (hereinafter sometimes simply referred to as “input luminance signal”). FIG. 22B is a diagram illustrating Yavg / Ymax, and FIG. 22B is a diagram illustrating the normalized input luminance signal Xin / Ymax and the normalized gain correction at the same pixel position p0 to pixel position p6 as in FIG. FIG. 7 is a diagram illustrating a luminance signal (hereinafter, simply referred to as “output luminance signal”) Xout / Ymax of an imaging signal Xb of FIG.

  As indicated by a broken line in FIG. 22A, the standardized input luminance signal Xin / Ymax is 0.6 at the pixel position p1, 0.7 at the pixel position p2, 0.8 at the pixel position p3, It is 0.7 at the position p4, 0.8 at the pixel position p5, and 0.6 at the pixel position p6.

  Therefore, as indicated by a solid line in FIG. 22A, when the number of taps n is 3, the normalized average luminance Yavg / Ymax is 0.66 at the pixel position p1 and 0.70 at the pixel position p2. The pixel position p3 is 0.73, the pixel position p4 is 0.76, the pixel position p5 is 0.70, and the pixel position p6 is 0.70.

When the maximum gain Gmax is three times, than the average luminance Yavg and expressions determined (3), the correction gain _{G k} is 1.29 times the pixel position p1, 1.25 times the pixel position p2, the pixel position p3 1.22 times, 1.19 times at pixel position p4, 1.25 times at pixel position p5, and 1.25 times at pixel position p6. Thus, the correction gain Gk of each pixel can be _obtained by obtaining the average luminance Yavg of each pixel.

  FIG. 22B is a diagram illustrating the normalized input luminance signal Xin / Ymax and the normalized output luminance signal Xout / Ymax at the same pixel position p0 to pixel position P6 as in FIG. . As shown by a broken line in FIG. 22B, the standardized input luminance signal Xin / Ymax is 0.6 at the pixel position p1, 0.7 at the pixel position p2, 0.8 at the pixel position p3, It is 0.7 at the position p4, 0.8 at the pixel position p5, and 0.6 at the pixel position p6.

Coordinates (M, N) pixel gain corrected output luminance signal Xout (M, N), using the coordinates (M, N) input luminance signal Xin (M, N) of pixels with a gain _{G k,} the following It can be obtained by equation (5).
_{Xout (M, N) = G} k × Xin (M, N) ... (5)

  As shown by a solid line in FIG. 22B, the standardized output luminance signal Xout / Ymax is 0.77 at the pixel position p1, 0.88 at the pixel position p2, 0.98 at the pixel position p3, It is 0.83 at the position p4, 1.00 at the pixel position p5, and 0.75 at the pixel position p6.

In general, when the input image is an RGB signal, the following expressions (6a), (6b), and (6c) hold.
Rb (M, N) = G _k × Ra (M, N) (6a)
Gb (M, N) = G _k × Ga (M, N) (6b)
Bb (M, N) = G _k × Ba (M, N) (6c)
Here, Rb (M, N) is an R signal (output R signal) obtained by correcting the gain of the pixel at coordinates (M, N), and Ra (M, N) is the pixel at the coordinates (M, N). R signal (input R signal) before gain correction, Gb (M, N) is a G signal (output G signal) after gain correction of a pixel at coordinates (M, N), and Ga (M, N) Is a G signal (input G signal) before the gain correction of the pixel at the coordinates (M, N), and Bb (M, N) is a B signal (output B) after the gain correction of the pixel at the coordinates (M, N). Ba (M, N) is a B signal (input B signal) before gain correction of the pixel at coordinates (M, N).

In general, when the input image is a YCbCr signal, the following equations (7a), (7b), and (7c) hold.
Yb (M, N) = G _k × Ya (M, N) (7a)
_{Cbb (M, N) = G} k × (Cba (M, N) -Cbof) + Cbof
... (7b)
_{Crb (M, N) = G} k × (Cra (M, N) -Crof) + Crof
... (7c)
Here, Yb (M, N) is a luminance signal (output luminance signal) obtained by correcting the gain of the pixel at coordinates (M, N), and Ya (M, N) is the pixel at the coordinates (M, N). It is a luminance signal (input luminance signal) before gain correction, and Cbb (M, N) and Crb (M, N) are color difference signals obtained by correcting the gain of the pixel at coordinates (M, N), and Cba (M, N N) and Cra (M, N) are color difference signals (input color difference signals) before correction of the pixel at coordinates (M, N), and Cbof and Crof are offset amounts when the color difference signal is signal-processed. .

Further, the formula (6a), (6b), as shown in (6c), by multiplying the same correction gain _{G k} into RGB signals uniformly, white balance in the local area, without displacement, improve the dynamic range can do.

  FIGS. 23A and 23B relate to the processing of the dark region (low luminance region) HD1 of the captured image in the imaging apparatus according to Embodiment 1, and FIG. 23A illustrates the pixel position from the pixel position q0. FIG. 23 is a diagram showing the normalized input luminance signal Xin / Ymax and the normalized average luminance Yavg / Ymax up to q6, and FIG. 23B shows the same pixel position q0 to pixel position q6 as in FIG. It is a figure which shows the standardized input luminance signal Xin / Ymax and the standardized output luminance signal Xout / Ymax.

As indicated by the broken line in FIG. 23A, the standardized input luminance signal Xin / Ymax is 0.1 at the pixel position q1 in the low luminance region HD1, 0.2 at the pixel position q2, and the pixel position. 0.3 at q3, 0.2 at pixel position q4, 0.3 at pixel position q5, and 0.1 at pixel position q6. Further, as indicated by a solid line in FIG. 23A, the normalized average luminance Yavg / Ymax is 0.16 at the pixel position q1, 0.20 at the pixel position q2, 0.23 at the pixel position q3, The pixel position q4 is 0.26, the pixel position q5 is 0.20, and the pixel position q6 is 0.20. Further, the correction gain G _k is 2.25 times at the pixel position q1, 2.14 times at the pixel position q2, 2.05 times at the pixel position q3, 1.97 times at the pixel position q4, and 2.7 at the pixel position q5. 14 times and 2.14 times at pixel position q6.

  As shown by the solid line in FIG. 23B, the output-corrected luminance signal Xout of each pixel is 0.23 at the pixel position q1, 0.43 at the pixel position q2, 0.62 at the pixel position q3, It is 0.39 at the position q4, 0.64 at the pixel position q5, and 0.21 at the pixel position q6.

  As can be seen from FIG. 22B, in the bright region DS1, the correction luminance is about 1.2 times and the output luminance signal is not so different from the input luminance signal. Thereby, the contrast of the pixel unit of a bright area is maintained. On the other hand, as can be seen from FIG. 23 (b), the correction gain is approximately doubled in the dark region HD1. This indicates that the signal level compressed at the black level becomes brighter and the contrast of the pixel unit in the dark region is also amplified.

As described above, obtains the correction gain G _k for each pixel by the average luminance YAVG, by a process of multiplying the correction gain G _k for each pixel in the image pickup signal subjected, dark regions HD1 is clearly (in FIG. 21 HD2), the bright region DS1 can improve the dynamic range as in the case of maintaining the same contrast (DS2 in FIG. 21).

  FIG. 24 is a diagram showing the occurrence frequency (frequency) at each luminance level of the imaging signal before gain correction in a histogram, and FIG. 25 is a histogram of the occurrence frequency (frequency) at each luminance level of the imaging signal after gain correction. FIG. 24 and 25 show the effect of dynamic range improvement of FIGS. 20, 21, 22A and 22B, and FIGS. 23A and 23B by histograms.

  As shown in FIGS. 24 and 25, since the bright region (DS1 in FIG. 20) of the image (input image) represented by the image signal before correction is a high luminance region, the captured image has a high luminance level. Distributed. Further, since the dark region HD1 of the image (output image) represented by the image signal before correction is a low luminance region, the captured image is distributed at a low luminance level.

  As shown in FIGS. 24 and 25, when the imaging signal is corrected, since the correction gain is small in the bright region DS1, the signal indicated by the two-dot chain line distributed in the high luminance level of the region DS2 is Although the signal is indicated by a solid line in the region DS2, the change is small. On the other hand, the signal indicated by the two-dot chain line distributed in the low luminance level of the region HD1 becomes a signal indicated by the solid line of the region HD2, and changes greatly.

  This is because the correction using the expression (3) improves the contrast at low luminance and high luminance, eliminates the blackout on the low luminance side, and the low luminance signal. Indicates that an image with good visibility and an improved dynamic range can be obtained by moving to a higher luminance side. Further, by performing the correction, the average luminance is distributed near the center, and the display quality can be improved even in a display device having a narrow dynamic range (for example, a liquid crystal display).

  Further, according to the imaging apparatus according to the first embodiment, the dynamic range of the pixel to be corrected is corrected based on the luminance distribution of the peripheral pixels of the pixel to be corrected, so that the reflection timing of the analysis result is shortened as much as possible. The dynamic range of the input image can be improved appropriately.

  Furthermore, according to the imaging apparatus according to the first embodiment, the dynamic range can be expanded in units of pixels without using an external frame memory, and it is not necessary to perform complicated calculations, thereby simplifying calculation and processing. Therefore, the configuration can be simplified, and as a result, the cost can be reduced.

As described above, in this embodiment, the exposure control means 10, the correction gain G maximum value of _k (correction maximum gain Gmax) corrects the exposure value in accordance with (the first exposure value from the second exposure value Based on the average luminance Yavg of each pixel of the luminance component of the imaging signal Xa output from the camera signal processing means 8 and the surrounding pixels when the corrected exposure value (second exposure value) is used. It calculates a correction gain G _k for each pixel Te, and generates an imaging signal Xb corrected by multiplying a correction gain G _k in the imaging signal Xa. The corrected maximum gain Gmax may be determined based on the average luminance of the dark part in the imaging screen as described above, or may be a predetermined value.

FIG. 26 is a diagram showing the effect of exposure correction (using the second exposure value E2 instead of the first exposure value E1) and gain correction.
An imaging signal Xa output from the camera signal processing means 8 when imaging is performed with the first exposure value E1 is indicated by a symbol Xa (1), and corresponds to the imaging signal Xa (1). The output Xb of the gain correction means 2 is indicated by the symbol Xb (1).
Further, the image pickup signal Xa output from the camera signal processing means 8 when the image is taken with the second exposure value obtained by reducing the first exposure value to ½ is indicated by the symbol Xa (2), and the image pickup signal A gain-corrected imaging signal (output of the gain correction means 2) Xb corresponding to Xa (2) is indicated by a symbol Xb (2).
In this case, it is assumed that the second exposure value E2 is ½ of the first exposure value E1.
When gain correction is performed on the signal Xa (1) obtained with the first exposure value E1, the correction gain is about 1.2 times, but when the second exposure value E2 is used, the correction gain G _k The contrast is improved by doubling.
Further, as can be seen at the pixel position (p5) or the like, when the first exposure value E1 is used, the bright part may be crushed and the contrast of the bright part may be reduced. By using E2, whiteout can be avoided.

  In this way, exposure control is performed so as to suppress white-out for an image that has been white-out conventionally by capturing an image using the second exposure value E2 and performing gain correction. Further, the signal shifted to the black side can be amplified by gain correction by exposure control. Contrast, image quality, and visibility can be improved for both a bright signal in a bright part and a dark signal in a dark part.

Embodiment 2. FIG.
FIG. 27 is a block diagram schematically showing the configuration of the gain correction means 2 of the imaging apparatus according to Embodiment 2 of the present invention (apparatus that performs the imaging method according to Embodiment 2). The gain correction unit 2 according to the second embodiment includes an offset detection unit 22, an offset subtraction unit 23, a multiplication unit 21, and an offset addition unit 24. The configuration of the imaging apparatus of the second embodiment other than the gain correction unit 2 is the same as that of the first embodiment shown in FIG.

The gain correction means 2 according to Embodiment 2 has means for adjusting the offset of the luminance signal (input luminance signal) Xin of the imaging signal Xa before gain correction.
The offset is when the image pickup by the image pickup device 5 is performed in the backlight state, the flare of the lens (not shown) of the image pickup apparatus affects the image, or the signal level is black due to the image pickup environment / condition of the subject. As a countermeasure, it is an amount added to cancel these influences (or an amount of deviation from the standard state due to such influences).

The detailed correction operation when the offset adjustment is performed will be described below. The offset detector 22 detects the minimum signal of the image signal Xa before correction, thereby obtaining an offset amount indicating the degree of black float of the input image. The offset amount Offset can be obtained by the following equation (8).
Offset = P × MIN (R, G, B) (8)
Here, MIN (R, G, B) indicates the minimum value of the input image RGB signal, and P is a positive real number that satisfies 0 ≦ P ≦ 1. The offset amount Offset is automatically detected by storing and storing the minimum value MIN (R, G, B) of the R, G, B signal of the image for one screen effective for correcting one frame or more before. Can be detected.

  FIG. 27 shows a configuration in which the offset detection unit 22 automatically detects the offset amount Offset from the imaging signal Xa, but a configuration in which the offset amount Offset is input from an external device can also be employed. In the external device, the performance of the external device can be improved by improving the correction gain for the signal to be corrected. Specifically, the external device performs advanced image processing such as biometric authentication of fingerprints, veins, faces, etc., shape authentication, character recognition, etc., and detects feature points of the subject (face in the case of face authentication) And having a function of performing authentication based on the detection result of the feature points. In the external device, it is possible to enhance the feature point signal by obtaining and setting the offset amount Offset from the region having the feature point and the result of detecting the signal level in the region from the image of the imaging device. It becomes. In addition, by increasing the signal level of the feature point, it is possible to improve performance such as detection accuracy and authentication rate of the external device.

The offset subtracting means 23 calculates the offset amount Offset obtained by the offset detecting means 22 from the input R signal Ra (M, N) at coordinates (M, N) and the input G signal Ga (M, N) at coordinates (M, N). ), Subtract from each of the input B signal Ba (M, N) of coordinates (M, N),
Ra (M, N) -Offset,
Ga (M, N) -Offset,
Ba (M, N) -Offset is output.

Multiplying means 21, the subtracted signal offset Offset from the offset subtraction means 23, by multiplying the correction gain G _k obtained by the correction gain calculation means 4,
G _k × (Ra (M, N) −Offset),
G _k × (Ga (M, N) -Offset),
G _k × (Ba (M, N) −Offset) is output.

The offset addition means 24 receives the multiplication signal from the multiplication means 21, adds the same offset amount Offset as the offset subtraction means 23, and
G _k × (Ra (M, N) −Offset) + Offset,
G _k × (Ga (M, N) −Offset) + Offset,
G _k × (Ba (M, N) −Offset) + Offset is output.

The processing of the offset subtracting means 23, the multiplying means 21, and the offset adding means 24 is summarized as shown in the following equations (9a), (9b), and (9c).
_{Rb (M, N) = G} k × (Ra (M, N) -Offset) + Offset
... (9a)
_{Gb (M, N) = G} k × (Ga (M, N) -Offset) + Offset
... (9b)
_{Bb (M, N) = G} k × (Ba (M, N) -Offset) + Offset
... (9c)

If you do not offset correction, the correction gain G _k ends up amplifying offset Offset, it decreases the correction gain to the signal desired to improve the contrast, resulting in conversion to totally free signal contrast. By performing offset correction, it is possible to increase the correction gain for a signal whose contrast is to be improved, and to realize a process with high contrast.

Note that the offset added by the offset adding means 24 (second offset) may be smaller than the offset (first offset) subtracted by the offset subtracting means 23. For example, the offset added by the offset adding means 24 is Offset1 = Offset1 = Q × Offset
(However, 0 ≦ Q <1)
It may be given by
By making the offset correction amount Offset1 of the offset adding means 11 smaller than the offset amount Offset of the offset subtracting means 10, there is an effect of reducing black float. In other words, the signal after offset correction is prevented by the offset amount Offset1, and the black floating before the offset correction is corrected, and the image quality without sharpness and without black signals can be improved. That is, an image with a black level is obtained. Processing for reducing the offset correction amount Offset1 of the offset adding means 24 compared to the offset amount Offset of the offset subtracting means 10 is expressed by the following equations (10a), (10b), (10c), and (10d).
_{Rb (M, N) = G} k × (Ra (M, N) -Offset) + Offset1
... (10a)
_{Gb (M, N) = G} k × (Ga (M, N) -Offset) + Offset1
... (10b)
_{Bb (M, N) = G} k × (Ba (M, N) -Offset) + Offset1
... (10c)
Offset1 = Q × Offset (10d)

  In the second embodiment, points other than those described above are the same as those in the first embodiment.

  According to the imaging device 12 according to the second embodiment, it is possible to detect the offset amount of the image, and by performing the offset correction based on the detected offset amount, the image quality can be improved by improving the contrast of the signal distributed in the low luminance. Improvements can be made.

  In the imaging apparatus according to the second embodiment, the subtraction amount of the offset amount before the correction gain multiplication processing and the addition amount of the offset amount after the correction gain multiplication processing can be made different. Signal tightness can be improved and image quality can be improved.

ここで、Ｇｍｉｎは、補正前の撮像信号Ｘａがその最大値であるときに乗算される最小利得を示している。 Embodiment 3 FIG.
In the imaging apparatus according to the third embodiment, the calculation formula for calculating the correction gain by the correction gain calculating means 4 is different from the formula (3) shown in the first embodiment. Specifically, the correction gain _{G k,} using the maximum gain Gmax, the minimum gain Gmin, the average luminance YAVG, the maximum luminance Ymax, obtained by the following equation (11).

Here, Gmin represents a minimum gain that is multiplied when the uncorrected imaging signal Xa has the maximum value.

Figure 28 is a graph showing a correction gain G _k output from the correction gain calculation means 4 for calculation of the equation (11), FIG. 29, a correction gain used in the third embodiment, normalized by the maximum intensity It is a graph which shows the value which multiplied the average brightness | luminance which carried out. In FIG. 29, the horizontal axis represents the average luminance Yavg / Ymax normalized by the maximum luminance, and the vertical axis represents the value G _{k obtained} by multiplying the average luminance Yavg / Ymax normalized by the maximum luminance by the correction gain G _k. X Indicates Yavg / Ymax.

Gmin is experimental, or the minimum value of the correction gain _{G k} obtained in statistical processing, compared with the correction gain _{G k} obtained by the equation (3), YAVG / Ymax is large (close to 1) the range Thus, the correction gain G _k obtained by the equation (11) is different in that it has a value less than 1. In the region where the average luminance is high, the signal level varies from pixel to pixel around the average luminance. The correction gain can be reduced with respect to the variation in the signal for each pixel as compared with that obtained by the expression (3), and the whitening of the high luminance portion can be prevented. It is possible to prevent the contrast signal in the high luminance portion from being lost due to white-out, and to compress and store the contrast signal with the correction gain Gmin.

  Note that the characteristic of the correction gain (the characteristic of the value obtained by multiplying the correction gain by the average luminance normalized by the maximum luminance) is not limited to the characteristic shown in FIG. A correction gain having the characteristics shown in FIG. By using such a correction gain, it is possible to improve the dynamic range optimum for the display device and the imaging device in consideration of the gradation characteristics of the display device (not shown) and the imaging device.

  Further, in the third embodiment, the points other than the above are the same as those in the first or second embodiment.

  According to the imaging apparatus of the third embodiment, it is possible to maintain the contrast information of the high-intensity signal with a bright distribution, and to prevent whiteout.

Embodiment 4 FIG.
In the imaging apparatus according to the fourth embodiment of the present invention, the content of the filter processing by the filter unit 32 is different from those in the first to third embodiments. In the fourth embodiment, the filter means includes, for example, an epsilon filter (ε filter) that is a nonlinear filter.

  FIG. 32 is a diagram showing an interval linear function when the filter unit 32 according to the fourth embodiment is configured by an ε filter that is a nonlinear filter. FIG. 33 is a diagram showing the luminance signal level after correction in the comparative example using a linear filter as the filter means 32. FIG. 34 is a diagram showing the luminance signal level after correction when the nonlinear filter of the fourth embodiment is used.

  As shown in FIG. 33, in the case of the comparative example using a linear filter as the filter means, the signal before the correction (imaging signal Xa) is smoothed, so that the region where the region luminance changes abruptly (for example, If the luminance level before correction in FIG. 33 changes abruptly between the pixel positions 5 and 6), the average luminance Yavg is affected by a rapid luminance change. Note that the average luminance (filter output) shown in FIG. 33 is a value obtained with the number of taps n = 5.

  At the pixel position 5, the average luminance is output at a higher level than the average luminance (filter output) from the pixel position 1 to the pixel position 4 due to the influence of the high luminance signal after the pixel position 6. It can be seen that as the average luminance increases, the correction gain decreases, and the signal level of the corrected signal (imaging signal Xb) at pixel position 5 (thick solid line in FIG. 33) decreases.

  Further, at the pixel position 6, the average luminance is influenced by the luminance signal before the pixel position 5, and becomes a small value. For this reason, the correction gain is increased, and the corrected output level (thick solid line in FIG. 33) is a large value. In a region (edge region) where the luminance change between pixels is large, the correction gain is increased, so that the contrast can be enhanced by an aperture effect in which the signal at the edge portion is enhanced. With the aperture effect, in the recognition device that performs image processing for the purpose of viewing photographs, displays, etc., and recognition processing using images with emphasized edges, the edge part is emphasized and the visibility is sharp A high-quality image with high display quality can be generated.

  On the other hand, an authentication apparatus that detects a feature point from a slight contrast difference needs to output a faithful signal without applying edge enhancement in the preprocessing. Therefore, when there is a region where the luminance changes drastically as described above, it is necessary to consider the luminance change in the average luminance.

Therefore, in the fourth embodiment, by providing the filter means 32 with a non-linear filter characteristic, it is possible to eliminate the above-described problem even when the change in luminance is severe. For example, a non-linear ε filter is used for the filter means 32. In general, the first-order ε filter is defined by the following equations (12) and (13).

  At this time, the function f (x) is an interval linear function with x as a variable, and is given by the following equation (14). y (n) represents the average luminance of the output of the ε filter of the Y signal, and x (n) represents the luminance of n pixels. Expression (14) calculates a difference value between the pixel (n) and the average pixel (± k pixels). If the difference value is within ε, the difference value is used to obtain an average value, and if the difference value exceeds ε, α (here, 0) is used. By performing such processing, the correction gain is obtained without the average value being dragged with respect to the amount of sudden change in luminance, such as when the luminance changes specifically due to edges or noise accompanied by a sudden change in luminance. be able to.

  Here, the function with α = 0 is the interval linear function shown in FIG.

  FIG. 34 shows the average luminance when a nonlinear ε filter is used as the filter means 32. Also at the pixel position 5 and the pixel position 6, the change (trend) of the luminance of the signal before correction is reproduced. Also, even with the signal level after correction, unlike the characteristic of FIG. 33, the change (trend) in the luminance of the signal before correction can be reproduced.

  As described above, by using the ε filter, it is possible to realize an optimum dynamic range conversion process while retaining edge information even when a sharp luminance change exists in a signal before correction.

  The ε filter can be composed of a comparator and a line memory, and has effects such as a reduction in hardware mounting area and a reduction in software processing time.

  In addition, the configuration using a non-linear filter such as a median filter or a stack filter may be used as long as it is a means capable of detecting the luminance level of the region without being limited to the ε filter.

  Further, in the fourth embodiment, points other than the above are the same as those in any of the first to third embodiments.

  According to the imaging apparatus of the fourth embodiment, the output of the filter means can hold edge information, and the Mach effect in the case of a sudden luminance change can be suppressed.

1 is a block diagram schematically showing a configuration of an imaging apparatus (an apparatus that performs an image processing method according to Embodiment 1) according to Embodiment 1 of the present invention. FIG. 6 is a block diagram illustrating an example of an exposure control unit used in Embodiment 1. FIG. FIG. 5 is a diagram illustrating a relationship among subject brightness, charge accumulation time, and gain in signal processing means in the first embodiment. 4 is a block diagram illustrating an example of a luminance filtering unit used in Embodiment 1. FIG. 3 is a block diagram illustrating an example of a correction gain calculation unit used in the first embodiment. FIG. 3 is a graph showing a correction gain used in the imaging apparatus according to Embodiment 1. 6 is a table showing correction gains used in the imaging apparatus according to Embodiment 1. 3 is a block diagram showing an example of gain correction means used in Embodiment 1. FIG. 6 is a graph illustrating a value obtained by multiplying a normalized average luminance by a correction gain used in the imaging apparatus according to Embodiment 1; 3 is a flowchart illustrating an operation of the imaging apparatus according to the first embodiment. It is a flowchart which shows the detail of the gain correction process (ST5) of FIG. It is a figure which shows an example of the relationship with respect to the maximum correction gain of the exposure correction value used in the exposure correction means 10 of Embodiment 1. It is a figure which shows the other example of the relationship with respect to the maximum correction gain of the exposure correction value used in the exposure correction means 10 of Embodiment 1. It is a figure which shows the further another example of the relationship with respect to the maximum correction gain of the exposure correction value used in the exposure correction means 10 of Embodiment 1. FIG. It is a block diagram which shows the other example of the exposure control means used in Embodiment 1. FIG. It is a figure which shows an example of the histogram of an imaging signal. It is a figure which shows the other example of the histogram of an imaging signal. It is a figure which shows the further another example of the relationship with respect to the maximum correction gain of the exposure correction value used in the exposure correction means 10 of Embodiment 1. FIG. It is a figure which shows the further another example of the relationship with respect to the maximum correction gain of the exposure correction value used in the exposure correction means 10 of Embodiment 1. FIG. 6 is a diagram illustrating an example of an image before correction processing by the imaging apparatus according to Embodiment 1. FIG. 6 is a diagram illustrating an example of an image after correction processing by the imaging apparatus according to Embodiment 1. FIG. (A) is a figure which shows the luminance signal (broken line) and the standardized average luminance (solid line) of the imaging signal before the standardized gain correction in the high luminance region shown in FIG. ) Is a diagram showing the luminance signal (dashed line) of the image signal before standardized gain correction and the luminance signal (solid line) of the image signal after standardized gain correction in the high luminance region shown in FIG. It is. (A) is a figure which shows the luminance signal (dashed line) of the imaging signal before the gain correction normalized in the low-luminance area | region shown by FIG. 20, and the normalized average luminance (solid line), (b) FIG. 21 is a diagram showing a luminance signal (dashed line) of a standardized image signal before gain correction and a luminance signal (solid line) of a standardized image signal after gain correction in the low luminance region shown in FIG. . It is a figure which shows the generation frequency (frequency) of each brightness | luminance of the imaging signal before a gain correction with a histogram. It is a figure which shows the generation frequency (frequency) of each brightness | luminance of the imaging signal after gain correction with a histogram. In Embodiment 1, it is a figure which shows the imaging signal obtained when using a 1st exposure value, and the imaging signal obtained when using a 2nd exposure value. It is a block diagram which shows an example of the gain correction means used with the imaging device (apparatus which enforces the image processing method which concerns on Embodiment 2) which concerns on Embodiment 2 of this invention. 10 is a graph illustrating a correction gain used in the imaging apparatus according to Embodiment 3. 10 is a graph illustrating a value obtained by multiplying a normalized average luminance by a correction gain used in the imaging apparatus according to the third embodiment. 10 is a graph illustrating a value obtained by multiplying a normalized average luminance by a correction gain used in another imaging apparatus according to the third embodiment. 10 is a graph illustrating a value obtained by multiplying a normalized average luminance by a correction gain used in another imaging apparatus according to the third embodiment. It is a figure which shows a section linear function in case the filter means of the imaging device which concerns on Embodiment 4 of this invention is comprised with an epsilon filter. It is a figure which shows the luminance signal level after correction | amendment of the imaging device of the comparative example using a linear filter as a filter means. It is a figure which shows the luminance signal level after correction | amendment at the time of using a nonlinear filter as a filter means of the imaging device which concerns on Embodiment 4. FIG.

Explanation of symbols

2 gain correction means, 3 luminance filtering means, 4 correction gain calculation means, 5 imaging device, 6 analog signal processing means, 7 A / D conversion means, 8 camera signal processing means, 9 timing generation means, 10 exposure control means, 11 Integrating means, 13 maximum gain determining means, 14 imaging signal generating means, 15 correction gain determining means, 31 luminance detecting means, 32 filter means.

Claims (13)

Imaging signal generating means for receiving light from a subject and outputting a first imaging signal corresponding to the light;
Gain correcting means for correcting the first imaging signal for each pixel and outputting a corrected second imaging signal;
A first exposure value of the imaging signal generating means for making the average value of the first imaging signal coincide with a predetermined target value is determined, and the first exposure value is further determined based on the maximum value of the correction gain. An exposure control unit that corrects to obtain a second exposure value, and controls the exposure of the imaging signal generation unit by the second exposure value;
Filtering is performed on the value of each pixel of the luminance component of the first imaging signal and the value of the surrounding pixels, and a correction gain is determined for each pixel based on the filtering output and the maximum value of the correction gain. Correction gain determining means,
The imaging apparatus characterized in that the gain correction means performs the correction by an operation including multiplication of the correction gain determined by the correction gain determination means and the first imaging signal. The imaging signal generating means is
An image sensor with variable charge storage time;
A variable gain processing signal processing means for amplifying a signal output from the image sensor;
The imaging apparatus according to claim 1, wherein at least one of the charge accumulation time and the amplification gain is adjusted by the second exposure value.   The imaging apparatus according to claim 1, wherein a maximum value of the correction gain is determined based on the second imaging signal.   2. The imaging according to claim 1, wherein the correction gain determination unit determines the correction gain so that the second imaging signal is a monotonically increasing function with respect to the first imaging signal. apparatus.   2. The filtering in the correction gain determining means is performed by an operation for obtaining an average value of a value of each pixel of a luminance component of the first image pickup signal and a value of surrounding pixels. Imaging device. The correction gain determining means has the following formula:

(Where Gk is the correction gain,
Gmax is the maximum value of the correction gain,
Yavg is a filtered output of the luminance component of the first imaging signal,
Ymax is a maximum value within a range of values that can be taken by the first imaging signal. )
The imaging apparatus according to claim 1, wherein the correction gain is determined by: The correction gain determining means has the following formula:

(Where Gk is the correction gain,
Gmax is the maximum value of the correction gain,
Gmin is the minimum gain that is multiplied when the first imaging signal is at its maximum value,
Yavg is a filtered output of the luminance component of the first imaging signal,
Ymax is a maximum value within a range of values that can be taken by the first imaging signal. )
The imaging apparatus according to claim 1, wherein the correction gain is determined by: The exposure control means obtains an exposure correction value based on the maximum value of the correction gain,
Based on the exposure correction value, the following equation E2 = Ke × E1 × (1/2) ^Kd
(However,
E2 is the second exposure value,
Ke is a predetermined constant,
E1 is the first exposure value,
Kd is the exposure correction value)
The imaging device according to claim 1, wherein the second exposure value is obtained by:   The imaging apparatus according to claim 8, wherein the exposure correction value increases as the maximum value of the correction gain increases.   The exposure control means uses the second exposure value only when the maximum value of the correction gain is greater than 1, and substitutes the second exposure value when the maximum value of the correction gain is 1 or less. The image pickup apparatus according to claim 1, wherein exposure of the image pickup signal generation unit is controlled using the first exposure value.   The imaging apparatus according to claim 1, wherein the exposure control unit adjusts the exposure correction value according to an image analysis result of the first imaging signal. The gain correction means includes
Offset subtraction means for subtracting a first offset value from the first imaging signal;
Multiplying means for multiplying the output of the offset subtracting means by the correction gain;
Offset addition means for adding a second offset value to the output of the multiplication means,
The imaging apparatus according to claim 1, wherein the second offset value is equal to or smaller than the first offset value. In an imaging method using an imaging device including an imaging signal generation unit that receives light from a subject and outputs a first imaging signal corresponding to the light,
A gain correction step of correcting the first imaging signal for each pixel and outputting a corrected second imaging signal;
A first exposure value of the imaging signal generating means for making the average value of the first imaging signal coincide with a predetermined target value is determined, and the first exposure value is further determined based on the maximum value of the correction gain. An exposure control step of correcting and obtaining a second exposure value, and controlling the exposure of the imaging signal generating means with the second exposure value;
Filtering is performed on the value of each pixel of the luminance component of the first imaging signal and the value of the surrounding pixels, and a correction gain is determined for each pixel based on the filtering output and the maximum value of the correction gain. A correction gain determining step,
The gain correction step performs the correction by an operation including multiplication of the correction gain determined in the correction gain determination step and the first imaging signal.

Images