JP2011199785A - Imaging apparatus and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a dynamic range enlarged image in which noise amplification during gradation conversion is suppressed.SOLUTION: An identical exposure and image-capturing setting unit has: a gradation conversion curve calculation section 14 for calculating a gradation conversion curve for a dynamic range enlarged image generated by combining a plurality of sheets of differently exposed and captured images; an amplification factor calculation section 15 for calculating a contrast amplification factor from the gradation conversion curve; a luminance range selection section 16 for selecting a specific luminance range in the dynamic range enlarged image based on the contrast amplification factor; and an identical exposure and image-capturing a time number calculation section 19 for calculating the number of times of identical exposure and image-capturing based on an exposure condition corresponding to the luminance range among exposure conditions during different exposure and image-capturing and based on the luminance range and the contrast amplification factor. Identical exposure and image capturing performed under the condition set by the identical exposure and image-capturing setting unit 9, an image combining unit 5 combines the dynamic range enlarged image and a plurality of sheets of identically exposed and captured images, and a gradation converting unit 7 performs gradation conversion for the obtained combination image.

Description

  The present invention relates to an imaging apparatus and method for generating a dynamic range expanded image by combining a plurality of captured images with different exposures.

  In general, the dynamic range of an image sensor mounted on an imaging device such as a digital camera is the ratio of the highest luminance to the lowest luminance of all subjects within the imaging range (frame) (hereinafter referred to as “scene”), that is, It may be narrower than the dynamic range. In such a case, in an image (captured image) obtained by imaging, whiteout occurs in a high luminance part and blackout occurs in a low luminance part. Such a captured image is different from the scene observed by the photographer.

  Therefore, for example, Patent Documents 1 and 2 disclose a technique for solving the narrow dynamic range of the imaging apparatus as described above. Patent Document 1 discloses a technique for obtaining a wide dynamic range image by acquiring a plurality of captured images with different exposure amounts for one scene and combining them to create a single image. Yes. This technology makes the dynamic range of the image substantially wider than the dynamic range of the scene by taking multiple shots, thereby preventing whiteout in the high-luminance area and blackout in the low-luminance area in the final image. Is. Further, Patent Document 2 obtains an entire imaging range image acquired with appropriate exposure for the entire imaging range and an attention subject image acquired with appropriate exposure for the attention subject when there is an object (attention subject) to be noted in the scene. Thus, by extracting a partial image of the target subject area from the target subject image and replacing the extracted partial image with the partial image of the target subject area in the entire imaging range image, both the target subject and the other subject are properly exposed. A technique for obtaining an image is obtained.

  In general, image processing such as gradation conversion processing is performed on a captured image obtained by imaging so that an observer can observe the image as a more natural image, and the image is stored or displayed. Has been done. At this time, in the gradation conversion process, the noise in the low luminance part is amplified due to the contrast amplification. Therefore, for example, Patent Document 3 discloses a technique for solving this noise amplification problem. In general, noise can be reduced by superimposing images taken under the same conditions. Therefore, in the technique disclosed in Patent Document 3, first, a contrast conversion factor is calculated by differentiating a gradation conversion curve used in the gradation conversion process, and imaging is performed under the same conditions based on the calculated contrast amplification factor. Set the number of images to be superimposed. Then, a set number of images are synthesized according to the contrast amplification factors. By this synthesis processing, gradation conversion in which noise amplification is suppressed is performed.

Japanese Patent No. 3394019 JP 2008-172395 A JP 2008-60855 A

In the noise amplification suppression technique disclosed in Patent Document 3, a contrast amplification factor is calculated using information of one image obtained by one imaging, and a composite number is calculated based on the contrast amplification factor. ing.
Therefore, when the technique disclosed in Patent Document 3 is applied to the technique for expanding the dynamic range disclosed in Patent Document 1 or 2, a plurality of images obtained under different exposure conditions are synthesized. It is necessary to calculate the contrast amplification factor using the information of one dynamic range expansion image created in this way, and to calculate the number of dynamic range expansion images based on the contrast amplification factor. In other words, a plurality of images obtained under the different exposure conditions require the number of sets obtained based on the contrast amplification factor. However, in order to obtain a large number of images and create a composite image, a large amount of calculation resources such as imaging time, calculation processing, and storage media are required.

  Therefore, the present invention relates to an imaging apparatus and method for generating a dynamic range expanded image by combining a plurality of captured images with different exposure conditions, and to suppress noise amplification during gradation conversion with a small number of images. The purpose is to provide.

  In order to achieve the above object, an aspect of the imaging apparatus according to the present invention includes a different exposure image acquisition unit configured to perform different exposure imaging with different exposure conditions a plurality of times to acquire a plurality of different exposure captured images, and the different exposure captured image. The exposure condition storage means for storing the exposure conditions at the time of imaging, the dynamic range enlarged image generating means for generating a dynamic range enlarged image by combining the plurality of different exposure captured images, and the dynamic range enlarged image A gradation conversion curve calculating unit for calculating a gradation conversion curve; an amplification factor calculating unit for calculating a contrast amplification factor from the gradation conversion curve; and a specific luminance in the dynamic range expanded image based on the contrast amplification factor Luminance range selection means for selecting a range, and exposure conditions corresponding to the luminance range are read from the exposure condition storage means. Light condition reading means, same exposure imaging number calculation means for calculating the same exposure imaging number based on the luminance range, the contrast amplification factor and the exposure condition corresponding to the luminance range, and an exposure condition corresponding to the luminance range The same exposure image acquisition means for acquiring a plurality of the same exposure captured images by performing a plurality of times of the same exposure imaging, and combining the plurality of the same exposure captured images with the dynamic range enlarged image to obtain a composite image. Image generation means for generating and gradation conversion means for performing gradation conversion on the composite image based on the gradation conversion curve.

  In order to achieve the above object, according to one aspect of the imaging method of the present invention, a plurality of different exposure captured images are obtained by performing a plurality of different exposure imaging with different exposure conditions by an imaging unit, and the different exposure captured image imaging is performed. The exposure condition at the time is stored in a storage means, a dynamic range enlarged image is generated by combining the plurality of different exposure captured images, a gradation conversion curve is calculated for the dynamic range enlarged image, and the gradation conversion curve Calculating a contrast amplification factor from the image, selecting a specific luminance range in the dynamic range expanded image based on the contrast amplification factor, reading out the exposure condition corresponding to the luminance range from the storage means, the luminance range, Based on the exposure condition corresponding to the contrast amplification factor and the luminance range, the number of times of the same exposure imaging is calculated, and the imaging means A plurality of the same exposure captured images are obtained by performing a plurality of times of the same exposure imaging using the exposure condition corresponding to the luminance range, and the plurality of the same exposure captured images are combined with the dynamic range enlarged image to obtain a composite image. And tone conversion is performed on the composite image based on the tone conversion curve.

  According to the present invention, in an imaging apparatus and method for generating a dynamic range expanded image by combining a plurality of captured images with different exposure conditions, it corresponds to a specific luminance range with a high contrast amplification factor obtained from the dynamic range expanded image. Since multiple tone exposure images are acquired under exposure conditions, and tone conversion is performed after synthesizing these images with an expanded dynamic range image, an imaging device that suppresses noise amplification during tone conversion with a small number of images And methods can be provided.

1 is a block configuration diagram showing an embodiment of an imaging apparatus according to the present invention. The figure which shows an example of the relationship between the dynamic range of an imaging device, and the dynamic range of a scene. The figure which shows an example of the relationship between the dynamic range of an imaging device, and the dynamic range of a scene. The figure which shows an example of the gradation conversion curve in case an input and an output are each 8 bits. The figure which shows the example of the calculation method of the gradation conversion curve based on a histogram. It shows an example of a relationship between the contrast amplification factor R for the input image luminance value I _in. The figure which shows the relationship between the contrast amplification factor R and the threshold value Z, and the specific area A. The block diagram which shows the structural example of a luminance range selection part. The figure which shows the relationship between the product of contrast amplification factor R and noise amount Noi, the threshold value L, and the specific area A. The figure which shows the example of calculation of the same exposure imaging frequency PF in the specific area | region A specified by setting the threshold value Z according to contrast amplification factor R. The figure which shows the example of the exposure conditions and imaging frequency of the same exposure imaging in case the specific area | region A contains several exposure conditions.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating an embodiment of an imaging apparatus according to the present invention. The imaging apparatus shown in FIG. 1 includes an imaging unit 1, an imaging control unit 2, an imaging setting unit 3, an image acquisition unit 4, an image synthesis unit 5, a synthesized image storage unit 6, and a gradation conversion unit 7. The different exposure imaging setting unit 8 and the same exposure imaging setting unit 9 are provided. The different exposure imaging setting unit 8 includes a dynamic range estimation unit 10, a scene luminance acquisition unit 11, a dynamic range calculation unit 12, and a different exposure imaging number calculation unit 13. The same exposure imaging setting unit 9 includes a gradation conversion curve calculation unit 14, an amplification factor calculation unit 15, a luminance range selection unit 16, an exposure condition storage unit 17, an exposure condition reading unit 18, and the number of times of same exposure imaging. And a calculation unit 19.

  The imaging setting unit 3 sets an exposure condition and outputs it to the imaging control unit 2. The imaging control unit 2 controls the imaging unit 1 based on the conditions input from the imaging setting unit 3. The imaging unit 1 includes an imaging optical system 1A, an aperture 1B, a shutter 1C, an imaging element 1D, and an A / D converter 1E, and is controlled by the imaging control unit 2 to perform an imaging operation. Further, the exposure condition can be changed by the imaging setting unit 3 and the imaging control unit 2, and the imaging unit 1 acquires one or more images with different exposures (hereinafter referred to as "different exposure captured images"). it can. In the present embodiment, the imaging device 1D in the imaging unit 1 is an RGB primary color system, and the gradation width of the signal by the A / D converter 1E is B bits.

  The imaging unit 1 as an imaging unit outputs at least one image (captured image) obtained by imaging to the image acquisition unit 4. The image acquisition unit 4 includes an image storage unit (not shown), and stores the captured image input from the imaging unit 1 in the image storage unit. The image acquisition unit 4 can store a necessary number of video signals having a gradation width of B bits. As described above, the imaging unit 1, the imaging control unit 2, the imaging setting unit 3, and the image acquisition unit 4, for example, perform different exposure imaging with different exposure conditions a plurality of times to acquire a plurality of different exposure captured images. It functions as an image acquisition means.

ここで、bitは撮像画像におけるデジタル信号変換時のビット精度であり、ｊ及びｋはそれぞれ映像信号をＢＶ値に変換する際の補正係数である。 On the other hand, in the different exposure imaging setting unit 8, the dynamic range estimation unit 10 acquires the dynamic range of the imaging device 1D included in the imaging unit 1, and among the RGB signals of the video imaged by the imaging device 1D. The minimum signal value V _min and the maximum signal value V _max are acquired. The dynamic range estimation unit 10 acquires imaging information related to exposure conditions such as ISO sensitivity ISO, aperture value F, shutter speed T, and the like in the imaging unit 1. The dynamic range estimation unit 10 uses these to calculate the minimum value BV _min ^ISO and the maximum value BV _max ^ISO of the signal value BV defined in APEX units by the following equation (1).

Here, bit is the bit accuracy at the time of digital signal conversion in the captured image, and j and k are correction coefficients for converting the video signal into a BV value, respectively.

ここでＮ及びＫはそれぞれ定数である。 The dynamic range estimation unit 10 further calculates a minimum luminance value B _min ^ISO and a maximum luminance value B _max ^ISO by the following equation (2) using the BV _min ^ISO and BV _max ^ISO .

Here, N and K are constants.

Then, the dynamic range estimation unit 10 regards the calculated minimum luminance value B _min ^ISO and maximum luminance of the captured image, assuming that the luminance range ΔB corresponding to the dynamic range of the imaging device 1D is regarded as the dynamic range of the imaging device. Using the value B _max ^ISO , the dynamic range ΔB of the imaging apparatus is calculated by the following equation (3).

In addition, the dynamic range ΔB of the imaging device calculated by the equation (3) may be a decibel value DR_ΔB as in the following equation (4).

  In addition, the dynamic range estimation unit 10 includes a dynamic range storage unit (not shown) that stores the dynamic range of the imaging device, images a dynamic range measurement chart or the like in advance to obtain an estimated value of the dynamic range of the imaging device, and You may memorize | store in the dynamic range memory | storage part. As a result, the dynamic range estimation unit 10 can read and use the pre-stored estimated value of the dynamic range of the imaging apparatus without calculating the dynamic range ΔB of the imaging apparatus every time a dynamic range expanded image is generated. .

The dynamic range estimation unit 10 outputs the dynamic range ΔB of the imaging apparatus thus calculated or stored in advance to the different exposure imaging number calculation unit 13.
The scene luminance acquisition unit 11 measures the luminance value E of the scene using an AE sensor (not shown) or the like and outputs it to the dynamic range calculation unit 12. The dynamic range calculation unit 12 calculates the dynamic range ΔE of the scene by the following equation (5) using the minimum value E _min and the maximum value E _max of the luminance value E input from the scene luminance acquisition unit 11.

The dynamic range ΔE of the scene calculated by the above equation (5) may be a decibel value DR_ΔE as in the following equation (6).

The dynamic range calculation unit 12 outputs the calculated dynamic range ΔE of the scene to the different exposure imaging number calculation unit 13.
The different-exposure imaging number calculation unit 13 is based on the dynamic range ΔB of the imaging device input from the dynamic range estimation unit 10 and the dynamic range ΔE of the scene input from the dynamic range calculation unit 12. By shifting the dynamic range ΔB, the number of the dynamic range ΔB of the imaging apparatus necessary for acquiring image information in the range of the dynamic range ΔE of the scene, that is, the number of different exposure images required (different exposure imaging) The number of times DF) is calculated by the following equation (7).

  FIG. 2 is a diagram illustrating an example of the relationship between the dynamic range ΔE of the scene, the dynamic range ΔB of the imaging apparatus, and the number of times of different exposure imaging DF. FIG. 2 shows that the dynamic range ΔB of the imaging apparatus is shifted for each different exposure imaging so that the ranges do not overlap in order from the highest luminance value range, and image information in a luminance range wider than the scene dynamic range ΔE is obtained. Show.

Alternatively, in the calculation of the different exposure imaging number DF by the different exposure imaging number calculation unit 13 described above, it may be calculated so as to include an overlapping luminance range X as shown in FIG. FIG. 3 is a diagram illustrating an example of the relationship between the dynamic range ΔE of the scene, the dynamic range ΔB of the imaging device, and the number of times of different exposure imaging DF. It shows that the range ΔB is shifted for each different exposure imaging to obtain image information in a luminance range wider than the dynamic range ΔE of the scene. In this case, the different exposure imaging count DF is calculated by the following equation (8).

  The different exposure imaging number calculation unit 13 outputs the different exposure imaging number DF thus calculated to the imaging setting unit 3. The imaging setting unit 3 sets imaging conditions (ISO sensitivity, aperture value, shutter speed) based on the different exposure captured image count DF input from the different exposure imaging count calculation unit 13, and the imaging control unit 2 Outputs the set imaging conditions. The imaging control unit 2 controls the imaging unit 1 based on the imaging conditions input from the imaging setting unit 3. The imaging unit 1 performs DF different exposure imaging under the control of the imaging control unit 2. The imaging unit 1 outputs to the image acquisition unit 4 DF different exposure captured images obtained by DF different exposure imaging.

The image acquisition unit 4 stores the DF different-exposure captured images input from the imaging unit 1 in an image storage unit (not shown). The image acquisition unit 4 outputs the stored DF different exposure captured images to the image synthesis unit 5. The image synthesizing unit 5 aligns the DF different-exposure captured images by a matching process using a known motion vector with respect to the image signals of the DF different-exposure captured images output by the image acquisition unit 4. To do. Next, the image synthesizing unit 5 performs an image synthesizing process for synthesizing pixel values at the same coordinates of the DF different exposure captured images, and generates a dynamic range enlarged image. The combining method may be simple addition or weighted addition as in the following equation (9).

Here, I is an image signal after synthesis, I _i is the i-th image signal, and α _i is a weighting coefficient. By performing weighted addition, it is possible to create an image closer to the scene observed by the photographer than in the case of simple addition. Thus, the image composition unit 5 functions as, for example, a dynamic range expanded image generation unit that generates a dynamic range expanded image by combining a plurality of different exposure captured images. In addition, when performing weighted addition, the image compositing unit 5 is, for example, a weight calculating unit that calculates a weighting coefficient based on luminance values of a plurality of differently-exposure captured images, and a plurality of sheets based on the weighting coefficient. It functions as an adding means for performing weighted addition of the different exposure captured images. The image composition unit 5 outputs the generated dynamic range expanded image to the composite image storage unit 6. The composite image storage unit 6 stores the dynamic range expanded image input from the image composition unit 5.

The composite image storage unit 6 outputs the stored dynamic range expanded image to the gradation conversion curve calculation unit 14 in response to a request from the gradation conversion curve calculation unit 14. The gradation conversion curve calculation unit 14 calculates a gradation conversion curve based on the dynamic range expanded image input from the composite image storage unit 6. That is, the gradation conversion curve calculation unit 14 functions as, for example, a gradation conversion curve calculation unit that calculates a gradation conversion curve for a dynamic range expanded image. This gradation conversion curve is a curve showing the relationship between the input image luminance value I _in and the output image luminance value Q _in the gradation conversion processing. FIG. 4 is a diagram illustrating an example of a gradation conversion curve when the input and output are each 8 bits.

  The gradation conversion curve is prepared in advance, stored in a storage unit (not shown) in the gradation conversion curve calculation unit 14, and can be calculated by reading out by a reading unit (not shown) in the gradation conversion curve calculation unit 14. . If a gradation conversion curve is created and stored in advance, it can be read out at the time of shooting, so the processing load at the time of shooting is reduced. As described above, the storage unit (not shown) in the gradation conversion curve calculation unit 14 functions as, for example, a gradation conversion curve storage unit that stores a gradation conversion curve set in advance in the imaging apparatus, and calculates the gradation conversion curve. A reading unit (not shown) in the unit 14 functions as, for example, a gradation conversion curve reading unit that reads a gradation conversion curve from the gradation conversion curve storage unit.

  Alternatively, the gradation conversion curve may be calculated from the dynamic range expanded image signal I. When the gradation conversion curve is calculated from the dynamic range expanded image signal I as described above, the gradation conversion curve calculation unit 14 includes, for example, a histogram calculation unit, a clipping unit, and a cumulative value calculation unit, although not particularly illustrated.

  The histogram calculation unit calculates a histogram as shown in FIG. 5A from the dynamic range expanded image signal I. This histogram shows the luminance value of the pixel on the horizontal axis and the frequency expressed by the number or the ratio of the pixels having the luminance value on the vertical axis for the target image. As described above, the histogram calculation unit (not shown) in the gradation conversion curve calculation unit 14 functions as, for example, a histogram calculation unit that calculates a histogram from a dynamic range expanded image.

  The clipping unit performs a clipping process using a predetermined frequency as indicated by a broken line in FIG. 5A as a clip value, and creates a histogram as shown in FIG. Thus, a clipping unit (not shown) in the gradation conversion curve calculation unit 14 functions as a clipping unit that performs a clipping process on the histogram, for example.

  The accumulation calculation unit calculates a gradation conversion curve by accumulating the pixel frequency from the clipped histogram. As described above, the accumulation calculation unit (not shown) in the gradation conversion curve calculation unit 14 functions as, for example, an accumulation value calculation unit that calculates the accumulation value of the histogram subjected to the clipping process. Alternatively, the gradation conversion curve may be calculated by accumulating the pixel frequency directly from the histogram calculated by the histogram calculation unit without performing the clipping process. In FIG. 5C, a solid line indicates a gradation conversion curve calculated from a histogram before clip processing, and a broken line indicates a gradation conversion curve calculated from a histogram after clip processing. In the present embodiment, any one of the gradation conversion curves indicated by the solid line in FIG. 5C and the gradation conversion curve indicated by the broken line may be used. When calculating a gradation conversion curve using a histogram for each scene, unlike the case of using a gradation conversion curve that has been created and stored in advance, an appropriate gradation conversion curve is obtained for each scene. Appropriate gradation conversion can be performed without over-emphasizing.

Generally when the gradation conversion processing, for example, FIG. 4 or FIG. 5 (c) to the input image luminance value I _in the tone conversion curve shown is given as found in the lower part, a sharp slope of the tone conversion curve is large When tone conversion is performed, noise is amplified. Therefore, in the present embodiment, this noise amplification is suppressed by combining a plurality of images captured under the same conditions.

Here, the principle of the noise amplification suppression method used in this embodiment will be briefly described. When the signal value of the image is S and the noise is N, the signal value S ′ and noise N ′ when the two images are combined are expressed by the following equation (10).

Thus, while the signal value is doubled, the noise is not doubled but is suppressed to double the root. Therefore, the S / N ratio of the combined signal when two images are combined is calculated as the following equation (11).

  The first term on the right side of the equation (11) is the same as the S / N ratio of the original one image signal, and the S / N ratio of the original one image signal is two by the amount of the second term on the right side of the formula. The S / N ratio of the image signal obtained by synthesizing these images is higher. Similarly, an S / N ratio is further improved in an image obtained by combining more images.

In the present embodiment, when noise is amplified by gradation conversion processing for an image with an expanded dynamic range, noise is synthesized by synthesizing a plurality of images (hereinafter referred to as “same exposure captured images”) captured under the same exposure conditions. Reduce.
Therefore, the gradation conversion curve calculation unit 14 outputs the calculated gradation conversion curve to the amplification factor calculation unit 15. The amplification factor calculation unit 15 differentiates the gradation conversion curve input from the gradation conversion curve calculation unit 14 to calculate the contrast amplification factor R. As described above, the amplification factor calculation unit 15 functions as an amplification factor calculation unit that calculates the contrast amplification factor from the gradation conversion curve, for example. Here, the contrast amplification factor R is a differential value of the gradation conversion curve, and represents the amplification factor of the output image luminance value with respect to the input image luminance value at the time of gradation conversion. Figure 6 is a diagram showing an example of the relationship between the contrast amplification factor R for the input image luminance value I _in.

The amplification factor calculation unit 15 outputs the calculated contrast amplification factor R to the luminance range selection unit 16. The luminance range selection unit 16 sets a threshold value Z related to the contrast amplification factor R input from the amplification factor calculation unit 15, and the input contrast amplification factor R and the set threshold value Z are expressed by the following equation (12). When the condition is satisfied, it is determined that the noise is amplified.

  That is, the luminance range selection unit 16 selects, as the specific region A, a luminance range that satisfies the condition in Expression (12). FIG. 7 is a diagram showing the relationship between the contrast amplification factor R and the threshold value Z and the specific area A. Thus, the brightness range selection unit 16 functions as a brightness range selection unit that selects a specific brightness range in the dynamic range expanded image based on, for example, the contrast amplification factor. That is, the luminance range selection unit 16 is, for example, a threshold setting unit that sets a threshold for the contrast amplification factor, and a selection unit that compares the contrast amplification factor and the threshold to select a specific luminance range in the dynamic range expanded image. Serves as a function.

  At this time, the brightness range selection unit 16 receives the aperture value, shutter speed, ISO sensitivity, and the like at the time of capturing the brightness range corresponding to the specific area A from the exposure condition storage unit 17 serving as a storage unit via the exposure condition reading unit 18. The setting value related to the exposure condition is read out. That is, the exposure condition storage unit 17 outputs the stored exposure condition to the exposure condition reading unit 18 in response to a request from the exposure condition reading unit 18, and the exposure condition reading unit 18 The exposure condition input from 17 is output to the luminance range selector 16. Thus, for example, the exposure condition storage unit 17 functions as an exposure condition storage unit that stores the exposure conditions at the time of capturing each of the different exposure captured images, and the exposure condition reading unit 18 corresponds to, for example, the luminance range. It functions as an exposure condition reading means for reading the exposure conditions to be read from the exposure condition storage means.

Further, the luminance range selection unit 16 reads the dynamic range expansion image stored in the composite image storage unit 6, calculates the noise amount in the dynamic range expansion image as follows, and uses this to select the specific area A You may do it.
In this case, the luminance range selection unit 16 includes a buffer unit 20, a local region extraction unit 21, a noise amount calculation unit 22, a calculation unit 23, and a luminance range specification unit 24, as shown in FIG.

ここで、α、β、γはそれぞれ定数である。 The buffer unit 20 stores the dynamic range expanded image signal I input from the composite image storage unit 6. The buffer unit 20 outputs the stored dynamic range expanded image signal I to the local region extraction unit 21 in response to a request from the local region extraction unit 21. The local region extraction unit 21 extracts, for example, a 5 × 5 local region centered on the target pixel from the dynamic range expanded image signal I input from the buffer unit 20. The extraction of the local region may be performed on the entire dynamic range enlarged image I or may be performed only on a designated region. The local region extraction unit 21 outputs information on the extracted local region to the noise amount calculation unit 22. The noise amount calculation unit 22 calculates the noise amount using the local region information input from the local region extraction unit 21. The noise amount N _ν at a certain target pixel ν is calculated by the following equation (13) as a function with respect to the average value S _ν of the signal in the local region, for example.

Here, α, β, and γ are constants.

In addition to the function of the equation (13), the noise amount N _ν may be calculated by using a look-up table (LUT) for the noise amount with respect to the input signal, and the standard deviation with respect to the signal in the local region is calculated as the noise amount N. _It may be _v .

The noise amount calculation unit 22 performs the above processing on the entire dynamic range enlarged image I or each pixel in the designated area as a pixel of interest, and outputs the calculated noise amount for each pixel to the calculation unit 23. Using the noise amount for each pixel input from the noise amount calculation unit 22, the calculation unit 23 calculates the noise amount Noi of the entire dynamic range expanded image signal by the following equation (14).

Here, M represents the total number of pixels. In calculating the noise amount Noi of the entire dynamic range expanded image signal, in addition to the simple addition of the equation (14), weighted addition by region may be used. Thus, the local region extraction unit 21, the noise amount calculation unit 22, and the calculation unit 23 function as a noise amount calculation unit that calculates a noise amount from a dynamic range enlarged image, for example. That is, the local area extraction unit 21 functions as, for example, a local area extraction unit that extracts a local area centered on the target pixel of the dynamic range expanded image, and the noise amount calculation unit 22 applies, for example, an image in the local area. For example, the calculation unit 23 functions as a calculation unit that calculates the noise amount of the entire dynamic range expanded image using the noise amount in the local region. .

The calculation unit 23 outputs the calculated noise amount Noi to the luminance range specifying unit 24. When the luminance range specifying unit 24 satisfies the condition of the following equation (15) using the contrast amplification factor R input from the amplification factor calculation unit 15, the noise amount Noi and the threshold value L input from the calculation unit 23 Judge that the noise is amplified.

  That is, the luminance range specifying unit 24 selects a luminance range that satisfies the condition in the expression (15) as the specific region A. FIG. 9 is a diagram illustrating the relationship between the specific area A and the product of the contrast amplification factor R and the noise amount Noi and the threshold L set by the threshold setting unit in the luminance range selection unit 16 (not shown). In this way, the luminance range specifying unit 24, for example, as a threshold setting unit for setting a threshold, and expands the dynamic range by comparing the value calculated using the contrast amplification factor and the noise amount with the threshold. It functions as a selection means for selecting a specific luminance range in the image.

  The luminance range selection unit 16 outputs the luminance range information of the selected specific area A to the same exposure imaging number calculation unit 19. The same exposure imaging number calculation unit 19 calculates the number of times of same exposure imaging performed under the exposure condition of the luminance range based on the luminance range information of the specific area A input from the luminance range selection unit 16. As described above, the same exposure imaging number calculation unit 19 functions as a same exposure imaging number calculation unit that calculates the same exposure imaging number based on the luminance range, the contrast amplification factor, and the exposure condition corresponding to the luminance range. .

FIG. 10 is a diagram illustrating an example of a method for calculating the same-exposure-captured image count PF performed under the exposure condition in the luminance range of the specific area A specified by the contrast amplification factor R with the threshold value Z set. The same exposure imaging number calculation unit 19 acquires the contrast amplification factor R from the amplification factor calculation unit 15, and a representative luminance value calculation unit (not shown) in the same exposure imaging number calculation unit 19 is in the luminance range of the specific area A. _A representative luminance value I _A that is an average value of the input image luminance values is calculated, and a representative luminance value amplification factor calculation unit (not shown) in the same exposure imaging number calculation unit 19 performs a contrast amplification factor R corresponding to the representative luminance value I _{A. Using A} , the number PF of the same exposure imaging is calculated by the following equation (16).

Here, C ₁ and C ₂ are constants and are integers set by the configuration of the imaging apparatus. In the present embodiment, C ₁ = C ₂ = 1. The same exposure imaging number calculation unit 19 outputs the calculated same exposure imaging number PF to the imaging setting unit 3. In this way, a representative luminance value calculation unit (not shown) in the same exposure imaging number calculation unit 19 functions as, for example, a representative luminance value calculation unit that calculates a representative luminance value from the specific luminance range, and similarly calculates the same exposure imaging number. A representative luminance value amplification factor calculation unit (not shown) in the unit 19 functions as, for example, representative luminance value amplification factor calculation means for calculating an amplification factor for the representative luminance value.

  The imaging setting unit 3 reads the exposure condition of the specific area A where the same exposure imaging is performed from the exposure condition storage unit 17 via the exposure condition reading unit 18. That is, in response to a request from the exposure condition reading unit 18, the exposure condition storage unit 17 outputs the exposure condition of the specific area A in which the same exposure imaging is stored to the exposure condition reading unit 18 and reads the exposure condition. The unit 18 outputs to the imaging setting unit 3 the exposure condition of the specific area A for performing the same exposure imaging output from the exposure condition storage unit 17. The imaging setting unit 3 sets the imaging condition by combining the exposure condition of the specific area A input from the exposure condition reading unit 18 and the same exposure imaging number PF input from the same exposure imaging number calculation unit 19. To do. The imaging setting unit 3 outputs the set imaging conditions to the imaging control unit 2. The imaging control unit 2 controls the imaging unit 1 based on the imaging conditions input from the imaging setting unit 3. The imaging unit 1 performs PF times of same-exposure imaging under the control of the imaging control unit 2. Through the above processing, PF times of same-exposure imaging are performed using exposure conditions in different exposure imaging corresponding to a specific luminance range in the dynamic range expanded image selected based on the contrast amplification factor.

  The imaging unit 1 outputs PF sheets of the same exposure captured image obtained by PF times of the same exposure imaging to the image acquisition unit 4. The image acquisition unit 4 stores PF sheets of the same exposure captured image input from the imaging unit 1 in an image storage unit (not shown). As described above, the imaging unit 1, the imaging control unit 2, the imaging setting unit 3, and the image acquisition unit 4 perform, for example, a plurality of times of the same exposure imaging using the exposure condition corresponding to the luminance range, and a plurality of the same number of images. It functions as the exposure image acquisition means for acquiring the exposure captured image.

At this time, as in the example shown in FIG. 11, if the specific area A is equal to or larger than the dynamic range ΔB of the imaging apparatus and includes a plurality of exposure conditions A ₁ and A ₂ , an exposure imaging count PF, for example, the representative luminance value I _A1 of the input image luminance values for the exposure condition a _1, the exposure condition a ₂ the representative luminance value I _A2 of the input image luminance values respectively calculated, the representative luminance value I _A1 And A _A2 and contrast amplification factors R _A1 and R _A2 , respectively, the number PF of the same exposure imaging is appropriately assigned to each of the number H of the plurality of exposure conditions as shown in the following equation (17). ₁ PF ₁ times (3 times in the example of FIG. 11), may be performed imaging (twice in the example of FIG. 11) PF ₂ times the exposure condition a _2. This allocation is set according to the configuration of the imaging apparatus.

  The image acquisition unit 4 outputs the stored PF same-exposure captured images to the image composition unit 5. The composite image storage unit 6 outputs the stored dynamic range expanded image to the image composition unit 5. The image composition unit 5 generates a composite image of the image signal of the PF same-exposure captured images input from the image acquisition unit 4 and the dynamic range expanded image input from the composite image storage unit 6. In this way, the image composition unit 5 functions as, for example, an image composition unit that composes a plurality of the same exposure captured images with a dynamic range enlarged image to generate a composite image.

  The image composition unit 5 outputs to the gradation conversion unit 7 a composite image of the generated PF sheets of the same exposure captured image and the dynamic range expanded image. Further, the gradation conversion curve calculation unit 14 outputs an appropriate gradation conversion curve to the gradation conversion unit 7 for the calculated dynamic range expanded image. The gradation conversion unit 7 performs gradation conversion processing on the composite image input from the image composition unit 5 based on the gradation conversion curve input from the gradation conversion curve calculation unit 14. The gradation converting unit 7 outputs the gradation-converted composite image as the final dynamic range expanded image. As described above, the gradation conversion unit 7 functions as, for example, a gradation conversion unit that performs gradation conversion on a composite image based on a gradation conversion curve.

  The output image from the gradation converting unit 7 whose dynamic range is expanded and gradation-converted is recorded on a recording medium (not shown) or output by an external output device such as a display device or a printing device via an interface (not shown). Or can be stored in a storage device or the like on a network via an interface (not shown).

  According to the imaging apparatus according to the present embodiment as described above, in the dynamic range expansion image generated by synthesizing a plurality of images having different exposure amounts, the same exposure of PF sheets for the specific area A having a high contrast amplification factor R. By acquiring and synthesizing the captured images, the effect of noise amplified by the gradation conversion process can be reduced with a small number of captured images, and it has a wide dynamic range similar to the scene observed by the photographer and An image with less noise can be provided.

  Further, as described with reference to FIG. 7, by determining the specific region A in which noise is amplified based on the contrast amplification factor R, it is possible to select a luminance range in which noise amplification is a concern with high accuracy. it can. Furthermore, as described with reference to FIGS. 8 and 9, by calculating the amount of noise and using this to determine the specific area A, it is possible to select a luminance range in which noise amplification is a concern with higher accuracy. . At this time, the noise amount of each pixel can be accurately calculated by calculating the noise amount for each local region.

  In addition, by performing the same exposure imaging only for the exposure conditions in which the luminance included in the specific area A is captured, the number of times and the number of imaging were obtained compared to the case where the same exposure imaging was performed for each of the different exposure conditions during the different exposure imaging. Image processing can be reduced. In order to execute the same exposure imaging, the exposure condition is stored in the exposure condition storage unit 17, and the condition is read out via the exposure condition reading unit 18.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiment, the problem described in the column of problems to be solved by the invention can be solved and the effect of the invention can be obtained. The configuration in which this component is deleted can also be extracted as an invention.

  DESCRIPTION OF SYMBOLS 1 ... Imaging part, 2 ... Imaging control part, 3 ... Imaging setting part, 4 ... Image acquisition part, 5 ... Image composition part, 6 ... Composite image memory | storage part, 7 ... Gradation conversion part, 8 ... Different exposure imaging setting part , 9 ... Same exposure imaging setting unit, 10 ... Dynamic range estimation unit, 11 ... Scene luminance acquisition unit, 12 ... Dynamic range calculation unit, 13 ... Different exposure imaging number calculation unit, 14 ... Tone conversion curve calculation unit, 15 ... Amplification factor calculation unit, 16 ... luminance range selection unit, 17 ... exposure condition storage unit, 18 ... exposure condition reading unit, 19 ... same exposure imaging number calculation unit, 20 ... buffer unit, 21 ... local region extraction unit, 22 ... noise Amount calculation unit, 23... Calculation unit, 24... Luminance range specifying unit.

Claims (10)

Different exposure image acquisition means for performing different exposure imaging with different exposure conditions a plurality of times to acquire a plurality of different exposure captured images;
Exposure condition storage means for storing an exposure condition at the time of capturing each of the different exposure captured images;
A dynamic range expansion image generation means for generating a dynamic range expansion image by combining the plurality of different exposure captured images;
A gradation conversion curve calculating means for calculating a gradation conversion curve for the dynamic range expanded image;
An amplification factor calculating means for calculating a contrast amplification factor from the gradation conversion curve;
Luminance range selection means for selecting a specific luminance range in the dynamic range expanded image based on the contrast amplification factor;
Exposure condition reading means for reading exposure conditions corresponding to the brightness range from the exposure condition storage means;
Based on the brightness range, the contrast amplification factor, and an exposure condition corresponding to the brightness range, the same exposure imaging number calculation means for calculating the same exposure imaging number;
Same exposure image acquisition means for acquiring a plurality of same exposure captured images by performing a plurality of times of same exposure imaging using an exposure condition corresponding to the brightness range;
Image combining means for generating a combined image by combining the plurality of the same exposure captured images with the dynamic range expanded image;
Gradation conversion means for performing gradation conversion on the composite image based on the gradation conversion curve;
An imaging apparatus comprising: The luminance range selection means includes
Threshold setting means for setting a threshold for the contrast amplification factor;
A selection means for comparing the contrast amplification factor and the threshold to select a specific luminance range in the dynamic range expanded image;
The imaging apparatus according to claim 1, further comprising: The luminance range selection means includes
Threshold setting means for setting a threshold;
A noise amount calculating means for calculating a noise amount from the dynamic range enlarged image;
A selection means for selecting a specific luminance range in the dynamic range expanded image by comparing the threshold value with a value calculated using the contrast amplification factor and the noise amount;
The imaging apparatus according to claim 1, further comprising: The noise amount calculating means includes
A local region extracting means for extracting a local region centered on the target pixel of the dynamic range expanded image;
A local area noise amount calculating means for calculating a noise amount for an image in the local area;
A calculation means for calculating a noise amount of the entire dynamic range expanded image using a noise amount in the local region;
The imaging apparatus according to claim 3, further comprising: The gradation conversion curve calculation means includes:
Gradation conversion curve storage means for storing a gradation conversion curve set in advance in the imaging apparatus;
Gradation conversion curve reading means for reading out the gradation conversion curve from the gradation conversion curve storage means;
The imaging apparatus according to claim 1, further comprising: The gradation conversion curve calculation means includes:
A histogram calculating means for calculating a histogram from the dynamic range expanded image;
Clipping means for performing clip processing on the histogram;
A cumulative value calculating means for calculating a cumulative value of the histogram subjected to the clip processing;
The imaging apparatus according to claim 1, further comprising:   The imaging apparatus according to claim 1, wherein the exposure condition includes an aperture value, a shutter speed, and ISO sensitivity. The dynamic range expanded image generation means includes
A weighting calculating means for calculating a weighting coefficient based on a luminance value of the plurality of different-exposure captured images;
Adding means for performing weighted addition of the plurality of different exposure captured images based on the weighting coefficient;
The imaging apparatus according to claim 1, further comprising: The same exposure image acquisition means,
Representative luminance value calculating means for calculating a representative luminance value from the specific luminance range;
Representative luminance value amplification factor calculating means for calculating an amplification factor for the representative luminance value;
With
The imaging apparatus according to claim 1, wherein the same-exposure imaging number calculation unit calculates the same-exposure imaging number using the representative luminance value and an amplification factor for the representative luminance value. A plurality of different-exposure captured images are obtained by performing a plurality of different-exposure imaging with different exposure conditions by the imaging means,
Storing the exposure condition at the time of capturing the different exposure captured image in a storage means;
A dynamic range expansion image is generated by combining the plurality of different exposure captured images,
Calculating a gradation conversion curve for the dynamic range expanded image;
Calculate the contrast amplification factor from the gradation conversion curve,
Based on the contrast amplification factor, select a specific luminance range in the dynamic range expansion image,
Read the exposure conditions corresponding to the luminance range from the storage means,
Based on the exposure range corresponding to the brightness range, the contrast amplification factor and the brightness range, the number of exposure exposure imaging,
A plurality of times of the same exposure imaging is performed by using the exposure means corresponding to the luminance range by the imaging means to obtain a plurality of the same exposure captured images,
Combining the plurality of same-exposure captured images with the dynamic range expanded image to generate a combined image;
An imaging method, wherein gradation conversion is performed on the composite image based on the gradation conversion curve.

Images