JPH07131718A - Picture synthesizer - Google Patents

Abstract

PURPOSE:To easily obtain an excellent picture by eliminating a pseudo contour after synthesis in the picture synthesizer synthesizing a picture with a wide dynamic range from plural picture data with a different exposure in one and same scene. CONSTITUTION:An object image by an image pickup means (a) is projected onto an image pickup element 3 through a lens 1, an optical low pass filter 2 and a color filter 3. In this case, a control circuit 4 controls the device so that plural picture signals of different exposures in one and same scene are obtained in one-image pickup operation and a synthesis means (b) extracts a common area in which luminance levels of picture data are within a predetermined range based on the standard picture data with standard exposure and non-standard picture data of a different exposure. After luminance levels of the common area are made coincident, one wide dynamic range picture is obtained through synthesis by replacing a luminance level of the non-standard picture corresponding to the area in which a luminance level of the standard picture is at the outside of a predetermined range with a luminance level of the area within the predetermined range.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus having a wide dynamic range.

[0002]

2. Description of the Related Art In recent years, solid-state image pickup devices such as CCD image pickup devices have been widely used as an image capturing means for television cameras, electronic still cameras and the like. However, the dynamic range of this solid-state image sensor is narrower than that of silver salt and the like, and the image quality is significantly deteriorated depending on the shooting conditions.

Therefore, as a method for expanding the dynamic range of a solid-state image pickup device, a plurality of images having different exposure amounts in the same scene are photographed, and the plurality of image data are combined by some operation to expand the dynamic range. Has been proposed.

Conventionally, as a means for synthesizing a plurality of image data having different exposure amounts in the same scene, Japanese Patent Laid-Open No. Sho 60-1985
As described in Japanese Patent Laid-Open No. 52171, there is a method of adding image signals. FIG. 8 is a conceptual diagram showing a dynamic lens enlarging method by this addition. In (a) and (b) of the same figure, the horizontal axis represents the illuminance of the subject, the vertical axis represents the output of the image sensor, the solid line (a) represents the input / output characteristics when the exposure amount of the image sensor is increased, and the dotted line (b). Is the input / output characteristic when the exposure amount is reduced. Here, by adding the respective output values of FIG. 8A, it is possible to obtain an output with an expanded dynamic range as shown in FIG. 8B.

As another means of synthesis, as described in JP-A-63-306777, etc.,
There is a method of cutting and combining images as shown in FIG. For example, when a subject having a wide dynamic range as shown in FIG. 9C is photographed with proper exposure, a person will be blackened as shown in FIG. 9A. Therefore, an image with an expanded dynamic range can be obtained by cutting out a person's portion from (b) of FIG. 9 photographed with a large amount of exposure and combining them.

[0006]

However, among the above-mentioned conventional examples, the synthesizing method by adding image signals is as shown in FIG.
As shown in FIG. 10B, each image sensor does not continuously share the light amount region as in (a) and is not necessarily linear with respect to the output light amount of the image sensor. There is a problem in that the brightness of a region in the middle of brightness is increased, resulting in an image with a bright brightness as a whole and lacking in contrast.

In another conventional example, which is a method based on image cut-out synthesis, standard image data taken by proper exposure is shown in FIG. 9 (a), and a non-sharpened area of the image is taken by another exposure amount. The standard image data is synthesized from the image shown in FIG. 9B. However, the luminance level of the image to be synthesized, that is, the output of the image sensor with respect to the change of the light amount of the subject is different from the luminance level of the synthesized image, and thus the synthesized image is synthesized. There is a problem that the area becomes a pseudo contour. In addition, if the brightness levels of the respective images are adjusted so as not to form a pseudo contour, there is a problem in that the effect of expanding the dynamic range is not very obtained.

The present invention has been made in view of the above problems, and it is an object of the present invention to provide an image synthesizing apparatus which can easily obtain a good image having a wide dynamic range without a pseudo contour after synthesizing. Has an aim.

[0009]

The image synthesizing apparatus of the present invention is configured as follows.

(1) In a case where one image is composed from a plurality of image data of different exposure amounts in the same scene, both image data are luminance from standard image data of standard exposure and non-standard image data of different exposure. After extracting a common area whose level is within a predetermined range and matching the brightness levels of the common areas, the area where the brightness level of the standard image is outside the predetermined range is defined as the brightness level of the corresponding non-standard image within the predetermined range. It is provided with a synthesizing means for synthesizing one wide dynamic range image by substituting the area in the.

(2) The apparatus of (1) above is provided with an image pickup means for obtaining image data.

(3) In the apparatus of (2) above, the image pickup means comprises a plurality of image pickup elements, a spectroscopic means for dividing an image into each of the image pickup elements in the optical path of image pickup light, and an exposure of each of the image pickup elements. And an exposure control means for changing the amount.

[0013]

In the image synthesizing apparatus according to the present invention, an image synthesizing apparatus for synthesizing image signals having different exposure amounts in the same scene obtained from a single or a plurality of image pickup elements to expand the dynamic range is used. Match the brightness level of the standard image data obtained with the brightness level of the non-standard image data obtained with another exposure amount, and then determine the blackout or whiteout areas of the standard image to determine the blackout or whiteout. It is designed to be replaced with non-standard image data that does not have any, and when combining using an addition method as in the past, the brightness level of the intermediate brightness region rises, and there is no loss of overall contrast. Also, it is possible to prevent the synthesized image from becoming a pseudo contour when using the synthesis method, and to expand the dynamic range. Natural synthesized image can be obtained.

[0014]

1 is a block diagram showing the configuration of a first embodiment of the present invention. The image synthesizing device of FIG. 1 is composed of a block of an image pickup means a and a block of an image synthesizing means b.
It is configured to combine one wide dynamic range image from a plurality of image data of different exposure amounts in the same scene.

In FIG. 1, 1 is a lens on which imaging light from a subject is incident, 2 is an optical low-pass filter, and 3 is CC.
An image sensor such as D, 4 is an image sensor control circuit for controlling the operation of the image sensor 3, 5 is an A / D converter for converting an image signal from the image sensor 3 into a digital signal, and 6 is a computer for image synthesis Thus, it constitutes a dynamic range expanding means for expanding the dynamic range. Reference numeral 7 is a memory (storing means) for storing synthesis information when synthesizing the wide dynamic range image, 8 is a data processing circuit, and 20 is a color filter.

The image synthesizing means b extracts a common area in which both image data brightness levels are within a predetermined range from standard image data of standard exposure and non-standard image data of different exposure, and the brightness of the common area is extracted. After matching the levels, the wide dynamic range image is synthesized by replacing the area where the brightness level of the standard image is outside the predetermined range and the area where the brightness level of the non-standard image is within the predetermined range. Is configured to.

Next, the operation will be described. A subject image (not shown) by the image pickup means a is projected onto the image pickup element 3 through the lens 1, the optical low pass filter 2 and the color filter 20. Here, the image sensor control circuit 4 is configured to sequentially obtain a plurality of image signals having different exposure amounts (aperture value, shutter speed) in the same scene by one shooting operation, and further store these plurality of image signals in a memory. Store in 7. After that, the computer 7 reads out from the memory 7 and synthesizes one wide dynamic range image at the same time, and at the same time, the memory 7 stores the synthesis information such as the pixel value necessary for the calculation at the time of synthesis. Further, when the user requests compression of the dynamic range width of the combined image data according to the characteristics of the output device, the data processing circuit 8 performs a compression operation based on the combined information stored at the time of combining. Data is output, and when there is no request from the user, the composite information is added to the composite information and then output, or the data is output as it is.

Next, the synthesis algorithm of the computer 6 of FIG. 1 will be described with reference to the conceptual diagram of FIG. In FIG. 2, reference numeral 9 denotes an image signal obtained from the image sensor 3 with an appropriate (standard) exposure amount.
Standard image data (called normal image data) in which the / D converter 5 has been converted into 8-bit data by a bar graph.
Similarly, 10, 10 'are non-standard image data (called bright image data) obtained by increasing the exposure, 11,
Reference numeral 11 'is non-standard image data (called dark image data) obtained by reducing the exposure, 12 is composite image data whose dynamic range is expanded by combining, and 13 and 1
3 ', 14 and 14' are points representing the values of arbitrary pixels.

The synthesis algorithm will be described below with reference to FIG. Here, it is assumed that each image data has been converted into 8-bit data by the A / D converter 5 in FIG. 1, and thus it is assumed that the pixel value has a value of 0 to 255. First, consider synthesis of the normal image 9 and the bright image 10. When synthesizing images of different brightness levels, it is necessary to align the brightness levels of the image data of each other. For this reason, if the reference point 13 is a pixel that is not crushed in both the normal image 9 and the bright image 10, the bright image 10 has a larger exposure amount than the normal image 9, and the value of this point is larger than that of the normal image 9. large. Therefore, 13, 1 in FIG.
There is a positional relationship such as 3 '. Therefore, in order to make the brightness levels of both image data the same, these reference points 13, 13 '
Make the values of the same. Then, the bright image 10 is slid as a whole and comes to the position of the bright image 10 'on the coordinates.

Similarly, the normal image 9 and the dark image 11
Also in the composition of, in order to match the brightness levels of both image data, if a pixel that cannot be crushed in both images is set as the reference point 14, the dark image 11 has a smaller exposure amount than the normal image 9, so the value of this point is It is smaller than the normal image. Therefore, there is a relationship such as 14 and 14 'in the figure, and in order to make the brightness levels of both image data the same, this reference point 1
If the values of 4 and 14 'are made the same, the dark image 11 is slid as a whole, and the dark image 11' is at the position on the coordinates.

By the above operation, the brightness levels of the respective images become the same. After that, 0-B of normal image 9
Up to (reference point 13) are replaced with A to B (reference point 13) of the bright image 10 ', and C (reference point 1) of the normal image 9 is replaced.
4) to 255, C (reference point 1) of the dark image 11 '
4) Replace with D. At the same time, the pixel values (A, 0, B, C, 255, D) used as the reference at the time of composition are stored in the memory 7 as a guide for compressing the image data according to the output device. By combining the three pieces of image data by such an operation, the image data 12 having an expanded dynamic range can be obtained.

Next, the calculation method of the operation of FIG. 2 on the computer 6 will be described with reference to FIG. In FIG. 3, reference numeral 15 represents a pixel value in an arbitrary area of the normal image data, and 16
Represents the pixel value of the same area as the area 15 of the bright image data, and 17 represents the combined value of the combined areas 15 and 16.

First, in order to make the brightness levels of the normal image and the bright image to be the same, a pixel which is not crushed in both image data, the reference point 13 in FIG. 2, is set. here,
If the normal image is 100 as the value of the reference point 13,
The reference point 13 'of the bright image is 200. Then, in order to match the brightness levels, 100, which is 200-100, is subtracted from all the pixel values of the bright image, and the result is slid. After that, the blackened area of the normal image (in the example of FIG. 3, the area where the pixel value of the normal image 15 is 0) is replenished with the value of the bright image to configure. Although the synthesis of the normal image and the bright image has been described above, the same applies to the synthesis of the normal image and the dark image.

The image data that has been subjected to the dynamic range expansion processing as described above is output by the output device. However, the dynamic range width of the output device may be narrower than that of the composite image data. , The output must be performed by compressing the dynamic range width of the composite image data. FIG. 4 is a diagram showing an example of compression of image data having a wide dynamic range, which represents the image data in a perceptual color space. In FIG. 4, reference numeral 30 indicates composite image data whose dynamic range has been expanded by the main image composition processing, and reference numeral 31 indicates image data 30 compressed in an arbitrary range. Definition of each coordinate is shown in FIG. Indicates. In addition, FIG. 4A shows the synthesized image data 30 linearly compressed, FIG. 4B shows the bright portion emphasized and compressed, and FIG. 4C shows the dark portion emphasized and compressed. d)
Indicates that the bright part is cut and compressed, and it is desirable that the dynamic range width can be freely changed and output in this way. Therefore, the operation of the data processing circuit 8 for compressing the image data having the expanded dynamic range according to the characteristics of the output device will be described below with reference to FIG.

In FIG. 5, reference numeral 24 designates image data whose dynamic range has been expanded using the present synthesis algorithm, and 2
5 and 29 are image data obtained by compressing the image data 24 into 8-bit data, and 26 to 28 are asymptotes. Here, as an example, the dynamic range of the composite image data is 8 bits.
A description will be given of a case where the output device has a value equal to or larger than the width and the dynamic range width of the output device is 8 bits.

The image data 24 of FIG. 5A is the same as the composite image data 12 of FIG. 2 and has a pixel value of 8 bi.
It has a width of t or more. In the image data 24, the bright image is replenished between A and B, and the dark image is replenished between C and D. Here, it is assumed that the output device has a dynamic range width of 8 bits, and in order to perform compression in accordance with this, the original standard image has a value between 0 and 255 on the graph of FIG. , A to B image data between 0 and B, and C to D image data between C and 25
If the compression is carried out for each of the five, it is possible to obtain an image having a brightness level substantially the same as that of the image data of proper exposure and having a dynamic range expanded more than that of the normal image 9 of FIG. Therefore, the value of the composition information (A, B, C, D, 0, 255) stored in the memory 7 at the time of composition is used. Hereinafter, an example of a calculation formula when the compression operation described in FIG. 5A is linearly performed on a computer will be shown.

(1) (A to B) → (0 to B) NewPix = (B-0) ÷ (B−A) × (OldPix−A) +0 (2) (B to C) → (B to C) ) As it is (3) (C to D) → (C to 255) NewPix = (255-C) ÷ (D−C) × (OldPix−C) + C As above, the compression was performed using the linear format. As shown in FIG. 5B, a curve 29 having straight lines 26 to 28 as asymptotes
By creating the image and performing the compression in a non-linear manner, the output value with respect to the light intensity becomes smooth, so that a more natural image can be obtained. Further, when it is difficult to create a curved line, compression may be performed by combining short straight lines to approximate the curved line 29. Further, it is desirable that these compression operations can be performed by a device other than the main imaging device. In such a case, as shown in FIG.
By adding the compression information (composite information) of the memory 7,
The compression operation as described above can be performed by a computer or the like.

In this way, by combining the brightness levels of the respective image data and then combining them, a natural composite image with a wide dynamic range in which the brightness balance is obtained without pseudo contours, and complicated calculation is required. I will not
The composition time is short and can be applied to movies.

When the dynamic range is expanded, the values of A, 0, B (reference point), C (reference point), 255, D and the like, which serve as a guideline for the expansion, are stored, and the coordinates thereof are stored. As a guideline, the compression can be easily performed according to the dynamic range width of the output device, and the dynamic range width can be freely changed and output from one wide dynamic range composite image according to the intention of the user.

The number of image data to be combined is not limited to three, and any number of two or more may be used.

FIG. 7 is a block diagram showing the configuration of the second embodiment of the present invention, and the same symbols as those in FIG. 1 indicate the same components. In FIG. 7, 18 is a first image sensor, 3 is a second image sensor, 19 is a third image sensor, 20 is a color filter, 21 is a color difference signal generation circuit for dividing a color signal into a luminance signal and a color difference signal, Reference numeral 23 denotes a prism (spectral means) that divides the image from the lens 1 into a plurality of image pickup devices 3, 18, and 19.

The operation will be described with reference to FIG. 7. A subject image (not shown) passes through the lens 1 and the prism 23, and further passes through the optical low pass filter 2 and the color filter 20 to be projected on the image pickup devices 18, 3, and 19. Here, the first image pickup device 1 is operated by the image pickup device control circuit 4 in one shooting operation.
8 is a color signal with an appropriate exposure amount, the second image pickup device 3 is a luminance signal with increased exposure (hereinafter referred to as a bright luminance signal), and the third image pickup device 19 is a luminance signal with reduced exposure (hereinafter, dark signal). (Called the luminance signal). Therefore, in this embodiment, the storage time of each image sensor is different,
For example, ND filters having different transmittances may be provided on the front surface of each image sensor. The image signals thus obtained are A / D
It is converted into a digital signal by the converter 5.

Next, the block of the synthesizing means b will be described. The synthesizing algorithm is the same as that of the first embodiment. That is, the brightness levels of a plurality of pieces of image data having different exposure amounts are combined according to the brightness levels of the image data taken with an appropriate exposure amount.

The color signal from the image pickup device 18 enters a color difference signal generation circuit 21, where a luminance signal Yn (hereinafter referred to as a normal luminance signal) and color difference signals R-Yn and B-Yn are generated. The difference between the luminance levels of the normal luminance signal Yn and the bright luminance signal Yb from the image sensor 3 (difference in pixel area between the reference point 13 of the normal image 9 and the reference point 13 'of the bright image 10 in FIG. 2) is calculated by the computer 6 Is calculated and stored in the memory 7. Similarly, the difference in brightness level between the normal brightness signal Yn and the dark brightness signal Yd from the image sensor 19 is also stored.

By temporarily storing the difference in the brightness level as described above, it is possible to simply normalize the difference by subtracting the difference from the bright brightness signal and adding the difference to the dark brightness signal from the next time. It is possible to match the brightness level of the brightness signal (bright image 10 'and dark image 11' in FIG. 2). Then, the signal having the brightness levels combined is combined by the computer 6 as the data 12 of FIG. 2, and the data processing circuit 8 changes the brightness level of the combined image to the bit width of the normal image (8 bits in this example). The color difference signal generating circuit 21 is compressed so as not to
When combined with the color difference signal from, it is possible to synthesize an image with an expanded dynamic range. At that time, the data processing circuit 8 operates in the same manner as in the first embodiment.

By performing the composition with the above-described configuration, it is possible to prevent the pseudo contour from being created by the composition. In addition, if multiple image data with different exposures are simultaneously captured using multiple image sensors and the difference in the brightness level of each brightness signal is stored once, it is possible to combine them by a simple calculation from the next time. Therefore, it can be applied to moving subjects and movies.

In each of the above-mentioned embodiments, the image synthesizing device in which the image pickup block and the synthesizing block are integrated has been described. However, even if both blocks are separated, it is not in accordance with the gist of the present invention. It's obvious.

[0038]

As described above, according to the present invention, for example, when a plurality of image data having different exposure amounts are combined to obtain one image data, the brightness level of the standard image data taken with proper exposure is obtained. By combining the brightness levels of the non-standard image data taken with other exposure amounts, and then combining,
Since it is possible to eliminate the pseudo contour of the combined image data and prevent an unnatural image in which the brightness balance is lost, it is possible to combine a natural image with a wide dynamic range. Further, since the composition algorithm is simple and complicated calculation such as region discrimination in composition is not required unlike the conventional art, the composition time is fast and it can be applied to a moving image.

Further, by storing in the memory the composition information such as the value of the pixel used as the reference at the time of composition, one image data whose dynamic range has been expanded can be converted into the dynamic range width and output characteristics of the output device. It can be easily compressed according to the user's preference and can be output with the dynamic range changed freely according to the user's intention. Therefore, various output devices can output good images with a wider dynamic range than standard images. can do.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a first exemplary embodiment of the present invention.

FIG. 2 is a conceptual diagram of a synthesis algorithm.

FIG. 3 is an explanatory diagram showing a calculation method of a synthesis algorithm.

FIG. 4 is an explanatory diagram showing an example of compression of image data.

FIG. 5 is a diagram showing a compression example of dynamic range expansion data.

FIG. 6 is a diagram showing an image format.

FIG. 7 is a block diagram showing a configuration of a second exemplary embodiment of the present invention.

FIG. 8 is a conceptual diagram showing a dynamic range expansion method by addition.

FIG. 9 is a conceptual diagram showing image cutout synthesis.

FIG. 10 is an explanatory diagram showing how the intermediate luminance range is raised by addition.

[Explanation of symbols]

 3 image sensor 4 image sensor control circuit (exposure control means) 6 computer (dynamic range expanding means) 8 data processing circuit (control means) 18 image sensor 19 image sensor 23 prism (spectral means) b image synthesizing means

Claims (3)

[Claims] 1. A method of synthesizing one image from a plurality of image data of different exposure amounts in the same scene,
From the standard image data of standard exposure and the non-standard image data of different exposure from it, a common area in which the brightness level of both image data is within a predetermined range is extracted, and after matching the brightness level of the common area, the standard image of the standard image A region having a brightness level outside a predetermined range is replaced with a region in which the brightness level of the corresponding non-standard image falls within a predetermined range, a combining means for combining one wide dynamic range image is provided. Image synthesizer. 2. The image synthesizing apparatus according to claim 1, further comprising an image pickup unit for obtaining image data. 3. The image pickup means includes a plurality of image pickup elements, a spectroscopic means for dividing an image into each of the image pickup elements in an optical path of image pickup light, and an exposure control means for changing an exposure amount of each of the image pickup elements. The image synthesizing apparatus according to claim 2, further comprising: