JP6739257B2 - Image processing apparatus, control method thereof, and program - Google Patents

Description

The present invention relates to an image processing device, a control method therefor, and a program, and more particularly to a technique for displaying a high dynamic range image on a display device.

In recent years, an image processing apparatus having a high dynamic range (hereinafter referred to as “HDR”) function has been developed. In an imaging device that includes such an image processing device and a display device such as an HDR-compatible monitor, it is possible to perform all processes from imaging to development and display in HDR.

The HDR image can represent a luminance range of about 100 times that of the SDR (Standard Dynamic Range) image. On the other hand, not all monitors (display devices) that display images are compatible with HDR, and conventional SDR type monitors are also widely used. The SDR type monitor has a smaller gradation expression capability (the number of gradations that can be expressed is smaller) than an HDR compatible monitor. Therefore, when the HDR image is displayed on the SDR type monitor, whiteout, tone jump, or the like occurs in the high-luminance region of the HDR image, and it may appear different from the actual image.

Therefore, there is a demand for grasping whether or not an HDR image with an intended brightness level is obtained in the process from shooting to storing and saving of a shot image even with an SDR type monitor. In response to this request, a technique has been proposed in which an input image is once converted into a luminance signal, and a region having a predetermined luminance or higher is converted into a color signal and colored to display the region with a color signal. (See Patent Document 1).

Japanese Patent No. 4181625

However, in the method described in Patent Document 1, since the image displayed on the monitor is converted from the luminance signal to the color signal, the appearance of the color may differ between the image being captured and the image actually displayed. Yes, further improvement is desired.

An object of the present invention is to provide an image processing device capable of improving the visibility of an HDR region when displaying an HDR image on a display device having a small gradation expression ability.

An image processing apparatus according to the present invention is an image processing apparatus including a conversion unit that performs a predetermined brightness conversion on an input image, and an output unit that outputs the image after the brightness conversion by the conversion unit to a display device. When the gradation expression capability of the display device corresponds to the first luminance range and does not correspond to the second luminance range wider than the first luminance range, the conversion means , A first image having a predetermined pixel value and a pixel value smaller than the predetermined pixel value in the input image, which is subjected to luminance conversion using a first gamma curve corresponding to the first luminance range. Is generated and the second region having a pixel value larger than the predetermined pixel value in the input image is subjected to luminance conversion by using a second gamma curve corresponding to the first luminance range. It is characterized in that two images are generated, and the output means outputs the first image and the second image in one screen of the display device.

According to the present invention, it is possible to improve the visibility of the HDR region when displaying an HDR image on a display device having a small gradation expression capability, and thus to improve convenience.

1 is a schematic block diagram of an image processing apparatus according to a first embodiment of the present invention. It is a figure explaining the brightness conversion performed in the brightness conversion part with which an image processing device is equipped. It is a figure which shows the 1st example of the input image to an image processing apparatus, and the output image from an image processing apparatus. It is a figure which shows the 2nd example of the output image from the image processing apparatus which concerns on 1st Embodiment. It is a schematic block diagram of the image processing apparatus which concerns on 3rd Embodiment of this invention. It is a figure which shows the example of the image alternately displayed on the display apparatus by the image processing apparatus which concerns on 3rd Embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, a first embodiment of the present invention will be described. FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to the first embodiment of the present invention. The image processing device includes a codec unit 102, a first brightness conversion unit 104, a second brightness conversion unit 105, a display device 106, and an output unit 150. The output unit 150 includes an image analysis unit 111, a highlight unit 112, a region extraction unit 113, a resolution conversion unit 114, an image synthesis unit 115, and an output selection unit 116.

The input image 101 is input to the codec unit 102, the first brightness conversion unit 104, the second brightness conversion unit 105, the image analysis unit 111, and the output selection unit 116. The input image 101 is an image captured by an image sensor (not shown), noise and the like caused by the image sensor are removed, and luminance conversion is performed using a gamma curve corresponding to HDR. The input image 101 may be an image stored in a storage medium included in the image processing apparatus after shooting, or an image input from an external device (various electronic devices including an imaging device such as a digital camera). It may be. When the input image 101 is input from an external device, the input image may be an image stored in a memory (a storage medium such as a memory card or a hard disk) of the external device, or in real time when the image is captured by the imaging device. It may be an image that can be acquired. The input image 101 is preferably 10-bit or more data compatible with HDR. This is because with a standard gradation number of 8 bits, a tone jump may occur in the luminance conversion, which may reduce the visibility. By providing a bit depth of 10 bits or more, it is possible to maintain a dynamic range compatible with HDR.

The codec unit 102 compresses and encodes the input image 101 in order to reduce the information amount of the input image 101 and store the information. The compression encoding method is not particularly limited, and a known method can be used. H.264 (MPEG-4 AVC) or the like is used. The stored image 103 compressed and encoded by the codec unit 102 is once written in a memory (not shown) and then stored in a storage medium (not shown). The first brightness conversion unit 104 and the second brightness conversion unit 105 perform brightness conversion by applying different gamma corrections to the input image 101, and output the brightness-converted image to the output unit 150. The details of the first brightness conversion unit 104 and the second brightness conversion unit 105 will be described later.

The output unit 150 performs a predetermined process on the images output from the first brightness conversion unit 104 and the second brightness conversion unit 105, and outputs the images to the display device 106 in comparison with the input image 101. To decide. The display device 106 is a known display device such as a TV monitor or an LCD panel. In the present embodiment, the display device 106 is assumed to be an SDR type display device having a maximum luminance value of about 100 [Cd/m ² ]. For data transfer from the output unit 150 to the display device 106, a known standard such as HDMI (registered trademark) can be used. The internal processing in the output unit 150 will be described later.

With reference to FIG. 2, the brightness conversion processing by the first brightness conversion unit 104 and the second brightness conversion unit 105 will be described. FIG. 2 is a diagram illustrating the luminance conversion performed by each of the first luminance conversion unit 104 and the second luminance conversion unit 105. The pixel values (luminance values) that the input image 101 can take are taken on the horizontal axes of FIGS. 2A and 2B, and the pixel value 201 indicates the maximum luminance value of SDR. In the dynamic range of the input image 101, an area having a pixel value of 201 or less is an SDR area (first area), and an area having a pixel value larger than the pixel value 201 is an HDR area (second area). Become. The vertical axis in each of FIGS. 2A and 2B represents the luminance range that can be displayed by the display device 106. In the present embodiment, since the gradation expression capability of the display device 106 is SDR (first brightness range), the brightness range is only SDR, and the HDR range (the first range of brightness) is wider than SDR (the first range of brightness). 2 luminance range).

FIG. 2A shows a gamma curve used for luminance conversion in the first luminance conversion unit 104. As described above, the pixel value of the input image 101 is divided into a plurality of luminance areas (here, two areas of the SDR area and the HDR area) with the pixel value 201 as a boundary. The first brightness conversion unit 104 uses the first gamma curve 202 to perform brightness conversion so that the dynamic range of the SDR region can be displayed correctly, and generates a first image. Specifically, the first luminance conversion unit 104 multiplies the pixel value of the input image 101 by a gamma curve that converts the HDR OETF to the SDR OETF in the SDR area of the input image 101. As a result, the input image 101 can be converted into a pixel value suitable for the SDR EOTF characteristic of the display device 106, and can be displayed on the display device 106 at a correct brightness level. OETF is an abbreviation for Optical-Electro Transfer Function. EOTF is an abbreviation for Electro-Optical Transfer Function.

A highlight process is performed on a HDR region in the input image 101 in a highlight unit 112 at a subsequent stage. Therefore, the first brightness conversion unit 104 converts the HDR region in the input image 101 into the maximum brightness value in the brightness range that can be represented by the display device 106 so as to saturate the brightness range of the display device 106.

FIG. 2B shows a gamma curve used for the luminance conversion in the second luminance conversion unit 105. The second brightness conversion unit 105 uses the pixel value 201, which is the maximum brightness value of the display device 106, as a threshold value, and uses the second gamma curve 203 to perform the brightness conversion so that the gradation of the HDR region can be expressed. And generate a second image. Specifically, the pixel value of the input image 101 is multiplied by a gamma curve that compresses the gradation of the SDR area of the input image 101 and converts the pixel value 201 so as to redraw the OETF in the HDR area. As a result, the gradation of the HDR area in the input image 101 can be displayed on the display device 106. The pixel value 201 of the input image 101 is converted into the pixel value 204. In the present embodiment, the compression is performed so that the gradation of the SDR area remains to some extent, but the OETF may be redrawn from the zero value (0) without leaving the gradation of the SDR area. In that case, since the image in the SDR area is painted black (blacked out), the subject disappears in the SDR area, but more gradations can be assigned to the HDR area.

Since the dynamic range of the HDR region is generally wider than the dynamic range of the SDR region, the second gamma curve 203 is set so as to draw a curve for compressing gradation. At this time, it may be possible to adjust which part of the HDR region the gradation remains. For example, when the gray level of the area close to the SDR area is emphasized, the curve of the second gamma curve 203 in the HDR area is set to have a steeper rise from the pixel value 201. On the other hand, in the HDR region, when the tone on the high luminance side is emphasized, the curve of the second gamma curve 203 in the HDR region is set to have a more gentle rise from the pixel value 201.

In the present embodiment, the brightness conversion of the input image 101 is performed by the first brightness conversion unit 104 and the second brightness conversion unit 105, but the brightness conversion of the input image 101 is performed by one brightness conversion unit. Good. That is, one luminance conversion unit performs luminance conversion using each of the first gamma curve 202 and the second gamma curve 203 to generate two images (first image and second image). It may be configured.

The description of the output unit 150 will be made. The image analysis unit 111 analyzes the input image 101 and extracts the dynamic range, the position and size of a main subject, and the like. In addition, the image analysis unit 111 calculates an optimum combination position for the image combination unit 115 to perform image combination. The highlight unit 112 performs a highlight process such as replacing an HDR region in the image output from the first brightness conversion unit 104, which cannot be expressed by the gray scale by the display device 106, with hatched display, for example. As a result, the visibility of the HDR region that cannot be displayed on the display device 106 can be improved.

The area extraction unit 113 extracts the HDR area from the image output from the second luminance conversion unit 105 and outputs the HDR area to the resolution conversion unit 114. In this embodiment, the area extraction unit 113 extracts a rectangular area including an area whose input pixel value is the pixel value 204 (see FIG. 2B) or more. However, when displaying the entire image output from the second brightness conversion unit 105 on the display device 106, the region extraction unit 113 performs the through process without extracting the rectangular region. The resolution conversion unit 114 performs resolution conversion on the image output from the region extraction unit 113 so that the converted resolution falls within the region calculated by the image analysis unit 111. The image composition unit 115 combines the output from the highlight unit 112 and the output from the resolution conversion unit 114 to generate a composite image. The composite position is the image area calculated by the image analysis unit 111, and the image analysis unit 111 calculates the composite position as an area in which the main subject and the HDR area are the smallest among the four corners of the image. The output selection unit 116 can select an image to be output to the display device 106 from the input image 101 and the combined image by the image combining unit 115. In the present embodiment, the output selection unit 116 outputs the composite image by the image composition unit 115 to the display device 106, and a mode of selecting and outputting the input image 101 will be described in a second embodiment described later. ..

Various processing units such as the first brightness conversion unit 104, the second brightness conversion unit 105, and the output unit 150 can be implemented by software (program) or hardware, and both software and hardware can be used. It may be implemented by combination with ware. Further, each unit forming the image processing apparatus may be a dedicated processor such as an ASIC that realizes all or part of the processing by a logic circuit.

Next, control executed by the image processing device when the output unit 150 outputs an image to the display device 106 will be described by comparing the input image 101 and the output image. FIG. 3A is a diagram showing the input image 101. Note that FIG. 3A is not an image displayed on the display device 106. The input image 101 is an HDR image, and the gradation of the entire image is maintained. FIG. 3B is a diagram showing the display image 301 according to the first example output to the display device 106 by the output unit 150. The display image 301 is an example of an image in which the output image from the highlight unit 112 and the output image from the resolution conversion unit 114 are simultaneously displayed on the display device 106. As described above, in the present embodiment, since the output selection unit 116 displays the output image from the image composition unit 115 on the display device 106, the display image 301 is also the output image from the image composition unit 115.

A region 303 in the display image 301 is a region in which gradation cannot be expressed because the display device 106 does not support HDR, and is a region shaded by the highlight unit 112 as highlight processing. A region 304 in the display image 301 is a rectangular region extracted by the region extraction unit 113 and subjected to the resolution conversion by the resolution conversion unit 114, and the image analysis unit 111 sets the lower right corner at the composite position of the rectangular regions (overlapping). (Display position). In the area 304, the subject (birds and branches, etc.) is displayed because the second luminance conversion unit 105 performs the luminance conversion so that the gradation of the HDR region remains. The areas other than the areas 303 and 304 in the display image 301 are the SDR areas output through the highlight part 112.

FIG. 4 is a diagram showing a display image 401 according to the second example, which the output unit 150 outputs to the display device 106. The display image 401 is another example of an image in which the output image from the highlight unit 112 and the output image from the resolution conversion unit 114 are simultaneously displayed on the display device 106. Like the area 303 in the display image 301, the area 403 in the display image 401 is an area in which gradation cannot be expressed because the display device 106 does not support HDR. It is an area shaded with. A region 404 in the display image 401 shows an output image from the resolution conversion unit 114. The area 404 is not subjected to the area extraction processing by the area extraction unit 113 for the output image from the second brightness conversion unit 105, and the resolution conversion unit is applied to the entire output image from the second brightness conversion unit 105. The output image after performing the resolution conversion process in 114 is shown. Since the area extracting unit 113 has not performed the area extraction processing, the ratio of the area 404 in the display image 401 is larger than the ratio of the area 304 in the display image 301. Therefore, although the area hidden in the input image 101 by the combining process increases with respect to the subject, it becomes easy to grasp the positional relationship of the surrounding subjects. In this way, the image synthesizing unit 115 may perform image synthesis by performing only the resolution converting process by the resolution converting unit 114 without performing the region extracting process by the region extracting unit 113.

As described above, according to the present embodiment, the images subjected to the luminance conversion suitable for the HDR region and the SDR region of the input image 101 are combined and displayed on the display device 106. As a result, when displaying an HDR image on the SDR type display device 106, the visibility of the HDR region can be improved, and the convenience for the user can be improved.

Next, a second embodiment of the present invention will be described. In the first embodiment, the output selection unit 116 selects the output image from the image synthesis unit 115 and displays it on the display device 106. On the other hand, in the second embodiment, the display image displayed on the display device 106 is switched according to the gradation performance that the display device 106 can display. Therefore, the overall configuration of the image processing apparatus in the second embodiment is the same as that of the image processing apparatus (FIG. 1) described in the first embodiment, but the following points are different from the first embodiment.

First, the display device 106 may be an SDR type or an HDR type, and outputs information including the resolution of itself (hereinafter referred to as “auxiliary information”) to the output selection unit 116. It has a function of transmitting to the unit 116. Secondly, the output selection unit 116 acquires the auxiliary information from the display device 106 and determines whether the display device 106 supports HDR based on the acquired auxiliary information. Thirdly, the output selection unit 116 determines whether to output the input image 101 as it is to the display device 106 or the output image from the image synthesis unit 115 to the display device 106 based on the acquired auxiliary information. , Outputs an image based on the determination result.

Specifically, the output selection unit 116 acquires auxiliary information from the display device 106 in order to detect whether or not the display device 106 supports HDR. A well-known method can be used as a method of acquiring the auxiliary information. For example, when communication according to the HDMI (registered trademark) standard is used, the auxiliary information transfer function based on EDID (Extended Display Identification Data) is used. When the gradation expression capability of the display device 106 is SDR (first brightness range) narrower than HDR, the output selection unit 116 selects to output the output image from the image synthesis unit 115. On the other hand, when the grayscale expression capability of the display device 106 corresponds to HDR (second luminance range) wider than SDR, the output selection unit 116 selects to output the input image 101 as it is. As described above, by switching the image displayed on the display device 106 according to whether or not the display device 106 is compatible with HDR, it is possible to display an image that matches the gradation performance of the display device 106, and the HDR region can be displayed. The visibility of can be improved.

Next, a third embodiment of the present invention will be described. In the first embodiment, the configuration in which the image combined by the image combining unit 115 is displayed on the display device 106 has been described. On the other hand, in the third embodiment, two images are alternately displayed on the display device 106 at regular intervals. FIG. 5 is a block diagram showing the basic configuration of the image processing apparatus according to the third embodiment of the present invention. The image processing apparatus according to the third embodiment is different from the image processing apparatus according to the first embodiment in the configuration of the output unit, but the other configurations are the same. Therefore, the components (blocks) of the image processing apparatus according to the third embodiment, which are the same as the components (blocks) of the image processing apparatus according to the first embodiment, will be denoted by the same reference numerals and will not be described. Omit it.

The image processing apparatus according to the third embodiment includes an output unit 510 having a highlight unit 112 and an output selection unit 516. The input image 101 is input to the codec unit 102, the first brightness conversion unit 104, the second brightness conversion unit 105, and the output selection unit 516. The respective processes of the first brightness conversion unit 104, the second brightness conversion unit 105, and the highlight unit 112 are as described in the first embodiment.

The output selection unit 516 outputs the output image from the highlight unit 112 and the output image from the second luminance conversion unit 105 when it is known in advance that the display device 106 is the SDR type as in the first embodiment. And are selected at predetermined time intervals and are alternately output to the display device 106. The same control as this is executed even when the output selection unit 516 acquires the information regarding the resolution performance of the display device 106 and as a result determines that the display device 106 is the SDR type. When the display device 106 is of the SDR type, the output image switching timing in the output selection unit 516 can be set to a time such that the user can confirm the subject in each image, for example, in units of several seconds (3 seconds, etc.). .. On the other hand, the output selection unit 516 outputs the input image 101 to the display device 106 when the display device 106 is the HDR type as a result of acquiring the information regarding the resolution performance of the display device 106.

FIG. 6 is a diagram showing an example of images displayed alternately on the display device 106. 6A shows a display image 601 output from the highlight unit 112 and displayed on the display device 106. FIG. 6B shows a display image 601 output from the second luminance conversion unit 105. The display image 602 displayed on the screen is shown. By alternately displaying the display images 601 and 602 at predetermined time intervals, the user can compare the SDR area and the HDR area at the same angle of view, and thus grasp the positional relationship between the SDR area and the HDR area. It will be easier.

As described above, in the present embodiment, the images subjected to the luminance conversion suitable for the HDR region and the SDR region are alternately displayed on the display device 106 every predetermined time. As a result, when displaying an HDR image on the SDR type display device 106, the visibility of the HDR region can be improved, and the convenience for the user can be improved.

Although the present invention has been described above in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various embodiments within the scope not departing from the gist of the present invention are also included in the present invention. included. Furthermore, each of the above-described embodiments is merely an example of the present invention, and the embodiments may be combined as appropriate. The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

104 first brightness conversion unit 105 second brightness conversion unit 106 display device 111 image analysis unit 112 highlight unit 113 region extraction unit 114 resolution conversion unit 115 image combination unit 116,516 output selection unit 150,510 output unit 202 First gamma curve 203 second gamma curve

Claims (10)

Conversion means for performing a predetermined brightness conversion on the input image,
An image processing apparatus comprising: an output unit that outputs an image after the brightness conversion by the conversion unit to a display device,
When the gradation expression capability of the display device corresponds to the first luminance range and does not correspond to the second luminance range wider than the first luminance range,
The conversion means performs brightness conversion by using a first gamma curve corresponding to the first brightness range in a first region having a predetermined pixel value and a pixel value smaller than the predetermined pixel value in the input image. The first image is generated, and the brightness conversion is performed using the second gamma curve in which the second region having a pixel value larger than the predetermined pixel value in the input image is associated with the first brightness range. Generate a second image by doing
The image processing apparatus, wherein the output unit outputs the first image and the second image in one screen of the display device. And the output means, the image processing apparatus according to claim 1, characterized in that the displayed superimposed on the first image by converting the resolution of the second image. The output unit cuts out, from the second image, a region generated by subjecting the second region in the input image of the second image to luminance conversion, and displays the region superimposed on the first image. The image processing apparatus according to claim 2 , wherein: The output means superimposes the second image on the image area after the luminance conversion of the first area in the input image using the first gamma curve in the first image. The image processing apparatus according to claim 2 or 3 . Conversion means for performing a predetermined brightness conversion on the input image,
An image processing apparatus comprising: an output unit that outputs an image after the brightness conversion by the conversion unit to a display device,
When the gradation expression capability of the display device corresponds to the first luminance range and does not correspond to the second luminance range wider than the first luminance range,
The conversion means performs brightness conversion by using a first gamma curve corresponding to the first brightness range in a first region having a predetermined pixel value and a pixel value smaller than that in the input image. The first image is generated, and the brightness conversion is performed using the second gamma curve in which the second region having a pixel value larger than the predetermined pixel value in the input image is associated with the first brightness range. Generate a second image by doing
And the output means, images processor means displays on the display device the first image and the second image alternately. The conversion means performs brightness conversion of the second region in the input image to a maximum brightness value in the first brightness range when generating the first image using the first gamma curve;
The output unit applies highlight processing to an image area whose brightness is converted to the maximum brightness value in the first image when the first image is displayed on the display device. The image processing apparatus according to any one of 1 to 5 . The output means obtains information on the gradation expression capability of the display device, and when the gradation expression capability of the display device corresponds to the second luminance range, the input image is displayed as it is. the image processing apparatus according to any one of claims 1 to 6, characterized in that output device. A method for controlling an image processing device for outputting an image to a display device, comprising:
Acquiring information about gradation display capability of the display device;
Outputting the input image to the display device when the gradation expression capability of the display device corresponds to the pixel value of the input image input to the image processing device;
When the gradation expression capability of the display device corresponds to a luminance range narrower than the pixel value of the input image, the first area having a predetermined pixel value and a pixel value smaller than the predetermined pixel value in the input image is set to the first area. The first image is generated by performing the luminance conversion using the first gamma curve corresponding to the luminance range, and the second region having the pixel value larger than the predetermined pixel value in the input image is generated. Generating a second image by performing a brightness conversion using a second gamma curve corresponding to a brightness range;
And a step of outputting the first image and the second image on one screen of the display device. A method of controlling an image processing device for outputting an image to a display device, comprising:
Acquiring information about gradation display capability of the display device;
Outputting the input image to the display device when the gradation expression capability of the display device corresponds to the pixel value of the input image input to the image processing device;
When the gradation expression capability of the display device corresponds to a luminance range narrower than the pixel value of the input image, the first area having a predetermined pixel value and a pixel value smaller than the predetermined pixel value in the input image is set to the first area. The first image is generated by performing the luminance conversion using the first gamma curve corresponding to the luminance range, and the second region having the pixel value larger than the predetermined pixel value in the input image is generated. Generating a second image by performing a brightness conversion using a second gamma curve corresponding to a brightness range;
And a step of alternately outputting the first image and the second image to the display device. A program characterized by causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 7.