RU2723676C2 - Processing of multiple sources of hdr image - Google Patents

Abstract

FIELD: image processing means.SUBSTANCE: invention relates to processing of combinations of multiple images from different sources and with different brightness characteristics. Result is achieved by device (301) for combining two images or two video images of images comprises: dynamic range setting unit (302), configured to establish dynamic range (combrng) of merging brightness, wherein dynamic range setting unit further comprises brightness anchoring unit (303), color transformation unit (310) configured to perform at least brightness conversion on at least one of two images or video recordings, wherein color transformation unit (310) comprises initial anchor brightness reading unit (311) and images merging unit (320), configured to combine two images or two video images of images to form at least one output image (im_o).EFFECT: providing universal unification of images or video recordings with different characteristics of dynamic range and brightness.15 cl, 18 dwg

Description

FIELD OF THE INVENTION

The invention relates to methods and devices for correctly processing combinations of multiple images from different sources and with different brightness characteristics (or video recordings of time-consistent images), at least some of which have a high dynamic range.

BACKGROUND

Until recently, the entire video signal was encoded according to the so-called low dynamic range (LDR) philosophy, which was also recently called the standard dynamic range (SDR). This means, whatever the captured scene, that the maximum code (for example, 8-bit lumens Y '= 255; or 100% of the voltage to excite the analog display) should, according to the standardized definition, correspond to the white visualization on the LDR display with peak lightness display PB_D (i.e., the lightest white color that the display can visualize), constituting, according to standard agreement, 100 nits. If people bought the actual display, which was slightly darker or lighter, it was assumed that the visual system of the viewer would adapt in such a way that the image would still look suitable, and not, for example, unpleasantly too light (in the case of, for example, a night scene in a horror movie). This is a relative visualization paradigm that always displays the maximum brightness color in the encoded image in the peak lightness of the display, whatever it may be. This will work if the actual display on the visualization / consumption side of the video signal or image is, in fact, not much lighter than 100 nits, but can give undesirable results if the display is much lighter, for example, 10 times lighter, i.e. has PB_D = 1000 nits.

Of course, it’s nice to work in concert for each television system with a strictly defined colorimetry, for a practical program, making this usually implied by maintaining the strict lighting control of the scene, since even in completely uniform lighting the reflection of various objects can already give a 100: 1 contrast ratio, and then everything there is still a difference in the illumination of different areas. Usually the lighting and exposure (by choosing the aperture setting), so that the white colors in the (well-lit part) of the scene displayed approximately in the code are white, i.e. the maximum luma code corresponding to the coding of peak luminosity PB_C, and usually can be cut off to this brightness value PB_C even for brighter objects or portions in the scene. Usually also cameras, in particular, earlier generations of digital cameras of the early 2000s, had problems simultaneously capturing very light and quite dark areas, i.e. for example, portions of the scene observed by a room window or a car window will usually be clipped to white (giving red, green, and blue additive color components R = G = B = max., corresponding to their square root values from the luma code R '= G' = B '= 255).

However, recently cameras with high dynamic range (and, importantly, new methods of capturing HDR images or, in general, creating images) have begun to display HDR displays, and the rest of the video image and / or image technology, for example, image encoding, image composition , etc., must remain coordinated with these new technical level requirements. Note that if in this application the dynamic range is specified primarily only with peak lightness (i.e., the lightest visualized brightness), it is assumed that the lower end of the range is pragmatically zero (whereas in practice this may depend on the observation conditions, for example , light reflection from the front plate of a display or movie screen, for example 0.1 nits), and that these additional details are not relevant for a specific explanation. In the event that nothing specific is done technically for the darkest colors of the image, it can be said that there is no specific reason why the display of a higher dynamic range, which will then be simplified to a display of higher peak lightness, must first of all render different dark colors of pixels from a lower dynamic range display, for example, in particular an SDR display. We also note that several methods for setting the dynamic range are possible, and that the most natural one usually used in the explanations below is the dynamic range of brightness visualized on the display, i.e. the brightness of the lightest color relative to the darkest (of course, in practice it is also possible to provide psycho-visual aspects, for example, one bright pixel in a dark scene may not be so expressive, but for some technical explanations this can be omitted below, and only the design of technical elements is explained, which is consistent with what people need). You can specify an HDR image as any image that may contain pixel colors, and, as said, in particular, lighter pixel colors that cannot be encoded in SDR image encoding. Since the colorimetry of such an SDR image or video encoding was fixed (with a Rec. 709 OETF video signal corresponding to sRGB OETF still photographs), it can be mathematically established that the 8-bit lum encoding can only match 1000: 1 dynamic range, i.e. from 0.1 nits to 100 nits, and nothing beyond. Therefore, if you want to be able to encode the brightness of the scene that is to be displayed on the display, for example, 2 times lighter than the normal white SDR scene, i.e. up to 200 nits, you need a new form of HDR image or video encoding (the new SMPTE 2084 is an example of OETF that allows you to encode much brighter scene brightness, or at least how you need to visualize them on the display under normal TV conditions, for example, up to 10,000 nit as lumens, even as 10 or 8-bit lumens, which can then be further processed as “normal” lumens, for example, for compressing a video signal, for example, HEVC compression). The human visual system can handle significantly higher dynamic ranges, and they also exist in various practical HDR scenes in the world (for example, a sunlit scene viewed from inside a cave or a dark room with a small window), therefore a higher dynamic range video processing circuit is required, for increased “lifespan” images. Usually the illumination of the internal space can be one hundredth of the illumination of the external space, therefore, knowing that black objects reflect about 1% of the incoming light, we can conclude that the dynamic range (DR) of 10000: 1 is necessary. But, if people are able to perceive one percent of the blackest colors in the night scene when viewed in the dark, the DR content of the encoded image can be on the order of 1,000,000: 1 (which some cameras can achieve and, for example, logarithmically measure). Of course, if, for example, a large cave is observed from inside, in principle, in some situations, you can artistically select the actual visualization that the surrounding cave forms as a cut-off minimal black, but in other scenarios there will really be a desire or need to encode all different gray values from the brightest to the darkest section of pixels in the image, for example, when some action occurs in the cave (and some partial views of the outer space in some forms of the cave may look rather strange when clipped to black, due to the inversion of the psychological perception of the foreground and background, they look like torn pieces of paper, which can be avoided by visualizing additional dark colors of the interior of the cave). Therefore, the visualized display range from 0.01 to 10,000 nits in practice can be a pleasant HDR range, but, as said, coding or visualization of even a couple of factors, 2 times larger than PB 100 nits, already qualifies as HDR technology, with the corresponding need to carefully observe how it needs to be handled in contrast to the traditional SDR video philosophy. We also note that the code system itself, by its nature, does not have a dynamic range if you do not associate an exemplary display with it, and this means that, for example, R '= G' = B '= Y' = 255 must correspond to PB 100 nits or 1000 nits, etc. A common mistake since ancient times is to think that the dynamic range of an encoded image is closely related to the number of bits used. Although this was true for linear encodings, for example, an ADC camera, which needed more bits to cover a wider range of photoelectronic pixel changes, and although it may be good to have at least somewhat greater encoding accuracy i.e. bits for higher dynamic ranges, the number of bits required also depends on the selected form of the luma code allocation function, which is also called the optoelectric transfer function (OETF). Therefore, the 10-bit encoded image Y'CbCr may be some HDR image, the colors of which are encoded to some value PB_C, as well as an SDR image with high accuracy. What is the type of image i.e. how lumens need to be visualized on a display with a PB_D high enough to be able to display all coded lumens as corresponding luminances will usually be determined by reading metadata, for example, the jointly coded value PB_C in the strings of the received image. PB_C is in fact also interpreted as PB_D of an ideal model display for which the image is color graded, i.e. on this display it will look optimal, not too dark and not too light.

Thus, encoding an image that is capable of encoding images with visualized luminances, for example, up to 1000 nits, is a good starting point in order to have good HDR quality, and the reader may keep this in mind if necessary to summarize part of the principle below. unless other features are mentioned.

Therefore, in practice, there are scenes that can have a very high dynamic range (for example, by capturing the internal space with dark objects, for example 1 nit, while simultaneously observing through the window sun-lit objects of the outer space with brightnesses above 10,000 nit), and since the displays improve (PB a couple of times lighter than 100 nits, with 1000 nits currently appearing, and several thousand nits are assumed to be PB), the goal is to be able to beautifully render these images, although this is important: not exactly identical to the original, but, at least very natural, or at least attractive. Therefore, HDR does not apply only to the lightest and darkest color, but also to all the brightnesses between them, i.e. in fact, this finally applies to psycho-visual appearances for people. Of course, technically, for some HDR processing technologies, for example, only coding, it is possible to formulate the necessary appearance with respect to the technical brightnesses to be visualized on the display, since this is what must happen by properly exciting the display so that the person sees the correct looking colors. But the applicant wants to emphasize (so that the reader understands the whole following story, and what this means), that when it is desirable to develop a universal technical HDR system, the old paradigm of the type of direct one-to-one connection of only coding should be canceled. In Rec. 709, the broadcaster can see on his standard (only existing) SDR monitor what brightness he captured, watching how they were visualized on his monitor, and would be visualized on all home TVs that will be the same everywhere, as it was only a single flavor SDR display with PB_D 100 nits. The only thing that needs to be done is to convert these luminosities to be visualized (measured by the camera) into luma codes (since such codes will make better use of the existing technology for transmitting technical information) by approximating the square root function, and then the inverse function (EOTF) on the receiver side will ensure that the viewer watching the decoded content on essentially the same TV in the same viewing environment as on the side of creation will perceive the program in the desired manner. This is still a kind of HDR10. EOTF of a different form is used than in Rec. 709, but still the reference colors rendered on the display, rendered on the side of the creation, are still rendered with identical brightnesses on the receiving side, only applying a fixed mathematical equation to the received lumens. But the real world of HDR is very variable, where different viewers having different displays with different PB_D, different number of lamps turned on in their viewing rooms, etc. And what was exactly what the HDR encoded HDR10 films were criticized for: since HDR films contain many pixels that are relatively dark, it will be difficult to see in bright ambient light (although important action often happens in relatively darker parts of the video image). And HDR10 does not offer a solution for this, hoping that display manufacturers will give it. However, the applicant believes that this is, in general, a complex subject, which depends on the type of image of the HDR scene, and should have a mechanism allowing the content creator to discuss the final (variable, i.e., not one-to-one directly connected inverted fixed math between some representations of the brightness and lumens of the desired colors) the colors of the HDR images displayed on the display.

The reader should also understand that since the viewer usually watches the content in a different situation (sitting in a dimly lit living room at night, or in a dark home or public cinema, instead of actually standing in a bright African landscape), there is no identity between the brightness in the scene and finally rendered on TV (or another display). In fact, no TV can even accurately visualize at least some objects, for example, the sun is 1 billion nits, and the viewer does not want to be blinded by it (when viewing a small screen, the angles or, thereby, a certain radiance, do not are the same as in the solid angle of the environment 4 * pi, so you should also take care of this). The applicant has decided this transition of suitable brightnesses displayed on the display from the relative RGB values captured by the camera, setting the reference grading of the HDR color to start with encoding and transmission. This set of images calibrated for HDR can be created at the request of the artist (it should be obvious to the reader that, although it can be noted that the artist, what should be the technical aspects of this invention, allowing the artist to transfer and manage his requirements), if he has an exemplary a monitor, for example, can sit in a typical viewing environment and perfectly calibrate its HDR image for a display with PB 5000 nits. But this is only half the story, namely, the ability (in general, in the new set of video signal requirements) to consistently determine an artistically attractive image of a view for an HDR scene (and the ability to correctly visualize this image when viewed on exactly the same PB_D 5000 nits display at those the same limitations of the observation situation). What happens if a consumer receiving HDR images of 5000 nits only has a 1000 nits display, will be explained below (display setting). This is a completely different topic, far from something that can be processed according to the direct one-to-one relationship approach, and one can begin to ponder what this single model system with one model PB_D display should be.

If this application speaks of “high dynamic range” images, that means one here has image content suitable for display on at least a display with PB_D 1000 nits (of course, the content may also be shown on the display 100 nits or 500 nits, but then not as perfect as it could be, for example, the lamps will be dimmer than ideally). Therefore, there will be content in this image that makes it visually interesting when displayed with a higher PB_D, i.e. there are some image objects that are significantly lighter than average (or, conversely, compared with the lightest color. which can be normalized to Y = 1; at least some objects that are much darker than typical SDR images). For example, an inner space scene can have lightnesses of up to 500 nits or even more in the original scene, and depending on whether there is a well-lit greenhouse or a section of a narrow tunnel a couple of nits or lower, and lightnesses of the outer space in the real world can be several thousand nitov. On the display, you can then visualize the internal space below 100 nits, and use any available display higher than optimal for objects of external space, for example, up to 500 nits if there is a film or a “gloomy day” program, and up to 1,500 nits if it is a sunny outer space, if there is a display that can visualize up to 1500 nits or more (PB_D> = 1500 nits). Therefore, even in visualization, there may be a relationship between the lighter and darker portions of the image (1500/2) / (100/2) or at least 500/100 (this is usually defined as areas that are differently lit, i.e. e. midpoints of areas that have up to 100: 1 and usually 30: 1 contrast within the area due to reflection from the subject). When linking this to lumi codes through a continuous function, these lumi will also be distributed separately, but less useful for indicating content (they will depend on the form of the OETF, and possibly on the technical requirements). When this application mentions images of a lower (than HDR) or lower dynamic range, we are talking about images (possibly a differently graded view of exactly what the camera’s original scene image, i.e. HDR images of the same scene), which have (or are actually associated with the PB exemplary display) peak lightness, which is at least one point (multiplier 2) lower, or usually at least 2-3 points (4 or 8 or more times ) The SDR image is consistent with the prescribed PB_C 100 nit standard. The contrast ratio of all objects in these is usually not much more than 100: 1 or maximum up to 1000: 1, i.e. Usually a significantly more uniform histogram of brightness or lumens is observed.

The second thing that is necessary if the dynamic range of brightness of the image pixel (which is subject to final visualization) for which you need to specify luma codes is known, how to actually distribute the codes, i.e. which code highlighting function to associate with the corresponding luma code Y '(not only 1023 in 10-bit coding, which can usually be displayed in PB_D of the corresponding model display, but, for example, what brightness should the luma code 743 correspond to) with the object or pixel brightness Y, or vice versa.

Over the past 2-5 years, a couple of companies have proposed different methods for encoding HDR video recordings (i.e. moving images or, in other words, sets of time-consistent images). There were several breakthroughs to the first products expected in 2016, so despite the complexity of encoding and processing HDR images, some angles can be cut off. This led to different coding methods, which were even based on different basic philosophies. This has not been finally decided, which means that all image processing systems should be able to take care of the differences, and that they should become very complex (of course, compared to the simplicity of the only colorimetric era of the SDR video signal, 5 years ago).

Dolby Laboratories Inc. started with a two-level system (i.e., the main image and the correcting image are required to create an HDR image, in accordance with the principles of scalability when compressing the video signal), which is somewhat more expensive for placement in cheap microcircuits, but built a single-level encoding and proposed a basic EOTF, the so-called the perceptual quantizer PQ, recently standardized as SMPTE ST. 2084.

The technology is nicely explained in: https://www.smpte.org/sites/default/files/2014-05-06-EOTF-Miller-1-2-handout.pdf.

The idea behind this is that the classic square-root (or more precisely Rec. 709) optoelectronic conversion or the OETF transfer function (the inverse of the quadratic EOTF), which has traditionally been used to convert scene brightnesses into codes with a camera lumas are not suitable for high contrast ratios between different areas that may occur in HDR images, regardless of how many bits are used (or at least for pragmatic quantities like 14 bits or less, for high dynamic ranges that may occur in real life, as reference graduations for scenes of 20 points or more). But this turned out to be a good incentive. Although originally constructed as a result of the random physical behavior of CRT electronic guns, the square root function models well how human vision converts scene brightness into visual luminance codes, so it’s nice to make the best use of the codes (with the following code corresponding to the following psycho-visual impression of luminosity, so not too much codes is wasted on but, which is not easy to perceive in any case, nor, worse, there are no sections where there are too few codes for the exact coding of smooth and accurate brightness gradients). Now we can summarize this curve, which gives for each successive interval of brightness (for example, one point brighter) an approximately equal number of lumina codes up to the maximum coded brightness, which Dolby usually assumes equal to 10,000 nits (which should really be enough for pragmatic visualization on the display , since some people already complain that it is too light, although the experiment also shows that it is possible to make expressive images even with large areas and even luminous areas above 20,000 nits, and everything, of course, depends on highlighting the brightness of various image objects over a possible range, and it’s not always necessary to fill it up to PB for each kind of image all the time; and those who complain of painfully light images that can be dangerous should understand that the full moon at night also has 5000 nits, and no one is blind when looking at the moon )

Therefore, Dolby standardized the reference EOTF, and you can use the inverse of it, which is OETF (or approximately the opposite when additional actions are needed) to specify the lum for all objects / pixels in the image, and then you can put these codes (usually a color coded classically like YCbCr, but you can also use RGB encoding), for example, to a Blu-ray disc, after applying regular HEVC encoding methods. Then, a standard HDR calibration on BD is carried out, which can be visualized, at the request of the artist, for example, on TV 5000 nits, which can decode it. Therefore, there is a mechanism for end-to-end, one-to-one provision for displaying HDR images (only) of a certain scene on the consumer’s display, in the consumer’s home. Of course, as mentioned below, for example, through BD, a qualified reader should understand that all other mechanisms for transmitting images or video, for example, satellite broadcasting, video delivery via the Internet, etc. also implied, since the principles presented are of great applicability. This EOTF 2084 has also been selected as a function of displaying luma in brightness for HDR10 encoding.

The British broadcaster BBC has proposed another way to encode HDR video. They use the standardized OETF (since they started with a camera view, not standardization on the display side of the EOTF model display), which has a slightly different formula. Their appearance and technology are nicely described in:

http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP283.pdf

They define an OETF that follows the classical form of the square root function Rec. 709 OETF to medium gray (Y '= 0.5) classic 100 nits Rec. 709 OETF, and for higher brightnesses it becomes logarithmic in order to be able to compress lighter colors above the white color of the scene object into the available codes. This was inspired by what cameras have been doing for a long time, when they are needed to create a good LDR output image for a scene that is only slightly more similar to HDR. That is, instead of approximately 200: 1 contrast ratio of pixel brightness, there are, for example, some areas that are too light, and an attempt is made to compress according to lightnesses that are 4 times brighter than white scenes that should be displayed in Y ' = 255, therefore, the visualized brightness Y = 100 nits. Although the white objects in the scene are then rendered grayish, this may be psycho-psychologically acceptable at times, since the brain also sees and interprets areas of lighter pixels in the full image. You can always display such bright scene brightness to display PB simply by lowering the exposure of the camera (for example, by changing the aperture), but then the face, for example, a news reader, may become too dark. What such cameras did, corresponds to a curve up to, for example, Y '= 80% or some other bend point, and then use a curve with a small slope, so it can only represent a few higher brightness of the scene in the remaining 20% of the codes. Or some camera-dependent logarithmic curve can be used that slightly reduces the darker pixel brightness (so maybe darkening the faces a bit, which doesn’t have to be a problem in a high contrast scene, because the viewer understands that the person is standing in a relatively dark area (maybe in shadows), and lightness can be properly adjusted, on a lighter display, in any case), but the release of some additional codes for really lighter pixel colors, which then usually appear somewhat coarser, for example, with soft clipping. Thus, there is a balance between the colors that are subject to the final visualization are fairly bright, and with acceptable visual quality for lighter colors. But the problem was that each camera performed a logarithmic display of the captured relative brightness of the scene in (of course, essentially absolute) luma codes in different ways, which was acceptable when the cameras were only very good LDR cameras, but more problematic for the camera, which is a really good HDR camera, which involves capturing complex HDR scenes of perhaps 20 points. Therefore, the BBC wanted to set a fixed curve for all practical purposes. Nevertheless, by applying this fixed OETF curve as a curve for displaying brightness in a reference graded HDR image as an input, some acceptable version of SDR is obtained, and when broadcasting this image, traditional displays can directly visualize the SDR image, and HDR displays can use fixed inverse curve for reconstructing the HDR reference image. And the advantage of the BBC method is that it requires a small change in technology, since everything works almost the same as that used for (but it can be overly deceitful and come with a price for payment, during production or at least some video signal combinations )

They can have different starting points, for example, Dolby can focus more on the professional film market, while the BBC can focus more on the practical requirements of television, for example, field shooting of a show, deployed television equipment, etc. But the main difference of interest for this patent application is that the BBC considered these pixel brightnesses (or actually colors) relatively represented, while Dolby considered them absolutely represented. This is the main difference in philosophy, because it determines whether you are standing on the side based on the scene or on the side based on the display of the gap that needs to be crossed in order to arrive at optimal visualization. This is the main difference in what brightnesses will be encoded in the two images to be combined, but in addition, this may affect how alternative the philosophy will indicate that combining may be required, from a substantial approach and point of view on the properties.

Therefore, the BBC achieved the goal by setting a basic OETF that set relative reference graded (or at least captured by the camera, also with some decisions in appearance, like choosing an exposure, and possibly a gamma curve) of HDR brightness and lumens, and suitable subject visualization of the display brightness should be calculated by applying some gamma of the system, which will depend on the PB_D display and the brightness of the observation environment. This means that if the creator of the display receives BBC-encoded content, which can contain brightness up to 12 times the white level of the scene, but, possibly, regardless of the actual content of the scene and the iris setting, the camera can perform either of two actions. He can use the BBC philosophy, and always display the PB_C code in the PB_D of the display, regardless of the PB_D of the display (and then maybe apply some gamma brightening and a contrast modulation function, depending on the actual PB_D, but at least white will be visualized in white, which has at least one advantage, consisting in the fact that, in the absence of such excessively bright areas, but only relatively darker pixel colors, as in a foggy scene, it is easily calibrated that such darker coded colors are also will be rendered darker than PB_D, but on the other hand, the lighter parts of the scene will be rendered with variable brightness PB_D; which will look good for combining image data, but is it?). Or you can imagine that this luminance data of an encoded image is good around or for exactly some absolutely brightness value PB_D, for example, 1000 nits, and then do your own internal reduction in the case when the TV has, for example, only PB_D 500 nits (by means of blind display gamma). Dolby, however, considered that it was necessary to set in absolute terms which brightnesses should be visualized, at least with a high-quality display of 5000 nits of Sim2 or a pulsar, and how this image was artistically generated does not matter, provided that you can set the required brightness objects in the reference range of the reference brightness EOTF 0-10000 nits. So, for example, if there is a dark face in a horror movie, in the absolute way it is usually said that this image should be visualized, for example, at 15 nits, which can be visualized on displays with both low and high dynamic range. However, in the relative method, the brightness of this person will scale, even not reaching 15% of 100 nits, will also mean 15% of 5000 nits, since the gamma function, depending on the PB display, will be somewhat correct for this, but still , it will not be precisely determined at what brightness the face will be finally visualized on the display. This suggests that the artist will like it if such a change occurs on the visualization side, so there is something to be said for more accurate absolute visualization (when using the full dynamic range of the display, of course, the viewer's sensitivity will adapt to this, but usually the subrange will be used HDR display, if the content requires rendering only, for example, up to 900 nits, at least for example 99% of the image pixels). Again, absolute brightness coding also does not have a complete answer, because if PB_D is actually present lower than PB_C coding, i.e. it is necessary to visualize the brightness so that the image is optimal, nevertheless, some reduction strategy is needed, which is better not to be too blind and against the needs of re-gradation of the appearance of a particular HDR scene or type of scene. The reader can imagine that such different philosophies lead not only to different EOTF definitions of the code, but probably also to different image processing, i.e. how the brightness of the pixels of the object will be allocated to different areas along the curve or in the corresponding range of reference brightness. And if this already leads to difficult disputes in the presence of a relatively simpler task of displaying some input image with a specific PB_C on some display of a lower PB_D, which can be represented as somewhat designed by will (for example, when using the so-called r-curve, due to its shape , with some amplification of lighter colors, and then for lighter luminances, it starts to get an ever smaller slope, effectively compressing them in smaller sub-bands of the SDR brightness range, i.e., for example, the first quarter of a circle or ellipse), because you can stretch, for example, a part of the increase for darker colors of the HDR luminance display curve in the SDR for a slight enhancement of the darkest colors, which will increase the luminosity of some typical object on this part of the input brightness range or curve, for example, someone’s coat, the reader can imagine that things can become more complex when it is desirable to harmonize the lightness of objects in different images, ra dynamic range, usually lit differently, and potentially differently encoded.

An applicant like Dolby also believes that it is best to set a reference EOTF that is set to 0-5000 nits. One reason is that the image captured from the bright African scene is mainly different in absolute value from the Norwegian autumn scene, therefore, although all kinds of mathematical color transformations (affecting at least brightness) can always be applied for the final obtaining a suitable kind of luminosity, why not encode this difference into lum values and their corresponding absolute luminances in the range of the reference luminance? (Mostly you can even design these lumens already correlating with psychovisual luminosities, then easily encode the artistic desires in suitable forms of an additional curve when grading from the reference initial HDR image corresponding to images of a different dynamic range that has neither HDR10 nor BBC.) But the most importantly, in the end there is only the final visualization of pixel colors (even if this can happen on displays with different PBs), and without at least the corresponding model display (specified not only by peak lightness, but also by its gray scale behavior, like EOTF ), an image is just a collection of numbers (which you can probably decode correctly), and nothing more. This concerns the final perception of the HDR image, and not so much how the construction to this point occurred, and what is also what graduators see on their model displays. One needs to be able on the creation side to make an explicitly defined initial image. Therefore, you need to start by standardizing the reference EOTF, which fixes this (otherwise, for example, the numbers of digital lumens in the computer that is used for color grading are undefined), and then you can build additional systems on top of this (even if you want to build some relativity into a system, at least, which then will be controlled relativity, determined on the basis of some jointly agreed reference brightness measurement system).

Now, summarizing here some of the new necessary basic background HDR views above, we can move on to a more practical situation, where there are different images for which the creators decided for some reason, they must be encoded in one way or another.

The difference in the behavior of the Dolby EOTF from the BBC EOTF (the applicant of the present patent built an EOTF very similar to Dolby PQ, or can use Dolby PQ) can be seen in FIG. 1. OETF 113 is the PQ EOTF (we showed only a portion of up to 1000 nits shown, because if we show a graph of up to 10,000 nits, the other two curves become undoubtedly small). BBC OETF 112 is a relative curve, so you can argue about what absolute brightness should be encoded, but it is assumed that it usually goes up to 500 nits (or maybe 1000 nits, but then it can start showing errors and become less suitable). Curve Rec. 709 can only encode absolute luminances of up to 100 nits, and it should be remembered that for various reasons in the distant future there will be a lot of SDR content (which, for example, is not easy to convert to some version of HDR). Relative lumens are shown on the y axis, since we want to avoid considering the bit depth, which is not so relevant for this application (we will assume that all encodings will use 10 bits for three color channels, so the maximum luma 1023 becomes 1023/1023 = 1.0 ) Obviously, when you receive an input luma of 0.5 (i.e. 512, or 128 in 8 bits) that can mean anything like the corresponding brightness, depending on the encoding method used (somewhere around 0.7 there may be a difference in 10 times, so a simple pixel luma replacement switching can suddenly produce a very dark or very light replaced pixel). In FIG. 1B, we show how the curves differ if we also set them at a relative input brightness of 0-1.0. Obviously, if you start with the lum encoded image, the visualized brightness can be calculated from the inverse curves.

Now, ignoring all practical problems like color errors after various coding qualities, costs of redesigning the microcircuit, etc., we can say that all three coding mechanisms work nicely if you only need to transfer a single set of HDR images, then what you need focus on the first phase of research, and development, and standardization.

But practical HDR processing systems are needed more, otherwise users are left with unpacked ends and issues that the system designer cannot do.

The applicant has constructed a coding system that cannot only process transmission (coding) and use (suitable visualization finally) a single standardized HDR video signal for a typical single kind of display in operating conditions (each needs to have a display with PB 5000 nits, or maybe everyone who does not have 100 nits is a traditional TV capable of buying only 1000 nits of PB TV), but which system can process various possible display types with different PB_D in the operating environment, thanks to the ability to calculate an optimally (re) graded image for each specific display, which technology the applicant calls customizability or setting display. Its main characteristics, and in particular decoder 251, are generally shown in FIG. 2 (as far as necessary for this application).

The applicant had an approach that any final kind of optimization depending on the observation environment can be carried out on the display side, although without the final gamut as the BBC offers, but through a mechanism that (although calculated by the receiver, for example inside a TV or BD player) is not only a specific TV ( i.e., its designer), and not only on the basis of the receiver side, factors like the actual peak lightness of the display and the lightness of the observation environment, but also depending on the image content itself, since the images are complex and therefore their optimization to the particularities of the visualization side. This crosses the old and difficult gap between creating content and using content. The receiving side can only be the side that actually knows what display capabilities are actually available, but the side of creation is only the side that definitely knows what the content is about, such as the art desired. If you can’t sell the director of a film or DOP instead of each TV to optimize the color appearance on the receiving side, you can at least let the creator set intentions parametrically, at least roughly, and the creator of the image can mathematically set: If I had a display with this PB_D, I would at least prefer to visualize different pixel brightnesses like this and thisʺ.

The applicant’s engineers objected that the content-dependent part of the image could be processed by a color calibrator allowing the artist (although it is possible to use the automatic image analysis functions very well, that is, to determine the brightness display functions based on the set brightness properties of the current image) to specify optimal color transformations according to it, which will at least change the brightness of the pixels of the object for (only) two types corresponding to different dynamic ranges or actually PB_C, at the two ends of the range of possibilities (peak lightness of the display). Therefore, an HDR image should correspond to a rather high PB_C, which the applicant selects 5000 nits (although the applicant can use another PB_C very well), and a low DR image can also be a 100Rit SDR grading, which satisfies for example 90% of the originally existing human market, using traditional SDR displays. Therefore, for a particular scene, the color grader can say: If I had only PB 100 nits, I should put the lamps in an image that I would not see rendered at 4000 nits in order to have nice-looking lamp objects in the rendered image, but this is not possible now, at 95-100 nits, when first of all it is desirable that there should be enough brightness in the display brightness range for visualization of objects of lower brightness in the room is quite bright, i.e. between, for example, 10 and 90 nits, so now, perhaps, I will put all the pixels of the lamp exactly 100 nitsʺ. The same can be done, for example, to make the night scene appropriately dark and the street light sources bright enough to visualize both 5,000 nits and 100 nits on the display. The reader should stop for a moment to ponder that it is not trivial to make a night graduation that looks properly nocturnal, i.e. the correct brightness for objects (for example, a black person sitting in a poorly lit area of the shadow), and then both in HDR and SDR (especially with additional restrictions, such as IC complexity), but this can be done, at least if not perfectionistly then acceptable (of course, if the display follows specifications approximately, and doesn’t make the image completely different).

Having obtained these two graded images (M_HDR, and M_LDR, the brightness distribution of which is usually determined by humans), one for the situation in which one has a high quality HDR display for image visualization and one for the traditional 100 nits SDR displays, the applicant wanted to avoid encoding and transmitting two complete sets of images, i.e. the need for double coding, which will require two channels, etc. And the applicant also wanted two different DR graded images to be encoded with existing DCT-based MPEG encodings, so we decided to set one of the two video image sequences parametrically with the color conversion functions (F_L) to be used to calculate images from a set of images that are actually transmitted (for example, although not necessarily, an SDR image sequence). Therefore, we can transmit a video signal through some system 250 (a qualified reader understands that our system can be used in many systems, for example, airborne broadcasting to a receiver that is a computer, either from a professional service that issues a video signal, or between consumer devices connected to a home network, or a professional secure cable communication system, the receiver being a professional device in a movie theater, or an Internet-based system, etc.) for example, HDR M_HDR reference images, such as HEVC encoded video signal S encoded by video encoder 203. The encoder also adds all parameters defining F_L color conversion functions as metadata, such as SEI messages or the like. Or vice versa, we can transmit the reference calibration LDR M_LDR, together with the function, which allows recalculating on the receiving side images of a higher dynamic range, such as M_HDR.

Therefore, on the creation (or transcoding) side, for example, there will be some color grading device 201 comprising a color conversion unit 202 configured to determine and apply various color conversion functions F_L as desired, to obtain an LDR reference image of the form M_LDR from a previously created HDR reference image M_HDR, for example 5000 nits PB_C usually (what can be done on the same color grading device at the same time, or otherwise determined earlier). As an example of image processing, the reader may refer to WO2014 / 056679, and the principles of the application are thus included in their entirety for countries that allow this. As an example of the conversion, you can depict a function that does not affect the color, but only on the brightness of the pixels (all of the considerations below will be focused on brightness conversion for ease of understanding), which bends the higher brightness slightly down and enhances the darker colors to a relative 0-1. 0 HDR input (x axis) to the output graph of the LDR function, so that they will look quite bright on a relatively darker 100 nits display (r-shaped curve). But more sophisticated functions that are customizable and potentially only locally fine-tune the brightness of some objects, as well as the conversion of color aspects, such as saturation, are also possible, provided that you can create two related optimal views: an LDR image and an HDR image.

Now the next thing that can be done with this coding technology is that if you don’t actually need to recalculate the M_HDR 5000 nits image, because you need M_MDR medium dynamic range images that have a properly graded view for the actually connected peak light display 280 PB_D = 1500 nit, for example, it is possible with our display adjustment algorithms to obtain correctly graded M_MDR images from received, for example, M_HDR images (or M_LDR in other embodiments / applications). The receiving side has a classic, for example, HEVC decoder 260, which outputs a normal (for example, linear RGB, or YCbCr) M_HDR image, and the necessary color conversion functions (F_L or its inverse mirror function) for changing M_HDR to M_LDR. Based on all this information, our color adjusting unit 271 in the color processing unit 270 (which, as we assume for this illustrative explanatory embodiment, is separate from the TV, for example, inside a set-top box, or an on-demand video provider server, etc. ., but it can also be located inside the TV) can not only determine the necessary optimal functions F * _L to determine the necessary image M_MDR from the received image, but also finally display this image of the average dynamic range M_MDR. Algorithms for adjusting the display to obtain a suitable F * _L function for converting SDR brightness to MDR based on the F_L function of converting SDR brightness to HDR or functions can be complex depending on which embodiment is desired, for example, to take into account the information of the observation environment, but for Understanding the present application, all these details are not required, so the reader can simply note that it will be a function of F * _L brightness for use to obtain from any incoming encoded image and an MDR image, for example, a gamma function.

Although this mechanism now allows optimal rendering of HDR images, not just for the only possible kind of HDR displays (for example, everyone who needs to buy a display with PB_D 1000 nits and cannot buy a display with PB_D 3000 nits still gets a nice looking image, at least without concern for colorimetric accuracy), but for any possible display, this additional technical expertise is still insufficient for a fully flexible HDR processing ecosystem. To this day, generalized technology has only focused on obtaining a single set of HDR video images, from a single captured HDR scene, correctly to various places of consumption, and how to properly visualize it on the only TV available. This was already a discouraging task to prepare for the new HDR video technology, so additional aspects were postponed until later.

But at present, programming a video signal is more complicated. You may wish to receive feeds from various places of any kind, and this facilitates the creation of video content (even for consumers), with simple portable devices (even live news coming from a helicopter or from some athlete taking pictures with GoPro installed on his helmet), and it is easy to transfer it through several means of transmission, for example, the Internet. And indeed, at least theoretically, since large deployments by consumers may still need to be taken off, televisions are becoming more and more like computers, with Internet connections, and capable of using or sharing various data sources presented as visual information. For example, we assume different uses of the video signal, for example, you can watch a news program in the background, and at the same time play some video game in some area displayed on the display, this can be the area of the same display or on the neighboring display.

This task started in the LDR area, but it was basically less difficult, since the entire video signal was set in the same Rec. 709, and more or less similarly lit and exposed. And even computer graphics, for example, the game was set similarly. But at present, a video signal or HDR image can be anything, since this is something that I would like to take into account, therefore, in the absence of a good strategy, the structure for processing and, in particular, combining multiple image (partial) contributions, can be assumed chaos in the future . As said, in the past, the entire video signal was set in the same LDR structure, i.e. all maximum lumina codes corresponded to the same visualized luminances of 100 nits, and luminances corresponding to lumas below the maximum can be decoded using a single existing Rec. 709 EOTF.

Some questions remain (in particular, in complex production, or external broadcasting and other field production), and the need for careful processing, since cameras may have some "younger" settings. Usually, white balance (i.e., the chromatic aspect is neutral, white colors, i.e. whether white objects look too bluish or too yellow, which may become noticeable when switching between different camera feeds) and exposure settings are usually important. When the exposure of capturing the same scene of one of the cameras is reduced, the camera will quickly get dirty as it gives an image that is too dark. Especially for professional multi-camera productions of the same show, which was noticeable, as in the news program, clipping to a live character can suggest different capture (especially if it is amateur video on a mobile phone), or at least that a completely different looking scene (for example, a night scene of Cairo, instead of a well-lit studio in the USA) was captured (intermittent or time-embedded image). Now there are two situations from which you need to make a choice. Of course, when switching between a night scene and a daytime desert scene, the viewer will assume some difference in the average lightness of the scene, or the brightness of the objects in the scene, although this should still be an acceptable difference, for example, with a desert scene that does not surprise the viewer or causes him to strabismus. But on the other hand, you can make the scene look night or bright by playing on the combination of the camera’s exposure and the final brightness of the visualization display. In the era of SDR, which sometimes led to strange images, if the operator must make an uncomfortable choice of exposure. For example, someone standing far from a window in a deep room will be much darker than objects near the window. Then the operator wanted to expose a person in a darker area to make it pleasantly colored, which could lead to the left side of the image and all objects in the room being completely clipped to white, forcing the viewer to observe an ugly “half-image”. But at least the colorimetry was acceptable. It was known that colors would come out like this, and that they would be exactly the same regardless of whether such images were mixed or just rendered. In the HDR era, they wanted to solve such problems by making all sections coded instead of clipped to white, for example, reducing exposure, depending on how much lighter exactly the bright parts of the room turn out to correspond to the brightness of the actor or presenter in a particular scene. But it will be unpleasant to return as a task, if you need to harmoniously mix video content, therefore, the need for the following technology implementation options.

In addition, more sophisticated more recent cameras have received additional settings, such as gamma black. It may not be easy to perfectly match two different cameras (if they are of the same brand, you can at least, for example, turn off or set all the “exotic” parameters similar), but at least it was the task of the camera shader, which is an engineer with a special eye for color differences (or CTO) to remotely control all of these colorimetric camera settings. This frees the operator of the actual camera from focusing on framing, which is already a rather difficult task, especially for some productions with actors jumping around the stage, etc. In any case, in the SDR structure, even if the actions are a little wrong, the fact that the creator of the content made sure that at least his main topic was acceptable coverage (i.e. at least 10% white, or approximately 1 point lower than average gray as lum or visual impression of luminosity), combined with a limited range of 100: 1, which roughly corresponds to various diffuse reflections, any incorrect combination with some brightness of an object incorrectly located compared to the brightness of other objects will usually not be too disharmonious (at least critical viewers may perceive that something is wrong, for example, black areas may become a little more milky gray, but nothing will look really powerfully bright, or shine with large differences in lightness, such as may happen in HDR production).

But now with the advent of the HDR era, there is no such single LDR reference range of 0-100 nits, and there are no cameras (and other systems) relatively simple operating in this range (i.e., usually performing non-extreme color conversions). As can be seen in FIG. 1, even if in the future we choose to use only one of the possible HDR codecs that are not obvious, there is a transition to some content specified by the LDR codec. And the difference between 5000 nits and 100 nits is so extreme that it can’t be called a “minor” change in appearance (sometimes only really indicated by professionals), but a very noticeable and potentially even extremely annoying change (and it can be assumed that if different devices, for example, the TV is in the future, for example, to act according to the preferences of the designer according to this vast range of brightness of this vast color space, so that variability can become even more extreme, and not at will).

The task may not be noticeable enough if you look at a single program from a single source. For example, if someone is watching HDR content with a high PB_D display in a dark environment, the colors may be a little too light, but at least his eyes can adapt. The differences always become very distinct, if there is some link to define the image, for example, in the presentation of the embedded image (PIP) of two videos.

Since the images can now come from anywhere, not everything is controlled by a director who decides on the type of images, nor shown independently in a managed structure, but only arriving together potentially sometimes for the first time on the display side (for example, an inserted designated advertising clip) There is an urgent need for a good structure to coordinate combining various initial HDR images.

Therefore, as said, most of the (new) HDR technology focuses on the ability to transmit all of the HDR image information by transmitting clearly defined lumi pixels, which can then be converted into visual brightness. If you have a display with a lower PB_D than PB_C content, you may need to do something because it is not possible to visualize the brightness up to PB_C on this display. You can perform linear compression (displaying white to white relative, i.e., visualize the pixels in the image with the PB_C luminance encoded in fact on the display with the PB_D luminance), but then with the full likelihood the rendered image will look suboptimal (for example, some of the darker parts of the image may look too dark, perhaps even with an invisible effect). Therefore, you need to try to make a more reasonable display of brightness, which supports darker brightness acceptable acceptable, for example, using the r-shaped curve. General gamma display methods will be possible, for example, inside a TV, but they will be blind, i.e. not adapt to HDR image information, and therefore substantially suboptimal.

According to US20150245004, two images may be mixed, of which at least one may be an HDR image and the other may be an SDR image. The application describes that there may be both mismatches in the color space (i.e., the width of the RGB triangle, since other additive primary colors are used in DCI P3 color definition than in sRGB the usual definition of video signal, but this chromatic aspect does not correspond to the explanation of this application, since it can be selected in addition to what is the primary essence here, namely, the brightness of the image pixels, and the corresponding dynamic range of image encoding), and the mismatch in the dynamic range. This application does not describe in great detail how video mixing should or can occur. [0029] mentions that mixing can occur in the color space of the display device, i.e. dynamic range, which in the title of this application ends in PB_D. In addition, the processing looks like a conversion to another TF transfer function, or gamma correction [0027]. Finally, there is a very universal principle [0036], according to which the composition unit can evaluate the dynamic range of video content, and which presumably corresponds to an additionally not described principle that statistical analysis of input images can be performed [0028]. One skilled in the art will interpret this in the light of existing knowledge. This is indeed the principle in optimizing the SDR video signal, which looks at the statistical properties of the input video signal, namely the end points of the brightness histogram, and optimizes this histogram, in particular for the available display range. For example, due to various reasons, it is not unusual that the black or darkest pixels in the captured image are brighter than they could (or perhaps should) be, for example, lum 30 instead of 0 (ignoring, again, at the moment, complex details about overfill protection ranges, display light leakage, lighting-dependent visibility problems, etc.). You can then stretch the histogram to have the lowest image luma of 0 instead of 30, which leads to cute images due to the increased contrast of the image structures, and the best black areas that can be especially pleasant to observe in a dark environment.

In fact, when a video mixer design is needed at first glance, in the absence of additional features of the invention, one of ordinary skill in the art can have the following two options to consider as explained in FIG. 13 (in addition to mixing the luma codes, which, as explained with reference to Fig. 1, although pleasant and easy to mix purely SDR images, are not so obvious or reasonable when there are differently defined images). If the principle says that a conversion is necessary due to different transfer functions, one skilled in the art will think of a naturally constructed mixing system, as in FIG. 13a. With this mixing, there is a range of natural brightness (1301), for example, up to 50,000 nits can be large enough, and in fact, with any definitions of the OETF code forms, it converts them to brightness. Both images containing (physically uniquely specified) brightness, of course, you can, in principle, mix them, but the question is whether this will be enough, from practical psycho-visual and artistic considerations. Secondly, if you map to the common preset dynamic range of the display (1302), with PB_D, for example 700 nits [so for simplicity we now skip the OETF stage and the reason is only about the corresponding brightness], then another natural mapping (for example, relative mapping paradigm) can be considered to adjust (based on атистstatistical properties of imagesʺ) images in the display capabilities (i.e. the dynamic range of the image in the dynamic range of the display). The natural way to do this is to stretch the ranges (i.e. corresponding to a white to white display), for example linearly. You can also apply some gamma display function, but the question is what should be the gamma coefficient, and why. But this creates results that are already exactly what in several cases we would definitely like to avoid. When stretched, for example, an SDR landscape image with a show of up to 700 nits, it can become unpleasantly light.

Furthermore, although these natural imaging systems are no longer perfect just for reasons about their endpoints, such as where the lightest color should be displayed, such methods completely ignore which, as said, are probably the most important aspect of HDR, namely (finely controlled) the distribution of all the brightness of objects between the endpoints of the dynamic range (and at present this seems like a missed understanding by many, if not most experts, who understand that HDR refers only to the white brightness value, any technology divided by black can achieve, although often an image that correctly visualized on a lower dynamic range display may look better than a poorly managed image on a higher DR display, for example, based on contrast and the HDR effect). It is easy to understand that the dynamic range is not related to technical endpoints, focusing on the example of the show on the reference calibration HDR PB_C = 5000 nits. For example, a calibrator displays the brightest part of the show in the sun at 800 nits, while another calibrator can only fit it at 550 nits, so in essence it looks like a situation with a lower dynamic range, since most of the codec features are not used for this image. In SDR grading, it can be assumed that typically the grader places the lightest pixel of the show at 100 nits. For graded MDR images, you can consider what brightness the lightest pixel should fall into, even when only rendering one image instead of a possibly very complex mixture of different image content. It is known with what final (mixed) brightness the show pixel should be finally visualized, whether it comes from an SDR image, where it is relatively encoded at a maximum of 1.0, or from “some” HDR image encoding, where, even ignoring the OETF shape, as it may not be the most important determination coefficient, the pixel of the show is some at first glance somewhat arbitrary-looking relative brightness, such as 0.126.

FIG. 14 illustrates another important point of view with methods that only relate to the dynamic range of any technical system, whether some coding corresponds to a theoretical model display, or an actual display, etc., like something with basically a degree defined by an endpoint, for example how blind brightness conversions to display gamma will work. In particular, in HDR, the illumination of objects should not be strictly controlled, as, for example, in the SDR news studio, but can, in principle, be anything. If someone goes to the sun after a forest glade, for example, through grass, you can visually perceive that the interior of the forest may already look black. This, of course, is not always because what will be black, because the trees are just brown, like something else, and the undergrowth is freshly green, but this is because the local lighting is, for example, 1/20, or even less than the local lighting is a free spot without trees that receives the full light of the sun and sky. And 1/20 begins to look like psycho-visual black for people until it takes a closer look.

An object like a T-shirt, therefore, can have different brightnesses, even in the same film (in other words, it can fall into the whole place, for example, in the range PB_C_HDR 1000 nits; note that the calibrator may decide to limit the change somewhat in its reference graduation, but we explain with a natural almost one-to-one relationship with the relative brightness of the original scene captured by the camera). In the sun, the T-shirt will be + - 5 times lighter than in the initial partial shade, where the T-shirt is mostly lit, a significant solid angle, with a blue sky. Indeed, 100 nits will also be artistically what can be assumed to render a “gloomy day” on a good HDR display, while 500 nits may look like really lit by the sun. Additionally, a person goes into the forest, the larger the area of the sky that can illuminate it is obscured by leaves, so the brightness of a particular object, in this case, a person’s T-shirt, will, for example, be 100 nits, therefore, 50 times darker than a light t-shirt, therefore, it most likely also looks somewhat blackish when rendering this particular HDR scene. Perhaps in the cinematic grading of HDR for home observation of brightness, it is possible to graduate more conservatively, for example, half of these values, but the distribution of different brightnesses can provide some point. So the question will now be, will any of the above relatively static mixing methods distribute these various visualizations of the T-shirt in the video being played back (with information of its own brightness), will always be “automatically” acceptable good, and if not, what should be done?

SUMMARY OF THE INVENTION

The aforementioned problem of the need for a method of acceptable coordination of combining images with a (potentially very) different dynamic range can be solved by means of a device (301) for combining two images or two video images (Im_HDR, Im_LDR), one of which is an image or video of a high dynamic range, moreover a device (301) for combining two images or two video images (Im_HDR, Im_LDR), one of which is an image or video of a high dynamic range, the device comprising:

- a dynamic range setting unit (302) configured to set a dynamic range (CombRng) of the brightness of the combining, wherein at least the maximum brightness (LMC) is determined based on at least one of: a maximum brightness of at least one of two images or two video recordings of images, and peak lightness of the display for visualizing two images or two video recordings of images, and the dynamic range setting unit further comprises a brightness anchor determination unit (303) configured to determine the brightness (anc) of the anchor in the dynamic range ( CombRng) the brightness of the union,

- a color conversion unit (310) configured to perform at least brightness conversion on at least one of two images or videos, the color conversion unit (310) comprising a brightness reading unit (311) of the source armature made with the ability to read at least one brightness (L_SA1) of the original anchor from the first source (350), which delivers the first image or video (Im1_LDR) from two images or videos, and the color conversion unit is configured to establish color conversion (FF_1 ) to be applied to the first image or video, the color conversion being dependent on the brightness value (L_SA1) of the source armature due to the property that the brightness (L_SA1) of the source arm is mapped to an output brightness close to the brightness (anc) of the arm; and

- block (320) combining images made with the possibility of combining two images or two video images to form at least one output image (Im_o). First of all, in order to avoid doubts, due to the combination of images, we offer various possible options for combining the image content in space or, in the case of video recording, possibly also in time. Combination should be understood in the broad Latin original sense as “unification”, and not in a more limited sense, for example, the blending that we reserve in this text for a weighted summation of two sources per pixel. The person skilled in the art understands that various embodiments of this invention solve the need for a good (harmonious in brightness or matching) video signal combination, which arises among other things in various associations of the embedded image (and, in general, this currently includes spatial interleaving of at least some pixels of the first image with some of the second image, for example, in a time-varying amount that corresponds to the transposition of the first image, as if it were a page and showing pixels of the underlying image, but this invention allows us to guarantee that the colors of the pixels, and in particular their perceived lightness, will correspond correctly), combination of mixing, for example, attenuation, or temporary scattering, for example, trimming to a commercial at regular points in time during the film, etc. If there is a replacement of the images of the spatial section by some other (in whole or in part) second image, and if there is a time-demonstration of the images of a different video sequence between or after the images of the first video signal, it is not necessary to have too much discrepancy in lightness (i.e., one of the images should not look excessively lighter than another, for example, in relation to its main column of the histogram or key objects), especially in the presence of also luminous content, etc. In any such scenario, you can set the dynamic range (CombRng) of the brightness of the combination, which is the dynamic range (empty view) of the combined image to be filled with the data of the combined image (for example, take for a temporary spread or sequence the largest range of both).

One skilled in the art should understand what the high dynamic range image is and why, namely, an image that is not a traditional LDR image specified for a peak brightness of 100 nits, i.e. which cannot be combined in the classic LDR mode (this at first glance looks like some mismatch between the brightness of the objects or image pixels specified in two different image encoding structures). It should be clear to the skilled reader that if the device receives only 2 still images (or, for example, 5), that usually the output image (Im_o) will be one still image. In the case of combining two or more videos of sequential images, an output video signal will be generated, i.e. a sequence of these images in one form or another, for example, in the case where each video signal has an image at each moment in time, the resulting video signal can be an alpha-mixed video signal with exactly the same number of images for the corresponding time instants as two original image sequences, or the resulting video signal may have images for different times, for example, have a length of 100% of the first video signal, + 50% of the second video signal, and its various images are highlighted in the mixture at some of the time points.

In order for the principles of this patent to remain readable and concise, we believe that the embodiments simply carry out transformations at brightness (the reader may, when reading, assume that the images are gray scale images), since the principles primarily determine how to handle differences in dynamic brightness range, and the viewer will have an impression of lightness. However, of course, a qualified reader understands that in fact color transformations will usually be carried out, for example, he can understand how brightness corresponds to the definition of linear colors R, G, B, and how you can create a pixel with the desired brightness by color converting its color, for example, linear RGB, as in WO2014 / 056679. In the case of expressiveness, we say below something for a video signal, one skilled in the art will understand that it can also be applied to still images. In some cases, a video signal may, due to real-time aspects, desire simpler processing methods, and for this purpose elegant quick mixing solutions are presented. Some of our best-implemented embodiments will work with the participation of a human artist, as people can best decide which HDR images are in question and what they need in different, mixed views, but other embodiments will work with automatic annotation through image analysis algorithms.

The device should be able to set the most appropriate dynamic range for combining brightness, which is the new dynamic range, which is partly the reason why the very complex task of combining becomes more solvable. As will be explained below, this should not be just one of the existing dynamic ranges, i.e. where it has already been invented and defined, namely, it does not require either the dynamic range of any of the source images, or the dynamic range of the final display on which the content should be visualized (even known during the merger, which is not true for all embodiments), the designated display (which may not be known in some embodiments, although some device on the production side may have a reasonable estimate of what a good model display can be, which is a typical example of a prospective display in the home, in a consumer’s home) This dynamic range of brightness of the combination should be such that the combined representation has the best visual quality or impact on the viewer. This may, of course, depend on various aspects of the content being asked. If the main content (which we will call the second image or video, as this facilitates the reading of the claims) is, for example, a film, then the creator of the film, for example, would not like it if his film was violated too much. For example, I would not want the viewer to be able to see well what is happening in the dark scene of the film after the light commercial. In such a scenario, since usually one of the two videos is distorted in its brightness properties to achieve greater harmony with the other input video signal, and therefore the combined video signal, the film is the most important (or leading) video signal, the brightness of which should, for the most part, remain undistorted ( if possible). In addition, the producer or sponsor of the ad would like to see his ad appear in the final visualization in the best possible way. The color conversion unit (310) and / or the combining unit (320) may implement a policy according to which the commercial cannot dominate the film, but, for example, the LDR commercial should also not be rendered too dark. For example, a face may not look very pretty if, in comparison with another related subject in the film, it looks almost black. As indicated above, a forest may look black compared to very bright areas illuminated by the sun if the human visual system adapts its interpretation of the meanings of gray in this bright area. Thus, if important (large) areas are formed in the HDR movie or television program (or game, etc.) that are very bright, for example, 2000 nits, and in particular, are located next to the combined SDR pixels, then the SDR object with a luminosity of approximately 20 nits may appear blackish, therefore some lightening may be desirable prior to combining. Also, visibility, for example, facial features or details of other objects, decreases with a decrease in the average brightness and contrast of the object. Although we have found that even with single images, some people may complain that in some situations the distribution of the brightness of the object (for example, the histogram of the brightness of the HDR rendering pixel) is not completely optimal, for example, too bright in some scenario, visual adaptation and the human brain are usually relatively are lenient when rendering a single image, because the brain can determine and regulate what all the objects in this single image should be. But if the object is now spatially composed in another image, the viewer is inclined to mark the differences as serious, even if some content is in the frame. Since the brain can consider some image content as a standard for distributing the brightness of the scene, and usually the scenes are not as disconnected as PIP, but, on the contrary, have a smooth change in illumination. As will be shown, at first glance it is not easy to find out which association has good lightness. At least, this is not possible during the creation of separate videos, because, for example, the creator of the film does not know which local commercial will be inserted (for example, by a cable TV operator), or even which second image the viewer will choose from some image source, for example, in PIP, when watching a movie (possibly streamed by its home computer or central home content management device). We also note that the visual importance of some, on the contrary, relatively bright object depends, among other things, on its size. In the presence of two images, the device can make more reasonable decisions on integration. It may be necessary to reduce the maximum visual brightness (LMC), for example, so that the HDR content is more similar to the first image, if it is, for example, an SDR image. This may depend on looking at the additional properties of the brightness of the pixel or object (where the object is a properly grouped set of semantically related pixels). For example, if the lightest objects are lamps, it is possible to reduce the lightness of these lamps in the dynamic range of combining (CombRng) for a while. This will not be the most spectacular HDR, but you can still use the HDR image to render pleasantly light lamps, and especially for connected displays with lower peak light brightness (LM_MDR) than the combined image (LMC), which cannot be accurately rendered in any way light-coded lamps, i.e. codec brightness. Therefore, there is still a movie with a high dynamic range, but with a dynamic range that is more suitable for the merging situation.

In fact, two typical alternative scenarios can be considered. In general, it can be argued that the dynamic range of the union, i.e. at least its higher brightness (LMC) is set by the device based on the needs of, on the one hand, the input of at least two images, and, on the other hand, if something touching an already installed, dynamic range (at least at least peak lightness) of the intended display on which the image aggregation is to be visualized (i.e., HDR needs appropriately presented image content of a typical HDR display). Thus, some embodiments provide for determining CombRng on the basis of only two images, if it can be said that nothing or in many ways is expediently related to how they should be configured for use (but, in some embodiments, there may already be some typical use: for example, if reference HDR images, for example, PB_C = 5000 nits, are to be displayed, for the most part, on displays with lower dynamic range, and then it is assumed to be significantly scattered with SDR image material, you can consider the creation, for example, for compressing a video signal for transmission, combined or combined (on the receiving side, in the sense that brightness conversion of some image information may still be required) of images in a lower dynamic range than PB_C = 5000 nits, for example, 1.5 times greater than the expected peak lightness typical display on the consumption side, n For example, 1000 nits, therefore, the dynamic range of the pool will have PB_C, for example 1500 nits). In the first typical example, an HDR film is considered, i.e. with a large dynamic range of two input videos of sequential images, and the SDR content must be converted in brightness, or at least correctly annotated by placing its one or more source anchor values in the shared metadata so that it harmonizes harmoniously (or can be made harmonious at the final presentation venue) where the HDR film is the lead content. However, an alternative typical scenario may already pre-convert the HDR content to some lower dynamic range, which is more compatible with either the expected typical display rendering and / or SDR content insertion. If there is an actual display for visualization, for example, 750 nits PB_D, such a system can, for example, convert (or annotate with the brightness of the anchors to determine the brightness transforms) the brightness of both videos (which can happen in completely different places and times of creation, which is the power of a consistent anchors) to make them harmonious on the DR union, for example, PB_C 1500 or 2000 nits, or perhaps even PB_C 1000 nits. Then this combined or combined content can be easily reduced to the required PB_D = 750 nits, a given dynamic range of the display.

In particular, since the applicant has constructed a structure in which (various) final (graded) images of a different dynamic range may not be transmitted in fact as real encoded images (i.e. with pixel colors permanently and fixed in color components of usually images, compressed by DCT), but as functions for calculating various possible pixel luminosities of such a secondary image from any primary image, these functions can be redefined, and thus more complex definitions can be transmitted (for example, specifications on how to calculate image brightness for at least some special image objects in various situations, for example, different dynamic ranges of association). Classical encoding of an image or video signal simply encodes and transmits the image to be used, i.e. with its brightness of pixels, since they should be visualized (!), but the applicant can transmit a completely different image (i.e. with the necessary spatial structure and texture of objects, but not the correct colorimetry, in particular, until the correct brightness of objects or image pixels) , and then the prescription of the method of how to finally create the correct brightness for the final image, which in this application will be a combined image defined by all the brightnesses of its pixels on the dynamic range of brightness of the CombRng combination.

Similarly, there may be reasons, for example, to increase the minimum (LmiC) of the combining range compared to the minimum (LmiH) of one of the images, which is the image of the dynamic range (with the dynamic range Im2_Rng), for example, because with the combination of a relatively bright LDR image (for example, if it was not very precisely obscured) the global (or local with respect to at least some dark parts of the HDR image) lightness of the combined output image is such that the darker parts that come exclusively from the HDR image cannot be very well observed. Those. by converting them color to lighter black areas of CombRng, they can be properly clarified (which will occur due to the choice of the combining range, and it is not necessary to additionally perform intelligent processing of the HDR image by itself, i.e., to brighten it properly even by simple color conversion to CombRng).

Therefore, the reader can understand how, at various points in the image processing chain, for example, in some place where the content is set, where the final content is set, by devices usually controlled by the content providers, before entering the chain of the image consumption image part, or some image consumption place mixing two types of content only under the responsibility of the end user, for example, the end user using his remote control, or under the control of a computer), a suitable dynamic range of association can be set, either by a person, potentially semi-automatically, or automatically. In general, such a device will look at both the image content (i.e. HDR quality, i.e. which PB_C has this or that content, and the distribution of various brightness of objects or pixels in this dynamic range ending in PB_C, and what kind of light objects [for example, HDR effects] are present, and is it easy to distort their brightness without creating very unpleasant changes in the view), as well as the need for the final use of these images, i.e. usually visualization on the display (if, for example, images are to be submitted to the installed consumer base of the SDR display, then lower CombRng may be in order). In automatic implementation, the algorithm will use heuristic models that determine which methodology the combiner people will use (i.e., those who create merged images and / or metadata for them, for example, specifying the LMC value and at least one anchor point, which will be the starting point of the anchor of at least one of the two images or videos to be finally merged). On the dark side of CombRng, a decision will be made about which darkest colors will still be relatively visible, by simulating such visibility based on aspects such as, for example, looking from the brightest areas of the lighter image objects, the alleged light leakage from the display, and masking depending on the ambient light of darker colors due to typical reflections from the front plate of the display, etc. A new element in this model compared to the single display and the assumed typical (for example, home theater, with dimmed light sources) observation environment is the introduction of secondary image content. But, in general, since this CombRng establishment usually happens first (before determining the appropriate anchor), this assessment can be crude (for example, inaccurate checking where the SDR content is entered and whether the lightest part of the SDR image, even potentially lightened, is geometrically after the dark part of the film, therefore, of potentially significant masking, although some embodiments may also take this into account when setting the CombRng higher and lower brightness, for example, by direct or iterative determination, the latter iteratively check how good the brightness of the armature is, and then, respectively suitable CombRng limit the brightness).

Since in various applications a rough combination may already be suitable (for example, if the viewer of the film wants to finally enjoy his film, he should not interrupt or combine it with any additional images, but if he does, he must confirm that he will always be present some distortion in the colorimetry of this leading video content, even if it is only psycho-visual in understanding the impression of a complete image of the combined image, and without really changing the brightness of the film compared to what they were before the combination), it may already be suitable to quickly calculate some heuristically lower brightness LmiC, based on typical content. For example, if the histogram shows that most of the SDR content falls between 10 nits and 100 nits, with approximately half of the display area of the combined image occupied by SDR pixels with a brightness above 50 nits, the device may establish that it does not need areas of deeper black than , for example, 0.01 or even 0.1 nits for this combination (since the typical display of the heuristic algorithm decides that darker image structures will not be well observed under such conditions). What makes this setting the combining range so interesting is that (although the original creator of the HDR movie content could have made the movie with the given ultra-black sections down to 0.0001 nits, when watching the movie in a dark room, for example, on an OLED display, which can visualize areas of very deep black), the combiner (for example, STB, or even the TV itself) can now decide to lighten the darkest colors of the HDR film somewhat by determining the appropriate display of brightness set between the HDR film (or, in general, leading content) DR and the dynamic range of the combination, and in particular the bottom of these dynamic ranges, before placing the pixels of the HDR film in the combined image. On the bright side of the brightness range, algorithms usually include determining to what extent clipping or contrast compression is still suitable for the initial HDR image (which will differ, for example, for lamps that do not need to have a visualized internal structure, as opposed to sunlit clouds, where, ideally, there will be sufficient contrast in the final rendered image, and thus any encoding of the image for this [so that, ideally, the definition of the final image is quite simple based on this encoded received image, which in its encoded brightnesses already indicates, for the most part, what the final visualized brightnesses of all pixels should finally be], therefore, less compression for the brightest colors of the image, defining these bright areas with a large subset of luma codes, whether already in the combined image, or previously combining I have a given HDR image before it is actually converted to pixels of the merged image). Therefore, depending on the needs, for example, an inserted advertisement (which should not look unpleasantly dark), inserted (supposed by the creator of the film, for example, when he knows his film on a particular channel, therefore, this is a potentially re-defined definition of the film, adjacent commercials will be offered) it may be necessary to compress the clouds in a slightly smaller portion of the light brightness not too high above the lightest brightness of the SDR image, at least how this SDR image is converted to combine. In order to still be able to customize this image in the most suitable way (and in general, i.e. at least some part of the SDR image and some important part or usually an object of the HDR image) and for various input, for example, SDR commercials, ideally, an HDR image indicates several important brightnesses for these clouds, for example 3, between which the combining device can switch, usually in the dark side, for different cloud areas (thereby also reducing contrast, for example, the dark gray part of a thundercloud). But this is when the brightness of the anchors comes into play, as explained below, with the present part of the principle, which simply explains that the range of the combination should not end with the LMC value so low that the appearance of the leading image begins to collapse (for example, if the film is dedicated to thunderclouds, which usually have nice different areas of gray, if some parts of the clouds are highly illuminated, for example, the edges of the clouds illuminated by the sun, and the lower edges of the clouds are not illuminated by the sun, but only by the light of the environment, therefore they may be significantly darker than the basic aspect of this film will be lost , if CombRng is such that these clouds need to be compressed in the higher CombRng sub-band, which is only, for example, 10 lumens, because even with a strong functional stretching in brightness as an attempt at display correction, this will never give a good visualization of the HDR of these clouds, what they deserve). But otherwise, the combining device may decide to carry out some qualitative reduction in the brightness and contrast of the HDR image, for harmonization (especially when from less demanding HDR scenes, such as a football match, some of which is in the sun, and some in the shade ) HDR images with images of a typical or actual lower dynamic range with which it will be combined.

Of course, although the dynamic range (at least a maximum, and possibly also a precisely defined nonzero minimum) is a very important (initial) definition of a property for the final images of a view, it still does not completely finalize how the lightness of objects or pixels will be located in this range. Simple methods of image processing really provide that you only need to know the upper and lower limits of a certain dynamic range, and then produce some display (which is often not everything, which is much more intelligent than just linear compression, i.e. displaying the white color of the input content the white color of the final range, and the black color of the input content to the black color of the final range). And which can even change significantly, especially when the use of the full potential of large dynamic ranges begins, with very critical scenes, both with respect to the brightnesses to be represented, for example, at the same time many areas of deep black in a cave, and many sunlit colors observed through a small crack leading outward, and even an object that needs critical control of its intra-object contrast, for example, a person observed through fog. But, as mentioned above, the inventor believes that for good control of the types of HDR images, and in particular, the view of the combined images, good control of all or at least many, or at least the most critical brightnesses between outside CombRng, therefore, something else is also necessary in order to have good HDR image processing systems, in particular, devices that can, without additional difficult participation by a person, create properly looking, finally combined images, for any situation that may arise in practice ( since it’s not possible to sell the same Hollywood director in every STB or TV so that he determines the appropriate final combined brightness for the viewer). It requires a quick mechanism that can still be nicely adapted by the creators of the content, no matter how critical or complex their needs may be, to obtain a good result of an acceptable high quality, at least in the most practical cases of combining image content.

Therefore, the second component to control the association sufficiently is that the device determines the brightness (anc) of the armature. These may be different values, but usually the semantically relevant brightness of a semantically important object. For example, it can be, in particular, a typical important object, illuminated in a certain way, giving brightness in the sub-band of the full range of the HDR image. This will be the brightness, which is supposed to be good, in a certain range of association, to determine other brightnesses around it (i.e., this brightness determines whether all the brightnesses have a suitable brightness, not too dark and not too light). In other words, if the brightness of the armature is correctly visualized, no matter how correct it is for a specific joining range, other brightnesses will not be bad either (and in Fig. 16 it will be illustrated how to control them with the help of additional technical components to further improve them, according to the desire of the creating artist).

In FIG. Figure 15 shows a typical HDR scene, the two sections of which have significantly different lighting, namely, a barn (Ins) with dim lighting (see the geometry of the scene in Fig. 15a), and a sunny portion of the outer space (Outs) with much brighter lighting. In order for the content calibrator to create a properly looking HDR image for a typical TV viewing (dim environment) from the physical brightnesses of the scene, it may be advisable to calibrate the brightness of the interior space, as in the SDR image, i.e. up to 100 nits for the brightest object in the interior. The brightness of the external space in the real world will be approximately 100 times lighter, up to 10,000 nits, but will be perceived as too light for television visualization (even if the viewer usually has such a modern display with PB_D 10,000 nits). Therefore, the calibrator can, for example, make this scene with lighter HDR brightnesses of the external space (in the sub brightness range of pixels of the external SDROU space) of the reference range HDR (DRH) up to 1200 nits for the brightest object (with the possible exception of some small specular reflections on metal), and with a typical average brightness of the outer space of 250 nits.

The power due to the brightness of the armature is illustrated in FIG. 15c. In this example, we do this simply, and assume that darker brightnesses, sub-brightness ranges of a pixel of the SDRIN internal space, can be visualized with the same brightness in all situations (with the formation of a stable, constant set of brightness in the union range, which, of course, does not always have place; for example, if the SDR commercial contains many bright colors, the device can use the bottom anchor point AncS_ins, which characterizes the internal pixels of the reference material of the original HDR image, in fact for some lifting of the darkest pixels, but we will explain the basic principles of our embodiments using an example scenario, where a single anchor point will be used for the correct luminance placement of the brightness of the pixels of at least this first HDR reference image in the union range (that is, with the second AncS_outs anchor point from a set of external, bright pixels). although any brightness can be used of a sample object, for example, with a typical light reflection spectrum, leading to a typical appearance of the gray value, it is usually nice to use some average light position of the anchor in the histogram column of all the brightnesses of the images corresponding to a portion of an acceptable similar illumination when it is desirable to determine the pixel brightness around such anc value . The reader may imagine that below, according to FIG. 16, because if for any reason the device wants to “re-illuminate” such a section, for example, to increase the illumination, then the brightness around it can be brightened continuously with the anchor point, just as real reflecting objects begin to glow more if the incident light increases. Note that for automatic determinations when calculating the illustrative value of anc one should be careful that, for example, the very bright brightness of the cloud-lit edges of the sun does not deviate too much from the result of an acceptable average, therefore, usually, if people can determine and transmit the brightness of the anchor together with encoded images, this may give the best results.

Therefore, the reader must understand that there are two kinds of anc values, namely, first of all, the result (i.e., the positions in the union range where the brightness of the images that correspond to the same semantic value of the definition of anc should finally go, i.e. are close to such a brightness value anc, but in the dynamic range (differently illuminated and differently encoded) of the source, when rendering, or written to the combined image), and the second (corresponding) value of ancS for all or at least most input images (in the case where annotated anc is missing in some images, the device should evaluate some value, for example, the SDR value, which should not be too unreasonable for the most typical SDR content). If such anc value is set (even semantically different from one of the union, but which may be associated with one of the union, for example, by multiplying by 4), then the input content can be harmonized with the structure of the union, and thus with all other content Images.

The reader can see how the value of anc (i.e., the resulting value anc of the union of the images, and the range that is prepared before the two images are actually mixed) is coordinated with a certain dynamic range of brightness of the union, i.e. usually at least its peak lightness, for example, to combine high-quality HDR PB_CS = 2000 nits. If the dynamic range setting unit (302) selects a high quality combining range (CmbRngSup), a high quality dynamic range is obtained that is close to the dynamic range of the HDR reference content. Therefore, it makes sense to also place the resulting value anc (of the semantic type:, position, where, on average, light, the second column, the colors of the outer space should fall into the combined image ʺ) at 250 nits (where the device follows the typical rules for visualizing the light area in television films, which also was the basis for selecting (approximately) 250 nits of the reference calibrator for the HDR reference image). This merging situation will lead to the fact that most of the relevant HDR brightnesses, also pixels of the external space, will be nicely visualized as assigned by the content creator, i.e. with the same luminances that are encoded in the HDR reference input image, and only, for example, specular reflections must be slightly damped to fall below 2000 nits. This is an example of the aforementioned first type of scenario: the aggregation range, for the most part, corresponds to the original HDR reference video content, which is leading and essentially rendered without distortion of brightness, and then you just need to harmonize the SDR content (which will be based on the corresponding 1 or more anc values for SDR image).

For the lower quality combining range (ComRngInf), the device may already take into account the necessary quality reductions, also for the HDR reference image. In this case, the device may need to lower the position of the Anc_Outs value, which can usually be done, for example, by heuristically assessing the needs for contrasts between sections and inside the section (even without reliance on specific images and scenes, it is already possible to determine some global values for good positions of the brightness of the anchors, which should work in concert for the upcoming various HDR movie frames). The second resulting position anc Anc_Outs2, should, depending on which value of the range remains above 100 nits or, in other words, above lower SDRIN brightnesses (for example, in points, 10 times more than 3 points to accommodate all HDR effects, whatever they were not, i.e. reflecting objects in bright sunlight, as well as lamps or specular reflections, and usually also taking into account the values that consumers would like to see for good quality HDR images, i.e. without large areas of objects that are too light or too dull) to be located in the brightness position in CombRng so that the light section or parts thereof sufficiently exceed 100 nits (to have a really sunny appearance, for example, a factor of 5-10 can be a good value, and this can be formulated as division of two values anc), and at the same time there is enough space for even brighter HDR effects, for example, small areas of specular reflection on objects, language of flames or explosions or laser beams, and all that a film can actually contain, that it can be graduated and visualized near this upper limit of PB_C_Inf 1000 nits. Some embodiments of automatic methods may work by calculating error estimates. For example, you can analyze textures in areas with complexity indicating the need for higher or lower contrast within the area.

Therefore, if only two images are known, for example, a movie created by a filmmaker (annotating with at least one anc value, and possibly already displaying brightness in a usually suitable CombRng) and a typical middle variant of the estimated SDR content, then the device can calculate at least an error for compressing an HDR image with a distortion of brightness, and distortion or disharmony of an SDR image when it is lightened again before combining, as shown in FIG. 17. Such distortion errors can be calculated as for scenarios of a typical average, for example, based on the probability of pixel brightness for one or more typical HDR scenes, evaluating how such a scene will need to be distorted (for the most part, in the brightest areas of the image), when reduced to CombRng with PB_Co1, PB_Co2, etc. should occur, and / or for actual images, i.e. taking into account the remaining pain, if the actual (optimal) display of brightness is used to display the brightness of all objects or sections from the natural representation of the image and its dynamic range, to the repainted image in the range of the combination. In FIG. 17 schematically explains how the error E_cmp can be associated with the compression of at least one part or object of the image, for example, cloud structures. Various practical embodiments representing the behavior of a human calibrator can, for example, extract texture, and by calculating texture measures and other spatial and / or statistical measures to locally estimate the area, how problematic it would be to downgrade to a smaller sub-range. For example, for clouds, the texture analyzer will detect the absence of sharp boundaries, and that the understanding of the cloud will mainly occur by (smoothly) distributing (many) gray values, and not by typical aspects of the geometric shape (for example, a linear pattern of the texture of a Scottish skirt allows more compression or contouring in smaller luma codes). Those. if a lot of precisely graded smoothly distributed gray values are mapped to a more limited set of irregular lightnesses, we can quickly assume that there will be some error that will dissuade the device from doing this too much, in particular if the leading image is of critical colorimetric importance, because, for example, the consumer should be able to enjoy the precisely graded quality of the film’s artistic color, which should not be reduced to often extreme and vulgar rough grading, for example, of some commercials (in addition to the basic principle of ensuring the brightness of anchors, some embodiments will allow the content creator to indicate with the help of additional metadata how much a large distortion is permissible between or around the anchors, for example, that the range between AncS_outs / 10 and AncS_outs * 10 is preferably not compress and stretch with a factor, for example, 2 or 3).

In FIG. 17 more specifically, an explanatory example of how errors can be formulated is shown to ensure brightness harmonization according to an embodiment based on such error equations. The HDR shed image and the SDR commercial for a soft drink are to be mixed, and in addition to their ranges, we show histograms of the brightness distributions of the scene images (hist_HDR respectively the histogram of the images of the low dynamic range commercial hist_SDR, with in the horizontal direction the number of pixels (N) Li in the total number image pixels that have a brightness equal to a certain value of Li). Compressing an HDR image in the lower CombRng subband to PB_comb, for example 1000 nits, results in an E_cmp error. This error can be balanced, for example, by an error to lighten the content of the SDR, or by an error corresponding to the incompatibility of this CombRng with the typical ranges of the intended displays. For example, assuming that everyone in the world will in any case have a display with a PB_D of 1000 nits, this E-cmp error can be normalized to zero, since this modification of the reference content should occur in any case. But still CombRng, for example, 1500 nits can be better, i.e. have a negative error indicating a higher quality compared to combining 1000 nits, with a very high weighting of the visual quality of the movie content. In addition to the errors that can be calculated to reject the luminosities of the SDR content, which may be less important (since, on the one hand, the SDR paradigm assumes that viewers are able to quickly adapt to different luminosities, but, on the other hand, the luminosity of a potentially “incorrectly” rendered image SDR, for example, defined by many experienced or inexperienced viewers as too light when rendering on its own HDR monitor with the lightest pixel, for example, 1000 nits, for this particular image, is now determined not by itself, but relative to the reference HDR image), you can evaluate the following error. The SDR luminance mismatch error (E_MM) can be determined, and this can be carried out, for example, as simply as the difference between the value of Anc_HDR CombRng, i.e. also, to which, ideally, the original image is displayed with the HDR image anc value (AncS_outs), and is actually displayed in the brightness (D_AncSDR) of the SDR brightness of the original AncB_SDR anchor. Those. the best harmonization in this example will correspond to the heuristic when D_AncSDR is equal to Anc_HDR. The reason this works is because this type of HDR image contains a fairly large pixel area of bright external space (for which reason its content creator set a higher anchor brightness, AncS_outs). When the viewer is already accustomed to lighter brightness when viewing this HDR movie scene, introducing approximately the same lighter pixels for (for example, PIP) an SDR commercial is much less unpleasant than when the user observes only dark brightness, for example, 5 minutes in a cave or basement ( and in this case, the introduction of such a bright commercial will probably surprise the viewer, or at least significantly prevent him from enjoying the rest of the film’s scenes, at least colorimetrically). This system works especially well if the SDR content creator uses a specific type of its anchor, namely, that can be so simple that it uses only two types: lightness above average and lightness below average. Even regardless of how luminances are actually distributed over the 0.1-100 nits SDR brightness range, the creator can indicate what kind of SDR image it is, and thus how it needs to be visualized, in particular according to this invention, how it can be harmonized with various capabilities of the sub-ranges of the HDR image. For example, an SDR image, being the normal content of a dark or dark scene (вет lightness below average ’), or it may be lumines that supposedly represent the sunny scene. It is assumed that the creator of the commercial wants to show it as a “sunny” scene, the colors of which look bright and bright, which are exactly bright and bright in combination, of course also depending on other image colors.

Now the reader can already understand the brightness power of the anchor in coordinating the images to be combined. Even in such a simple system, based on the fact that the creator of the content of the commercial is interested in making his image visualized “sufficiently light” (but probably not interested in annoying the viewer with visualizations of his part of the full image content, which are too light), but, of course , also (in a larger, the same, or perhaps slightly smaller number of scenarios) the desired image quality is also important the creator of the HDR film, the combination can already work directly acceptable when the HDR movie has only “darker” content in the case, the creator of the HDR content includes only lower AncS_ins. Then the device understands that at least this scene in the film does not have (too many, important) highlights, and thus the viewer will adapt to darker content, for example, in this non-restrictive simple explanatory example, only in the subband SDR SDRIN. Thus, the device can create a harmonious union if in this situation it maps AncB_SDR (despite the fact that it is assumed to be “light SDR content”) to or near the lower Anc_ins. By proximity we mean a fixed in the device or adaptable usually multiplicative fraction of the value of anc higher or lower than the value of anc, for example, from 1/3 or 1/2 to, respectively, 2 or 3 anc. The degree of proximity can be determined (either at the time the device performs the union, or on the side of creation, for example, with clearly transmitted brightness proximity boundaries) based on various aspects of the implementation, such as, for example, allowable brightness errors of objects during display, type difference or variability of anchors, etc. But obviously, if AncB_SDR is, for example, 20% 100 nits, even if it is displayed in 3x 25 nits (i.e., the position of proximity of the upper limit of Anc_ins), then the union is much more harmonious (since perfectionism usually does not need to be achieved, but a pragmatically acceptable working system, balanced between the accuracy of the appearance and the complexity of implementing the system in practice) than with linear stretching, in which the SDR content is directly displayed from white to white, i.e. from 100 nits to 1000 nits, which will render an average AncB_SDR of 200 nits (!), which probably has all the SDR image objects that are too light to be attractive (note that, in principle, anc can be set even lower than the actual histogram brightness, for example, the mode i.e. the most common value of the histogram brightness, but in any case, the actual brightness will be visualized similarly overly bright).

Returning to FIG. 15b, we show that it is interesting how the anchors, and their placement in different relative positions in any CombRng, will relate to different brightness displays (influence on them). The TL_CS function is a brightness display for displaying the initial brightnesses of the HDR reference image in brightness in CombRng (i.e., the readiness to perform image merging, be it pixel replacement, alpha blending, etc.). The applicant has found it very useful to carry out brightness transformations in a relative coordinate system, i.e. input brightness and output brightness or lumens ending in 1.0 (in the case when lumens are used as a vertical coordinate, the display is specified with OETF, for example, some fixed OETF definition like SMPTE 2084, or Rec. 709 for backward compatible image transmissions) . For lumas, this is very useful, since it bypasses the consideration of how many bits the vertical axis should be quantized (1.0 is the highest value of the lumas code, i.e. the lightest color to be encoded). For luminance axes (for example, input), one can consider whether the remainder of the relative brightness processing is, but in any case, the necessary curves can be set for any necessary transformation of the image into any dynamic range (since this version can always be mathematically equated with its corresponding absolute brightness structure ) We see that if we want to keep the absolute brightness of darker pixels identical, in a relative coordinate system with a lower peak lightness, this corresponds to raising this part of the TL_CS brightness display function above the diagonal, and we can clearly see how the remaining higher CombRng subband leads to some compression that is not too excessive for the high-quality pooling range. But for a lower quality combining range, the compression of lighter HDR objects should be more substantial, since it is also observed from the form of the corresponding brightness display function to this lower combining range, namely TL_CI. In any case, this situation of peak lightness of the merging range, and a good anchor point, explains that, in general, when any HDR frame occurs, some compression of light colors will be necessary, which will be archetypal approximations of this form (specific possibilities of embodiments are described in detail below ) From the combination of the two functions, you can see that if you need to convert relatively from higher to lower CombRng, you will need to relatively lighten the darker pixels, but which visually primarily correspond to the compression of the lighter portions of the HDR (a relative percentage sub-range, which is, in addition to the full available range, good measure of coding or rendering quality of a part of the image). In any case, in this way, the applicant defines his brightness transformations in his most typical actual embodiments, and in particular, a very useful property, which will be explained in FIG. sixteen.

Methods of processing an HDR video signal (actually only encoding) like HDR10, which consider only a fixed representation of HDR images, do not even need to transmit one fixed brightness conversion function (since they can put the entire distribution of the brightness of an HDR object in image brightness, which does not require OETF transmission lum, if this OETF is selected fixed). Technologies that have a simplified view of the distribution of brightness of an HDR object, i.e. its ability to re-calibrate to other dynamic ranges, such as the mixed BBC loggamma, can use a fixed function to transmit, for example, as a LUT.

The applicant wanted both the creator of the content and the user (whether it is the TV manufacturer and / or the end viewer) to be able to release the full HDR quality (hierarchically), i.e. build a system in which the content creator can indicate his desire for HDR image content step by step (from coarse to accurate, depending on the degree of criticality, or the time and budget spent), and so that the video consumption side can also decide on the desired accuracy for following these specifications, or whether the final system wants to give the images some microaromatics of color processing, for example, deviating slightly from the contrasts of objects specified in the fine adjustment function (s).

In FIG. Figure 16 shows an example of the power of the anchors, when combined with one of the preferred systems of the applicants, allowing, for example, usually a human calibrator to hierarchically set his needs for brightness gradation, with a set of sequential functions.

It will consider that the input and output ranges are quantized perceptually uniformly (approximately logarithmically). This will create a set of visually equidistant levels of lightness, for example, ultra-dark, ultra-dark, very dark, dark, etc., up to extremely light. One can imagine that a calibrator can set his gaze on the image by roughly equating the brightness of objects with one of these levels, for example, a good-quality HDR image should have a lamp brightness in the sub-range of extremely bright brightness, for example, between 90% and 100% PB_C. Of course, the lower dynamic range of the output signal, in this case, the luminance pooling range (L_Cmb_out) may not stretch to the end to a level that looks like extremely bright luminances, but then the calibrator will resort to displaying the pixel luminances of this lamp to the highest available level, for example very bright. This will form part of the form of the brightness display function, and the reader can understand how a calibrator (or also an automatic system using heuristic analysis of the image) can get some full form of the brightness display function.

Now just specifying two such axes will be a typical at first glance (albeit fairly direct) way of displaying the brightnesses specified in one dynamic range into output brightnesses in the second, for example, a smaller output dynamic range (which will be expressed by the function that is the diagonal in this graph) . Indeed, “half acceptable” output brightnesses will be formed, since this strategy will display the white color of the white sections of any source image in the lightest possible (encoded or visualized) color of the output dynamic range, and black in black, which, due to the logarithmic definition, would also be acceptable distributed gray values between black and white from the input to the output image.

But this gives rather nondescript images with irregular lightness and contrast, not to mention that it allows the artist to nicely adapt the artistic needs to the composition of any particular HDR scene (although the lightest and darkest pixel of our barn scene, for example, when encoding PB_C = 5000 nit can have the same meaning as in an image, for example, a table lamp in a dark room with several objects in the background, of course the composition and semantics of the images for these two images will be very different, leading to very different needs regarding grading and re-gradation of different brightnesses objects or pixels).

Therefore, the applicant is already for the sole use of the image, i.e. either reconstructing the approximate initial reference HDR image, for example, 5,000 nits, or adjusting the display to obtain, therefore, an optimal MDR image for driving, for example, a display with PB_D 750 nits, a luminance display detection system has been invented that is capable of setting at least , two consecutive functions. First of all, the coarse-grading function F_CRS_MDR globally re-allocates the luminosities of all perceptual subbands so that they have the best initial appearance for the real HDR image and information about it. We see some roughly r-shaped curve for which the brightest half of the brightness of the HDR pixels of the scene is compressed in a small subrange of the output brightness, for example, the brightness of the SDR images to excite the SDR display, or, for example, to better match the principles of combining, MDR 1000 images nits when the HDR reference image has a PB_C of 5000 nits. Depending on the needs of the scene, the graduator can adapt this form. For example, if there are only some mirror portions of a pair of pixels, or lighting lamps in this higher half of the brightness, the calibrator can, in principle, even set a function that cuts off to 1.0 for the higher half of the input brightness, which will not significantly degrade visual quality of this particular scene. But if the higher half contains sunlit clouds, even a slight decrease in the local slope of the curve can lead to significant quality errors. We should not forget that we are talking about relative brightness axes, therefore, when displaying, for example, the SDR range, there may not be many luma codes and corresponding brightnesses for correct visualization of very dark brightnesses, average brightnesses and very bright brightnesses simultaneously (considering for example, 3 differently lit areas of the image, for example, with a dark kitchen in 3D areas closer to the viewer where the black person is hiding, with the middle room, which is usually lit, and again, with solar objects observed through the windows, as schematically shown in FIG. .16b; and another typical example of two sections is shown in Fig. 16c, with a light showcase on the street at night, i.e. other semantics than in the barn example, but with subbands that can be processed in the same way or not, according to the decision of the content creator) .

FIG. 16c may be “simple” (in terms of the complexity of the HDR effects) to be able to sufficiently calibrate, and by adjusting the grayscale function of the brightness display, specifying only the coarse grading function F_CRS_MDR.

But for a more complex scene of three sections (R1 indicates the bright outside space of the street in daylight with objects like houses seen through the windows, the middle section of R2 is lit normally or dimly, i.e. with typical SDR brightnesses, and in R3 the light sources are turned off (i.e., with dark luminances) it may be more difficult to acceptably display all luminance subbands and the corresponding lightness of objects in small subbands, especially lower dynamic ranges (e.g., SDR range). In this case, the calibrator may find it optimal for using an additional curve, namely, the fine adjustment curve F_FINGR, which should be applied to the relative brightnesses obtained as a result of a rough calibration, i.e. virtually equivalent to a rough curve deviating from this shown shape. But such a strategy can also be applied to “simpler” HDR scenes, for example, illuminated from inside a window display at night in FIG. 16c. To make the display case and its object sufficiently expressive contrasting and bright, so that they look really luminous (consistent, but when visualizing this single image by itself, for example, on the HDR 5000 nits display, on the HDR 1000 nits display, and at the same time it is acceptable to approximate and pleasant enough as limited display capabilities allow, on the SDR display with PB_D 100 nits), and also make the relatively dark (at least in the HDR reference grading) pixels of the night street object sufficiently bright and visible, the grader can create the F_CRS_MDR function, for example, relatively flat on the higher end. This can provide an acceptable view of the shop window and the dark street, i.e. already acceptable general view of the image. But a display case may contain specific objects that do not come out in close connection with any aspect, for example, their brightness, contrast, or color, for example, a flame in a display case, or, possibly, some highly illuminated part of a white neck or mannequin shirt, etc. d. Thus, the calibrator can set a correction curve (i.e., an exact calibration curve F_FINGR) that, in some part of the brightness range, for example, corresponding to the brightness of this shirt, for example, reduces lightness, for example, to obtain greater intra-object contrast or color saturation or any function , which is realized as an effect (there may even be various spatially localized brightness displays that are designed to avoid changing other objects with similar brightnesses in other parts of the scene, but which demonstrate that the additional complexity of our system is beyond the scope of understanding of this application).

therefore, it is interesting that the crude function quickly makes all the brightness of the objects already approximately correct, with the easy action of the calibrator (for example, in one of our embodiments, he just needs to turn two knobs that determine the slopes of the upper part of the light colors starting with 1.0 and the lower part of the function starting at 0.0, and then our encoder automatically determines a smooth rough grading curve that can then be used by various HDR image combining devices of the present invention and the application). But then “real calibration” can begin, by fine-tuning the brightness of the various pixels of the image objects, and, consequently, the corresponding intra- and inter-object contrasts, allowing the calibrator to determine any shape of the fine adjustment curve as desired.

A normal decoder, or display tuner, will simply use these two curves as is. Moving in the coding direction (in the case where a backward compatible system is used, which transfers the HDR image information in fact, like traditional SDR images displayed on the display), first applying the coarse and then fine adjustment function creates an optimally looking SDR image corresponding to this HDR reference graded image a particular HDR scene, which was the encoder input. On the decoding side, in order to reconstruct the approximation of the HDR reference image from the received SDR image (s), the inverse functions of this exact and coarse calibration function are used, since this was actually the relationship between these two optimal functions that were set optimal in the data typical visualization situations. Those. with an HDR display, HDR image visualization, and with an SDR display viewed under typical viewing conditions, visualization of SDR images (and with a PB_D display that deviates significantly from these two PB_C graded images, our display setting is used to create an intermediate image dynamic range, appropriately corresponding to the reference HDR image, i.e., sufficiently approximating its appearance brightness of the object, at least as far as lower display capabilities allow).

But now we have a situation in which we need to combine content, i.e. usually it may be necessary to re-modify the brightness of at least one and possibly all of the images to make them harmonious. Separation of calibration in such two curves should be very powerful, since this allows you to separate the needs associated with the object, for example, sufficient local contrast, from the total compression of various sub-bands in the available (even if optimized for combining) output dynamic range.

Therefore, the coarse function given by the brightness of the anchor, even if it consists of linear segments, can now be used as an alternative to coarse sub-bands, since this is obviously a better grading for the mixture than the original coarse function F_CRS_MDR for a single visualization, for example, an HDR movie, in the dynamic range with substantially the same peak lightness as the selected combining range. Now, various embodiments may correlate the shapes of the various sub-segments of the F_CRS_MDR curve, but this is usually not required. It is important that the combiner can apply fine adjustment features to various newly highlighted optimized levels (i.e., by promoting various semantic sub-sections of the HDR reference image, such as clouds, or the interior of the cave into different CombRng subbands), while essentially maintaining accurate graduation curve shape. Thus, it is possible to shift the deviation curve to a new position (along the optimal rough curve to display the input HDR brightness in the joining range, that is, F_Cmb), realizing a new, now combined optimally fine adjustment curve F_FINCMB. This can be done, for example, simply by equating the reference brightness, i.e. shifting on vertical lines, the multiplicative values of the deviation.

Now the reader should begin to understand that such anchor systems are very powerful for harmonizing different content, even if it is difficult with respect to the desired brightness gradation, but at the same time in a relatively simple, therefore, feasible way.

Now, first, we describe a simpler embodiment in which the device can independently determine such brightness of the armature in the HDR image, and, accordingly, in the combining range. Finally, it is important to have the brightness of the anchor in CombRng, but this can be easily equated in some embodiments where it is in the HDR image (and less easily in other embodiments, in which case the creator of the HDR image content can explicitly code it together as L_S2A1 ) For example, we will explain the principles with an important example of having a face (of course, people are important, so in many images or videos there will be at least one actor or presenter, etc., and they can usually be reasonably well covered by the content creator, although, in the LDR area, it is never exactly specified what will be acceptable well, as well as what is really necessary). But a qualified reader should understand that our principles regarding the brightness (s) of the anchor are universal, so you can have other brightness of the anchors for other images that may not even have a face, for example, the brightness of medium gray. The reader must understand that in the LDR era, the medium gray was unique and very accurate (the coding average, which is usually visualized as an averaged looking gray, and in the initial scene of reflectivity, the object corresponds to roughly 18% white, or incoming lighting, which will also be a consequence of that visualization of relative white on white corresponds to approximately 18% PB_D), but it is not completely unique to HDR (since there may be a medium gray reflecting object in a relatively dimly lit image, or even a dark portion of the image, and the same gray object may be located in relatively bright illuminated image or area, in particular, if you want to again force the use of HDR content in a direct LDR jacket, very accurately producing lighting and capture, but at the same time generously illuminate scenes, generously artistically graduate content, etc.).

It is beneficial if some of these brightnesses of the anchors are standardized (so that each device quickly understands that each particular brightness situation of any input image), but even if the first image is set with a different type of anchor than the second image, or the image anchors are different (if only one) anchor, which the device considers optimal for setting CombRng, then these various specifications can still be roughly connected. For example, if black is not precisely defined (and, of course, just like in HDR, there may be different white areas and light colors, there may be different black areas), the device may assume, depending on the situation (i.e. PB_D or other features display, and possibly the observation environment), which is supposed to be visualized, for example, at 2.5 points, or 4 points below what the device considers its main medium gray (whether this is the first brightness of the anc anchor, or auxiliary secondary). Black areas can be described (i.e., encoded video signal) and processed, for example, visualized, with less or higher accuracy. If the content creator takes care of what happens in the black areas, for example, because there is some critical action of the criminal hiding in the bushes, which should be “half perceived” i.e. neither too noticeable nor missed by the viewer, he must annotate the image or video with additional descriptive metadata for this sub-band or color processing mode. And, in particular, the creator must specify at least one brightness of the anchor for black sections. If he doesn’t or doesn’t really care, for example, since in the real HDR scene there is only some background texture, for example, photographs in a dark basement that can be visualized with more or less the same semantic quality, whether they are good or not very well observed, then the creator can rely on typical optimizations of the receiving side or the intermediate side of the black areas where the imaging device can visualize them with different brightness levels, provided that it still looks acceptable black (i.e. deep black, good black or milky black).

In a simple example that allows the user to easily grasp the main points of our invention, the brightness (anc) of the anchor in the dynamic range (CombRng) of the brightness of the union indicates that the brightness of the face must be in the combined (output) image to make it look suitable. This, of course, will depend on whether the face is well lit, highly lit, or shaded. This allows you to avoid that the actor or person, for example, in a commercial, or BD comments, or talking on a videophone, etc., was much brighter than in the film. Of course, the device does not have to exactly align the two brightnesses of the faces, but it must visualize one with a brightness shift (for example, if the actor in the leading HDR film is deliberately made very dark, but the rest of the HDR film is (significantly) lighter, the commercial does not have to be dampened up to brightness features of the actor of this film; note that some embodiments of the combining device may additionally make decisions using heuristics, for example, how long the dark scene of the film lasts, for example, marking the elapsed time, since the previous significantly different brightness anc or type, for example, the scene in in daylight, and some embodiments may even annotate the brightness of anc with additional metadata, for example, anc at the beginning of a movie scene having a second data element indicating how long the dark scene will last, but we will describe the important points of the embodiments discussed here, based on that some creators of the conte They won’t want to have difficulty creating a large number of annotations, and will simply perform a simple selection action somewhere in the image, or in its other representation, for example, a histogram, using a pointer to set the real value of anc, which then of course can show in user interfaces, for example, showing all the brightness in the image in the range or near the value of anc in the red pseudo-color, as a check of a person's choice of the value of anc).

Thus, the device knows where the good visualization brightness of the face (anc in this explanatory example), depending on the specifics of the situation, should actually be (suppose at the moment a Caucasian face with spectral reflectivity averaged over wavelengths of about 36%, and, of course , other skin types, for example, 5% reflectance, will fall, in this local light, to the corresponding brightness positions CombRng). The reader may not go into further details and more easily penetrate into various key aspects based on the fact that CombRng in this explanation is precisely the dynamic range of the HDR image (Im2_Rng in Fig. 4), and the device knows where the most important face brightness is located, this range. As explained above, various alternative definitions of the optimal CombRng may occur in various embodiments of the device, which depend on at least some of, on the one hand, the brightness characteristics (the encoded range of brightness, the content is actually in the range, etc.) input images, as far as they are already known or amenable to assessment, or, on the other hand, the actually established or intended use of the image content, for example, the brightness of the display on which the content is to be visualized in a surveillance environment (i.e., in fact, a typical appearance of universal, medium or verification images, and the specific images present). This is already difficult, since it’s not natural for people to have the same color or brightness (not to mention that in some films faces can be painted in blue, etc.). when studying different faces even in LDR images, one can see that they can contain lumas of pixels even to the limits of the LDR range in principle (i.e., in very contrast capture, or grading, at least some pixels of the darker part of the face are close to zero, for example, in capturing backlight, and cutting off selections up to 255; sometimes, for example, with consumer content, you can even see half of the face eaten off as clipped white), although many faces have lower contrasts between the usually lit and shadow area, i.e. . are more normal. Here you can divide the content into well-lit images and specially-lit images, for example, in real-life field shooting (currently even include images made by non-professionals, for example, in a news program, and faces can really be poorly lit in the night scene) . We can say that in such difficult situations it would be very good to refuse accurate visualization or combination, but, on the other hand, it can also be said that the more abnormal (deviating from the practice of good capture) is some image content, which is the frame in the first place , the more you can benefit from its subsequent regularization, for example, using the appropriate anc value (for example, if the operator is in an unfavorable situation, for example, cannot illuminate a person who for one reason or another should be in the darker part of the scene , and it turns out that he has to use a camera with less DR capabilities than might be required to optimize his image frame imperfectly, but then, at least using the anc mechanism, he can quickly annotate (exotic) features of this image, for example, as simple as using a stylus on the display of your camera, or on your a portable computing device prior to the transfer of content, for example, to a TV program production center, etc.). It is assumed that with the advent of better cameras and other image processing devices, for example, cameras that can capture dark night scenes, as if they were day scenes, in the distant future, the complexity of image processing (HDR) will only increase, only because then there will be different standards and levels of production, for example, also some people still delivering poor quality news content using old mobile phones. Therefore, the best solution would be to create a technology that can handle all these situations nicely. For example, a movie about bandits can have poor lighting, for example, cinematic black, with contrasting faces, and strong discharge, for example, in hair or on a bald face. Talk shows can be less contrasted, sometimes with shadows almost invisible to the amateur, and a woman can be illuminated with more attractive, more uniform lighting. It should also be noted that this, for example, is not the initial ratio of face illumination 4: 1 (the lightest to the darkest half of the face) in the scene when capturing, which is important, but, on the contrary, in the encoded image (how much the contrast is standard graduated). This contrast can usually be subject to some display of the primary brightness in the luma in the camera, and if the image is obtained by scanning the image on a film or film, the final contrasts may depend on aspects such as development, pushing, etc.

We have identified this aspect as a range of facial colors, for example, in the illustrative example in FIG. The 4 colors of the faces in the LDR image were softly lit (hence, the small range of brightness R_f), and the faces in the HDR image were more rudely lit because it was, for example, a horror movie, resulting in a larger range of R_f2; for example, a person can walk along a corridor illuminated by rare point sources of light, which makes his face darker or brighter, around a certain average value). It also matters how, geometrically, the face is illuminated. If, for example, there is a small white spot on the bald head and the picture is incorrect, too light, mixing with another HDR film in CombRng, you can hear a remark that it looks like a lamp sticking out of the head (and the viewer may get a false impression, when such a person quickly flashes through the image, and potentially he loses part of the story, or at least is distracted by the wrong part of the image). But a small spot of a couple of pixels is not necessarily the main problem. This can usually be ignored to determine a typical complexion. When rendering on HDR displays, it may seem that a person is illuminated by an extremely bright lamp, but at least the spot is small (for example, a pair of pixels). If the same thing happens with his whole nose, his eyes, or his entire half face, this can be even more confusing.

Therefore, due to the ability to indicate the brightness of the face (for example, L_SA1 in the first, for example, the LDR image) in the content, the creator can also easily and quickly indicate not only where the brightness of the face is located, but also, potentially, the type of brightness distribution. For example, if there is only a small selection in his hair, he can use the average of the darker pixels of the actual face as L_SA1, and some pixels rendered brighter will look good. If, on the other hand, half (or more than half) of the face is highly lit, he can use this average or other characteristic brightness value as L_SA1, and then the remainder will be visualized darker, and this will usually be the best way to combine this content with whatever content nor united. In this case, the creator can still indicate through additional metadata that the face is actually an atypically brightly lit face, for example, X points above normal lighting. Such information can advantageously be used by the combining device to determine with what brightness (above or below the position anc in CombRng) this person should then be visualized (approximately). Normal illumination of a Caucasian (white) face can be taken from the reflection spectrum, which gives an average reflectivity of 36% over wavelengths. Therefore, one point is higher than 18% of the average gray (which makes them comparable values). As stated above, in the classic LDR structure, this medium gray will also be visualized at approximately 18% of the peak lightness of the display, for example, 12%. Of course, other races need to be rendered slightly darker depending on their skin type, because otherwise it can be difficult to actually determine the type of skin of an actor in the film for those who may wish for this, which may be confusing. For example, black skin can reflect about 5%, i.e. be 3 points darker than typical Caucasian. As for visualization, you can visualize a face somewhat brighter in the LDR range than 36% of the peak lightness, equal to 100 nits, to have pleasantly “sunlit” faces. For example, the creator of the image can indicate in the first metadata field the “average” luminaire code or the reference brightness (using EOTF luminaires can be calculated in absolute brightness) of this face object, for example, 853, or 80% in normalized lum encoding, and it can indicate in the second field, that it’s the brightness of a face like ʺ2SBʺ, in the sense that it is illuminated two points brighter than other places in the scene and its image, etc. The reader understands that the same can be achieved by other encoding mechanisms, for example, you can also encode the second anchor point L_S2A1pkS, which is k points lighter brightness, still containing the complexion (for example, the lightest, or 95 percent, or the lightest still in the face, not in the selection of hair), and the third anchor point L_S2A1mkS, indicating some darker end of this face in this HDR image or video image frame. Therefore, anchor points of various types can be indicated in a standardly understood manner (in accordance with various standards implementing the system), for example, or with a set of the listed types, for example, aceFace36ʺ, ʺFace18ʺ, ʺFace72ʺ, ʺFace5ʺ, ʺMiddleDimGreyʺ, ʺMiddleBlackʺ, ʺMiddle, data annotation, for example ʺ face ʺ + ʺ 1 point above ʺ + ʺ dim ’(in which a face means a reflection of 36% of the average amount of light present in this part of the image with a specific kind of lighting, and more importantly, a specific sub-range of the dynamic range of the display to be used to visualize these pixels , to give a correctly visualized scene appearance: “dull” will be medium light colors, which, for example, are usually visualized on most or all displays, in the range of 10-100 nits, below which at least one section of black colors should exist, and preferably to ʺ black colors ’, which is slightly worse lit, i.e. dim lighting is up to 10 times weaker (usually 1-10 nits of brightness), and a deep black area, up to 1/100 of normal, dim lighting (i.e. with luminances of 0, x to 1 nits), which will usually be used in the visualization of the HDR art scene, for example, for the deep ghostly scenery that can be seen in the castle when the hero walks along it with only candles. Above dim lighting, you need to be able to get good results, to provide pleasantly controlled high-quality HDR scenery, by adding a section for “bright colors” (where the sunny external space will be visualized, for example, approximately 10 times for watching TV, where in real scenes it will be 100 times better illuminated, that is, with brightnesses of 100-1000 nits), and one for "ultralight", where you can visualize, for example, lamps, explosions, etc., i.e. effects for which it is interesting to have modern displays with HDR, with a higher sub-range, for example, 1000-10000 nits. Management, whether to adjust the display (to completely bridge the gap between scene-based image encoding and display-based, by, after absolutely encoding the image structure, re-introducing some correct display relativity level), or, as in this application, to correctly combine images, which allows you to accurately differentiate control in 5 scenes, should be sufficient. In fact, if you study the types of HDR scenes that exist in real life or that artists can create, then often you see two images of the site, for example, the store at night in FIG. 16c (one uniformly illuminated image of a region, of course, is usually encoded in SDR), and it is already somewhat less likely to collide with images of three regions, as in FIG. 16b, and when the full power of 5 different lighting modes is required, a really complex HDR scene is created (perhaps in some films appearing only at the beginning as a commercial demonstrating the ultimate HDR capabilities). But now the reader will understand that our principles are not limited to less or more complex options for implementation. If the content calibrator indicates a single anc value, it is already possible to harmonize the content of different images with this anc value. This will usually happen if there is one single critical object in the image, for example, a single actor or host (but, as said, although this actor, even in very contrasting lighting, will have one single typical lighting and brightness, it can theoretically take values between 0 nits and PB_C). If nothing special is required for the type of anchor, it is assumed that the combining device will work with a normal anchor associated with the average gray value and normal visualization brightness, usually “dull”. Therefore, content creators annotating only 1 anc value usually use an illustrative value for the average gray position (in the local illumination of this area of the scene and its image) of its actor, for example, it is in the sun in a bimodal histogram, and then annotate the type, for example, as “Light”, so that even with this small amount of information, the combining device can be nice to determine what to do (for example, in the case where the film should not be broken too much, a decision may be made to visualize the “light” actor in the commercial as a dull PIP inside the film in In any case, or still giving the impression of lightness, you can visualize the bright actor slightly above the dim level, for example, at (light + dim) / 2, for example, from (30 + 300) / 2 nits; if the standard allows additional metadata to be transmitted, and this metadata is full, creators of content, such as a commercial, may indicate that they don’t want to reject it too much from their preferred “light” visualization, but on the other hand, for various reasons, devices can ignore this, for example, because the end viewer indicated using the UI that he wants to watch a movie, i.e. preference should be given to reducing interference errors of light inserted content in the histogram of the film).

Of course, if the calibrator annotates the additional values of anc, he can, for example, set his “deep dark”, and then the combining device can take this into account in its final combination, are they prepared for poorly finally visualized black areas in a light observation environment (i.e. when unification is still carried out in a theoretically perfect structure of an ideal visualization system until the settings are displayed in the real system).

Thus, based on the brightness (anc) of the anchor indicating where to approximately put the complexion (for simplicity of explanation of this embodiment, it is assumed that the HDR colors of the faces already at that brightness level, so that obtaining facial colors from the HDR image is a trivial task , which does not require complicated considerations for displaying brightness), and a method for identifying the location of at least the primary complexion in the LDR image (brightness of the original L_SA1 anchor), the device can relatively easily place all the face colors of both images correctly combined in the image of the union output image quality. And even for the simplest embodiments, the surrounding colors (the faces from which to start) and all other colors of the image will get relatively well automatically in the vicinity of this brightness (anc) of the anchor. The philosophy for other colors is also relatively simple, since it does not require a more detailed study of visualization for these technical scenarios. If all other colors fall in the range of 2-2.5 points around medium gray (for example, white is usually 90% or 5 times lighter in linear brightness) this will give a good image. Imprints often cannot get areas of very deep black, and also practical LDR television was usually limited to 32: 1 (peak white versus the deepest still acceptable black areas), due to the reflection of light from the front glass of the TV. 5 points of the content of the image object, means that, making sure that the darker colors and, in particular, the important black areas are 2-2.5 points from the average gray, we can conclude that the darker side of the well-lit LDR scene also has acceptable brightness. All this, in principle, does not mean anything more in the processing of HDR images, in particular for layout and visualization, therefore, an improved mechanism is required, and, in particular, due to complexity (images can contain anything), some guidance from a person’s side of what is semantically relevant, and thus of what will be a good association. More complex embodiments allow better coordination of the choices of how to correctly position the brightness of other objects around the main object, located by the brightness of the anchor anc, which in this simple example to understand is the face. You can annotate, for example, an anchor point as a range limit, etc. For the avoidance of doubt, by a video signal, we mean simply a temporary sequence of images, and we are not limited to a specific strategy that should be applied similarly for the entire film, but, on the contrary, can be applied only to a part of this film, for example, a frame of images of the same scene (i.e. ., for example, CombRng can differ at different points in time, of course, if then a new commercial is combined with other characteristics of the brightness of the object). Since most of the characteristics of the embodiments of our invention will be similar, at least for spatial associations, whether still images or videos are used, in order to avoid tedious less readable formulations, it can be used in cases where the distinction is not required by either of the two embodiments, a video signal or images, in the explanation, and the reader must understand that he can also read another. The definition of a function that maps the first luminance (the original anchor) to the second luminance (the anchor of the join range anc) should be readily apparent to the reader for various embodiments. As said, for example, you can build a simple color conversion unit for the combining device so that the two-part linear function raises a specific input brightness (of the original image) from the diagonal to the desired position, and then connects the lines with the maximum and minimum of the input and output ranges (0, and [ PB_C_image_1; LMC]), but if there is a function of a specific form for reprocessing the brightness of the images so that they are more suitable for different dynamic ranges or lightness, etc., you can also, for example, linearly scale this function so that the point on for ancS, it fell on the output value of anc on the y axis. It should also be clear how similar systems can be built that do not need to use precise alignment of anchors, but work with it (not too large) near this anc value. A suitable proximity can be established as any union between what the content creator could indicate in the source metadata (for example, this anchor can be visualized with a deviation of 20% of its exact value) and that the union device can make a decision depending on the situation (as said, though for example, a person in a commercial can, in a simple calculation, leave the calculation for visualization, for example, 150 nits, the combining device can, according to various program rules, make a decision to deviate somewhat, for example, it can decide what PIPs who want to be located in a very important central position of the image, you need to lighten it a bit, for example, so that the combining device, which can only be a device having all the information of all parts of the image to be combined, and possibly even the visualization system and the environment on which it should be displayed the combined image would make a decision optimal or acceptable for a true full presentation of the content of the combined image). If nothing else is specified, a pragmatically acceptable proximity can be 2 times lower or higher than the ideal anchor position.

Finally, it should also be noted that since the applicant's embodiments may work with functions indicating the final appearance of the images, in various embodiments it is not necessary that the actual device (e.g., a combining device) actually create a merged image. For example, if the combining device is STB, it can send an already fully optimally combined image to TV, for example, HDMI-encoded HDR10, if this standard is enough for what is needed for this image, and then TV renders it directly as a mute display. But the STB can also simply determine and prepare the situation, and then send all the necessary data: at least some image data, and some transformations (which, becoming optimal FF transforms to calculate the combined image, should be sufficient, but some other options for implementation also should send anchor values, maybe even additional anchor brightness values defined by STB on TV). In this case, the TVs have all the information for the unique implementation of the association, i.e. in the PIP union, he will not only select pixels, but also apply FF_1 to the received corresponding pixels of the SDR image until they are saved in the buffer of the combined image, and FF_2 to the brightness of the pixels of the HDR image, to obtain the correct brightness of these pixels in the combined image. The simplest embodiments of the combining device can determine only one good anchor position. For example, depending on what the TV shows, how it is configured (for example, by user control of lightness), etc., it can only determine the average value of the gray dim light anchor, which should, under the present condition, render nice-looking images, for example, anc_1D = 30 nit. Of course, this will not provide the best control for combining images, but if the first content indicates with its anchor (for example, the average value of black brightness) that the action takes place on a dark night, and the second content indicates that it is a light and vulgar commercial, this simple embodiment of the combining device it should already be able to harmoniously harmonize both images around its anc_1D value. Of course, the best, more complex embodiments of the combining device will determine what would be a good combining situation for several varieties of content, and have given optimal anc values for the 5 aforementioned values of anc brightness types (from deep black to ultralight). Then the device will be able to powerfully harmonize any incoming content, based on its type of the indicated situation or HDR effects, checking its initial values anc (and potentially additional metadata, for example, re-grading functions, for example, indicating how, if necessary, black areas can be turned gray ), to these different lighting ranges of CombRng, in any situation (for example, if the merging is done or prepared for the proposed additional processing, for example, a visualization system that has low light colors, for example, insufficient peak lightness of the display to expressively visualize ultra light, it’s enough lighter than lighter colors, and in this case, the union may be inclined to some form of contouring for the brightest pixels of the image; which in itself would not be the best theoretical combination of the colors of the image pixels, but a good combination for such lower quality systems). Therefore, the brightness of the anchor in the pooling range is usually defined as at least one good brightness for determining other brightnesses around it, and is determined by the pooling device (based on the general needs of visualizing an image of good quality and / or features of the actual or estimated content and / or display capabilities ), and the source anchors perform similar actions in the dynamic range of the original image, and are usually determined according to what the creator considers important for his images, i.e., at least what specific important objects and sections of pixels around him (at least , based on brightness) should be visualized and used under good control, according to various possible usage scenarios, in particular, image associations. You can understand how to set the appropriate brightness of the anchor for a specific object, i.e. of the spatial and temporal part of the images, and it should be clear that the anchors may need to be redefined when the film is replaced by a new (frame) image (s), from another scene, for example, the first scene occurs in the living room of the house under daylight on a rainy day, and the next the scene is in the basement illuminated by a single light source, namely, a lamp in the middle of the basement (both scenes will have similar objects that can be indicated by an anchor, for example, an actor’s shirt, but, of course, will have different graded brightnesses in the reference images, and need different visualizations in the final MDR images for changing displays).

Advantageously, the color conversion unit (310) is configured to determine the color conversion (FF_1) so that the output brightness (LF1_o) determined by applying the color conversion (FF_1) to the input pixel color brightness of the first image or video (Im1_LDR) equal to at least one brightness (L_SA1) of the original anchor, equal to the brightness (anc) of the anchor. Therefore, one way of coordinating the brightnesses of at least two images (or videos) to be combined is to formulate a combination strategy in such a way that the selected brightness of the anchors for all images are the same and set equal to the value of the suitable brightness (anc) of the anchor in the dynamic range of the combination output image (Im_o), in which the final combination of images (for example, PIP or alpha blending) is assumed. One skilled in the art understands how a device can calculate a function having the property: FF_1 (L_i = L_SA1) = anc, and then display for all the brightnesses around anc, respectively, L_SA1. In particular, since it is possible to work in the color space of linear brightness (and reasonable choices regarding the placement of the brightness of the object on the axis of brightness, for the most part, have already been made by the decoder 251 working on at least one of the images, since the content creator created the features different graded images for the dynamic range of their images, and the corresponding functions of the gradation conversion of brightness, if transmitted), often you can use only a linear display (or sometimes simple non-linearity in unevenly distributed contrasts, for example, a gamma function). As explained with reference to FIG. 18a, it is possible to understand how it is necessary (with a linear dimming coefficient) to reduce the brightness according to the input coding of the image with the same coefficient for darker images, and a similar slightly different mapping to the input brightness of the images is lighter than the original anchor L_SA1.

In the case of a linear function, it is easy to understand that the function can be defined (for example, in the coordinate system normalized to 1.0, where the linear input brightness, for example, LDR images, and the normalized brightness in CombRng are plotted along the x axis), namely, as L_out = FF_1 (L_in) = C * (L-L_SA1) + anc. C - constant contrast, which the device can further optimize to make the two images more harmonious. It is possible to determine, based on the semantics of two images, for example, a histogram, and the spatial-brightness distribution (for example, small forms at the top of the image, that is, lamps, etc.), the ratio of the average lightness of the images, etc. For example, if an HDR film consists mainly of a dark basement in which a person walks in the shadows, and part of the high lightness consists of only one lamp, then the relevant contrasts (most of the HDR image) will not be very high. Therefore, if combining with a very contrasting LDR commercial is required, it may be necessary to slightly reduce the contrast of this commercial to better match the lower contrasts that the visual system perceives and therefore expects in dark environments (for example, you can limit the lightest part of the advertising LDR, by applying the slightly inclined portion FF_1 for these lightest relative colors, whatever semantics they contain, whether they are sun-lit objects of the outer space or lamp pixels, as in Fig. 18c. On the other hand, an increase in C is slightly higher than 1 can help increase the visibility of the commercial, even if it is not shown with maximum or HDR average lightness in an HDR film that is very contrasting (i.e., you can control the contrasts in the local sub-band e HDR CombRng, to which all possible brightnesses of the SDR images will be highlighted, and t .d.). Of course, the reader understands that the triad can also perform other FF transformations of brightness (color), for example, a function that has a relatively high contrast just below the point (L_SA1, anc), but then begins to decrease its slope, aligning at some threshold black areas L_out = L_b, etc. d. In fact, in principle, any function can be determined by the union device, provided that it approximately maps the brightness of the original anchor to the brightness of the anchor, but usually the resulting functions will be relatively simple (and, for example, a piecewise linear function usually gives good results, at least , on less critical images of the HDR scene), or until the creation side has conveyed features for following the combining device, for example, the form of partial brightness display functions for re-grading various brightness sub-bands, or the embodiment of the combining device has significant internal image analysis blocks that allow it to offer more beautiful-looking associations (by identifying and optimizing various inter-pixel, or inter-section contrasts, texture rendering quality, etc.).

Advantageously, the color conversion unit (310) is configured to determine a color conversion (FF_1), so that the ratio of the second output brightness (LT2_o), which is determined by applying the color conversion (FF_1) to the second input brightness (LT1_i), to the output brightness (LF1_o), a constant number (C) times the ratio of the second input brightness (LT1_i) to the brightness (L_SA1) of the original armature. As stated above, simpler embodiments can determine a fixed value of C for the entire (partial) brightness range (input brightness of the image to be processed), but you can also make C a variable function of the input brightness L_i (C = CF (L_i)). Partial brightness ranges can be set by the combining device (again, either according to the accepted additional metadata, or independently) in one way or another, for example, you can determine the length of the main column of the histogram, or you can determine the brightness range in the LDR image between ancS / k and k * ancS , where k, for example, is 4, and the remaining LDRs are displayed according to the two dashed partial functions of FIG. 18c, etc. As an example of more variable FF functions, it is possible to increase the contrast of the brightest parts of the LDR image (above the constant L_it, for example, as a percentile of a histogram, etc.), in case information is important (for example, highly illuminated faces), but, alternatively , you can also reduce the contrast for this bright area if this is only less relevant information above the main action, so that, for example, the device can slightly increase the average brightness of the first mixture, for example, an LDR image (for example, by displaying in anc + d_anc), but do not dominate the HDR movie with too lightly visualized highlights in the LDR image with brightness display in CombRng. This allows, among other things, to make the brightness more consistent with the intended person - the calculated distribution of lightness. Therefore, LT2_o can be selected, for example, as a percentage above LF1_o (for example, equal to anc), for example 150% LF1_o, or k points above LF1_o, and then set C1 for this brightness interval, etc. A reasonable adaptation of such a (at least one or more) sub-section of the usually less important one of the images (for example, the SDR commercial) can significantly improve the harmonious appearance of the full association.

Advantageously, the color conversion unit (310) comprises a deviation determination unit (312) configured to determine, based on at least one brightness (L_SA1) of the source armature, the brightness offset (d_anc), and wherein the color conversion unit is configured to the ability to determine the color conversion (FF_1), so that the output brightness (LF1_o) determined by applying the color conversion (FF_1) to the input brightness of the pixel color of the first image or video (Im1_LDR) equal to at least one brightness (L_SA1 ) of the original anchor, equal to the brightness (anc) of the anchor plus the offset (d_anc) of the brightness. As said, there may be significant differences between aspects related to the brightness of the object, such as, for example, face lighting (for example, very contrasting), and the ratio of the brightness of the face to the rest of the image. Therefore, the device may make a decision, it is more advisable not to visualize the face of the LDR (or the object of the first image anchor of any dynamic range, or the section and the corresponding brightness sub-range) with exactly the output brightness anc, but, instead, for example, 2 times brighter. This can, among other things, be determined on the basis of the type of illumination of the face (which, for example, is usually indicated by coding the type of anchor), for example, if the brightness (L_SA1) of the original anchor is defined as the average brightness of a very brightly lit half of the face, and the face must be visualized in the final image with a large size, the device may decide to determine a negative d_anc, for example 50%. Heuristic considerations underlying these program rules and mathematics may consider, for example, that light areas may overshadow surrounding darker areas, or be too noticeable and attract attention, or simply annoyingly bright compared to the rest of the image, which may be the main content, that the viewer watched, such as a movie, etc. Considerations may also evaluate the luminosity composition of an HDR movie image. For example, this image may consist of a dark portion of the interior space and a lighter portion of the exterior space observed through the open garage door. If the PIP of the relatively lighter LDR commercial is supposed to be placed in the dark part, it should be darkened (i.e. a relatively large d_anc, for example, to make the average brightness of the face in LDR, or the brightness of additional anchors characterizing the LDR image, weighted in The single illustrative brightness for the LDR image, in the middle between anc and the local average brightness of this dark portion of the second, HDR film image) had the contrast between the two images, and the local contrasts observed from there by the brain in neighboring objects of these two images were not excessive. But if a light PIP is to be presented near part of the solar outer space of an HDR film, it may even need to be lightened with positive d_anc. Therefore, in the general case, these deviations of d_anc will be calculated on the basis of an ideal combination (for example, two faces) that would be in an ideal situation (for example, if the faces were surrounded in both images by a medium gray background), and the actual situation, at least of the main image (i.e., in this example of an HDR movie into which the LDR commercial is inserted), as globally (i.e. what kinds of objects does it contain, with average brightness and size, and, potentially, additional semantic characteristics , for example, of internal complexity with a measure of texture, etc.), and locally in the place where the composition was formed (for example, in PIP of small size X, Y). Therefore, the offsets will be determined, in the general case, on the basis of what is harmonious for the two images, and in various embodiments of what will depend on the features of the images themselves (contrasts, such as the initial position of the anchor and the content of the object in this proximity, and the content beyond the limits of this proximity, geometric size, etc.), and the combination (for example, whether the PIP is located in a small less critical place in the lower right part of the image, or how the mixture will appear through partial information in the mixing mixture, etc.). The simpler embodiments will only use a relatively simple harmonization structure, which is not so bad in itself (especially if the value anc + d_anc can be optimized to reduce, for example, the risk of excessive contrast, if this is determined for the commercial), but more advanced associations can look more precisely where and how exactly the information of the second image is located, or even additional properties of the association.

Advantageously, the color conversion unit (310) is configured to read at least one brightness (L_S2A1) of a second source armature obtained from a second source (351) that delivers a second image or video (Im1_HDR) from two images or videos, and however, the deviation determination unit (312) is configured to determine the brightness offset (d_anc) also based on at least one brightness (L_S2A1) of the second source armature. Above, we described a simple embodiment in which the luminance structure of the main image (which we call image 2 in our wording of the claims) has a known luminance structure, and thus it is easy to convert and form the base link into CombRng. This is a useful way to combine in the case where the HDR image is the leading image that will be minimally distorted, therefore, this is, for the most part, secondary content that is converted to it. This can happen, for example, if it is known that the HDR image is of the type created to ensure that all Lambert reflective objects (for example, in a studio for holding a talk show) have a given brightness, which always falls within a strictly specified range of 0-400 dynamic range nit Im2_Rng for Im_HDR (and which correspond, for example, to 0-300 nit range in CombRng, or the same range 0-400 to CombRng), and everything above is, for the most part, selections, for example, small specular reflections on metals. In this scenario, there is a specific variation of the HDR scene in which the lower part is created for the most part similar to the production of LDR television, and the HDR effects are, for the most part, light sources for which their exact brightness is not too critical (and are not, for example, objects through a window that is not only clearly visible, but these objects of external space may even need to have specific brightness to convey some artistic mood). In this case, the task, for the most part, is to coordinate the LDR commercial with the HDR talk show lighting situation, for the most part, up to 400 nits, and also coordinated to some extent with what is in bright areas (for example , you can use the aforementioned contrast mechanism with C multiplicative constants, or any general definition and application of the brightness display function, to brighten small portions of the selection (identified or suspected) in the LDR commercial to better match the brightnesses of these allocations actually present in HDR talk shows )

But, in general, HDR content can be anything. For example, his faces may not be lit up to approximately 25% 400 nits strictly, but fall on the entire axis of brightness if the actor runs, for example, along a dark corridor in a shaft with sparse lamps. Thus, it is beneficial if each content creator annotates its content in such a way that also the HDR image will indicate where one or more of its faces fall on its axis of brightness, with at least one brightness (L_S2A1) of the second source anchor. Then it is more specific that both images will be in harmony in their final combination, because the device can, by reading L_S2A1, check where the brightness of the faces is in the second HDR image (see the corresponding explanatory example in Fig. 4). Then the HDR image can be optimally transformed for the best final merging situation (in particular, CombRng), for example, given that, most likely, the film will show the external space instead of the dark environment, etc. For example, the FF function can modulate other face lighting to a lesser extent, in particular, if, for example, the observation situation is such that the visualization of the texture of a dark object will deteriorate compared to the optimal cinematic viewing of a film.

It is also advantageous if the color conversion unit is configured to establish a color conversion (FF_1) for the first image or video recording also depending on the first functions (F1_L) of color re-display, which specify a change in the brightness distribution of objects in the first image or video recording (Im1_LDR) for displaying this first image or video from the dynamic range associated with encoding the first image or video into a dynamic range that differs from the peak lightness by at least 2 times, or the color conversion unit is configured to establish color conversion (FF_2) as applied to the second image or video, also depending on the second functions (F2_L) of re-displaying colors, which specify a change in the brightness distribution of objects in the second image or video (Im2_HDR) to display this second image or video from the speaker The optical range associated with the encoding of the second image or video is converted into a dynamic range that differs from the peak lightness by at least 2 times. As explained above, in our basic structure for encoding a video signal or HDR image, we indicate with transmitted color conversion functions (at least specifying the brightness conversion, but sometimes also the necessary saturation conversion, for example, to increase the saturation of objects that need to be darkened as LDR, so that they remain lively looking; functions that indicate the appearance for dynamic ranges that differ, usually at least 2 times in PB_C, for example, how to convert one extreme or average graduation, for example, PB_C 5000 nits to another, for example, PB_C 100 nits), how the distribution of brightness of objects should change (which is, in general, a difficult optimization to lower capabilities of a display or codec with a lower PB, taking into account the semantic features of the HDR scene) when moving from a higher dynamic range to a lower one ( or vice versa). Those. when converting from a reference graded image of HDR of 5000 nits, for example, to an image of 100 nits, or some image of the average dynamic range between them, for example, optimal for driving a display of 1200 nits. Usually, the creator can determine how he would like to change the appearance, for example, gradually darken the selection and dark areas.

The power of the union structure is that a decision can be made on how to formulate a new kind of functions that display not in the dynamic range of another (single) optimal visualization, but in the dynamic range of the association, depending, inter alia, on how it is desirable to control the colorimetry of the association (which itself may depend on various external settings of the parameters, and the parameters quantitatively express, among other things, the desire of the creator, owner or distributor of the content, the viewer, if he tries to carefully watch the movie, for example, or even the manufacturer of the display). Of course, associations of the full form may vary depending on the particular association and what is in it, but this does not mean that the re-grading functions themselves do not contain interesting needs or the desired re-grading (whether in the lightest sub-range, for example, above 20% Lum, there are clouds that require careful re-grading, or less important objects), which also the merging device may want to follow to some extent.

Now, if the above-mentioned simple linear FF functions are used to display the input brightnesses of the images around the brightness of the anchor anc, the details of F_L may not be required. But more complex mappings can change on a linear strategy, and, in particular, do it according to what the creator of the content in the F_L functions implicitly requires for this particular image of this particular HDR scene, in this brightness sub-range. For example, if the creator wants darker brightnesses to quickly fade into black, the color conversion unit (310) can take this into account when determining the optimal FF function, for example, he can stretch some of the contrast for brighter areas (for example, around anc complexion) by more quickly clipping to black a larger percentage of LDR brightnesses than a linear function would do. Therefore, the device can evaluate the functional form of at least the brightness conversion function of the set of functions F_L (which, for ease of understanding by the reader, can be assumed to be only one function of arbitrary shape, which displays the input brightness normalized to 1.0 to the output brightness normalized to 1.0), and use part of this form to design the final function form for display in CombRng. It should be noted that in such functions various parts of the image desired for re-gradation can be set, either by themselves, or in connection with other possible brightnesses in other parts of other images, etc. For example, you can set for geometric images of the area that the brightness present there should not be, for example, excessively amplified, even if, for example, the remainder of this image are pixels with the same brightness, which can be enhanced to a greater extent. Any such form of partial function can, in principle, be indicated (for example, the previous embodiments developed by the applicant make it possible to indicate a subset of pixels in the image satisfying the property of hitting a specific geometric area, for example, a rectangle, and have brightnesses between L_min and L_max, and then you can specify the desired behavior of the partial display function, for example, for the brightnesses in this section, for example, HDR images that fall between L_min + k and L_max-l, with constants k and l. For example, you can specify a single or parameterized increasing function for these luminances L_out = B * L_in + O, where B and O can be a function of various properties, for example, LMC, or the average brightness of the combined output image surrounding the rectangle in which the data from the HDR image is located, etc. Any behavior of the divided re-gradation can be indicated in functional specifications before final merging, or even collapse after To create a merged image (for at least a portion of the full content to be merged).

Advantageously, if the dynamic range setting unit (302) is configured to set the dynamic range (CombRng) of the brightness of the combination depending on the brightness in at least one of the two images or videos. Not all embodiments are necessary or may take into account the particularities of the final observation environment. For example, the output image may be mixed in some reference control room for transmission to various users. The final situation may not be known, and may vary significantly (if one viewer looks, for example, in a train, and the other in the cinema, which he arranged in his attic). In this case, it is beneficial if the combination is optimized based on the characteristics of both (or all, when mixing more than 2) images, since this will always be relevant. Of course, the reader will understand that if used well, the union paradigm is further scalable. For example, the first content creator may mix two image sources, but this does not mean that below the line additional content cannot be mixed. And what can happen both after and according to the assumption of the first creator, and at least some of the embodiments of the combining device should be considered for all such situations. For example, the first content creator may first discuss what should happen, so he can determine CombRng (and / or anc) that is optimal for his content, but perhaps already assuming that some entity will add some image content further below the line with some typical additional characteristics (for example, a thicker tape with news information at the bottom, or in the situation of a cinema hall or store or museum, a secondary projector projecting some secondary image information in addition to or partially through the primary visualized image content, etc., with the secondary content will have a dynamic range, average lightness, dynamics, for example, the temporary evolution of average lightness, etc.). The secondary content mixer may then decide whether it is better to follow the instructions of the first content creator, for example, to save CombRng, or perhaps it is better to fine-tune it, provided that there is now additional information regarding the final merged image or its presentation. Two classes of embodiments are possible. In a simpler class, the two images are already inseparably mixed at this stage (for example, the final PIP composition), and the final receiving side can only optimize this full image according to its visualization features (display PB for visualization on the display, average lightness of the environment, etc. .). Those. for ease of understanding, for example, this combined image is taken and applied as a function of the square root of the ego luminosities to obtain the final output luminances to be visualized. This may be sufficient for many scenarios, since the mixed image may already have harmonized lightnesses, and the observation environment may not differ too much from the intended one.

More advanced implementation options will allow you to re-determine the brightness distribution of the object / pixel of two images on the receiving side, and in some embodiments, the implementation may include re-definition of the original images, at least to some extent (for example, their main coding, or at least initial brightness grading; which is the aforementioned possibility of destruction). For example, some embodiments may specify, for example, combining PIP in some brightness distribution mode, but jointly encode CombRng features, and FF functions to bring at least one of two images to correctly harmonized luminances in CombRng. For example, you can presumably just put the PIP LDR in an HDR environment image with brightnesses linearly scaled to PB 5000 nits, which, without proper processing on the receiving side, will of course give a terrible looking pool. But the receiving side receives all the necessary information to make the integrated visualization correct, for its part, i.e. with harmonized PIP image. This can be done by appropriately changing the brightness of the pixels in this rectangle, taking into account everything that is known (i.e., for example, partial brightness display functions adopted for this section, or special additional anc values for this rectangular section, etc. etc.; or what can be determined for its part, for example, that such a section looks significantly bright compared to surrounding pixels, etc.). One of ordinary skill in the art will now understand that these FF functions themselves are not F_L functions for re-grading to another dynamic range, for example, not just functions for acceptable display of a single LDR image appropriately in the dynamic range of an MDR display with PB for example 1500, but, on the contrary, with the functions by which the creator side indicates (or the combining device, wherever it is installed) that this will give a good mixture for the LDR image harmonized with the HDR image, i.e. tailored to the two images together.

Advantageously, the dynamic range setting unit (302) is configured to set the dynamic range (CombRng) of the brightness of the combination depending on the peak lightness of the display on which at least one output image (Im_o) is to be displayed, and preferably also on the characteristic the lightness of the observation environment. If an embodiment of our device is included in the end device or system of a visualization site, for example, a TV or a computer, it may be advisable even to additionally set the association according to the particulars of observation. If the LDR image looks excessively light when observed by itself in a dark environment, it is likely to be too light for this dark observation environment when combined with a film, in particular, since this film can already be optimized for this particular observation situation (see in Fig. 3, a suitable image automatically converted to MDR in the form Im2_MDR is calculated, for example, for a display with PB 1200 nits observed in a dull environment, for example, from Im_HDR 5000 nits, which is accepted by all viewers who are subscribed to the same content or watching the same broadcast program). Partial images can then be mixed in their appropriate dynamic range MDR (Im1_LM and Im2_HM). This will be especially important when, for example, the darkest pixels of a certain image are difficult to observe in lighter observation environments, in particular, if the viewer is distracted (possibly even spatially adjacent) by light content from another combined image, and then, based on, for example, additional anc values for darker pixels in at least one of the images (for example, the leading image) can significantly improve the optimality of the final combination. A qualified reader will understand that, of course, with the combined presentation of two images, optimization should usually be different from optimizations of a single image, and, depending on the situation as a whole, that is, what the viewer is ready to perceive (of course, devices that perform optimization will be pragmatically constructed more or less accurately based on the complexity that you can afford given the market price of the device; some simpler options for implementation may already work quite acceptable). When watching a single LDR commercial that is too light, even in a dark observation environment, the viewer will have to, for the most part, adapt his vision to what is seen on the screen. But when combining the two images, it is not known in advance exactly what human vision will adapt to, and, therefore, according to what luminosities and luminosities, the brain will see the combination of visualized pixel brightness (there is a possibility that it will not be able to see the leading film at the beginning), therefore, the desire to have a system where the content creator can at least indicate the minimum necessary information and guidance (practical technical systems always balance between the complexity, for example, the labor spent by the content creator, and the cost of IC needed to perform the calculations, regarding the need to have the ability to process at least some of the minimum necessary color transforms so that the system behaves at least half acceptable and not quite bad; and then the various options for implementation, usually standardized, will decide how far to go with which additional con specific management functions in these scenarios).

Therefore, a qualified reader understands that it is advantageous to determine CombRng (and also usually the brightness of the anchor, and also the specific forms of the FF functions in the case of non-linearity, or the contrast coefficient C in the case of linearity) depending on at least the capabilities of the display (its PB can be sufficient in most situations, but additional features may be included, for example, if it is not fully used for reasons of energy conservation, etc.). And if the devices are connected to assess the brightness of the viewer's surroundings, you might want to do more. For example, a light meter may be used, but it may be better to use a camera somewhere in the room of the receiving side, for example, connected to a display that can be calibrated to a brightness meter for various parts of the imaged observation environment, and is already present on some TVs. On this basis, you can roughly assess how the viewer will perceive the image. Simpler or more advanced versions can be used, for example, a camera that checks various brightnesses on a section behind (see around) TV up to 1 TV width in each direction.

Advantageously, if the block (303) determining the brightness anchor is configured to determine the brightness (anc) of the anchor depending on at least one of: the dynamic range (CombRng) of the brightness of the combination, the brightness in at least one of two images, or videos, the peak brightness of the display on which at least one output image (Im_o) is to be visualized, and the characteristics of the brightness of the observation environment. You must also set at least one brightness of the anchor anc, from which the brightness distributions of both images can be harmoniously distributed. The reader should understand that various embodiments may define CombRng and anc in any order. For example, some embodiments may first set, for example, depending on the capabilities (actual or estimated) of the display and the characteristics of the lightness of the viewing environment, how good CombRng will be, for example, usually for the situation of the current movie, or even for each frame of images, etc. d.

For example, some embodiments may, prior to the start of the film, analyze the first images of the film, or a characteristic set of images selected throughout the film. Or a movie can have coded metadata from which the device can set, even before decoding the first image, how acceptable at least the original CombRng will be. For example, metadata may indicate that the average lightness of the film is 200 nits in the lower range of up to 400 nits, and with the most extreme sequence, for example, a shot in the desert, falling to (i.e. which is preferable to visualize on any MDR display that has sufficient opportunities for this) the brightness of diffuse objects, for example, sky or sand, up to 900 nits (and an average of 600 nits). Then, an embodiment of the image combining device may decide what should be reserved in CombRng (in particular for serving displays with different PBs) a fixed eclipse region, for example, up to 200% of these 900 nits, which would be the most excessive requirement for this film, such so that more than 100% of the brightness can be used for all selections (whether it be small areas of specular reflection, a lamp, or even darkening areas containing image details, for example, a very brightly lit part of the scene). Of course, more advanced specifications can help determine exactly which region of lighter brightness should be in CombRng, also by coding what kind of HDR effects are typical in a program or film, or part of it. Working with a pair of such subbands is powerful and versatile (whether they indicate, for the most part, the needs of the content, or the possibilities of upcoming visualization, or both), especially if you add a couple of good brightness of the anchors for these ranges.

Having determined the optimal CombRng, the device can on this basis determine where the brightness of the armature, for example, complexion, should be located. For example, if half the range corresponds to well-lit Lambert objects, then it may be decided to use 40% of these 50% peak brightness as the point of brightness of the face anc. However, some embodiments may work differently. Taking into account all the visualization characteristics, the device can determine the appropriate brightness anc for faces or medium gray Lambert objects of the main area, so that it looks bright enough for the viewer. Then around it you can build a suitable CombRng. For example, it looks at the most demanding HDR image, and sees light objects (see lamps) up to 20 times brighter than faces. Then it can decide to put the maximum of the range in this brightness value, or consider, for example, 80% of this value sufficient for the combined visualization of two sets of images, and then higher values in Im_HDR, respectively, Im2_MDR (whichever was used in the combination in a particular embodiment) will usually be clipped to the maximum LMC dynamic range of the Association. This can be very useful when it is desirable to harmonize the content, for example, with less dynamic range capabilities, especially if, for example, the HDR movie is not the leading content, but, for example, PIP on the computer display, which, for the most part, shows other portions of the image, for example, with information, etc. Thus, simple embodiments will make the object 2 times lighter than the complexion or, in general, any brightness of the anchor in any image (i.e., the first and second HDR image, or the LDR image when the image LDR is used in the union) twice as lighter also in the corresponding pixels of the image when ready to add the union, or image (Im_o) to the image. Or, more advanced color transformations (or combinations, some of which can also weight or change the brightness of pixels in the mathematics of their combination) can make some brightness darker or lighter. And for similar reasons, the device can come to the lower limit of LmiC brightness for CombRng, for example, slightly brighter than in at least one of the original images, so that the combined image (which can now have much more bright objects, than at least one of the originals) doesn’t look either too dark or too contrasty, etc.

Additional options for implementation can be implemented as appropriate devices (either small, part of the IC, or large, like the entire consumer or professional system) or methods, for example:

a method for combining two images or two video images (Im_HDR, Im_LDR), one of which is an image or video of a high dynamic range, the method comprising:

the establishment of a dynamic range (CombRng) of the brightness of the Association, characterized by at least maximum brightness (LMC), and the establishment further comprises determining the brightness (anc) of the armature in the dynamic range (CombRng) of the brightness of the Association;

applying a color conversion (FF_1) to at least one of two images or videos consisting of at least a brightness conversion, the color conversion being set to the brightness value of the original armature (Im1_LDR), which is read from the first source (350) which delivers the first image or video (Im1_LDR) of two images or videos; and

the combination of images, and the colors of their pixels are in the dynamic range (CombRng) of the brightness of the Association for the formation of at least one output image (Im_o).

The method of combining two images according to claim 10, wherein the color conversion (FF_1) is determined so that the output brightness (LF1_o) determined by applying the color conversion (FF_1) to the input color brightness of the pixel of the first image or video (Im1_LDR) equal to at least one brightness (L_SA1) of the original anchor is equal to the brightness (anc) of the anchor, or the brightness offset from the brightness (anc) of the anchor by a certain difference (d_anc).

A method of combining two images in which the color conversion (FF_1) is determined so that the ratio of the second output brightness (LT2_o), which is determined by applying the color conversion (FF_1) to the second input brightness (LT1_i), to the output brightness (LF1_o), a constant number (C) times the ratio of the second input brightness (LT1_i) to the brightness (L_SA1) of the original armature. As explained above, other brightnesses that occur in the images around the brightness of the armature can either simply be displayed around the brightness of the armature, or in a more precisely defined way.

A method of combining two images in which the color conversion (FF_2) of the second image or video image is determined based on at least one brightness (L_S2A1) of the second source armature obtained from the second source (351), which delivers the second image or video (Im1_HDR) from two images or videos. It is possible to determine the different brightnesses of the anchors of not only the first, but also the second image, so that their various ranges of lightness (for example, well-lit, chiaroscuro, deep shadows, lamps, etc.) can be optimally coordinated when displayed on each other in the output output image range CombRng.

A method for combining two images in which at least one of the color transforms (FF_1, FF_2) to be applied to the corresponding one of the at least two images or videos is determined at least partially based on at least , one received function (F1_L, F2_L), which indicates how to convert the corresponding image from the dynamic range corresponding to the one being encoded into a dynamic range that is at least two times higher or lower. As said, these functions determine how exactly the lightnesses of all objects in their scene should look in any dynamic range (at least 2 times different from its natural dynamic range, i.e. PB of the model display, which corresponds to the EOTF, which sets the codes lumens Y 'of the received input image). Therefore, this information can also be used if necessary to re-calibrate, for example, to a slightly lower dynamic range (for example, CombRng is slightly less than the MDR range Im_MDR). This will most elegantly provide coordinated brightnesses of at least one image or video used independently, but it can be a good starting point for rough determination of the initial set of pixel brightnesses of the displayed image in CombRng as well for the content representations of the combined image. Additional fine-tuning can occur, either somewhat simplified, for example, heuristic linear stretching, determined by the device or method of combination, or more advanced, taking into account the specific necessary brightness coordination in the images in their objects or brightness sub-bands (or reasonably determined according to, for example, object segmentation algorithms and the corresponding brightness ranges, or more approximately, by, for example, dividing CombRng into 5 equal or unequal sub-ranges, and the same for the two brightness ranges of the input image and their respective sub-ranges), since they are set by the content creator in the form of characteristics of the F1_L functions, accordingly, F2_L, each of which is verified and used in any embodiment.

The method of combining two images according to any one of the preceding paragraphs, in which at least one of the dynamic range (CombRng) of the brightness of the combination and the brightness (anc) of the anchor is determined based on at least one factor from the set: properties of objects depending on the brightness distribution in at least one of the two images, information summarizing the brightness characteristics of at least one of the two images, the dynamic range of the display on which the combined at least one output image (Im_o) is to be visualized, and lightness measures for the observation environment in which the output image is to be observed.

All of the above can be included in various systems and forms, whether consumer devices, or professional systems, of which some components can be located on servers in other countries connected via the Internet, etc.

It may be advantageous to supplement the image with metadata, for example, the CombRng specification (for example, its higher brightness LMC, and its lower brightness limit LmiC), and / or the typical brightness of the anchor, for example, the face, and possibly also at least one a color conversion function indicating how to display the brightness of the image pixels in CombRng.

It is also advantageous if the content creator annotates its images with at least one semantically relevant (and, preferably, pre-agreed type) brightness of the armature, so that when receiving lighting choices, the images can be understood by a device wishing to combine different images.

Advantageously, each of the methods can be implemented in the form of a computer program that can be transmitted through some physical medium, for example, a computer program product containing program code that allows the processor to execute code in such a way that when the code is executed, all the steps of any of the above described embodiments of the method are carried out .

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the method and apparatus according to the invention are apparent from and will be explained with reference to the implementations and embodiments described below, and with reference to the accompanying drawings, which are merely non-limiting specific illustrations representing more general principles, and where the strokes are used to indicate that component is optional, components without strokes are not necessarily important. Strokes can also be used to indicate that elements that, according to the explanation, are important, are hidden inside the object, or for intangible things, such as allocations of objects / sections (and how they can be shown on the display).

In the drawings:

FIG. Figure 1 schematically illustrates how in the (recently begun) HDR era, various different ways of encoding videos were proposed (in 2015), which will lead to images with very different brightness characteristics that are not easy to couple;

FIG. 2 schematically illustrates the basic components of how an applicant’s HDR coding technology can encode an HDR image, together with another kind of lower brightness LDR (or color grading), the lightness of objects of the same HDR scene, and how this information is sufficient to calculate a suitable view with the correct lightness of objects images on any available HDR display with any peak lightness (PB_MDR);

FIG. 3 schematically illustrates a basic embodiment of the present device combining images or videos with HDR capabilities showing the main units;

FIG. Figure 4 schematically illustrates how a suitable color conversion to optimally selected dynamic range combinations occurs before the combination of video signals or images;

FIG. 5 schematically demonstrates what problems can arise in the absence of careful colorimetric preparation of the brightness of pixels (or, in the general case, colors) of two images, but, on the contrary, combines them in a simplified mode for one of the possible unions, which is a composition of an embedded image;

FIG. 6 schematically illustrates how an apparatus can determine some illustrative suitable color conversion (FF) function, or at least brightness, for at least one of at least two images to be combined;

FIG. 7 schematically shows which algorithms can be used developed by the authors of the present invention for artistically optimal automatic re-grading, if desired by the creator of the content, of the image from the first dynamic range to another second image, the dynamic range of which (at least its peak lightness) can differ by several to two;

FIG. 8 schematically shows some more advanced examples of how a device can adjust functions (FF) for color-converting images to be combined into a combining range (CombRng) based on semantic features of the distribution of brightness of an object in an HDR scene, as shown;

FIG. 9 schematically shows one application example of an embodiment of the apparatus and method described herein, namely, in the initial production of multiple video recordings, for example, talk shows for TV;

FIG. 10 schematically illustrates a second example in which an embodiment of a device and method is used in a system at an intermediate node of a video signal transmission network, for example, a transmission control room for serving a local area (the reader should understand that other embodiments of the combining device may be located in other places in the processing circuit videos or images, for example, in the home of the end user of the video image);

FIG. 11 schematically illustrates how embodiments of a device can determine CombRng, and highlight the original brightness of images in it, starting from a good position of the brightness value for the brightness of the armature;

FIG. 12 schematically shows further explanation of examples of how embodiments of the device can determine CombRng, and highlight the original brightness of the images in it, starting from a good position of the brightness value for the brightness of the armature;

FIG. 13 schematically shows how to combine pixels from two images in a simplified manner if the technical aspects of the present invention are not used, but which can lead to poor quality of the combined images;

FIG. 14 schematically shows how, even in one image capturing method, in particular, encoded according to one technical method for encoding an HDR video signal, it is possible to have very variable brightnesses for the same object, in particular if it is most desirable to use the HDR image forming potential most generously;

FIG. 15 further schematically explains how various ranges of combining can be determined, and what can be meant in practice, in particular for establishing the associated appropriate brightness of the armature (anc), and hence typical brightness display functions for harmonizing partial content from at least two images or videos;

FIG. 16 schematically shows, in particular, how it is possible to harmonize very powerfully when an earlier part of the video processing chain is specified, the encoding of the video signal itself, in a certain way of rough and precise re-grading of different image brightnesses;

FIG. 17 schematically illustrates how automatic versions can formulate the definition of various variables based on various algorithms for determining factors such as errors, distortions of the image content and / or disharmony of the content, which in the case of graduation and annotation of the image by a person can be determined by the person who created the content; and

FIG. 18 schematically illustrates some additional examples of a useful embodiment of a luminance conversion function for determining harmonized luminances for combining images.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1 and 2 are explained above, and in FIG. Figure 2 shows how it is preferable to encode an HDR video signal, or a more precisely graded LDR image and a graded HDR image (usually PB 5000 nits) of the same scene that can be decoded (by applying the brightness conversion function to the received LDR image) to obtain an optimally re-graded image for the peak display brightnesses anywhere in or even outside the range of dynamic ranges covered by the LDR and HDR grades as extremes.

FIG. 3 explains at a higher level some of the basic components that underlie most embodiments of our HDR image, or video signal, device or harmonic combining method. The situation with the PIP example is described below, although the same methods can also be used, for example, for coordinated dimming of the lightness of calibration in the time sequence of an interleaved video signal from different sources. For example, you can set a good CombRng level for the LDR commercial, and adapt the brightness in the HDR images gradually to it (various additional metadata can be sent regarding the temporal evolution of various videos). A qualified reader will understand that after the situation of very different and differently set brightness (in addition to luma codes) of two images was converted to having corresponding related, similar brightness (to be combined), you can also start mixing content in a more complex way, for example , alpha blending, or replacing part of a face with some computer-generated face structure with appropriately coordinated brightnesses, etc. For example, if the brightness of faces (for example, on either side of a differently lit face) displayed in CombRng of the first image is x and y, and of the second image x + e1 and y + e2, with e1 and e2 rather small deviations, then you can apply mixing equations, for example, L_out_comb = alpha * L_im1_comb + (1-alpha) * L_Im2_comb, in which L_im1_comb and L_Im2_comb are, respectively, the pixel brightness of the first and second image after color conversion (with the corresponding FF function to be used) in CombRng, and alpha is a real number between 0.0 and 1.0, and L_out_comb is the final brightness for this mixed pixel in the output image Im_o, i.e. A merged image that has CombRng as its dynamic range.

Readers can understand that if the creator has the fullest possible control over the original images (in particular, you may even need to know the features of the assigned visualization environment), you can, of course, customize the combined image by any means until it becomes completely satisfactory as desired. But the present embodiments take into account many situations where it is not so simple or possible, for example, since there is no time to carry out all the adjustments manually, or some input or variables specifying the optimal form is not yet controlled, or even unknown. Something can always change to the image processing situation, so that is what the most advanced HDR processing technology should take into account. Or, at least, our options for implementation allow universal applications in which colorimetric fine-tuning can be carried out at various times, for example, according to new desires, and it is possible to retain some of the information of the original image. This is important, there remains something, at least the most important aspects of the images at each stage. Of course, the end user may decide to completely repaint the video signal, but then all the efforts of the content creator are bypassed and lost, and that is actually not useful to anyone (even if, for example, the manufacturer of TV wants to add some of his own specific taste to render the image, be sure to do this, ignoring everything that the image refers to, in particular, what the content creator has set for the image; but, on the other hand, the receiving side should not feel that it says absolutely nothing about the colorimetry of the image, even if it clearly looks inappropriate for the situation).

The decoder 251 is again similar to that explained in FIG. 2, i.e. it can process gradings (HDR, LDR) of an image according to our explained HDR encoding method (retrieve brightness display functions, determine new re-graded MDR images from them, etc., according to various possible global or local to the image embodiments). Of course, the reader understands that this is only an explanation of one of the possible implementations, and the device is usually able to process (i.e. decode and combine) all types of video signal or HDR or LDR images, i.e. Im_HDR can also be set according to the PQ EOTF, or the BBC-HLG method, etc. In this example, however, it is assumed that from some (second) source 351 images (for example, a satellite broadcast channel, or connecting to a video storage repository over the Internet, or even a connected camera, etc.), the second one of the two images for mixing is obtained (for example, the main, most important (leading) program is an HDR film, the cat is precisely artistically color-coded), and for example, it is a grading of HDR PB 5000 nit of which images are converted with PQ EOTF, and then encoded by DCT, and encoded according to some profile and HEVC level. Metadata that contains F2_L functions is also transmitted to lower this HDR image to a 100 nit LDR image. The decoder can create the optimal image for the connected display, for example 2500 nits, MDR image, Im2_MDR. This will be the correct graduation in this film, viewed independently, without combining with any other (first) image data, scattered in space or in time.

In FIG. 7 summarizes how, according to one of our embodiments, it is possible to perform such a rearrangement from the form of the first dynamic range to the second. Suppose L_in is the normalized [0,0, 1,0] HDR input brightness that corresponds to the actual (to be displayed on the display when rendering the HDR image) brightness of 0-5000 nits. L_out is the LDR brightness, normalized, but corresponding PB = 100 nits (i.e. SDR brightness). The function 702 that maps this HDR to this LDR image is an example of F2_L color in particular brightness conversion (we can assume for a simple understanding that there is only a grayscale image, but we have shown that you can also apply the necessary color conversion on RGB colors in this way) . Suppose we want to output the optimal re-grading function 703, which corresponds to the PB_MDR of the MDR display, for example 3400 nits. The diagonal line 701 will be a necessary display of brightness if it is necessary to map the input image of 5000 nits to an image optimal for the display PB = 5000 nits, since the HDR image is already optimally graded for this display (i.e., it is symbolically transformed by an identical transformation). Set the direction of the DIR output, for example, vertical. The scaled function for the PB display is 3400 nits, it should give the value F * (Ls) for any input brightness Ls between F (Ls), i.e. by any optimally defined lowering function F (F2_L) occurs at this brightness point, and Ls. This point can be defined as Ls + FI (PB_MDR, Ls) * (F (Ls) -Ls), where FI is some function giving values between 0 and 1, and in a simpler scenario it is only the PB_MDR function, which can also be recorded multiplicatively as k (PB_MDR) * Ls. Which function is used depends on the philosophy of re-grading the embodiment used, and the reader may imagine various more or less complicated embodiments depending on, for example, features of the observation environment (or even taking into account the preferences of the viewer), but the details are beyond the scope of the explanation of this application which is already quite difficult (the reader just needs to understand that some re-grading function can be installed and applied to the input image to obtain a correctly re-graded image with the correct brightness of the images of objects for the MDR display).

As said, the first image can be anything, but we will explain the principles of the LDR commercial, the principles of our embodiments are similar for other content. This 2500 MDR may not be a suitable grading (i.e. 2500 may not be a suitable PB) when mixed with a commercial of LDR 100 nits, at least due to a large difference in the dynamic range of the two images, but also due to an extreme distribution of brightness of objects in the LDR ad (i.e. 2500 nits may not be the best LMC value for CombRng, but then another value can be chosen). The high dynamic range should be understood not only as “high”, i.e. bright, but also in terms of "dynamic", i.e. where on the axis of brightness, the creator highlighted various semantic objects and / or sections in the image (s). In contrast to the popular misconceptions that show an image with only high lightness, it can lead to a rather flat LDR-like appearance, and even annoyingly light, so more stringent control of all or many between gray values is desirable. For example, the creator of an ad could make the LDR ad lighter and more vulgar to give it a high visual impact on 100 nit LDR screens. But this influence may even be too high in the dynamic range of 2500 nits (in addition to the PB_D 5000 nits on the display for the viewer who bought such a display), therefore, for example, CombRng, taking into account both, the need to show nice light lamps in the film, and the extreme graduation of the LDR, may for this case it is necessary to be, for example, LMC 1500 nits. The device can use methods, for example, measuring the global contrast of an LDR image, for example, counting the number of pixels above k% = for example, 80% brightness in an LDR image, high and extreme average brightness frequencies, or changing the number of pixels above k%, etc., and from here to determine an extreme measure, and from here to determine, for example, at least how light the brightest parts of this LDR commercial can finally become when displayed in CombRng, and for example, from here to determine which LMC should be (for example, the lightest pixel of pixels LDR can become 300 nits, depending on the non-linearity used, or depending on the required degree of lightening to combine the brightness of the faces, and the lightest lambert pixels of the HDR film (i.e. identified in the lower range of this film, for example, as 5x the coded middle gray of this main section for an HDR film) can, for example, become k times larger or smaller, i.e., by for example, 200 or 450, and light bulbs can become 5 times 300 nits, i.e. LMC = 1500 nits).

A qualified reader understands that there are various ways to obtain CombRng (or anc), for example, it can be determined mainly by an HDR film, and then determine how to coordinate the LDR image with the HDR film displayed in this CombRng (according to the examples described above), or as presented here, it can be determined mainly or to a greater extent by the characteristics of the LDR image, ensuring the safety of sufficient HDR image quality when determining how far the brightest HDR objects can go beyond the lightest or average colors in the LDR image, or iteratively, until an acceptable optimum for two will not be found, etc. It all depends on the needs of a particular situation, and a specific embodiment of a device combining behavior. Which of the two images is of the highest importance may be present in different ways, for example, the viewer may, in some embodiments, use his remote control to indicate that he wants the minimum interference of his film from any of the images will be combined, or the creators of the content can give instructions on its content, about which the device of association finally makes a decision, etc.

Returning to FIG. 3, the first image source 350 delivers a first image, which is assumed to be an LDR image Im1_LDR, i.e. standard encoded eg Rec. 709 (of course, there may be a decoder block that is not shown for it). This may, in the case of a recent video signal, be annotated with its own F1_L functions for conversion to HDR, or may not be annotated if it is an old SDR video signal. Those. it can have F1_L functions linked in its metadata, allowing artistically optimal conversion to other dynamic ranges than its natural 100 nit DR. In the case of missing functions, the combining device can always apply on average acceptable working functions, and ratings. Again, the source 350 of the image may be various sources, for example, a security camera on the front door of the viewer's house may have a PIP image when someone rings the doorbell, etc.

As explained, according to one of the various possible algorithms, the dynamic range setting unit (302) analyzes the situation, for example, the brightness characteristics of the first and second images, and sets a suitable dynamic range (CombRng) of the brightness of the combination, in the above example with LmiC = 0 nits, and LMC = 1500 nit. Block (303) determining the brightness anchor will determine the brightness (anc) of the anchor. For example, it reads from the first image source in the metadata associated with the first image the brightness (L_SA1) of the original anchor of the type “face brightness”, this value is equal to, for example, 45 nits. Knowing that for at least one of the face images seem important, it is necessary to establish that there will be a good complexion anc in CombRng (of course, other options for implementation can decide first on anc, without even looking at what initial brightness of the anchors were set , or seeing that the brightness of the anchor was, for example, the middle gray in the light subassembly of the outer space, it can determine for CombRng the general middle gray, and the d_anc position, where to place the brightness of the bright outer space compared to the general neutral color rendering in CombRng. depending on the complexity of the embodiment, a typical embodiment may want to check what brightness value of the second source anchor L_S2A1 of type facial color indicates what facial colors are currently in this frame of the film. On the normalized axis of brightness, this will of course be a lower value, since a higher range is used for HDR effects, such as brightly lit objects or a part and, or lamps. Nevertheless, if the acceptable value in the range of diffuse objects up to for example 400 nits is 180 nits, and we find the value is only 30 nits, it is known that we have either a face that is dark in nature, or a dimly lit face, or both. In any case, even if this person should look darker than in the commercial, it will be much darker if it was decided to put the value anc on, for example, 180 * 1500/2500. Therefore, we may want to calculate anc value, which is slightly closer to 30 nits, the position LDR brightness of the face on or around this value, for example, 0.8 * 180 * 1500/2500 nits, and also have HDR colors of the faces displayed closer to this value anc, e.g. 0.3 * 180 * 1500/2500. This ensures that we have properly blunted faces for the LDR image, and properly darkened faces, as they should be according to the story, for the HDR film, and both coordinated according to the optimal anc value. Of course, the same can be done taking into account other or additional brightness of the anchors, for example, medium gray, in particular, for the area illuminated in HDR.

By setting CombRng with the brightness of the anc anchor, the only thing left to do as a whole is to optimally display two images in this dynamic range according to one specific brightness value, which is determined because it should fall at or near the value of anc. One way to do this is to linearly highlight the luminance, with the optimum contrast value C, what the slope of the line of the luminance display function is, and then highlight the luminance by calculating the linear equation until clipping occurs (and the clipping value can be controlled together with the value C; note that it is possible to cut off to some value in CombRng, i.e. below the maximum LMC value). Display down to CombRng in this example can also be done taking into account the same math that decoder 251 uses, namely using at least a brightness display (F2_L) with a shape that encodes the desired behavior for the lighter and darker parts images defined usually by a human calibrator. For example, when CombRng is known, it is possible to have pixels corresponding to the second image displayed by the decoder 251 directly from the ImR_HDR HDR image 5000 nits. The color conversion unit (310) is configured to do all the necessary color transformations of the pixel, in particular at least a suitable brightness control, so that the combining unit (320) can apply simple algorithms (for example, simple additive weighting or pixel replacement or block switching, since all colorimetric optimization has already been performed). The luminance reading unit (311) of the source anchor will at least read one brightness of the original anchor associated with one of two images, for example, the brightness of the face of the LDR image, or a good average value for black areas in the displayed scene, etc. as shown above, it can in some embodiments also read several initial brightnesses of anchors that characterize particular areas of brightness in images (for example, a particular gray value, or a gray value associated with a common semantically important object, for example, the average brightness of a light or dark sky , etc.), from both or, generally, all images. Since in some less simple scenarios it is undesirable to display the brightness of two images on the basis of equating only one similar brightness of the armature (anchored to anc), but it is advisable to reasonably position the output brightness compared to anc, the deviation determination unit (312) is configured to determine a suitable offset ( d_anc) brightness. As presented, this can be, for example, since the commercial contains a well-lit and contrasting face, and the horror film contains a gloomy dark face, and, for example, if anc is determined near the brightness of the face of a dark film, then a positive enough d_anc offset is desirable in order to where to display the face colors of the commercial of the face LDR. The intermediate images correctly with the brightness display Im2_HM and Im1_LM arrive at the combining unit (320), which can then, for example, be as simple as a pixel switch, which depends on the position (x, y) of the pixel or places the displayed HDR color (i.e. e. Im2_HM) in the output image Im_o, or the color LDR Im1_LM, either as linear RGB or recalculated in Y'CbCr, etc.

FIG. 4 illustrates everything additional to the selected example, showing displays on dynamic brightness ranges. Im1_Rng is the dynamic range of the LDR 0-100 nits of the first LDR image, i.e. having coded luminances that should fall between the minimum brightness Lmi_L = 0 nits and the maximum brightness LM_L = 100 nits. Im2_Rng is, for example, the dynamic range of an HDR image with LMH = 5000 nits (the MDR image can be optimized for display, in which case LMH will be 2500 nits in the above example). It follows that there is no need for CombRng to be somehow equal to or close to Im2_Rng or the display range of Di_Rng. This display range is shown for comparison, but it may not even be used in combination in some embodiments, but then instead Im_o will go to the color conversion unit to configure the display to display the combined output image Im_o set to, for example, 3000 nits, dynamic range of the display, for example, 1400 nits, or any possible values (the person skilled in the art understands that there may be scenarios where the LMC is higher than LM_MDR, for example, if the combination occurred in another device, for example, in the cable end, and a specific the user has an HDR display with a limited PB, for example 950 nits; or, in the case where mixing occurs at the location of the viewer, and the viewer has a high PB display, for example 7000 nits, but is currently receiving HDR content, such as a news program with PB = 1200 nits, or less, and an LDR image, then LMC can be defined as significantly lower than LM_MDR = 70 00 nits). Then the tuning functions can, for example, be for the content of the HDR movie (i.e. F2_L), since the LDR commercial is already quite harmonized with it. Of course, more complex embodiments can directly optimize both images with a dynamic display range. The reader will understand that situations are possible where CombRng cannot even be, for example, the dynamic range of the display, for example, at the place of production or transmission of the video signal, where the features of the TV viewer are not even known. This is one of the difficulties that the HDR market that needs to be served has been created. In the LDR era, all televisions were approximately the same (with a PB of about 100 nits), so video processing was simple, but now some viewers may have TV HDR 1000 nits, others TV HDR 7000 nits, others traditional TV SDR 100 nits, while others like view images on their iPad or mobile phones, with PB, for example, 500 nits (before starting to reveal all the details of various image content). It should be understood that only with their large deviation, these dynamic display ranges cannot all trivially always be the most suitable CombRng for combining two images and optimal presentation of both. But, and especially, if you need to finally visualize, for example, on a TV or a monitor 1000 nits, it would be most optimal if you increase and combine LDR to the end to 5000 nits; image 2 the dynamic range of HDR, if you then need to significantly decrease again, and the risk , especially when using the F2_L functions, to lose some important part of the lighter colors of the LDR?

We explain that a little further from FIG. 5, which uses the PIP 501 example (but similar problems arise, for example, with temporary masking and adaptation, etc.) in the main portion 500 of the HDR film.

Classic LDR combiners can combine in Y'CbCr (i.e. lum + 2 chroma coordinates), non-linear R'G'B ', linear RGB, or, in principle, any color space (although this may be less typical for systems that must run at video speed but also want to save computing resources). Suppose we have a gray value image.

In the case of a simplified pixel-by-pixel switching of the luma codes Y 'pixels, which will lead to very poor results, since they are defined in such different ways for HDR and LDR images (this basically becomes the main problem of misunderstanding the processing device). For example, everything observed through window 510 in the PIP LDR (for example, the director’s comment) can be clipped to the maximum luma code, Y '= 255, or normalize to 1.0, since it is too light for LDR and in any case not interesting in this single video signal. When rendering, the combined Im_o image in lum encoding, where some of the normalized lum pixels of the original HDR film (for example, a dark night scene), the pixels are replaced with normalized lum LDR images, by decoding it for example with PQ HDR EOTF, PIP will look excessively light (window colors will be visualized as, for example, PB_D = 4000 nits, where you can alternatively, in principle, have pleasantly harmonized colors of the object of the outer space, or at least less annoyingly light clipped colors). This will lead to a re-irradiation 502 also referred to as blurriness or light distribution. If this is not already due to the physics of the display (for example, a low amount of LED backlighting, mutual reflections on the front plate of the display), then often the shining in the human eye can also make very bright areas on the HDR display difficult to observe without using your hand to cover the bright area. In short, the viewer does not like this, and he would like another if the technology allows it. At the very least, it is clear that those PIP LDR luminances and dark portions of the night scene in the HDR movie are not well harmonized. But the opposite can also happen. If PIP follows the sun in an HDR movie, or another very bright part, it may be too dim. Which can also be ugly, for example, colors that are obviously assumed to be white look too grayish. If, as shown on the right, we reduce the brightness of the PIP face, bringing it closer to the brightness of the dark face of the HDR, since the brightness of other objects in the LDR image is related to the complexion (in terms of contrast), they will also become more properly dimming. Therefore, all colors should look relatively nicely coordinated.

One can naively think that all problems can be solved by highlighting the proper LDR sub-band for the PIP image (i.e., the brightness sub-range is 0-100 nits, since it gets into HDR CombRng, whatever it is), which corresponds to its correct decoding with Rec 709 maximized to LDR PB of 100 nits, and not (relative) PB of the display, for example 3000 nits, and then bringing these LDR pixels to a maximum of 100 nits in the dynamic range of the display brightness (assuming that CombRng is set equal to the existing DR display). But as said, although in some situations this may be a good choice, this leaves a solution that depends on the actual dynamic range of the display (observed relatively, this becomes a random ratio). For a very bright scene illuminated by the sun, it can happen that next to the PIP LDR there is a maximum of 100 nits, on the 3000 nits display there are neighboring objects with, for example, a brightness of 1500 nits. This makes PIP quite disappointingly dark, and its colors will look gloomy rather than bright, and that’s not what the person who paid for the ad was counting on. The same can happen when the dynamic range of the HDR (Im2_Rng) is combined in the image. Therefore, it should be understood that suitable mixing is not trivial, i.e. good structure as required in the present invention. Also, if there are two HDR images, even if the EOTFs are not different, you can be sure how the content creator used this available range, i.e. colorimetrically built his complete system (content production rules, etc.). Even if the PBs of the two encodings are not too different, for example, 5000 from 4000 nits, the creator has to make significantly different decisions (for good reasons, for example, so that his specific story with nicely constructed HDR scenes looks spectacular, while the other creator who creates the news show, easily convertible to other dynamic ranges, for example, for viewing on portable displays, while images still need to have some interspersed HDR in them), for example, the lower range in which Lambert reflective objects, for example, scenes of interior space, fall. The first creator can set them up to 300 nits (in the 4000 nits system), while the creator of the second image can decide for his scene, for example, the internal space is some kind of space station, bring up to 900 nits (in the 5000 nits system). Sometimes it is as simple as if you wanted light, or unrealistically dark space stations. Even if you follow very strict rules regarding lighting or at least the brightness of visualization of faces, for example, 50% diffuse white (i.e. 300 or 900 nits), without proper harmonization of the faces in the pool, it can look significantly, even mysteriously, different, perhaps even shining. And the same thing can happen with many other colorimetrically critical HDR image objects that may contain, for example, a pool may look somewhat shining and illuminated from the inside, or just gloomy ordinary water, or dark like a night scene, or like a foggy day, or someone it can even take care of the brightness on the lit poster on the front surface of the diner, especially if it refers to other objects in the scene, and the story and the transmitted mood, etc.

FIG. 6 shows two possible color conversion examples in CombRng. We have as the input image one of the images to be combined with the relative input brightness L_in. As explained, the device needs to define a color conversion function (FF_1) that allocates the output brightness L_out_comb in CombRng to all possible input brightnesses. Having this function, for example, loaded into the LUT, you can then start processing incoming pixel colors for successive images. The first simple linear definition gives a linear function 601, defined by two parameters anc_Fc for the brightness L_SA1Fc of the complexion in the input image, and a multiplicative constant C of contrast. This function may be cut off in black or white, which may or may not be desirable (again, depending on how simple the decision-making side wants it, whether it is the final creator of the content, or the manufacturer of the device). In more complex embodiments, we can improve after that. A brightness display function 602 shows how it is possible to nicely configure a desired display based on only a pair of important anchor brightnesses. It will, in the general case, although not much more difficult to use, produce significantly better results. Perhaps this image of the LDR commercial had too many bright pixels, for example, in the background. When applying the specific rules of photographic composition, you can optionally photograph the main person against a darker or lighter background, for example, an average gray background, or a very light or white background, or potentially even against a dark background. Often in the video signal, which will depend on the actual background (for example, if the video signal reports a fashion photographer who intends to photograph her model on a white background, the video signal will show it talking against white walls or the backdrop to be photographed). In studios, the background can be built. In a weather forecast, a weather forecaster can stand opposite (usually a virtual green screen) visualization of a photograph filed by a viewer, not a professional. Or in a movie, someone can sit opposite a bright public screen, for example, an LED screen, for some reason, etc. This is another example of the need for coordination, somewhere, since a pleasantly coordinated LDR photo in the LDR weather forecast would have been shown in the past, but soon viewers will be able to send HDR photos to show what else is being created in LDR mode, or someone can now a couple of years send a very low quality LDR photograph for display therefore mixed in an HDR program. In any case, for curve 602, it is assumed that there will be many too light pixels in the LDR image to resemble PIP. It may not only be necessary to reduce the output values of function 602 for some white, but also, for example, start soft clipping for the lightest input brightness. Therefore, curve 601 will give too bright results. The device (301) did this by reading the three types of brightness of the original anchor in the input image, and highlighting them with the corresponding brightness of the anchors in CombRng output image. Namely, the content creator also set a good reference level for diffuse white, i.e. what brightnesses in their image correspond to Lambert reflective objects, which are white (i.e. with a reflectance of, for example, 90%), and which are illuminated on average. The device will highlight the anc_DW brightness for this LDR image for the diffuse white portions of the LDR image in CombRng, which is clearly lower than what the linear brightness conversion curve 601 will give. Therefore, the combining device can now intelligently decide how to convert the brightness of such bright pixels in the image, whatever it defines as the specific needs for these brighter pixels in the combining image. A similar suitable behavior can be set at the dark end, for example, in this case, to avoid clipping, by specifying the brightness of the black anchor L_SA1RBk in advance in the content, and determining the suitable brightness of the black anchor anc_RBk during combining. Then the device can, for example, decide to continue the function linearly to absolute black (i.e., 0.0) from this point anc_RBk, and make it increase with a very small slope over anc_DW. Or it can optimize black areas for a specific visualization situation, with knowledge of L_SA1RBk the brightness of the black source anchor.

Let us explain a couple of possible more complex examples in FIG. 8, namely, how some of our embodiments of the device and method can not only construct piecewise continuous functions based on different brightnesses of the anchors, but also determine the shape of the parts of the curve (FF) of the brightness conversion based on the re-gradation behavior specified in the color conversion functions ( F2_L, and if any, F1_L) as shared with the image from the creation side in the metadata.

It can be assumed that the determination of the function (FF) between the three brightnesses of the anchors occurs again as before (by linear or non-linear interpolation), but that the device intends to determine the brightness conversion behavior of the improved brightness conversion function 810 in FIG. 8B, based on the knowledge obtained by the creator of the graduation of the content from the received metadata (F2_L). F2_L is now the brightness conversion (for the second image, but the same can be understood for all other images to be combined in the output image of the union) of the second image, from its natural dynamic range (DR1) to another predetermined dynamic range (DR2), t .e. the input brightness L_DR1_im2 is associated with the output brightness L_DR2_im2, which is a different kind of dynamic range for the same HDR scene depicted). From the graph you can see that there is a main part (between anc_DW and anc_RBk) that is well represented (will be visualized) with good enough bright and contrasting brightnesses (for the merging situation), but that black areas (below L_SA1RBk) can be easily reset, at least , for some species, corresponding to some dynamic ranges, because the function quickly drops to zero, and cuts off. Those. in this example there will be no very important objects in this sub-band of the darkest brightnesses. The device can use this to determine its suitable function 810, for example, quickly resetting the function for these dark black areas, and can determine this based on how dark or contrast it wants the combination, or at least part of the second image in this merge image (for example, based on the number of pixels with brightness below L_SA1RBk in this second image, or after a suitable merge, for example, scaling, in the output image Im_o, etc.). Those. the combining device looks at the coding re-gradation behavior of the received two graded images (i.e., how the darkest pixel luminances are processed when re-grading from a reference, for example, graded PB_C 5000 nits image, to 100 nits SDR grading, in this part of the form of the received brightness display function) , and defines a partial form, which is similar to some extent.

We also illustrated the opportunity for the lightest pixels. Violation of the continuity of the curve F2_L indicates that there are two sections, for example, Lambert objects in the scene of the inner space, and light colors that represent the lamps (or similar situations). Especially if we have lamps, this means that you can generously highlight their final brightness, depending, among other things, on the dynamic range on the MDR and / or CombRng display. Therefore, the device can determine a suitable average L_RefB level for these pixel luminances, and finalize the shape of curve 810 in this way. In FIG. 8A shows how the adopted brightness display function F2_L looks to display between the received grades of the reference HDR and SDR, with the corresponding shape and behavior of the light partial (802) and dark partial (801) functions.

A qualified reader understands that the present various embodiments are applicable in many systems to many integration scenarios, and with various components in various combinations, but we will explain some illustrative additional features in the two figures.

In FIG. Figure 9 shows an example television production system. This may be production in a multi-camera studio (first camera 901 and second camera 902), and some live content may come from a location, etc. It can also be field production, with a mobile studio, etc. The CTO must mix camera feeds, and other video signals, some of which may be HDR, and some, for example, LDR, and HDR videos, of course, may differ from LDR videos that are the same (the universal uniform standard Rec. 709 that existed) , the presence of various types and characteristics. It is desirable to determine the optimal association, on the association determination apparatus 903, which may be located in the production control room. Although it defines the association, we show in this example that this association does not necessarily lead to a fixed Im_o (on the contrary, the technical director only needs to determine the appropriate association, but the features of this association can be transmitted in different ways to different receiving places in addition to the video transmission pipeline moreover, the receivers may even want to redefine the union differently, for example, slightly different from the one preferred by the technical director; i.e., the association can be transmitted as a rough image and function and other metadata, before the association). In this case, the image signal 910 with metadata can be transmitted, which contains two images (for example, with scaling functions to determine the resolved combination of PIP), and with a specific CombRng, anc, and possibly also with color transforms FF_1 and FF_2 for displaying images in CombRng, so that any receiver (e.g., transmission receiver 904, at some intermediate node, e.g., a transmission control room) can actually apply the final join. For potential redefinition (at least one of CombRng, anc, FF_1, or FF_2), the production side may also carry additional metadata on the video signal, for example, does it contain many bright scenes that need additional 3x lightness above average lightness a movie (i.e. how it is desirable to display the brightness of this content in the final CombRng), features regarding the brightness of the object or section, or semantic information, for example, the brightness range and / or location of the sky, etc.

In FIG. 10 shows another possible embodiment, namely an intermediate combining device (1001). This could be, for example, in the transmission control room of a cable provider or similar. He can receive the image (s) from the main production (created in Fig. 9, and transmitted either as final images or initial images with sufficient metadata for them to be properly combined according to the director), namely Im_Prod, and for example, the source of the local commercial ImLocComm . The intermediate combining device (1001) can potentially also perform dynamic range conversion, for example, for converting 5000 nits Im_Prod to output images encoded with PB = 3000 nits, which may be better suited for the purpose, for example, additional distribution over some predefined video transmission system . The latter, of course, is more interesting if the device 1001 outputs two separate image streams for further mixing, and in this example it is assumed that a final suitable image stream is generated (only for display settings by final receivers), which usually already have an LMC other than 5000 nits.

The reader can also understand how similarly the consumer’s stronghold system can combine HDR images or video recordings, for example, a film together with images received through a computer and integrated into an IC set-top box, personal computer, etc., or how it can be used in video conferencing rooms, etc. Im_o in FIG. 3 can be sent to a TV, one of the images can be taken to a Blu-ray disc through a BD player, etc.

In FIG. 11 and 12 show a couple of possibilities for determining CombRng. Log_base2 units are plotted along the brightness axis, since this gives a better correlation with the person’s visual lightness and impressions about the lightness of the images during visualization. Algorithms for determining various pixel luminances are also predominantly formulated in such a logarithmic representation; in fact, some of our preferred embodiments for encoding a video signal also work in such a perceptually uniform luminance representations. In these embodiments, the device has already determined a good anc_FC value. For example, a device suggests that an LDR value of approximately 30 nits would be a good value, also for HDR imaging. Now the device will look at the situation of the distribution of brightness of the two images, combined with the information of the union, to achieve good choices LMC and LmiC. The device will have rules for this, which can be selected and changed, for example, by the viewer controlling the settings. In the first two examples, the image of the higher dynamic range (Im2, shown by a thicker line) is the main image, and the image of the lower dynamic range (not necessarily SDR) Im1 is PIP, for example, occupying 1/4 of the area (which is one of the control parameters , which can indicate the visual predominance of objects or Im1 plots compared to objects in the Im2 scene. In these examples, the device will want the HDR Im2 to prevail, so it first determines CombRng and places Im2 there, and according to the prescription, and then harmoniously places it Im 1. In these embodiments, the complexion has already been selected for the optimal final complexion anc_FC, so the question remains about highlighting the remainder of the brightness, both extremes of CombRng will affect this, and from there the increasing color conversion functions to match the brightness coordinate with other brightnesses, and the variability permissible for changing the form of these functions after as the effect of lightness requires it. In FIG. 11A, HDR Im2 has a histogram consistent with a typical face scene when rendering around anc_FC, i.e. the face in Im2 is usually lit. We see from the long tail to the black areas that there are dark areas in the scene, for example, it may be an unlit room somewhere behind the actor. It may be that some of the colors of the pixels will be invisible in some visualizations because they are indistinguishably dark, but at least they are encoded and calculated by the device, so that any final visualization device (for example, TV, which sets up the display for this MDR range, and vendor-specific lightening treatment for the darkest areas) can use this data appropriately. We see, moreover, that there is a mode of light brightness HiLaIm2, which can correspond, for example, to a lamp in which we would like to see some structure ideally (for example, how a lamp is reflected in its metal lamp). The image, to be PIP, Im1, has a characteristic with a high peak pixel count HiModIm1 near white (the lightest brightness in this code). This may be because a person (who also has an acceptably large area, as can be seen from the mode near anc_FC, where the typical complexion should be in normally lit images) is standing in front of a large bright screen, for example, a light box that is near white or even cut off potentially to maximum white. This peak is more likely to be too bright and outrageous in the union. However, in this first example, since the complexion is in normal positions in both images, and there is an acceptably large number of brighter colors already in the HDR image (both modes of diffusely lit objects, for example, under the main set of lamps, and some HLaIm2 highlights, which 100 times brighter than the face), you can put HiModIm1 in its normal position (i.e., the same number of points above the anchor in CombRng, encoded in Im1 when decoding into a linear representation of brightness, which is also a normal decoded value in this example about 100 nits, because the brightness of the face is in its normal position LDR), because there will be a normal brightness, which can also be seen arising in Im2, i.e. it will not be too disharmonious. This also means that the Im2 selections can remain in their normal positions (since re-matching or re-coordination is not required, since all consistent colors fit into the current HDR image brightness distribution profile, which has its selections where they arise for this image), t .e. the device can select the LMC HiLaIm2 brightness mode as the upper limit, or the peak lightness of the HDR Im2 range in the absence of brightness up to PB. This is not necessarily the case, but may be an acceptable choice if additional features are not known, for example regarding typical use additionally under the HDR image processing chain line). In FIG. 11B shows a similar situation (the same Im1 is PIP), but now Im2 has a different histogram (for example, a later frame of the movie). Here, faces are subject to visualization (i.e. if they are visualized on their own without combining with other image content) relatively dark, for example, 4 nits. For example, a person may stand in the shadows. In this example, the device does not want to make the faces in PIP so dark, but still wants more coordinated darkness for Im1 faces as well. The device decided to take the anc_FC setting one point above the brightness of the faces of the HDR image, i.e. 8 nits, and render Im2 faces one point below anc_FC (since they must remain dark to maintain the artistic purpose of this movie scene), and (depending on, among other things, the PIP size) to render faces of the LDR Im1 commercial in CombRng 1 point above anc_FC (still a distinctly lighter part of the image, but not too disharmonious light). When maintaining the relations of all the brightnesses of objects in Im1 (linear display through the brightness of the anc_FC anchor), this will be already lower than the brightness of a large section of HiModIm1 to below 60 nit and not about 100. Yet, when the device decides on the influence of lightness (which it can determine purely at luminances, for example, by calculating the CHL contrast between some reference color of an important part or object Im2, for example, the complexion anc_FC, and this brightness HiModIm1, or by also spatial analysis, for example, by weighing a larger and closer selection region Im1 as more seriously contrasting with the important, for example, central region of Im2), it can still consider that the large bright region of Im1 is too light in comparison with the rather dark appearance of most Im2. Therefore, it can make a decision to reduce the brightness of this HiModIm1 mode by 1 point, bringing it up to 30 nits, so as not to be too distracted by the small PIP in the upper right corner. As shown above, the device can do this in different ways, for example, by reducing the contrast coefficient C for converting the brightness in CombRng to higher than anc_FC, or some non-linear function that can be calculated based on some desired maximum LAMX position for the lightest possible pixels in Im1, potentially additionally determined by placing some brightness of the diffuse white anchor, etc. Note also that the device in this embodiment also performed some harmonization processing in the black luminance subband. Since the black portions of the HDR image go very deep (and it can be assumed that several displays can visualize this, at least when the algorithm is applied before optimizing the display, when implemented with a connected display, or even in the display, of course, the device can take into account which display can actually render with sufficient visual quality, i.e. visibility), the device can also deepen the black areas of Im1 so that both sets of black areas are more harmonious. This can be done by distributing, for example, all the luminances below encoded for Im1 as LA_Bk (some black separator), for example, by stretching the local contrast by 2 times. From FIG. 11B, it can be seen that in this example, maintaining the brightest portions of the HDR image may still be useful, i.e. The LMC is determined again to the maximum value necessary for the correct visualization of Im2 on displays that can do this, since the task, for the most part, was to coordinate Im1 on a sub-section of brightness that falls much deeper. Note that the device in this example chose to limit LmiC slightly higher than the darkest colors in Im2, for example, X points below anc_FC, because it decides that the colors that are too dark will be ignored by the viewer in the union anyway. Color transformations displayed in CombRng can take this into account and lighten the darkest colors to bring them to CombRng.

In FIG. 12A, an example is given in which Im1 (i.e., in our simple explanation, the commercial) becomes the predominant image. Maybe viewers don’t want their film to shrink when commercials begin, but PIP can be, for example, the second channel to watch, or even contain the originally viewed film Im2 switches to a small PIP, if something important starts on the second channel, the viewer also wants to watch (for example, this image of Im1 corresponds to the beginning of the news, which the viewer does not want to miss, because an important news item is assumed). Now suppose that Im1 is an SDR image, but someone (for example, a content creator, for example, a commercial, can set the following in metadata ʺ visualize the brightest brightness at 500 nits, on PB displays at least 1000 nits ’, or the viewer can choose some custom setting that indicates how dynamic or conservative he wants to see the brightness of his content, even if the SDR) made it subject to visualization at higher dynamic ranges as very bright, since there is a room for this on lighter displays. Therefore, if decoded normally according to Rec. 709, the histogram Im1Nat will be obtained, but in reality, it will turn out in CombRng, the histogram Im1adj to be visualized with values for Im1 in the combination of up to 500 nits (at least when the creator of the desired SDR content is completely accompanied by the combination device, because, for example, it is set equal to free content watch mode, paid by the owner of the commercial). Then FIG. 12A provides an embodiment of a calculation where, for example, a younger viewer set his system to dynamic mode. In order to be able to expressively contrast with the large bright area coming from Im1, the device needs to install (at least in this intermediary, indicating CombRng whether the display can be adjusted) light lamps, for example, up to 8000 nits, i.e. you must set the LMC to 8000 nits (even when it was, for example, 4000 nits in the original Im2 decoding). In FIG. 12B shows an embodiment that is different from the first possibility because the viewer has set his system as conservative (because he does not like to see excessively bright images). Now this is another consideration of harmonization, where the lightness of the SDR content is leading, and quite bright, so the reader sees that a different result of the output of LMC CombRng is made. Since there are already so many pixels from Im1 that are very bright that contributes to the specific or perceived overall high lightness of the combined Im_o image, the device may wish to reduce the brightness of the HDR HiLaIm2 light mode, and therefore LMC only up to 2000 nits (despite that the film was, for example, a reference display, graded for grading PB_D 10000, and transmitted as HDR images PB_C 5000 nits, for example, highlighting the luma SMPTE EOTF 2084). Although now there is less contrast between the lightest lamps of the HDR image and the bright large portion Im1 (2 points instead of 5 or more), i.e. an HDR film will lose some of its interspersion when a PIP pops up, at least the overall image is not too bright. The viewer may desire this kind of association.

Also, if the CombRng range is set to anc_FC, the device can look at various aspects of pixel brightness and objects in both images, for example, set one or more corresponding typical brightnesses. For example, it can look at the percentage of pixels in both images k times higher than the average gray, or more than l times lower than the peak brightness of the decoded images, and decide from these values in both images what good values of several typical brightnesses from which it starts at least the maximum brightness possible in the LMC pool. If the embodiment of the device uses one typical value of the brightest brightness of the objects for each image, it can determine how they will relate when displayed in CombRng, i.e. for example, what brightness contrast they will have, and offer a more harmonized contrast. Of course, additional values, for example, illustrative values of the average gray of most colors in the scene (average gray may be some actual reflection coefficient of 10% in some typical scene lighting, but can also be used to indicate, for example, mode, most value, or average, etc. (dp of pixels in the image), can help determine what the various intra-regional contrasts will be in the combined image, for example, the lightest parts of Im2 versus the middle gray part of Im1, etc., and to choose harmonious relations between them. For example, some embodiments of the device may always be configured to select some typical contrast values, with light, dark, and middle areas highlighted to some subsections of typical CombRng brightness (which may, of course, differ depending on whether the LMC is approximately 1000 nits, 2000 nits, 5,000 nits, or 10,000 nits), and these values can deviate from (for example, displaying k points higher in CombRng) when some characteristic type of image is found, for example, a night scene, a well-lit news program, evening in nature, and etc. When determining a good anc value for, for example, a good face or lightness of gray (or areas of bright light), the device can measure in which observation environment the viewer is sitting. For example, the front camera in a TV can summarize the characteristics of the brightness of the environment in one or more characteristic values that can be used in the equations to scale the value of anc_FC, etc. (for example, 1 or more points up or down), and the contrast required for other brightness compared to the brightness of the anchor, etc. For example, the camera can see that the viewer (by detecting the face) sits on the couch in some local lighting, and can detect the second areas that are the walls and other objects behind, usually less well-lit. If a single characteristic value of the lightness of the environment needs to be derived according to the embodiment, the lightness of the surrounding walls will be estimated higher when calculating (for example, a * surround_average_L + b * local_spot_average_L), since local lighting will usually come from above and the influence of the less experienced environment where the viewer is located ( for example, which the brain sets as deep black in this environment, with the display forming part). From these various explanations, it should be clear to the reader that in various embodiments, there may be different ways to obtain all CombRng, one or more brightness anc, and the shape of the brightness display function, and in fact at the end, a harmonious output image for each situation.

In FIG. 18 shows only some explanation of the universal typical capabilities of certain color conversion functions FF to be applied to the brightnesses of at least one of the images or image parts to be mixed (or some or all of them having the corresponding FF_x functions), and from these examples the reader can present additional opportunities, following the general principles of our invention and its various embodiments. For example, in FIG. 18a shows two simple ways of defining a function FF_1 made up of two parts (in this example, ending at the exact endpoints of a range that can be deviated from in other scenarios). If the result of applying the brightness function (LF1_o) to the brightness of the input image equal to the original anchor (L_SA1) should be exactly equal to anc, we get from this internal program of the device where the function was obtained from. In the case where the output brightness of LT2_v can be close, it is possible to determine a similar slightly different two-segment function FF, which passes through a point with the value Y anc / k, where k is, for example, 1.5, 2, 3, or 4.

FIG. 18b shows that complex functions can be built around this, for example, a normally / dimly lit middle gray anchor, for example with an S-curve for black sections. The combining device can offer such a part of the brightness display function, for example, usually, referring to the brightness grading functions (for example, the fine adjustment function) which was transmitted according to our encoding principles to obtain one of the dynamic range images from another one actually received. If the combiner sees this S-type behavior for black areas, it can understand that it seems to be some area of important objects in the middle of the black areas, which should have good strong contrast, while the deepest black areas look mostly ignored, since the function is shown that they can almost be clipped to the minimum black (0, or any Min-Black happening). Therefore, somewhat more complex embodiments of the combining program will follow this behavior in its specific FF form, but now the function changes shape, since the display does not pass between the reference brightness range of HDR 5000 nits and the SDR range of 100 nits, but, for example, 500 nits CombRng (therefore it is not necessary to convert a little black, according to similar requirements, for the most part, making sure that the middle black areas are best visualized, but it will be somewhat different). Similarly, for part of the function for light colors, the combining device may decide to follow the shape features corresponding to the brightness sub-bands of the critical image object to a more accurate or lesser extent. In FIG. 18c shows an example where the combining device controls the contrast C of a certain “middle range”, for example, the SDR image to be combined, and also in this example, the device decided not to stretch the upper part of the FF function to the maximum to CombRng (output image Im_Cmb), as according to the embodiment , which can lead to too bright visualization of the SDR pixel (the reader must understand that the histograms of brightness of an HDR image usually have far-spaced columns, where the column is the brightest brightness, for example, only containing 20 small mirror inserts of 20 × 5 pixels, whereas the SDR image contains closely related histograms, with many brightnesses around the maximum PB_C = 100 nits, for example, potentially even large areas of the image clipped to white scenery of the outer space, and, of course, in a relatively dimmer or darker HDR film, these areas of the combined pixels should not be rendered so as bright as PB_Comb = 2000 nits, so as not to interfere with the movie). Obviously, previously, embodiments of a combining device may only have a couple of such pre-programmed rules in their software or hardware, but embodiments in the future will perform sophisticated analysis to arrive at a completely balanced combination of images. Therefore, various different options for implementing the system can be built on the basis of the basic processing device to correctly mix images or videos, for example, so that a person can ask, long to the (often unknown) actual implementation of the mixing, in an elegant, not too time-consuming way, what his content needs, by choosing a good value for the brightness of the anchor of its content. Various embodiments may occur in a mixing device, for example, where the device uses the same type of anchor (although some devices may also decide to use a different type of anchor, and convert the harmonized display of two images from their types of anchor to the available brightness of the dynamic mixing range). A qualified reader can understand that in practice the first situation can be implemented, for example, in a mixing device that operates in a certain subordinate mode, checking that at least the main video signal, for example, setting what its critical brightness of the anchor face was, then sets that a good brightness of the face will be in the dynamic range of mixing (for example, already taking into account the final characteristics of the visualization), and then it will convert the second image in brightness (even if it does not contain a face), to harmonize with this brightness of the important face in the first image (i.e. as presented in the dynamic range of the association). An example of the second strategy, in which the mixing device is the leading one, for example, if the user watches photos on TV and sets some standard for this (setting the brightness of the anchor as an illustrative unit value for the entire typical lightness with which the visualizations of the photos should (approximately) be consistent with; and usually also representing the dynamic range, for example, he specified a not too light (soft view) dynamic range of combining already with photos the brightness displayed in this range, until any other image data can come in and mix harmoniously, and can also specify that the photos in currently they use, for the most part, half of this range, i.e. all of our majority of their pixels have brightness below PB_Comb_PhotView / 2, despite the fact that there is some place in the dynamic range for combining lighter pixels, there should be a desire for secondary content ) The mixing of the content of the secondary image (the primary content is photos where it is already displayed in a certain brightness display) will then be performed by the device, referring to the value of the anchor in the content (which may differ from the type of anchor for viewing photos, but according to the formula, the calculation of the mixing device is associated with it, for example anc_face_inmix = 1.3 * anc_photviewapplic), and the set dynamic range for combining photo viewing with brightnesses between, for example, 0 or k nit and PB_Comb_PhotView. It is very useful to have a universal single (capable in all situations) system for mixing HDR image content, which can handle various situations, and determination in different versions will depend on what is optimal for the situation and what the device already knows at any time.

For example, in the case when the final visualization system is not yet known, the best thing a mixer can do if it is necessary to harmonize the contents of two images is to at least watch how they are best mixed (for example, does one of the images have excessively more areas ( possibly large) high lightness, which must be muted to enjoy other image content). In this case, the final stage of display optimization (for example, on the consumer’s TV) may need to perform more complex optimization, taking into account the images already indicated as they should (in general) mix optimally. However, according to embodiments where the visualization environment is already known, and in particular where it has strong desirables, the mixer can balance more to the final look (i.e. visual quality needs a large coefficient with a specific Combrng display, and for example sacrifice a somewhat perfect the only visualization of one of the images when performing another display). In the case of visualization capabilities of primary importance in determining CombRng, usually at least some typical considerations will be applied to the intended properties, for example, HDR images (for example, merge device software rules determining that most of the “universal” typical HDR images are likely will be located (+ -) below, for example, 500 nits, and light objects should be visible if they are highlighted as displayed brightness, for example, up to 1500 nits. If the display then, for example, only has the capabilities of PB_D = 1000 nits, this will correspond to some error non-optimal visualization of good quality HDR images (i.e., light bursts such as 1400 nits will be somewhat too dull.) In this situation, although the actual PB_D of the available display can have a big impact on determining PB_C CombRng to mix the image content before rendering, the combining device is still can take a solution for determining that PB CombRng is slightly higher than 1000 nits (and apply the final display setting of the mixed image to display it in the range displayed on the display), for example, 15% higher i.e. 1150 nit. Of course, ideally, in the best embodiments, the CombRng optimization, and also a suitable anc point in this dynamic range of association, will take into account, and balance, all known factors (i.e., the need for perfect content visualization, and all practical requirements and limitations mixing phases of the current HDR image). The optimal definition of the point anc can usually look at what is available (possibly) after the previous task of optimal CombRng (but according to the above, various other factors can be provided, for example, what kind of content semantically contain images, and in some embodiments, it is even possible what the end viewer expects for such image content, etc.). Therefore, various embodiments can set anc differently, going from a fixed setting in the device, which will (whatever is specified as the anchor types of the original image) always connect everything with suitable face brightnesses, according to a system that has various options like anc and selects the closest to what was determined in at least one of the source images to be combined (i.e., the brightness anc in CombRng of the same type as indicated in at least one of the images to be combined, for example , good complexion, but in CombRng, i.e., optimal for the situation of combining, which, it should be understood, differs from that for visualizing each image independently), i.e. usually a leading image, to determine in the on-line mode of the device what would be the appropriate value of anc, taking into account all the physical features of the situation (for example, the distribution of the brightness of the content pixel and the display capabilities), etc. The latter variant of the embodiment is good if, for example, the complexion should be determined optimally taking into account the specific limitations of the visualization, for example, how dark the display can show dark objects well above which the brightness of the face should fall, which can be especially critical for faces in dark areas source images. For clarity, some embodiments of the device can work by determining a certain anc value (for example, good average brightness), although the two currently incoming images can be set with anc brightness such as complexion, since in general the only thing that is necessary for the device to work , they can be connected by the device (i.e. that the typical face brightness in the input image can be displayed in the vicinity of the anc value of the “medium type” CombRng, since the device decides that sufficiently good images will result if the faces, for example, are 50% brighter than these, in particular, first of all and independently established by the well-suitable value of anc. Therefore, the anc types of the source image and CombRng must be connected or related.anc can optimally and elegantly generalize all colorimetric complexities of the image, and in many situations only one point may be required coordination to combine anyway. For completeness, it is obvious that the value of anc in the dynamic range means that there are no endpoint brightnesses, but there is information additional to this. Regarding the establishment of a dynamic range of brightness, a qualified reader can understand how this may include determining a higher brightness, and for embodiments that systematically do not accept the lowest brightness as zero nit, also setting a lower brightness, again, depending on balanced the needs of known factors, for example, the amount of relevant image content in darker areas of the image, the capabilities of the visualization system to display dark colors, etc.

The algorithmic components disclosed in this text can be (in whole or in part) put into practice in the form of equipment (for example, parts of specialized ICs) or in the form of software running on a special digital signal processor, or a universal processor, etc.

A person skilled in the art will understand from our view which components may be optional enhancements and may be implemented in conjunction with other components, and how the (optional) steps of the methods correspond to the corresponding means of the devices, and vice versa. The word "device" in this application is used in the broadest sense, namely, as a group of tools that can achieve a specific goal, and therefore, for example, can be (a small circuit part) IC, or a special device (for example, a device with a display), or part network system, etc. ʺConfigurationʺ is also to be used in the broadest sense, therefore it can contain, among other things, a single device, part of a device, a collection (parts) of interacting devices, etc.

The term "computer software product" should be understood as encompassing any physical implementation of a set of instructions that allow a universal or special purpose processor, after a sequence of loading steps (which may include intermediate conversion steps, for example, translation into an intermediate language, and the final processor language) for input commands to the processor, and to perform any of the characteristic functions of the invention. In particular, a computer program product may be implemented as data on a medium, such as a disk or tape, data present in memory, data transmitted through a network connection, wired or wireless, or program code on paper. In addition to the program code, the characteristic data necessary for the program can also be implemented as a computer program product.

Some of the steps necessary to complete the method may already be present in the functionality of the processor instead of those described in the computer program product, for example, the steps of inputting and outputting data.

It should be noted that the above embodiments illustrate but do not limit the invention. When a person skilled in the art can easily implement the mapping of the presented examples to other parts of the claims, for brevity we have not fully disclosed all of these options. In addition to combinations of elements of the invention combined in the claims, other combinations of elements are possible. Any combination of elements can be implemented in a single special element.

Any reference position in parentheses in the claims is not intended to limit the claims. The word "containing" does not exclude the presence of elements or aspects not listed in the claims. The use of the name of the element in the singular does not exclude the presence of many such elements.

Claims (21)

1. Device (301) for combining two images or two video images (Im_HDR, Im_LDR), one of which is an image or video of a high dynamic range, the device comprising: a dynamic range setting unit (302) configured to set a dynamic range (CombRng) of the brightness of the combining, wherein at least the maximum brightness (LMC) is determined based on at least one of: a maximum brightness of at least one of two images or two videos of images, and peak brightness of the display for visualizing two images or two videos of images, moreover, the dynamic range setting unit further comprises a brightness armature determining unit (303) configured to determine the brightness (anc) of the armature in the dynamic range (CombRng ) the brightness of the union, a color conversion unit (310) configured to perform at least brightness conversion on at least one of the two images or videos, the color conversion unit (310) comprising a brightness reading unit (311) for the source armature made with the ability to read at least one brightness (L_SA1) of the original anchor from metadata from the first source (350), which delivers the first image or video (Im1_LDR) from two images or videos, and the color conversion unit is configured to establish color conversion (FF_1) to be applied to the first image or video, the color conversion being dependent on the brightness value (L_SA1) of the source armature due to the property that the brightness (L_SA1) of the source armature is mapped to an output brightness close to that of (anc) Anchors and an image combining unit (320) adapted to combine two images or two video images of images to form at least one output image (Im_o). 2. The device according to claim 1, in which the color conversion unit (310) is configured to determine a color conversion (FF_1), so that the output brightness (LF1_o) determined by applying the color conversion (FF_1) to the input pixel color brightness the first image or video (Im1_LDR) equal to at least one brightness (L_SA1) of the original anchor is equal to the brightness (anc) of the anchor. 3. The device according to claim 2, in which the color conversion unit (310) is configured to determine the color conversion (FF_1), so that the ratio of the second output brightness (LT2_o), which is determined by applying the color conversion (FF_1) to the second input brightness (LT1_i), to output brightness (LF1_o), a constant number (C) times the ratio of the second input brightness (LT1_i) to the brightness (L_SA1) of the original armature. 4. The device according to any one of the preceding paragraphs, in which the color conversion unit (310) comprises a deviation determination unit (312) configured to determine, based on at least one brightness (L_SA1) of the original armature, the brightness offset (d_anc) and wherein the color conversion unit is configured to determine the color conversion (FF_1), so that the output brightness (LF1_o) determined by applying the color conversion (FF_1) to the input color brightness of the pixel of the first image or video (Im1_LDR) equal to at least one brightness (L_SA1) of the original anchor is equal to the brightness (anc) of the anchor plus the offset (d_anc) of the brightness. 5. The device according to claim 4, in which the block (310) color conversion is configured to read at least one brightness (L_S2A1) of the second source armature obtained from the second source (351), which delivers the second image or video (Im_HDR ) of two images or videos, and the deviation determination unit (312) is configured to determine the brightness offset (d_anc) also based on at least one brightness (L_S2A1) of the second source armature. 6. The device according to any one of the preceding paragraphs, in which the color conversion unit is configured to establish a color conversion (FF_1) in relation to the first image or video recording also depending on the first functions (F1_L) of the re-display of colors that specify a change in the distribution of brightness of objects in the first image or video recording (Im1_LDR) for displaying this first image or video from the dynamic range associated with encoding the first image or video into a dynamic range that differs from the peak lightness by at least 2 times, or the color conversion unit is configured to setting the color conversion (FF_2) in relation to the second image or video also depending on the second functions (F2_L) of re-displaying colors, which specify a change in the brightness distribution of objects in the second image or video (Im2_HDR) to display this second image recording or video recording from the dynamic range associated with the encoding of the second image or video recording to a dynamic range that differs from the peak lightness by at least 2 times. 7. The device according to any one of the preceding paragraphs, in which the dynamic range setting unit (302) is configured to set the dynamic range (CombRng) of the brightness of the combination depending on the brightnesses present in at least one of the two images or videos. 8. The device according to claim 7, in which the dynamic range setting unit (302) is configured to set the dynamic range (CombRng) of the brightness of the combination depending on the peak lightness of the display on which at least one output image (Im_o) is to be visualized , and, preferably, also on the characteristics of the lightness of the observation environment. 9. The device according to any one of the preceding paragraphs, in which the block (303) determining the brightness anchor is configured to determine the brightness (anc) of the armature depending on at least one of: the dynamic range (CombRng) of the brightness of the combination, the brightness in at least one of two images or videos, the peak brightness of the display on which at least one output image (Im_o) is to be visualized, and the characteristics of the brightness of the observation environment. 10. A method of combining two images or two video images (Im_HDR, Im_LDR), one of which is an image or video of a high dynamic range, the method comprising the steps of: set the dynamic range (CombRng) of the brightness of the combination, and at least the maximum brightness (LMC) is determined based on at least one of: the maximum brightness of at least one of two images or two video images, and peak lightness a display for visualizing two images or two video images, the setting further comprising determining the brightness (anc) of the armature in the dynamic range (CombRng) of the brightness of the association; apply color conversion (FF_1) on at least one of two images or videos, consisting of at least a brightness conversion, and the color conversion is set based on the brightness value (L_SA1) of the original anchor, which is read from metadata from the first source (350) which delivers the first image or video (Im1_LDR) of two images or videos, the color conversion having the property that the brightness (L_SA1) of the source arm is displayed in an output brightness close to the brightness (anc) of the arm; and combine two images or two video images with the colors of their pixels being in the dynamic range (CombRng) of the brightness of the combination to form at least one output image (Im_o). 11. The method of combining the two images of claim 10, in which the color conversion (FF_1) is determined so that the output brightness (LF1_o) determined by applying the color conversion (FF_1) to the input color brightness of the pixel of the first image or video (Im1_LDR) equal to at least one brightness (L_SA1) of the original anchor, equal to the brightness (anc) of the anchor or equal to the brightness (anc) of the anchor plus a certain difference (d_anc). 12. The method of combining two images according to any one of the preceding paragraphs, in which the color conversion (FF_1) is determined so that the ratio of the second output brightness (LT2_o), which is determined by applying the color conversion (FF_1) to the second input brightness (LT1_i), to the output brightness (LF1_o), a constant number (C) times the ratio of the second input brightness (LT1_i) to the brightness (L_SA1) of the original armature. 13. The method of combining two images according to any one of the preceding paragraphs, in which the color conversion (FF_2) of the second image or video image is determined based on at least one brightness (L_S2A1) of the second source armature obtained from the second source (351), which delivers a second image or video (Im1_HDR) of two images or videos. 14. The method of combining two images according to any one of the preceding paragraphs, in which at least one of the color transformations (FF_1, FF_2) to be applied to the corresponding one of the at least two images or videos is determined at least , in part based on at least one received function (F1_L, F2_L), which indicates how to convert the corresponding image from the dynamic range corresponding to the one being encoded to a dynamic range that is at least two times higher or lower. 15. The method of combining two images according to any one of the preceding paragraphs, in which at least one of the dynamic range (CombRng) of the brightness of the combination and the brightness (anc) of the anchor is determined based on at least one factor from the set: properties of objects, depending on the brightness distribution in at least one of the two images, information summarizing the brightness characteristics of at least one of the two images, the dynamic range of the display on which the combined at least one output image (Im_o) is to be visualized , and lightness measures for the observation environment in which the output image is to be observed.

Images