RU2431939C1 - Method of displaying two-dimensional on-screen menu on stereo video sequence - Google Patents

Abstract

FIELD: information technology. ^ SUBSTANCE: stereo frame is split into two colour images. The images undergo colour calibration. Histograms of pairs of initial colour images and pairs of colour-calibrated images are compared, wherein the scalar value of the difference between histograms is calculated and the minimum difference between histograms of two pairs is selected. A given stereo frame is classified as an on-screen menu if the minimum difference between histograms of two pairs is greater than a first predetermined threshold T1, the value of which is given in the range t1êê[0Ç5]. A given stereo frame is classified as not being an on-screen menu if the minimum difference between histograms of two pairs is less than a second predetermined threshold T2, the value of which is given in the range T2êê [150Ç200]. Geometric elements on images are sought for, wherein additional operations are performed if the minimum difference between histograms is greater than or equal to the first predetermined threshold T1 and greater than the second predetermined threshold T2. ^ EFFECT: recognising an on-screen menu formed by a third-party device for correct display of said menu on a three-dimensional display. ^ 7 cl, 6 dwg

Description

The invention relates to a method for processing video data, and more particularly to a method for determining the nature of distortion in a stereo video sequence displayed in stereo mode, and can be used to determine the type of on-screen menu that is superimposed on the stereo video sequence.

The on-screen menu is an image superimposed on a picture on the screen containing controls and is often used in modern television, VCRs and DVD players to display information about sound settings, current channel, time, etc. When playing a two-dimensional video sequence, displaying the on-screen menu is not a problem, but in 3D mode, displaying the on-screen menu can cause many visible distortions and causes serious problems due to errors in the algorithm for estimating depth and further generating the image. This leads to user dissatisfaction with the quality of three-dimensional (3D) and 3D-ready displays.

Currently, the on-screen menu is used in most devices that can be connected to a TV. But in the case of a stereoscopic (3D) TV, the on-screen menu must be formed in a special three-dimensional format. Currently, DVD devices and Blue-ray players cannot distinguish between mono video sequences and stereo video sequences, i.e. the on-screen menu is displayed in mono format for both cases. When a stereo sequence with such a monoscreen menu is analyzed by the built-in units of a stereoscopic TV, a lot of distortion occurs.

It should be borne in mind that the modern market for television devices is developing through the use of new image reproduction technologies, in particular, on the basis of new types of stereoscopic (also known as three-dimensional or 3D) displays. In this regard, it is necessary to solve numerous problems related to the processing of stereo video sequences for such three-dimensional displays. Over the past five years, a large number of inventions on this subject have been published. One of the problems that has arisen with the advent of stereoscopic TVs is the correct display of the on-screen menu in the case of stereo video sequences. If the on-screen menu is displayed by the TV itself, then information about the current type of video sequences and the on-screen menu itself will be displayed correctly. But if the on-screen menu is opened by another device, for example, a DVD player, then information about whether this video sequence with the on-screen menu superimposed stereo video sequence or not is not available and the on-screen menu is usually displayed in mono mode. A mono-screen menu is formed as a single-color or multi-color rectangle with text and pictograms. The on-screen menu can be of various levels of transparency and different colors. Adding a simple mono-screen menu leads to distortion during video processing. For this reason, there is a need to apply some “smart” procedures to identify these cases.

Published European patent application EP 2091241 [1] describes a method for displaying transparent on-screen menus for home theaters based on the detection of movement changes within consecutive video frames of the displayed video. This technical solution allows users to easily view videos in the background while displaying an on-screen menu.

US patent application 20040027267 [2] describes a system for selectively reproducing images for viewing stereo images or mono images, which system includes a stereoscopic system for processing stereo images, a mono / stereo mode switch and a display capable of displaying mono and stereo images. The stereo image processing system is configured to generate signals for controlling the display based on the input video signals, the state of the mono / stereo switch, and the type of display. This allows you to determine and adjust the nature of the on-screen menu for various video display modes.

US patent application 20090060280 [3] proposes a stereoscopic image system that includes an image preprocessing module for preprocessing the right and left images and a stereo pair matching model for stereo matching of the right and left images to obtain a depth map of the right and left images at low resolution and distance information high-resolution right and left images after detecting an object within a range of distances based on distance information at low permission. US patent application 20080089577 [4] relates to the formation of a three-dimensional vector object representing one of the characteristics within the scene by analyzing two-dimensional vector objects representing this characteristic in a stereo pair. Two-dimensional vector objects are analyzed using stereoscopic vision algorithms to form a three-dimensional vector object. The results of the analysis make it possible to obtain data concerning the three-dimensional position of the corresponding points of two-dimensional vector objects. A three-dimensional vector object is formed on the basis of the results of the stereo image analysis. A three-dimensional vector object can be compared with a three-dimensional model of a digital point. A three-dimensional vector object can also be compared with another three-dimensional vector object formed from a stereo pair and captured under other conditions. This method, the disadvantage of which is the difficulty of its practical implementation, is closest to the claimed solution and therefore is selected as a prototype.

The problem to which the claimed invention is directed is to develop a simpler and more reliable way to identify the on-screen menu on a stereo video sequence and determine its nature. At the same time, the new method should provide the ability to detect the presence of a mono-screen menu on stereo video sequences in various formats and not distort the display device (DVD player or TV) in cases where the on-screen menu is correctly displayed for stereo video sequences.

The technical result is achieved by choosing an approach based on the application of the Yes / No detection algorithm (in the specialized literature the terminology "OSD / non" is often used, that is, "on screen display / non") to the on-screen menus for evaluation, there is whether the OSD on the current frame. If such a menu is detected, it is proposed to turn off the stereo mode and show the video sequence in mono mode until the OSD disappears. Moreover, the inventive method provides for the following operations:

- divide the stereo frame into two color images;

- colorize images;

- compare the histograms of a pair of initial color images and a pair of color-calibrated images, while calculating the scalar value of the difference of the histograms and choose the minimum difference of the histograms of two pairs;

- classify this stereo frame as an on-screen menu if the minimum histogram difference of two pairs exceeds the first predetermined threshold T ₁ , the value of which is set within the range of T ₁ ∈ [0 ... 5];

- classify this stereo frame as not being an on-screen menu if the minimum histogram difference of two pairs is less than the second predetermined threshold T ₂ , the value of which is set within the range T ₂ ∈ [150 ... 200];

- search for geometric elements in the images if the minimum histogram difference is greater than or equal to the first predetermined threshold T ₁ and greater than the second predetermined threshold T ₂ by performing the following steps:

- calculating the differences of two pairs of images, namely a pair of original images and a pair of color-calibrated images, by applying the formula

- первое первоначальное изображение,

- второе первоначальное изображение,

и

- цветокалиброванные изображения, W - ширина и Н - высота этих изображений, R, G, В - цветовые компоненты;Where

- first initial image,

- second initial image,

and

- colorized images, W - width and H - the height of these images, R, G, B - color components;

- calculate the minimum difference D = min (D ₁ , D ₂ ) between the differences of the pairs of the original and color-calibrated images;

- calculate the binary image by setting the threshold of the calculated minimum difference, using the ratio

,

,

while choosing the value of the threshold T depending on the magnitude of the difference of the histograms:

,

,

where const ₁ > const ₂ , and the constants are selected from the corresponding ranges of values, namely: const ₁ ∈ [25 ... 40], const ₂ ∈ [15 ... 35];

- perform the morphological processing of the calculated binary image by applying to D _{T the} operation of opening and subsequent closing, while the structured element SE _opening for opening D _T has a size of 1 by 2 and consists of "1":

SE _opening = [1 1]

where D _O is the binary image of the application of the opening operation to D _T , and the closure is performed to connect the parts of the horizontal lines that were preserved during the opening, while the structured element SE _closing for closing has a size of 1 by 5 and also consists of "1" :

SE _closing = [1 1 1 1 1],

where D _M is the binary image of the result of applying the opening operation to D _O ;

- count the number of geometric elements in the final binary image D _M ;

- compare the value obtained in the previous step with the predefined threshold T ₃ , the value of which is set within the range T ₃ ∈ [1 ... 5], and if this value is greater than the predefined threshold T ₃ , then this stereo frame is classified as an on-screen menu; otherwise, this stereo frame is not classified as an on-screen menu.

In the case where the histogram difference value is not too large or not too small, search for geometric elements in the images by calculating the image difference, calculating the binary image by setting the threshold of the calculated difference, morphologically processing the calculated binary image, taking into account non-zero elements of the final image and comparing this a number with a predefined threshold.

Further, the invention is explained in detail with the use of graphic materials.

Figure 1. The main steps of the method for identifying an on-screen menu.

Figure 2. Explanation of splitting a video frame into two images and color calibration of images.

Figure 3. Explanation of the division of images into blocks.

Figure 4. Explanation of intermediate images obtained during the execution of the process depicted in figure 1.

Figure 5. The process of calculating the number of geometric elements in a binary image obtained after setting the threshold for the image difference.

6. A variant of the device for implementing the inventive method.

The diagram in FIG. 1 describes the method in more detail. The input stereo video sequence can be presented in various formats: left-right, top-bottom, staggered, etc. In step 101, the stereo input frame of the stereo video sequence (stereo pair) is divided into two color images, this is that half of the stereo pair is copied to one memory buffer, and the second half to another memory buffer. These images, i.e. the first image and the second image can be reduced in the process of copying by the closest proximity method, this means that only every nth element of each nth row is copied instead of copying each element. The number n can vary from 2 to 10, depending on memory resources. In this embodiment, n is 8. In step 102, one of the two resulting images is colorized (adjusted in color) to the other image (then a pair of colorized images consists of one original and second colorized image), using, for example, the method stretching or histogram matching (both methods are described, in particular, in the published work of Gonzales and Woods Digital Image Processing, Prentice Hall, Upper Saddle River, New Jersey 07458, ISBN 0-201-18075-8) [5], and save in dop tional buffer memory (2 illustrates steps 101 and 102.) In step 103 all the three pictures, namely the two primary color images and one image tsvetokalibrovannomu horizontal blur applied. The blurring procedure is also described in detail in [5]. In this case, blurring is applied using a filter with an F kernel of size 1 divided by m:

Such a filter, applied line by line, allows for quick processing, since images are stored in memory line by line. At step 104, the pair of initial images is compared using the calculated histogram difference ΔH ₁ . First, each image (the original first and the original second) is divided into 9 parts of the same size (three horizontally and three vertically), and 9 parts of the image are then compared (see Figure 3 for an explanation).

,

,

для первого изображения и

,

,

для второго изображения. Все гистограммы сортируются по возрастанию. Это помогает преодолеть большую разность из-за сдвига освещенности. Затем разность гистограмм для текущей части изображения ΔH_part вычисляют в соответствии со следующей формулой:For each part of the image, three histograms are calculated (for each of the three color components designated as R, G and B), for each of the two images,

,

,

for the first image and

,

,

for the second image. All histograms are sorted in ascending order. This helps to overcome a large difference due to a shift in illumination. Then the histogram difference for the current part of the image ΔH _{part is} calculated in accordance with the following formula:

The maximum histogram difference ΔH ₁ between all parts of the images is calculated in accordance with the following formula:

At step 105, the procedure of step 104 is repeated for a pair of colorized images. As a result, the histogram difference ΔH _{2 is} calculated. Ultimately, in step 106, the total histogram difference is selected as the smallest between ΔH ₁ and ΔH ₂ :

In step 107, ΔH is compared with a predetermined threshold T ₁ . This means that if the difference between the histograms is small enough, then the current frame is classified as non. If the difference is large enough, i.e. more than some predetermined threshold T ₂ , then the current frame is classified as "OSD". This condition is checked at step 108. Otherwise, when it is impossible to draw a conclusion based on the difference in the histograms, the following steps are used for analysis (109-112).

, второе первоначальное изображение обозначается как

и цветокалиброванные изображения обозначаются как

и

. Ширина и высота этих изображений одинакового размера обозначается как W и Н соответственно, цветовые компоненты обозначаются как R, G, В. Затем вычисляют две величины, D₁ и D₂:At step 109, the difference between the mappings between the frames is calculated. Let the first initial image be denoted as

, the second original image is denoted as

and color calibrated images are denoted as

and

. The width and height of these images of the same size are denoted as W and H, respectively, the color components are denoted as R, G, B. Then two values are calculated, D ₁ and D ₂ :

Then, in step 110, the minimum image difference is calculated

In step 111, a threshold for the image difference D is set:

,

,

The threshold T is selected depending on the magnitude of the difference between the histograms:

where const ₁ > const ₂ . At step 112, the morphological operation of opening and subsequent closing to D _{T is} applied. Opening operation is the dilatation of erosion of a set. This operation is used to remove small dots and blotches. The structured element for opening D _T has a size of 1 by 2 and consists of "1":

Closing is done to connect parts of horizontal lines that are preserved during the opening process. The structured element for the closure has a size of 1 by 5 and also consists of "1":

After morphological operations, a search for geometric elements in D _{T is} performed (step 113). This process is illustrated in FIG. In step 114, N is compared with a predetermined threshold T ₃ . If N is greater than T ₃ , then the current frame is classified as “non” (step 115). Otherwise, the current frame is categorized as “OSD” (step 116).

Figure 4 shows the intermediate images obtained during the execution of the process depicted in figure 1.

5 discloses in detail the process of searching for geometric elements (Step 112). At step 501, i, j, N are set by some i _start , ≥0.0 and 0, respectively. At step 502, the num (number) is set to 0. At step 503, the condition is checked whether num is greater than the predetermined constant S. If not, then perform the next step 506. If D _M (i, j) is 1 (condition 506 ), then num is increased by 1 (see step 507). Then check condition 508, i.e. whether the process has reached the end of the current line. If not, then i is increased by 1 (step 504), and the process proceeds to condition 503. If the result 503 is positive, then N is increased by 1 (see step 505). At step 510, it is checked whether the method has reached the last valid image line. If not, then i is increased by 1 (see step 509). If yes, then at the next step 511, the resulting N. is output.

It should be noted that the values of T1, T2, T3, K, L, const ₁ , const _{2 are} set by some constants. These constants can be selected from the following ranges:

T ₁ ∈ [0 ... 5], T ₂ ∈ [150 ... 200], T ₃ ∈ [1 ... 5], K∈ [1 ... 5], L = [5 ... 15], const ₁ ∈ [25 ... 40 ], const ₂ ∈ [15 ... 35], S∈ [10 ... 30].

The inventive method is implemented using devices known from the prior art. In particular, in the simplest version (see FIG. 6), such a device consists of a processor (or a plurality of processors) 601, a memory 602, and a data bus 603, through which data are transferred from memory to the processor and back. At the beginning, the original stereo frame is recorded in the memory, and the device starts processing, the response is also recorded in the memory and can be read from there.

The invention can be used in stereo television for preprocessing video streams to display images in three-dimensional mode, in particular in cases where the TV is connected to a DVD or Blue-ray player, and when it is impossible to determine the type of video sequence.

It should be noted that specialists should take into account the possibility of other embodiments of the invention, not beyond the scope of the claims. Thus, the drawings and description can be considered only as illustrations of the claimed solution.

Claims (7)

1. A method for detecting a two-dimensional on-screen menu on a stereo frame, characterized in that it involves the following operations:
- divide the stereo frame into two color images;
- colorize images;
- compare the histograms of a pair of initial color images and a pair of color-calibrated images, while calculating the scalar value of the difference of the histograms and choose the minimum difference of the histograms of two pairs;
- classify this stereo frame as an on-screen menu if the minimum histogram difference of two pairs exceeds the first predetermined threshold T ₁ , the value of which is set within the range of T ₁ ∈ [0 ... 5];
- classify this stereo frame as not being an on-screen menu if the minimum histogram difference of two pairs is less than the second predefined threshold T ₂ , the value of which is set within the range of T ₂ ∈ [150 ... 200];
- perform a search for geometric elements in the images, while if the minimum histogram difference is greater than or equal to the first predetermined threshold T ₁ and greater than the second predetermined threshold T ₂ , perform the following steps:
- calculate the differences of two pairs of images, namely a pair of original images and a pair of color-calibrated images by applying the formula

,
Where

- first initial image,

- second initial image,

and

- colorized images, W - width and H - the height of these images, R, G, B - color components;
- calculate the minimum difference D = min (D ₁ , D ₂ ) between the differences of the pairs of the original and color-calibrated images;
- calculate the binary image by setting the threshold of the calculated minimum difference, using the ratio

, while choosing the value of the threshold T, depending on the magnitude of the difference of the histograms:

, where const ₁ > const ₂ , and the constants are selected from the corresponding ranges of values, namely:
const ₁ ∈ [25 ... 40], const ₂ ∈ [15 ... 35];
- perform morphological processing of the calculated binary image by applying the opening operation and subsequent closure to d _t , while the structured element SE _opening for opening d _t has a size of 1 by 2 and consists of "1":
SE _opening = [1 1]

,
where D _O is the binary image of the application of the opening operation to d _t , and the closure is performed to connect the parts of the horizontal lines that were preserved during the opening, while the structured element SE _closing for closing has a size of 1 by 5 and also consists of "1" :
SE _closing = [1 1 1 1 1],

,
where d _m is the binary image of the result of applying the opening operation to D _O ;
- count the number of geometric elements in the resulting binary image d _m ,
compare the value obtained in the previous step with a predetermined threshold T ₃ , the value of which is set within the range of T ₃ ∈ [1 ... 5], moreover, if this value is greater than the predetermined threshold T ₃ , then this stereo frame is classified as an on-screen menu; otherwise, this stereo frame is not classified as an on-screen menu. 2. The method according to claim 1, characterized in that the color calibration is performed by the method of stretching the histograms. 3. The method according to claim 1, characterized in that the color calibration is performed by the method of comparing histograms. 4. The method according to claim 1, characterized in that the images are blurred before comparing the histograms. 5. The method according to claim 1, characterized in that the images are divided into blocks and histograms are calculated in each block, then the sum of the absolute differences for all color channels and all blocks is calculated; choose the maximum sum of the absolute differences as the scalar value of the difference of the histograms. 6. The method according to claim 5, characterized in that each histogram is sorted in ascending order to calculate the sum of the absolute differences. 7. The method according to claim 1, characterized in that the geometric elements are short horizontal segments in a binary image, the length of which varies between 10-30 non-zero elements of the binary image.

Images