RU2573737C2 - Multi-sample resolving of re-projection of two-dimensional image - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: method includes identifying a plurality of two-dimensional image samples for each pixel in re-projection, generated by shifting the re-projection pixel relative to the image by a value based on depth data, each two-dimensional sample during re-projection having the same width as re-projection pixels; determining a plurality of sample coverage values for each re-projection pixel; resolving the final value for each re-projection pixel by combining each two-dimensional sample.

EFFECT: high quality of rendering a two-dimensional image during re-projection.

22 cl, 8 dwg

Description

Cross reference to related applications

This application relates to patent application No. 12/986814, which belongs to the same copyright holder and is simultaneously being examined by the patent office (receipt No. SCEA10052US00), entitled “DYNAMIC ADJUSTMENT OF PREDETERMINED THREE-DIMENSIONAL RE-PROJECTION SETTINGS BASED ON SCENE CONTENT January 7, 2011” .

This application relates to patent application No. 12/986827, which belongs to the same copyright holder and is simultaneously being examined by the patent office (receipt No. SCEA10053US00), entitled “SCALING PIXEL DEPTH VALUES OF USER-CONTROLLED VIRTUAL OBJECT IN THREE-DIMENSIONAL SCENE January 7, 2011,” .

This application relates to patent application No. 12/986854 (the receipt No. SCEA10054US00), which belongs to the same copyright holder and is entitled "MORPHOLOGICAL ANTI-ALIASING (MLAA) OF A RE-PROJECTION OF A TWO-DIMENSIONAL IMAGE", filed 7 January 2011

Technical field

Embodiments of the present invention relate to a multi-sample resolution method for three-dimensional re-projection of a two-dimensional image.

State of the art

The ability to perceive a two-dimensional image in three dimensions through the use of numerous different technologies has become quite popular over the past few years. Providing a depth aspect for two-dimensional images potentially creates a greater sense of realism of any depicted scene. This use of three-dimensional visual presentation has greatly enhanced the viewer's experience, especially in the field of video games.

There are a number of ways to three-dimensionally represent this image. More recently, a method has been proposed for projecting two-dimensional image (s) into three-dimensional space, known as rendering based on depth image (ROIG). In contrast to previous proposals, which were often based on the concept of "stereoscopic" video, i.e. receiving, transmitting and displaying two separate video streams, one for the left eye and one for the right eye, this new idea is based on a more flexible joint transmission of a monoscopic video image (i.e. a single video stream) and the associated depth information for each pixel. Based on the presentation of this data, one or more “virtual” views of the 3D object can then be generated in real time on the receiver side using so-called ROIG methods. This new way of rendering a three-dimensional image has several advantages over previous methods.

Typically, two methods are used to present the viewer with two separate images to create an illusion of depth. The system commonly used to project 3D images onto a screen uses two separate synchronized image projectors for the left eye and the right eye. Images for both eyes are projected onto the screen simultaneously, but with orthogonal polarizations, for example, vertical polarization of the image for the left eye and horizontal polarization of the image for the right eye. The viewer wears special polarized glasses for 3D viewing, having respectively polarized lenses for the left and right eye (for example, a vertically polarized lens for the left eye and a horizontally polarized lens for the right eye). Due to the polarization of images and lenses, the viewer perceives only the image for the left eye with the left eye and only the image for the right eye with the right eye. The degree of illusion of depth is partly a function of the offset between the two images on the screen.

In 3D video systems, images for the left eye and the right eye are displayed on the video screen, but not exactly at the same time. Instead, the images for the left eye and the right eye are displayed alternately. The viewer uses glasses with an active shutter that closes the left eye when displaying an image for the right eye and vice versa.

The perception of 3D video may depend to some extent on the characteristics of human vision. For example, the human eye has a discrete number of light receptors and, nevertheless, people do not distinguish pixels, even with side vision. Even more surprisingly, the number of photosensitive cones in the retina of the eye can vary significantly among individuals - up to 40 times. Despite this, people perceive colors in the same way - we see mainly with our brain. The human vision system also has the ability to determine the alignment of objects depending on the width of the cones (visual acuity). This explains why signs of spatial distortion (i.e. visual disturbances) are more noticeable than color errors.

Using this fact, computer graphics hardware manufacturers are making significant efforts to compensate for signs of distortion by providing color accuracy for spatial continuity. Many methods are supported by hardware using weighted color samples, similar to the integrating capabilities of digital cameras.

Undoubtedly, all signs of distortion will eventually disappear due to an increase in display resolution and sampling frequency. Signs of distortion can also be controlled at low resolution by calculating and averaging multiple samples for each pixel. Until now, for most image presentation algorithms, these solutions may not always be practical from a practical point of view, given the limitations of hardware.

This is within the context, which shows embodiments of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGA-1G are block diagrams illustrating the effect of the distance between the corresponding three-dimensional pixels on the depth perceived by the viewer.

FIGA-2B are block diagrams illustrating the effect of depth separation in three-dimensional re-projection of a two-dimensional image.

FIGA-3G are flowcharts illustrating a method for multi-sample resolution of three-dimensional re-projection of a two-dimensional image, in accordance with the prototype.

FIG. 4 is a flowchart illustrating a multi-sample resolution method for three-dimensional re-projection of a two-dimensional image, in accordance with an embodiment of the present invention.

FIGS. 5A-5B are flowcharts illustrating a multi-sample resolution method for three-dimensional re-projection of a two-dimensional image, in accordance with an embodiment of the present invention.

FIG. 5B is a block diagram illustrating the alignment between pixel samples in a multi-sample three-dimensional re-projection of a two-dimensional image, in accordance with an embodiment of the present invention.

FIG. 6 is a block diagram illustrating an apparatus for multi-sample resolution of three-dimensional re-projection of a two-dimensional image, in accordance with an embodiment of the present invention.

FIG. 7 is a block diagram illustrating an example implementation of a cellular processor of a device for multi-sample resolution of three-dimensional re-projection of a two-dimensional image, in accordance with an embodiment of the present invention.

FIG. 8 illustrates an example of an immutable computer-readable information storage medium with instructions for implementing multi-sample resolution of three-dimensional re-projection of a two-dimensional image in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

INTRODUCTION

Three-dimensional re-projection of a two-dimensional image can be observed by displaying two separate views (i.e., one for each eye) so that the viewer creates an illusion of depth. Looking at the three-dimensional image, the left eye and the right eye bring together two corresponding pixels (i.e., a pixel for the left eye and a pixel for the right eye) to simulate one pixel having depth.

FIGA-1G are block diagrams illustrating the effect of the distance between the corresponding three-dimensional pixels on the depth perceived by the viewer. The pixel size has a fixed border, determined by the resolution of the image, which is inevitably limited by the hardware of the visual display (for example, a three-dimensional TV). As the separation of the corresponding pixels increases, the location of the perceived depth gradually becomes farther from the screen.

FIG. 1A shows two corresponding pixels that overlap. These two pixels will create a perceived depth 101, which is located on the surface of the screen (for example, a visual display). As the corresponding (i.e., neighboring) pixels move away from each other in FIG. 1B, the perceived depth 101 increases and moves slightly toward the screen. As the further uniform distribution of the corresponding (i.e. separated by the pixel width) pixels from each other in FIG. 1B, the perceived depth 101 appears to be shifted further into the screen. When the pixel for the left eye is located to the right of the pixel for the right eye in FIG. 1G, the perceived depth will appear closer to the screen. It is important to note that the perceived depth is not linearly proportional to the distance between the respective pixels. Thus, a slight increase in the separation of the respective pixels can lead to a significant increase in the perceived depth associated with these respective pixels.

The relationship between the separation of the respective pixels and the perceived pixel depth makes smooth transitions of depth values quite difficult. This is due to the fact that each pair of corresponding pixels is projected onto a specific depth plane, determined by the separation of pixels. FIGS. 2A-2B are flowcharts illustrating this effect, referred to in this application as depth separation.

A three-dimensional visual artifact, known as depth separation, occurs when trying to position a three-dimensional element 201 with an inclined depth, as illustrated in FIG. To a greater extent than the perception of an element 201, such as located on one diagonal plane, the element will be perceived as located on multiple planes 203 that are parallel to the screen, as illustrated in FIG. 2B.

In an ideal situation, simply increasing the resolution of the displayed image could solve the issue of separation of depth. However, due to the fact that the process of three-dimensional re-projection is limited by the capabilities of the hardware, this solution is not feasible. Instead, the effects of depth separation are minimized by mixing / combining colors or other values of adjacent three-dimensional pixels. The result is a muted waveform 205 to a greater extent than a straight diagonal plane, as shown in FIG. Being imperfect, this form provides the viewer with a much more natural representation of the three-dimensional image.

To blend 3D pixels to create smoother pixel depths, a smoothing solution is used. Prototype anti-aliasing solutions include temporary anti-aliasing, which requires the storage of additional buffer data. For example, in the case of 3D images, this means storing either buffers previously re-projected in color and depth, or two image color buffers for the left eye and the right eye. For 720P images, this requires a memory capacity of about 7 megabytes. Considering that 3D games already need extra storage buffers for an image for the second eye, this only increases the load on already filled memory.

Another prototype solution, known as full-screen anti-aliasing (PES), also requires the use of high-capacity buffers and increases the memory load. PES includes increasing the resolution and then applying it to gradually lower. In addition to the remaining separation of depth, the serious question of reduced efficiency of higher resolution arises.

An additional solution is known as multi-sampling anti-aliasing (MSS), which is similar to PES, but the color value is selected at a lower resolution than the resolution of the sample. The result is a significantly cheaper implementation in terms of processing load and memory. Typically, MVS requires permission before performing additional image processing work. This usually means that, due to the subsequent processing of the original image, all additional sample data is lost due to the use of time re-projection. Even if the sample information is stored, a depth separation is observed (due to the fixed positions of the sample already considered). The memory requirements for PES and MVS are more changing than for temporary smoothing.

FIGS. 3A-3G show a multi-sample smoothing method of a prototype for multiple sampling along with the problems associated with the prototype. Each three-dimensional pixel of a three-dimensional image (i.e., a left view or a right view) can be represented by a memory segment. Typically, a pixel memory segment contains only one sample. By way of example, and not by way of limitation, this pattern may be a pixel, a pixel element, or some group of pixels from a corresponding two-dimensional image intended for re-projection in three dimensions. A sample can be characterized by a set of values that describe its color profile. In this case, in the re-projection scheme, which does not implement the multi-sample method, the color profile of a three-dimensional pixel can be characterized by one sample that it contains.

In the multi-sample method, it is assumed that each pixel memory segment may contain more than one sample. In a typical multi-sample design, the positions of the samples are assigned within a three-dimensional pixel, and the sample is assigned to this position if it is covered by the current geometry. FIG. 3A shows a series of three-dimensional pixels 301 implementing a 2X multi-sample circuit. Black dots represent the positions of patterns assigned to each three-dimensional pixel 301 for use during multiple sampling. In a 2X multi-sampling scheme, each three-dimensional pixel 301 can be characterized by 2 patterns from a corresponding two-dimensional image. While the example shown is limited to using two samples for each three-dimensional pixel, it is important to note that any number of samples can be used in a multi-sample design. In addition, depending on the direction of the re-projection offset, the samples can be used from any direction. If re-projection offset occurs only in the horizontal direction (i.e., using parallel-offset cameras), samples can be used with horizontal scanning of a two-dimensional image. This method requires a relatively small amount of memory, since scanning only one line needs to be saved at a time.

A multi-sample design assigns samples from a two-dimensional image to the positions of the samples for each pixel in three-dimensional re-projection. However, the actual process of combining multiple samples to produce a single color profile is known as resolution. Both steps are necessary to achieve a muted waveform for the diagonal planes described above. A typical resolution process assumes that the coverage of each sample is proportional to the total number of samples for each three-dimensional pixel. Thus, in a 2X multi-sampling scheme / resolution, each sample in a given three-dimensional pixel is given 50% by weight, and the values are added together. Then, if the left swatch is 100% red and the right swatch is 100% blue, the three-dimensional pixel will assume a color profile of 50% red and 50% blue (or some kind of magenta). Similarly, if a multi-sampling scheme / 4X resolution is used, each sample will have 25% weight. Other blending combinations may be used, but based on a fixed position of the sample point from the pixel that was resolved.

The problem associated with the multi-sample method in the prototype is the actual position of the sample within a given three-dimensional pixel. FIG.3B-D illustrate this problem in the context of a multi-sample scheme / resolution 2X. In our example, each sample from a two-dimensional image will be an individual pixel from this two-dimensional image. Each pixel of three-dimensional re-projection 301 may be represented by two different patterns. The positions of the samples within each three-dimensional pixel are fixed.

FIG. 3B, after the samples of two black pixels 303A, 303B are followed by samples of black pixels 305A, 305B in a two-dimensional image, which is re-projected to the right. The first black pixel 303A covers the position of the left pattern in the first (leftmost) re-projection pixel 301A in line 301. The second black pixel 303B covers the right pattern in the first re-projection pixel and the left pattern in the second re-projection pixel 301B. The first white pixel 305A covers the position of the right pattern of the second re-projection pixel 301B and the position of the left pattern in the third re-projection pixel 301C. The second white pixel 305B covers the position of the right pattern in the third re-projection pixel 301C and the position of the left pattern in the fourth re-projection pixel 301D. For the purposes of this example, the position of the right sample of the fourth re-projection pixel 301D may be assigned an initial value (for example, without color or white), since the value of the corresponding pixel from the two-dimensional image is not assigned. Assigning 50% weight to each position of the sample results in the resulting black color of the first re-projection pixel 307A, since both positions of the samples are covered by the black pixel values. The second re-projection pixel 307B is resolved as gray (i.e., 50% white and 50% black), since one sample position for this pixel is covered by a black pixel, and the other position is covered by a white pixel. The third pixel 307C in the allowed line 307 pixels is resolved as white, since both positions of the samples are covered by the values of the white pixel. The fourth pixel 307D in the allowed row of 307 pixels is resolved as white since one position of the sample is covered by a white pixel 307D and, in this example, another position of the sample was initiated for the white value. It is noted that in typical MVS implementations all points of the samples should be covered with some initial values, for example, black, white or gray. In particular, if the buffer is initially cleaned for white, then samples 305A, 305V may be ignored.

In FIG. 3B, pixels 303A, 303B, 305A, 305B were slightly shifted to the right. However, the offset is small enough so that the coverage of the positions of the samples in line 301 does not change. In particular, the positions of the samples in the first re-projection pixel 301A are covered by black pixels 303A, 303B, the positions of the samples in the second re-projection pixel are covered by black pixel 303B and the white pixel 305A, the positions of the samples in the third pixel 301C are covered by white pixels 305A, 305B, and one position the sample in the fourth pixel 301D is covered with a white pixel 305B, and the other position of the sample is not covered and is initiated for white. As a result, the pixels 307A, 307B, 307C, and 307D in the allowed row of the pixel 307 are again resolved as black, gray, white, and white, respectively, as in FIG. 3B. Since the re-projection of pixels 301A, 301B, 301C and 301D has fixed sample positions, small offsets in the re-projection of a two-dimensional image will not change the profile of the re-projection pixel values when the two-dimensional pixels 303A, 303B, 305A and 305B are shifted to the right.

However, if the two-dimensional pixels 303A, 303B, 305A and 305B move slightly further to the right than in FIG. 3B, the coverage of the positions of the samples in the re-projection pixels 301A, 301B, 301C and 301D changes dramatically. In particular, as shown in FIG. 3G, as a result of a slight right shift, both positions of the samples in the first and second re-projection pixels 301A, 301B are now covered by black pixels 303A, 303B, respectively, both positions of the samples in the third and fourth pixels 301C, 301D are covered, respectively, with white pixels 305A, 305B. As a result, the first two allowed re-projection pixels 307A, 307B are allowed as black, and the second two allowed re-projection pixels 307C, 307D are resolved as white.

FIG. 3B-3G shows that in a typical 2X multi-sampling scheme / resolution, different values in the allowed pixels will be only when there is a discrete bi-dimensional image offset in re-projection. Instead of allowing gradual conversion of the value of the resolved three-dimensional pixel with each gradual offset, a typical multi-sampling scheme / resolution allows only discrete offsets of the values of three-dimensional pixels, such as color values. Since human eyes can accurately match colors, this leads to an increase in the planes of perceived depth (i.e., a greater separation of depth).

An increase in the number of samples will allow a more gradual transition of the profile of the values of the three-dimensional pixel, also leading to more depth planes perceived by the viewer. The visual artifact is not too bad in one line of scanning (the viewer simply perceives that the entire area has shifted slightly). However, in a full image that uses multiple line scanning, the problem is magnified. With multiple scan lines, the viewer can note the depth offset of one line, while the other line remains at the same depth.

A significant increase in the number of sample positions for each three-dimensional pixel will lead to a decrease in the number of visual artifacts perceived by the viewer. However, a significant increase in the number of samples is not feasible, since the system memory is limited. For example, a typical 720p buffer using RGBA costs about 3.5 M. Adding a second sample position to a three-dimensional pixel doubles the amount of memory required.

Embodiments of the present invention are aimed at limiting the number of two-dimensional samples required for a given three-dimensional pixel, and at the same time at creating a more gradual transition of the color profile in the three-dimensional pixel for offsets in re-projection. Embodiments of the present invention may use more than two samples. However, it was noted that in embodiments with a re-projection offset only in the horizontal direction, the visual improvement was negligible when four samples were used instead of two. Without limitations from any execution theory, this can be explained by the fact that in the test embodiment, all the pixels were the same size as after re-projection. In a typical scene, there are several types of samples that would not overlap in such a way as is necessary to use more than two samples, in the case when we allowed the use of a maximum of 4 samples. Embodiments with re-projection of pixels in both horizontal and vertical directions can provide improved visual quality using more than two samples. In embodiments of the present invention, values for each pixel in three-dimensional re-projection are calculated based on the “coverage” value of the respective samples. In essence, coverage values can be used to determine the weight to apply to each value of a two-dimensional sample in calculating the value of the corresponding three-dimensional pixel in re-projection.

EMBODIMENTS

FIG. 4 is a flowchart illustrating a multi-sample resolution method for three-dimensional re-projection of a two-dimensional image, in accordance with an embodiment of the present invention. Before re-projecting a two-dimensional image 401 in three dimensions, the samples must be identified for use with the corresponding three-dimensional pixels 403. Three-dimensional re-projection is actually a combination of multiple separated views (ie, view of the right eye and view of the left eye) that combine to create an illusion of depth . For each type of three-dimensional re-projection, it is first necessary to determine which two-dimensional patterns correspond to each three-dimensional pixel for this particular type. As discussed above, a pattern refers to a pixel, pixel, or some other group of pixels within a two-dimensional image. A three-dimensional pixel can be characterized by any number of samples, depending on the type of re-projection performed for the two-dimensional image. Also, any two-dimensional pattern can overlap more than one three-dimensional pixel.

Since the number of samples in each three-dimensional pixel may be more than the maximum allowed, there must be a choice of which samples to use. When re-projecting, patterns can be selected to use foreground objects. This can be done by controlling the depth for each sample, or, in the case of cameras with parallel displacement, choosing the order of traversal of the samples of two-dimensional pixels to use the initial or last recorded sample. Preservation of multiple samples can be achieved by choosing standard retrieval rules from the cache, such as PPPO (first-come-first-served).

To enable easy selection of samples in our embodiment, we designate two types of samples: “leading” and “closing” samples. Leading Pattern — A pattern that intersects the left edge of a three-dimensional pixel. Similarly, a trailing pattern touches or intersects the right edge of a three-dimensional pixel. A pattern that does not intersect any of the edges is classified as closing. In embodiments where the width of the two-dimensional pixel is the same as the width of the three-dimensional pixel when re-projected, the touch or intersection of the edge is guaranteed. It is noted that other embodiments do not require that the width of the two-dimensional pixel be the same as the pixel width in re-projection. As an example, in FIG. 5A, the pattern of the black pixel 503 is a trailing pattern in the first pixel of the row of pixels 501 and a leading pattern in the second pixel in the row. In cases where four samples are used, multiple leading and multiple closing samples can be used and the amount of coverage can be adjusted based on other leading and closing samples. Observations showed that doing this had little visual effect.

After determining which two-dimensional samples correspond to each three-dimensional pixel, 405 coverage values for each sample should be determined. The amount of coverage of the sample refers to the area of the pixel covered by the corresponding two-dimensional sample. Depending on the direction of the re-projection offset, the samples can be used from any direction. By way of example, and not by way of limitation, a three-dimensional re-projection system that uses parallel-offset cameras will use samples from scanning in the horizontal direction of a two-dimensional image, since re-projection is offset only in the horizontal direction. Unlike the prototype, where the positions of the samples were used to resolve the value of a three-dimensional pixel, embodiments of the present invention track the actual amount of coverage of each sample associated with this three-dimensional pixel. The coverage of one sample may be affected by another sample. The primary result of this is manifested in the form of overlapping samples. Since the depth value will usually be different between two two-dimensional pixels when re-projected, the pixels will often overlap. As mentioned above, information about foreground objects is usually more important than background objects.

It is noted that the identification of samples at 403 and the determination of coverage values at 405 can occur simultaneously, and the selection of samples can be based on the value of the coating (as well as the choice to save a sample having a higher coating value).

FIG. 5B illustrates the effect of overlapping two different patterns 512, 514 and the concept of a “lead” and “trailing” pattern. The area between the two dashed lines represents the area of overlap between the two samples. The black pattern 512 is the lead pattern for the second pixel in line 501. The gray pattern 514 is the trailing pattern for the second pixel. In the example shown in FIG. 5B, the gray sample 514 also represents a sample that is closer to the viewer. In this case, the coverage of the leading sample 512 can be reduced by overlapping the amount of coverage with the gray sample. It is possible to completely eliminate another sample from the coating of one sample (or reduce the coating of another sample to zero).

After determining the coverage values for all samples corresponding to all three-dimensional pixels for the final pixel value (for example, color), a resolution in accordance with these coverage values can be performed in 407. For a given three-dimensional pixel, each sample corresponding to that pixel will be weighed in accordance with its coverage value and then combined to obtain the final pixel value. The final pixel value may be the final value for any data channel used to determine the pixel in the image. By way of example, and not by way of limitation, the final pixel value may be the final color value. However, embodiments are not limited to embodiments in which the final value is a color value.

In the case where the weights of the samples used to resolve the final pixel value are not 100% together, typical filling of re-projection holes may occur. By way of example, and not by way of limitation, coverage values can be divided by the total coverage of a three-dimensional pixel, which helps to resolve any inconsistencies. If samples for a given three-dimensional pixel are completely absent, such a pixel may be allowed to pass through the hole filling step. In certain embodiments, the method shown in FIG. 4 may include an optional hole filling step shown in 408. Typically, hole filling can adjust the final value for any pixel for which the total weighted coverage value for all samples used for the final value less than 100%. By way of example, and not by way of limitation, the step of filling holes may include copying the background values of a neighboring pixel to a given pixel, as this is usually less noticeable. In some embodiments, this can be done as part of the resolution of the pixel value at 407 through a specific processing order during resolution.

After resolving the values of the three-dimensional pixel for three-dimensional re-projection, the re-projected image can be displayed, as indicated in 409.

It is noted that coating values for samples can be temporarily stored in several different ways. As an example, coverage values may be stored in arrays. For example, in the case of RGBA data stored as a 32-bit value, there must be 32-bit re-projection pixel values (since 2 samples are used). This does not prevent the use in lists of embodiments of the invention of lists or indexes to reduce the amount of occupied memory. However, with lists of two samples, it may be more likely to use the extra memory rather than reduce it. This can be fixed, for example, by using variable-sized lists.

It is noted that in the case of three-dimensional stereoscopic images for the left eye and the right eye, pixel values for images for the left eye and for the right eye can be resolved as described in FIG. 4. In this case, there is a separate set of samples for each target re-projection (i.e., for the left eye and for the right eye), and each target must be resolved independently. For efficiency, both goals can be resolved simultaneously, but also independently. It is noted that pixel values for stereoscopic images (for example, interlaced images from various types, as used in displays with step lens arrays) can also be resolved independently. If required, such images can be resolved independently simultaneously, for example, using a parallel processing system and software with a suitable configuration.

Images for the left eye and the right eye can be displayed sequentially or simultaneously depending on the nature of the display. For example, images for the left eye and the right eye may be displayed sequentially in the case of a 3D television display used with active shutter glasses. Alternatively, images for the left eye and the right eye can be displayed simultaneously in the case of a 2-projection type display used with passive glasses for 3D viewing having differently colored or differently polarized lenses for the left and right eye.

FIGS. 5A-5B further show the use of a coating in a multi-sample resolution of three-dimensional re-projection of a two-dimensional image. As an example, and not by way of limitation, FIGS. 5A-B illustrate a 2X multi-sample resolution method for three-dimensional re-projection of a two-dimensional image according to an embodiment of the present invention. In a 2X multi-sample scheme, a given re-projection pixel in line 501 may have up to two two-dimensional patterns associated with it. In FIG. 5A, immediately after two black pixel patterns 503A, 503B, two white pixel patterns 505A, 505B in a two-dimensional image are re-projected to the right. It is noted that in re-projecting a typical scene, one can reasonably expect that most of the pixels will not shift so that they remain next to the adjacent pixel. Some overlap of the offset pixels is allowed, as there are gaps between the offset pixels. Then, the coverage value for each two-dimensional sample 503A, 503B, 505A, 505B is determined. In the example shown in FIG. 5A, the first black sample 503A has 50% coverage in the first re-projection pixel 501A. The second black sample 503B has 50% coverage in the first re-projection pixel 501A and 50% coverage of the second re-projection pixel 501B. The first white sample 505A has 50% coverage in the second re-projection pixel 501B and 50% coverage in the third re-projection pixel 501C. The second white sample 505B has 50% coverage in the third three-dimensional pixel 501C and 50% coverage of the fourth re-projection pixel 501D. For the purposes of this example, it is assumed that the re-projection pixels were initiated to have no samples or to have zero coverage. Using the coverage values as weights and applying the general coverage described above to the fourth re-projection pixel, we get the allowed re-projection pixels 507A, 507B, 507C, 507D in the allowed re-projection line 507, having values of black, gray, white and white respectively.

However, the situation will be different if the coating is only slightly different. In particular, in FIG. 5B, two-dimensional pixels 503A, 503B, 505A, 505B are additionally re-projected to the right, which leads to different coverage values for two-dimensional pixels 503A, 503B, 505A and 505B. In the example shown in FIG.5B, these coatings are as follows. The first black sample 503A has 75% coverage in the first re-projection pixel 501A. The second black sample 503B has 25% coverage in the first re-projection pixel 501A and 75% coverage of the second re-projection pixel 501B. The first white sample 505A has 25% coverage in the second re-projection pixel 501B and 75% coverage in the third re-projection pixel 501C. The second white sample 505B has 25% coverage in the third three-dimensional pixel 501C and 75% coverage of the fourth re-projection pixel 501D. Again, for the purposes of this example, it is assumed that the re-projection pixels (including the fourth re-projection pixel 501D) are initiated to have no patterns or to have zero coverage. Using the coverage values as weights and applying the total coverage to the fourth re-projection pixel, we get the resolved re-projection pixels 507A, 507B, 507C, 507D in the re-projection allowed line 507 having black, gray, white, and white values, respectively. The darker gray value for the second allowed re-projection pixel 507B is the result of a higher relative coverage of the second re-projection pixel 501B with the black pixel 503B compared to the coverage with the white pixel 505A.

While the prototype method uses the position of the samples within a three-dimensional pixel to determine the final value of a three-dimensional pixel, the method of the invention uses the coating of a sample of a three-dimensional pixel in the resolution of its final color. This provides the ability to more accurately perform the resolution step compared to the prototype. As discussed above, the prototype allows the implementation of transitions of pixel values only for three-dimensional pixels, where there have been discrete offsets in re-projection. The method of the invention, on the other hand, resolves the final value of each three-dimensional pixel using the coating sizes of the samples, which allows smooth color transitions responsible for any re-projection offset.

Thus, embodiments of the present invention not only solve the problem of depth separation during three-dimensional re-projection, but also provide a more natural appearance of three-dimensional images.

In the case of using an autostereoscopic display, such as a stepped lens matrix, to display a final three-dimensional image, it is expected that the offset between the corresponding images of the observation points and the images of other observation points will be objectively small. Embodiments of the present invention also help to reduce the perceived jump when moving between points of observation. In 3-dimensional prototype systems, various objects tend to “lock in” in their positions, while the viewer moves between the observation points. Embodiments of the present invention provide a more natural displacement of all objects in a three-dimensional scene when changing the position of the viewer.

It is noted that FIGS. 5A-5B show an example where the samples 503, 505 are displaced horizontally, but not vertically, relative to the three-dimensional pixels 501. However, embodiments of the invention may include embodiments where the samples are displaced vertically or where the samples are displaced both vertically and horizontally .

FIG. 6 shows a block diagram of a computer device that can be used to implement a multi-sample smoothing three-dimensional re-projection of a two-dimensional image. The device 600, as a rule, may include a processor module 601 and a memory 605. The processor module 601 may include one or more processor cores. An example of a processing system using multiple processor modules is a cellular processor, examples of which are described in detail, for example, in the document Cellular Broadband Engine Architecture, available at http://www-306.ibm.com/chip/techlib/techlib.nsf/ techdocs / 1AEEE1270EA2776387257060006E61BA / $ file / CBEA_01_pub.pdf, which is incorporated into this application by reference.

The memory 605 may be in the form of an integrated circuit, such as RAM, DRAM, ROM, and the like. Memory 605 may also be the main memory available to all processor modules. In some embodiments, the processor module 601 has local storage devices associated with each core. Program 603 may be stored in main memory 605 in the form of processor readable instructions that can be executed on processor modules. Program 603 may be configured to perform multi-sample resolution of three-dimensional projection of a two-dimensional image. Program 603 can be written in any processor-readable language, such as C, C ++, JAVA, Assembly, MATLAB, FORTRAN, and a number of other languages. The source data 607 may also be stored in memory. Such source data 607 may include information regarding the identification of patterns to be used and the coverage values of the samples. During program execution 603, portions of the control program and / or data may be loaded into memory or local storage of processor cores for parallel processing by multiple processor cores.

The device 600 may also include well-known functional components 609, such as input / output (B / B) elements 611, power supplies (PIs) 613, a clock generator (TG) 615, and a cache 617. The device 600 may optionally include a memory device large capacity 619, such as a disk drive, CD-ROM drive, tape drive or the like, for storing programs and / or data. The device 600 may optionally include a display unit 621 and a user interface 625 to facilitate interaction between the device and the user. By way of example, and not by way of limitation, the display unit 621 may be in the form of a finished 3-D television displaying text, numbers, graphic symbols, or other visual objects as stereoscopic images, perceived by means of glasses 627 for 3-D viewing.

The user interface 625 may include a keyboard, mouse, joystick, light pen, or other device that can be used in conjunction with a graphical user interface (GUI). The device 600 may also include a network interface 623 to enable the device to communicate with other devices through a network such as the Internet.

System 600 components, including processor 601, memory 605, support functional components 609, mass storage device 619, user interface 625, network interface 623, and display unit 621, can be operatively connected to each other via one or more data buses 629. These components can be implemented in hardware, software, firmware, or in some combination of two or more of them.

There are a number of additional ways to enable parallel processing by multiple processors in the hardware. For example, it is possible to “deploy” processing cycles, for example, by copying a control program in two or more processor cores and each processor core executing a control program to process various data blocks. This can avoid the delay associated with setting up the loop. As applied to our invention, multiple processors can simultaneously identify two-dimensional patterns that correspond to pixels in three-dimensional re-projection. Additionally, multiple processors can simultaneously determine the coverage of samples of three-dimensional pixels or finally determine their colors. The possibility of parallel data processing saves valuable processing time, which leads to the possibility of creating a more efficient and optimal system for multi-sample resolution of three-dimensional re-projection of a two-dimensional image.

One example, among others, of a processing system capable of processing in parallel in three or more processors is a cellular processor. There are a number of different processor architectures that can be categorized as cellular processors. As an example, and not as a limitation, FIG.7 illustrates the type of cellular processor. Cellular processor 700 includes a main memory 701, a single common processor element (OPE) 707, and eight synergistic processor elements (SPE) 711. Alternatively, the cellular processor may be configured using any number of SPEs. 7, the memory 701, OPE 707 and SPE 711 can communicate with each other and the input / output device (B / B) 715 through the connecting bus ring type 717.

The memory 701 contains raw data 703 having characteristics common to the program described above. At least one of the SPEs 711 may include, in its local memory (ROM), instructions 713 for multi-resolution three-dimensional re-projection of a two-dimensional image and / or part of the source data that are intended for parallel processing, for example, as described above. OPE 707 may include in its L1 cache commands 709 multi-resolution three-dimensional re-projection of a two-dimensional image, having characteristics common to the program described above. Commands 705 and data 703 may also be stored in memory 701 for access to the SPE 711 and the PES 707, if necessary. It should be noted that any number of processes involved in the invented method for multi-sample resolution of three-dimensional re-projection of a two-dimensional image can be processed in parallel using a cellular processor.

By way of example, an OPE 707 may be a 64-bit PowerPC Common Processing Unit (OPB) with corresponding cache elements. OPE 707 may include an optional vector multimedia extension unit. Each SPE 711 includes a synergistic processor unit (SPB) and a local storage device (LZU). In some embodiments, the local storage device may have a capacity of, for example, about 256 kilobytes of memory for programs and data. SPBs are less complex computing units compared to SDBs, since they usually do not perform system management functions. SPBs may be able to perform the procedure with one stream of commands and multiple data streams (OIMP), and usually process the data and initiate all the required data transfers (relative to the access parameters set by the OPE) to perform the assigned tasks. SPBs allow the system to implement applications that require a higher computational density and can effectively use the provided instruction set. A significant amount of SPB in the system managed by the OPB allows for cost-effective processing in a wide range of applications.

By way of example, a cellular processor may be characterized by an architecture known as a Cell Broadband Engine (ASHD) architecture. In an ASHS-compliant architecture, multiple PES can be combined into a PPS group, and multiple PES can be combined into a PES group. For purposes of example, a cellular processor is shown as having one SPE group and one PES group with one SPE and one PES. Alternatively, a cellular processor may include multiple groups of common processor elements (OPE groups) and multiple groups of synergistic processor elements (SPE groups). ASHSD compatible processors are described in detail, for example, in the document Cellular Broadband Engine Architecture, which is incorporated into this application by reference and is available at https://www-306.ibm.com/chips/techlib/techlib.nsf/techdocs/1AEEE1270EA277638725706000E61BA /$file/CBEA_01_pub.pdf.

According to another embodiment, instructions for multi-sample resolution of three-dimensional re-projection of a two-dimensional image can be stored in a computer-readable storage medium. By way of example, and not by way of limitation, FIG. 8 illustrates an example of an immutable computer-readable storage medium 800 in accordance with an embodiment of the present invention. Storage medium 800 comprises computer-readable instructions stored in a format that can be extracted, interpreted, and executed by a computer processing device. By way of example, and not by way of limitation, a computer-readable storage medium may be a computer-readable memory, such as a random access memory (RAM), or a read-only memory (ROM), a computer-readable disk for storing information for a stationary disk drive (e.g. hard disk drive) or removable disk drive. In addition, the computer-readable storage medium 800 may be a flash memory device, computer-readable tape, CD-ROM, DVD-ROM, Blu-Ray, HD-DVD, UMD, or other optical storage media.

Storage medium 800 comprises instructions for multi-sample resolution of three-dimensional re-projection of a two-dimensional image 801. Commands 801 for multi-sample resolution of three-dimensional re-projection of a two-dimensional image can be configured to implement multi-sample resolution in accordance with the methods described above. In particular, multi-resolution resolution commands 801 may include two-dimensional image identification commands 803, which are used to determine one or more samples from a two-dimensional image that correspond to a three-dimensional re-projection pixel.

Multi-resolution resolution instructions 801 may also include instructions 805 to determine the area that a given two-dimensional pattern occupies in a three-dimensional pixel. For three-dimensional re-projection, this allows the displacement of parallel cameras; this sampling is performed only in the horizontal direction (i.e., samples are inhomogeneous in the vertical direction for any three-dimensional pixel). It is important to note that multiple sampling can be performed for any number of different directions.

Teams 801 for multi-sample resolution may additionally include commands 807 resolution of the final color of the pixel, which are configured to determine the color of the three-dimensional pixel by combining the weighted values of the coverage of the sample. For example, if a given three-dimensional pixel has a red sample with 80% coverage and a blue sample with 20% coverage, then the resolution commands for the final color of the pixel 807 can resolve the color of the three-dimensional pixel as a shade of magenta with a higher intensity of red due to its higher coverage. It is important to note that any number of weighing methods can be used to resolve the final color based on sample coverage values.

The multi-sample resolution instructions 801 may optionally include hole filling instructions 808, which can set a final value for any pixel for which the total weighted coverage values for all samples participating in the final value are less than 100% when executed.

Teams 801 for multi-sample resolution may additionally include commands 809 display three-dimensional re-projection of a two-dimensional image after performing multi-sample resolution.

Although embodiments have been described in which stereoscopic 3D images are observed using passive or active glasses for 3D viewing, embodiments of the invention are not limited to such embodiments. In particular, embodiments of the invention can be applied to stereoscopic 3D video technologies that are not based on tracking head position or using passive or active glasses for 3D viewing. Examples of such “point-free" stereoscopic 3D video technologies are sometimes considered autostereoscopic technologies or autostereoscopy. Examples of such technologies include, but are not limited to, step-based lens technologies. A stepped lens is a matrix of magnifying lenses designed in such a way that when viewed at slightly different angles, various images are magnified. Different images can be selected to provide a three-dimensional viewing effect when observing a stepped screen at different angles. The number of generated images increases in proportion to the number of viewing points of the screen.

In particular, in a video system using step lenses, images of re-projecting an object from slightly different viewing angles can be generated from the original 2-dimensional image and depth information for each pixel in the image. Using re-projection methods, different types of images from progressively different viewing angles can be generated from the original 2-dimensional image and depth information. Images representing various views can be divided into stripes and displayed alternately on an autostereoscopic display with a screen between the step lens array and the viewing point. The lenses that make up the stepped lens can be cylindrical magnifying lenses combined with strips and, as a rule, are twice as wide as the strips. The viewer perceives various types of scenes depending on the viewing angle of the screen. Various views can be selected to provide an illusion of the depth of the displayed scene.

In addition, although certain embodiments of the present invention can solve depth separation issues in the case of three-dimensional re-projection of two-dimensional images and include generating more than one image for re-projection, the embodiments are in most cases more applicable to non-3D re-projection cases. The multi-sample resolution of re-projected pixels described here can provide higher quality re-projected images due to the offset gradient. Such multi-sampling resolution can be achieved, for example, during a typical rasterization of a re-projected image.

In addition, in some three-dimensional implementations, it may not be necessary to generate two or more images. For example, in the case of stereoscopic imaging, it may not be necessary to generate both images for the left eye and the right eye through re-projection. Alternatively, only one new image can be generated. For example, it is possible to start with color and depth information for each pixel of the image for the left eye and generate the corresponding image for the right eye through re-projection (or vice versa), obtaining sufficient images for display using a stereoscopic display. This includes generating only one re-projected image.

While the full description of the preferred embodiment of the present invention has been given above, it is possible to use various alternatives, modifications and equivalents.

Although the present invention has been described in sufficient detail with reference to certain variations, other variations are possible. Therefore, the nature and scope of the attached claims should not be limited to the description of the preferred options contained in this application. Instead, the scope of the invention should be determined with reference to the attached claims, together with its full scope of equivalents.

All features disclosed in this description (including all accompanying claims, abstract and drawings), can be replaced by alternative features that serve to achieve the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is only one example from a general series of equivalent or similar features. Any feature, preferred or not, can be combined with any other feature, preferred or not. In the claims below, a singular expression or term means one or more, unless otherwise expressly indicated. Any element of the claims that does not explicitly state “means for” the fulfillment of this feature should not be interpreted as the phrase “means” or “step”, as indicated in 35 USC § 112, ¶6. In particular, the use of the “step” in the claims in this application is not intended to use the provisions of 35 USC § 112, ¶6.

The reader’s attention is drawn to all papers and documents that are submitted with this description and which are open for public inspection with this description, and the contents of all papers and documents are incorporated into this application by reference.

Claims (22)

1. The method of multi-sample resolution re-projection of a two-dimensional image, in which re-projection covers a group of pixels, including:
a) the identification of a plurality of two-dimensional image samples for each pixel in re-projection, wherein each such re-projection pixel is generated by the offset of the re-projection pixel relative to the two-dimensional image by the offset value based on the depth data of the two-dimensional image, and each two-dimensional sample when re-projected has such same width as re-projection pixels;
b) determining a plurality of sample coverage values for each re-projection pixel, in which the coverage amount of each sample is determined by the pixel offset in re-projection relative to the two-dimensional image, wherein the coverage amount of each sample identifies the pixel area covered by the corresponding two-dimensional sample after the displacement; and
c) resolution of the final value for each pixel of the re-projection by combining each two-dimensional sample associated with the pixel, each such two-dimensional sample being weighted according to its corresponding sample coverage value,
wherein a), b) and c) are performed for at least an image for the left eye or an image for the right eye, for stereoscopic display. 2. The method according to p. 1, characterized in that the identification of each sample in a) is performed using horizontal scanning. 3. The method according to p. 1, characterized in that a) includes the identification of at most two samples for each pixel in the re-projection. 4. The method according to p. 1, characterized in that b) further includes determining whether a particular pattern associated with the pixel in re-projection has a leading pattern in the neighboring pixel. 5. The method according to p. 4, characterized in that b) further includes determining whether the particular pattern associated with the pixel in the re-projection closes the pattern in the adjacent pixel. 6. The method according to p. 1, characterized in that a), b) and c) are performed for both images for the left eye and for the right eye, for stereoscopic display. 7. The method according to claim 1, characterized in that the two-dimensional image is an image for the left eye or an image for the right eye, for stereoscopic display, a), b) and c) are performed for the other of the images for the left eye or the right eye. 8. The method according to p. 1, characterized in that the stereoscopic display is an autostereoscopic display. 9. The method according to p. 1, characterized in that b) includes determining the overlap between two or more samples, and that c) includes reducing the amount of coverage for a sample of two or more samples having a depth farther from the viewer by the amount of coverage overlap with one or more other samples from two or more samples. 10. The method according to p. 1, characterized in that it further includes adjusting the final value for any pixel for which the total weighted coverage value for all samples contributing to this final value is less than 100%. 11. Device for multi-sample resolution re-projection of a two-dimensional image, including:
CPU;
memory; and
machine-readable instructions stored in memory and executed by a processor, where machine-readable instructions are configured to implement a multi-sample resolution method for re-projecting a two-dimensional image, wherein re-projecting covers a group of pixels, including:
a) identification of a plurality of two-dimensional image samples corresponding to each pixel in re-projection, wherein each such re-projection pixel is generated by the offset of the re-projection pixel relative to the two-dimensional image by the offset value based on the depth data of the two-dimensional image, and each two-dimensional sample when re-projected has Same width as re-projection pixels
b) determining a plurality of sample coverage values for each re-projection pixel, in which the coverage amount of each sample is determined by the pixel offset in re-projection relative to the two-dimensional image, wherein the coverage amount of each sample identifies the pixel area covered by the corresponding two-dimensional sample after the displacement; and
c) resolution of the final value for each pixel of the re-projection by combining each two-dimensional sample associated with the pixel, each such two-dimensional sample being weighted according to its corresponding sample coverage value,
wherein a), b) and c) are performed for at least an image for the left eye or an image for the right eye, for stereoscopic display. 12. The device according to p. 11, characterized in that it further includes a three-dimensional stereoscopic visual display, configured to display re-projection after multi-sample resolution. 13. The device according to p. 12, characterized in that the method further includes:
d) displaying re-projection on a three-dimensional stereoscopic visual display. 14. The device according to claim 11, characterized in that the identification of each sample in a) is performed using horizontal scanning. 15. The device according to p. 11, characterized in that a) includes the identification of at most two samples for each pixel in the re-projection. 16. The device according to p. 11, characterized in that b) further includes determining whether a particular pattern associated with a pixel in re-projection has a leading pattern in a neighboring pixel. 17. The device according to p. 16, characterized in that b) further includes determining whether the particular pattern associated with the pixel in re-projection has a trailing pattern in the adjacent pixel. 18. The device according to p. 11, characterized in that a), b) and c) are performed for both images for the left eye and for the right eye, for stereoscopic display. 19. The device according to p. 11, characterized in that the stereoscopic display is an autostereoscopic display. 20. The device according to p. 11, characterized in that the two-dimensional image is an image for the left eye or an image for the right eye, for stereoscopic display, a), b) and c) are performed for the other of the images for the left eye or right eye. 21. The device according to p. 11, characterized in that b) includes determining the overlap between two or more samples and where c) includes reducing the amount of coverage for a sample of two or more samples having a depth further from the viewer by the amount of overlap of the coating with one or more than other samples of two or more samples. 22. The device according to p. 11, characterized in that the method further includes adjusting the final value for any pixel for which the total weighted coverage value for all samples contributing to this final value is less than 100%.

Images