JP2000253422A - Method for generating three-dimensionall image from two-dimensional image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a three-dimensional(3D) image having no breakdown by generating pixels around a boundary between a subject image and a background image by interpolating them in a time base direction and generating pixels other than that around the boundary by interpolating them in a space axis direction in the case of generating a 3D image from a two-dimensional(2D) image. SOLUTION: A 2D video signal and depth information are stored in memories 102, 103 for plural frames and a subject boundary detection part 106 detects a boundary between a subject image and a background image. In the case of generating parallax information around the boundary, a corresponding pixel retrieving part 109, a corresponding pixel calculation part 107 and an interpolating image generation part 110 are constituted so as to generate pixels for obtaining parallax by retrieving them from the front/rear frame of the 2D video signal and obtaining interpolated pixels by pixel operation, and in the case of generating parallax information except the neighborhood of the boundary, constituted so as to obtain interpolated pixels by pixel operation in the frame of the 2D video signal.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for generating a three-dimensional image from a two-dimensional image, and more particularly to an interpolation process of an image necessary for generating a three-dimensional image when depth information of the two-dimensional image is given. It is something that was devised.

[0002]

2. Description of the Related Art While the development of consumer stereoscopic displays is currently progressing, the amount of software for consumer stereoscopic images is small, and it is necessary to newly produce stereoscopic images in order to display stereoscopic images.

Therefore, as a method of utilizing the assets of the conventional two-dimensional video, a method of converting the conventional two-dimensional video into a three-dimensional video has been proposed. A three-dimensional image generating apparatus for converting a two-dimensional image into a three-dimensional image is described in, for example,
No. 17.

In this method of generating a parallax image, an image of a current frame is presented to one eye, and an image delayed by a predetermined frame is presented to the other eye. When the subject is moving in the horizontal direction on the two-dimensional moving image, this movement becomes parallax and is perceived stereoscopically. However, when the subject is deformed, there is a problem that the amount of deformation is perceived as unnecessary parallax.

As another method of generating a parallax image, there is a method of generating a parallax image by interpolating a two-dimensional image in a spatial axis direction and enlarging / reducing the image. However, if interpolation processing is performed near the boundary between the subject image and the background image in the spatial axis direction, the subject and the background may be mixed and an unnatural image may be formed.

Another method for generating a parallax image is as follows.
There is a method of presenting an image of the current frame to one eye and presenting an image delayed by a predetermined frame to the other eye. When the subject is moving in the horizontal direction on the two-dimensional moving image, this movement becomes parallax and is perceived stereoscopically. However, when the subject is deformed, there is a problem that the amount of deformation is perceived as unnecessary parallax.

[0007]

As described above, when a parallax image is generated by interpolating the image in the spatial axis direction, it is inappropriate at the boundary of the subject. In addition, when the image delayed by the frame is used as the parallax image, the image is inappropriate when the subject is deformed between frames. In addition, when a frame-delayed video is used as a parallax video, there is no flexibility because an arbitrary parallax cannot be set in the video.

Accordingly, the present invention devises a pixel interpolation processing method when a stereoscopic video signal is generated from a two-dimensional video signal, for example, for a portion requiring a sense of depth, such as a boundary between a background image and a subject image. An object of the present invention is to provide a method for generating a stereoscopic image from a two-dimensional image that can improve the stereoscopic effect.

[0009]

According to the present invention, a two-dimensional video signal and its depth information are provided, and the two-dimensional video signal is converted into a left or right stereoscopic video signal according to the depth information. When the conversion processing is performed, the boundary of the subject image is detected, and when generating the parallax information near the boundary, the parallax information is generated by interpolation from the front and rear frames of the two-dimensional video signal. Is generated by an interpolation operation within a frame of the two-dimensional video signal, and the generated parallax information is added to the two-dimensional video signal.

This prevents the image near the boundary of the subject from becoming an unnatural image. Further, by interpolating the inside of the subject image from within the frame, it is possible to prevent the influence of the image change between frames.

[0011]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a configuration diagram showing a first embodiment of the present invention. The input unit 11 is supplied with the compressed two-dimensional video signal, and is guided to the two-dimensional video signal decoding unit 12. The two-dimensional video signal decoded here is supplied to the depth estimating unit 13. The depth estimating unit 13 obtains, for example, depth information of a subject image and a background image. There are various methods for obtaining depth information.

First, the background image is identified, and the background image and the subject image are discriminated. Next, find the average motion vector of the background image,
Next, a motion vector of the subject image is obtained. Next, a relative motion vector of the subject image is obtained from the average motion vector of the background image and the motion vector of the subject image. Next, the scalar amount of the relative motion vector is obtained, and the scalar amount is converted into depth information. It is determined that the depth is closer to the front as the scalar amount increases.

As a method of discriminating between the background image and the subject image, a portion where the contrast is lower than the threshold is determined as the background image, and the other portions are determined as the subject image. As a method for obtaining a motion vector, for example, a method defined in the MPEG standard is used.

The depth information obtained as described above is input to a depth information memory 103, and the decoded two-dimensional video signal is input to a video memory 102.

The three-dimensional image generation unit 101 includes the image memory 10
2. Depth information memory 103, video memory interface (video memory I / F) 104, depth information memory interface (depth information memory I / F) 105, subject boundary extraction unit 106, corresponding pixel calculation unit 107, motion vector detection unit 108 , A corresponding pixel search unit 109, and an interpolated video generation unit 110.

Next, the operation of each block will be described in detail.

The three-dimensional image generator 101 receives a two-dimensional image signal and depth information corresponding to the image. The two-dimensional video signal is stored in the video memory 102, and the depth information is stored in the depth information memory 103 for a plurality of frames. Here, for example, after five frames are stored as shown in FIGS. 2A and 2B, the third frame is read out.

The two-dimensional video is input to the subject boundary extraction unit 106 via the video memory interface 104,
The depth information is stored in the depth information memory interface 10.
5 to the corresponding pixel calculation unit 107.

The subject boundary extraction unit 106 detects the boundary (contour) of the subject image on the two-dimensional video. FIG. 3 shows an example of the boundary of the extracted subject image. The boundary of the subject image can be obtained from a single still image by extracting edges using primary differentiation and secondary differentiation filters, and by dividing and integrating regions such as K-means clustering. And the like. As a method for obtaining from a plurality of continuous frames, a motion vector may be obtained, and a boundary of the subject image may be obtained from a spatial change in the direction and magnitude of the vector. In the case of the present invention, since the depth information on the two-dimensional video is known, the boundary of the subject image may be obtained from the spatial change of the depth information. Further, the above-described methods may be used in combination. The extracted boundary information of the subject image is supplied to the corresponding pixel calculation unit 107.

The corresponding pixel calculator 107 calculates the positional relationship between the pixels of the generated parallax (stereoscopic) video signal and the pixels of the two-dimensional video signal according to the depth information. For the stereoscopic video signal, one for the left or the right is specified in advance.

This positional relationship will be described with reference to FIG. In FIG. 4, white circles indicate pixels on the horizontal line of the input two-dimensional video arranged in the depth direction based on the designated depth information. Black circles indicate pixel arrangements based on depth information on the display surface. Pixel of this black circle (light-emitting part)
Therefore, it is necessary to generate an image using black circle pixels so that the white circle ○ pixel is perceived. For this reason, the pixels from the white circles to the black circles are generated by interpolation.

When performing the interpolation operation, spatially adjacent pixels are used in the case of the same subject. At the boundary of the subject image, it is necessary to display the pixels of the background portion that do not exist in the current frame. Therefore, corresponding pixels are searched for from the front and rear frames and used.

For example, a case where the pixel of the black circle 7 'is generated will be described.

FIG. 5 shows this explanatory diagram. Pixels adjacent to the 7 'pixel correspond to 8 and 9. Since the pixels 8 and 9 are determined to be the same subject, they are generated by linear interpolation of the pixels 8 and 9. For this reason, the corresponding pixel calculation unit uses the address of the adjacent pixel (in this case, 8
And 9) and the distance (a and b) or the ratio of the distance (a / b) from the pixel 7 ′ to the interpolated video generation unit 110, respectively.

Next, a case where the 9 'pixel is generated will be described.
This explanatory diagram is shown in FIG. Pixels adjacent to 9 'pixel are 9, 10
However, since a subject boundary exists between the pixels 9 and 10, the pixels 9 and 10 are determined to be pixels of different subjects. For this reason, the 9 'pixel is not generated from 9 and 10. For example, if there is a 9p pixel that is located in the background with respect to 9 and is the same subject as 10, a 9 ′ pixel can be generated by linear interpolation between 9p and 10 pixels. By using the pixels of the preceding and succeeding frames in this manner, it is possible to obtain a video with less failure.

For that purpose, it is necessary to search for 9p pixels. However, since this 9p pixel does not exist in the current frame, it is necessary to search for the 9p pixel from the previous and next frames. The corresponding pixel calculation unit 107 notifies the corresponding pixel search unit 109 to search for a pixel between 9 and 10. The corresponding pixel search unit 109 searches the front and rear frames for pixels necessary to obtain parallax at the boundary of the subject.

In the present invention, a motion vector is used to determine whether a pixel exists in the front and rear frames. FIG. 7 shows the relationship between the motion vector and the parallax image.

In FIG. 7, reference numeral 701 denotes a current frame;
2 indicates a previous frame, and 703 indicates a subsequent frame. The subject image 704 moves relatively to the right with respect to the background, and the subject image 705 moves relatively to the left. The subject image 704 gradually moves to the right in the preceding frame, the current frame, and the subsequent frame. The subject image 705 is
The previous frame, the current frame, and the subsequent frame are gradually moving rightward.

At this time, it is considered that the parallax video signal is generated such that the subject images 704 and 705 are located in front of the background image.

In the case of a right eye image, subject images 704 and 705
Needs to be shifted to the left with respect to the background to generate a parallax image. By shifting to the left, the background parts 707 and 70
Eight areas result. This part is interpolated from the front and rear frames. As for the background region 707, the subject image 704 is moved rightward with respect to the background image, so interpolation is performed from the previous frame.

On the other hand, with respect to the background area 708, since the subject image 705 is moving leftward with respect to the background image, interpolation is performed from the subsequent frame. In the case of the left eye image, the subject image 704
And 705 must be shifted to the right with respect to the background image to generate a parallax image. By shifting to the right, the background part 7
Areas 10 and 711 occur. The background area 710 is interpolated from the subsequent frame. The background area 711 is interpolated from the previous frame.

Next, the operation of the corresponding pixel search unit 109 will be described. The corresponding pixel search unit 109 uses the motion vector to determine whether the corresponding pixel exists in the previous and next frames. Therefore, the motion vector at the boundary between the subject image and the background image is first obtained by the motion vector detection unit 108. The motion vectors in this case are the motion vector of the foreground area and the motion vector of the background area located on both sides of the boundary.

For obtaining the motion vector, for example, a block matching method is used. However, in the case of the rectangular macroblock shape used for the conventional motion vector detection, there is a high possibility that an error occurs near the boundary of the subject image. Therefore, the shape of the macro block is determined in consideration of the boundary shape of the subject image. For example, as shown in FIG. 8, the block shape is determined so that a certain number of pixels in the same area around the designated pixel are included in the block. As a method of block matching, a motion vector may be obtained by matching of parallel movement of a block, or a motion vector may be obtained by matching employing an affine transformation in consideration of deformation between frames of a block. The obtained motion vector is output to the corresponding pixel search unit 109.

The corresponding pixel search unit 109 searches and outputs the address and frame of the pixel corresponding to the parallax portion from the foreground motion vector, the background motion vector, and the boundary shape. This principle is shown in FIGS.

FIG. 9 shows the pixel arrangement in the horizontal direction of the image, in which black circles indicate pixels belonging to the foreground area, white circles indicate pixels belonging to the background area, and the current frame, one frame later,
The relationship between corresponding pixels after two frames is shown. Here, it is assumed that the boundary of the subject image exists between the pixels a and b. The motion vector of the foreground area is 0.5 pixels left, the motion vector of the background area is 1.0 pixels right, and the relative motion vector (foreground-background) is 1.5 pixels left.
When the motion vector is constant between the frames, the pixel a has a motion vector of 0.5 pixel leftward when compared with the frame after 1 frame and after 2 frames, so the position of c after 1 frame and the position of g after 2 frames Position.

Since the motion vector of pixel b is 1.0 pixel to the right, it is at position f after one frame and at position k after two frames.

Since the boundary exists somewhere between the pixels a and b, the range where the boundary may exist is indicated by the hatched portion. Here, e is a pixel in the background area that is not present in the current frame but exists one frame later, and h, i, and j is a pixel in the background area that is present two frames later. Since d is within the range of oblique lines, it is undefined whether the pixel belongs to the foreground area or the background area. FIG. 9 shows that when the direction of the relative motion vector of the foreground is toward the foreground, pixels in the background area necessary for generating parallax between the pixels a and b exist in the rear frame. The number of pixels depends on the relative vector and the number of frames.

FIG. 10 shows that the motion vector of the foreground area is 1.
The motion vector of the background region is 0 pixel right direction and the motion vector of the background region is 0.5 pixel left direction, and the relative motion vector (foreground-background) is 1.5 pixel right direction. The pixel a has a motion vector of 1.0 pixel to the right, so it is located at a position c before one frame and at a position g two frames before. Since the motion vector of pixel b is 0.5 pixel to the left, it is at the position of f one frame before and at the position of k two frames before.

Here, there are d and e as pixels in the background area that exist one frame before but not in the current frame, and h, i and j as pixels in the background area that exist two frames before. FIG.
When the relative motion vector of the foreground is oriented toward the background from 0, it can be seen that there are pixels in the background area necessary for generating parallax between the pixels a and b in the front frame.

FIGS. 11 and 12 show two-dimensional horizontal and vertical pixel relationships between frames. In FIG. 11, a black circle indicates a pixel belonging to the foreground area, and a white circle indicates a pixel belonging to the background area. Here, when the pixel a in the foreground area and the pixel b in the background area have the motion vector shown in the figure, the pixels a and b become c after one frame.
It corresponds to d, and to e and f after two frames. The shaded area indicates the range where the boundary exists. Here, the pixels in the background area that do not exist in the current frame but exist after the first and second frames are indicated by thick circles.

Leftward of d after one frame, f after two frames
Since there are pixels in the background area that do not exist in the current frame to the left, it can be seen that there are pixels in the background area necessary for generating disparity information between the pixels a and b in the subsequent frame.

FIG. 12 shows a case where the boundary shape is different from that of FIG. Each motion vector is the same as in FIG. In the case of the boundary shape in FIG. 12, the left direction of d after one frame and the left direction of f after two frames become the foreground area or the range where the boundary exists. There are no pixels in the background area required for generation. Therefore, it can be seen that whether or not a pixel exists depends not only on the horizontal component of the relative motion vector but also on the boundary shape.

FIGS. 13A and 13B show the case where the pixels corresponding to the pixels a and b near the boundary of the current frame are located at positions c and d after t frames, respectively. Here, the position of d is between pixel arrangements. Therefore, the pixels required to generate parallax information
Since e and f are also between pixels, e can be generated by interpolating from g and h, and f can be interpolated from i and j by referring to distances m and n between pixels.

A flow for searching for a pixel required for generating parallax in consideration of the examples shown in FIGS. 9 to 13 will be described below.

In FIG. 14, in step 1401, a relative motion vector is calculated by subtracting a background motion vector from a foreground motion vector. In step 1402, it is determined whether or not the direction of the relative motion vector is the foreground direction (the method in which the foreground motion vector is oriented). In the case of the foreground direction, the process proceeds to step 1403, and in step 1403, a frame after a predetermined number from the video memory 102 is selected as a search target frame for generating disparity information. When the direction of the relative motion vector is the background direction (the direction in which the background motion vector is directed), the process proceeds to step 1404, and in step 1404, a predetermined number of previous frames are selected from the video memory 102 as search target frames for generating disparity information.

In step 1405, it is determined where the pixel located on the background side of the boundary in the current frame moves on the search target frame. This displacement amount is a value obtained by multiplying the background motion vector by a predetermined number of frames. In step 1406, it is determined where the existing range of the boundary in the current frame moves on the search target frame. This displacement amount is a value obtained by multiplying the foreground motion vector by a predetermined number.

In step 1407, it is determined whether or not a pixel required for generating disparity information exists on the search target frame. From the pixel position and the boundary position on the boundary background side on the search target frame calculated in steps 1405 and 1406, the pixels belonging to the background region in the horizontal direction from the pixel position on the boundary background side to the boundary on the search target frame Is searched for, and if present, the address of the corresponding pixel is output (step 140
8).

For example, in FIG.
Two frames later, and if the pixel position on the boundary background side is f, addresses g, h, i, and j will be output. If the corresponding pixel is located between the pixels, the address of the adjacent pixel and the distance between the addresses are output. In the case of FIG. 13, since the corresponding pixels are located at e and f, respectively, g and h
, The addresses of i and j and the distances m and n are output. As a result of the search, if there is no pixel belonging to the background area necessary for generating the disparity information on the search target frame, the fact is output (step 1409).

FIGS. 15 and 16 show a second method of searching for pixels necessary for generating parallax information. In FIG. 1, pixels required for parallax generation are searched from one search target frame.However, considering that an image is deformed between frames and an error is included in a motion vector, a frame close to the current frame in a time axis is considered. It is desirable to search for a pixel necessary for generating disparity information from. FIG. 15 illustrates this search method. In FIG. 15, the boundary shape of the subject and each motion vector are the same as in FIG. In FIG. 15, a thick circle indicates a pixel in the background area that does not exist in the current frame but exists one frame later in the figure one frame later, and two frames that do not exist one frame later in the figure two frames later. The pixel of the background area that exists later is shown.

Considering that pixels necessary for generating disparity information are selected from a frame close to the current frame, g, h, i, and j pixels after one frame, and k, l, and m pixels after two frames Becomes

FIG. 16 shows a flow of a second method of searching for pixels necessary for generating disparity information. The difference from the example of FIG. 14 is that one frame after or one frame before is used as a search target frame instead of a frame after or after a predetermined number as a search target frame. If a pixel required for generating disparity information exists on the search target frame, the address of the pixel is output. After outputting the address of the pixel necessary for generating the disparity information, the search target frame is updated again one frame later or one frame earlier than the current frame. Here, pixels that are not present in the previous search target frame but are present in the current search target frame and are required for generating disparity information are output. The above processing is repeated until the number of updates reaches a predetermined number.

The address of the searched pixel is output to the corresponding pixel calculating section 107. The corresponding pixel calculation unit 107 determines where a pixel required for generating parallax is located on the current frame. FIG. 17 is shown for this explanation. FIG.
Is a case where pixels required for generating disparity information exist in the previous frame. A black circle indicates that pixels of the current frame are arranged according to the depth information, and a cross indicates that pixels of the previous frame are arranged according to the depth information. c indicates a pixel required for generating the specified parallax information as a result of the search. When the pixel corresponding to a on the current frame is at the position b in the previous frame, the displacement amount of a and b can be used to determine where c on the previous frame is located on the current frame. . FIG. 17 shows that the pixel c is located at the position d on the current frame.

After the upper position in the current frame of the pixel required for generating the parallax information is obtained, the same as the information for generating the pixel of the parallax information (image), as in the case where the adjacent pixels belong to the same subject image. The address and the frame position of the adjacent pixel in the subject and the distance from each pixel are obtained and output to the interpolated video generation unit 110 (explanatory FIGS. 5 and 6).

If there is no pixel required for generating the parallax information, the pixel required for generating the parallax information is replaced by a neighboring pixel in the same subject, and the pixel is output to the interpolation image generating unit 110 as an adjacent pixel. . This explanatory diagram is shown in FIG.
FIG.

The interpolated video generation unit 110 generates each pixel of the parallax image by linear interpolation according to the pixel address and the distance input from the corresponding pixel calculation unit 107. For example, to generate a pixel z of a parallax image, the values of adjacent pixels are x and y, respectively.
When the distance to z is a and b, respectively, z = {x × b / (a + b)} + {y × a / (a + b)}.

However, if there is no pixel necessary for generating parallax information, it is replaced by a nearby pixel in the same subject.

FIG. 19 shows a configuration diagram of the second embodiment of the present invention. 1 are indicated by the same reference numerals. The difference from the first embodiment is that the relative motion vector average calculation unit 190
In step 2, the average of the horizontal components of the relative motion vector (the motion vector of the foreground area-the motion vector of the background area) is calculated, and the frame stored in the video memory and the depth information memory is controlled with reference to the calculated value. .

The second embodiment will be described with reference to FIG. When the horizontal component of the relative motion vector is small, it is unlikely that there is a pixel required for generating disparity information between temporally adjacent frames. For this reason, the interval between frames stored in the video memory and the depth information memory is increased. On the other hand, when the horizontal component of the relative motion vector is large, there is a high possibility that pixels required for generating disparity information exist between temporally adjacent frames. Therefore, the interval between frames stored in the video memory and the depth information memory is shortened. Accordingly, when only a certain number of frames can be stored in the video memory and the depth information memory, it is possible to select and store a frame suitable for generating disparity information.

FIG. 21 shows a configuration diagram of the third embodiment of the present invention. 1 are indicated by the same reference numerals. The difference from the first embodiment is that the depth information difference average calculation unit 2102
Is to calculate the average of the difference between the depth information and control the frames stored in the video memory and the depth information memory with reference to the calculated value.

A third embodiment will be described with reference to FIG. When the difference between the depth information is large (FIG. 22C),
The area of the parallax region required for generating the parallax image is also large.
For this reason, the frame interval stored in the video memory and the depth information memory is widened so that a large number of pixels required for generating parallax are included. On the other hand, when the difference of the depth information is small (FIG. 22A), the area of the parallax region required for generating the parallax image is also small. Therefore, the interval between frames stored in the video memory and the depth information memory is shortened. When the difference of the depth information is small, the influence of the deformation between the frames of the subject is suppressed by interpolating the pixels required for generating the parallax from the temporally close frames. Accordingly, when only a certain number of frames can be stored in the video memory and the depth information memory, it is possible to select and store a frame suitable for generating disparity information.

FIG. 23 shows a configuration diagram of the fourth embodiment of the present invention. 1 are indicated by the same reference numerals. The difference from the first embodiment lies in the operation of the subject boundary calculation unit 2302. The subject boundary calculation unit 2302 calculates not only the position of the boundary but also the range where the boundary exists (the width of the boundary). The range where the boundary exists is determined with reference to the spatial frequency components near the boundary.

FIG. 24 shows the relationship between the spatial frequency near the boundary and the range where the boundary exists. When the spatial frequency component near the boundary is low, the pixels in the vicinity are expected to have a mixed value of the two regions. Therefore, it is inappropriate to use this pixel when generating a parallax image by interpolation. Therefore, when the spatial frequency component near the boundary is low, the range where the boundary exists is widened (the hatched portion in FIG. 24D),
When the spatial frequency component near the boundary is high, the range where the boundary exists is narrowed (the hatched portion in FIG. 24B). Pixels in the range where the boundary exists are not used for interpolation calculation for generating disparity information, so that pixels in which two regions are mixed are not used for generating disparity images.

[0064]

As described above in detail, according to the present invention,
When generating a stereoscopic image from a two-dimensional image, by generating pixels near the subject boundary in the time axis direction, and generating pixels other than the vicinity of the boundary by interpolating in the spatial axis direction, It is possible to provide a stereoscopic image without failure.

[Brief description of the drawings]

FIG. 1 is a configuration diagram showing a first embodiment of the present invention.

FIG. 2 is an explanatory diagram showing operations of a video memory and a depth information memory.

FIG. 3 is a diagram showing an example of extracting a subject boundary from a two-dimensional video.

FIG. 4 is a diagram showing a relationship between a two-dimensional image arranged with designated depth information and a pixel arrangement on a display surface.

FIG. 5 is a diagram illustrating a case where a parallax image is generated by interpolating pixels other than those near the boundary of a subject.

FIG. 6 is a diagram illustrating a case where a parallax image is generated by interpolating pixels near a boundary of a subject.

FIG. 7 is an explanatory diagram showing a relationship between a motion vector and a parallax region.

FIG. 8 is a diagram showing an example of the shape of a block used for block matching.

FIG. 9 is a diagram showing an example in which the relationship between a motion vector and a frame of a subject boundary is shown in a horizontal direction.

FIG. 10 is a diagram showing an example in which the relationship between a motion vector and a frame of a subject boundary is shown in a horizontal direction.

FIG. 11 is a diagram showing an example in which a relationship between a motion vector and a frame of a subject boundary is shown in a two-dimensional direction.

FIG. 12 is a diagram showing an example in which a relationship between a motion vector and a frame of a subject boundary is shown in a two-dimensional direction.

FIG. 13 is a diagram showing an example in which a relationship between a motion vector and a frame of a subject boundary is shown in a two-dimensional direction.

FIG. 14 is a flowchart illustrating a method for searching for a pixel required for generating parallax.

FIG. 15 is a diagram showing an example in which a relationship between a motion vector and a frame of a subject boundary is shown in a two-dimensional direction.

FIG. 16 is a flowchart illustrating a method for searching for a pixel required for generating parallax.

FIG. 17 is an explanatory diagram showing a method of arranging searched pixels three-dimensionally on a current frame.

FIG. 18 is an explanatory diagram illustrating interpolation processing when there is no pixel required for generating parallax.

FIG. 19 is a configuration diagram showing a second embodiment of the present invention.

FIG. 20 is an explanatory diagram showing a method of controlling a frame stored in a memory according to the magnitude of a relative motion vector.

FIG. 21 is a configuration diagram showing a third embodiment of the present invention.

FIG. 22 is an explanatory diagram showing a method of controlling a frame stored in a memory according to the magnitude of a difference in depth information.

FIG. 23 is a configuration diagram showing a fourth embodiment of the present invention.

FIG. 24 is an explanatory diagram showing a method of controlling a range where a boundary exists according to a spatial frequency component near the boundary of the subject;

[Explanation of symbols]

101: stereoscopic image generation device, 102: video memory, 10
3: depth information memory, 104: video memory interface, 105: depth information memory interface
106: subject boundary extraction unit, 107: corresponding pixel calculation unit,
108: motion vector calculation unit, 109: corresponding pixel search unit, 110: interpolation video generation unit, 201: video memory, 2
02: depth information memory; 1901: stereoscopic video generation device; 1902: relative motion vector average calculation unit; 1903
... video memory interface, 1904 ... depth information memory interface, 2101 ... stereoscopic video generation device, 2102 ... depth information difference average calculation unit, 2103 ...
Video memory interface, 2104 depth information memory interface, 2301 stereoscopic video generation device,
2302 ... A subject boundary extraction unit.

Claims (8)

[Claims] When generating a stereoscopic video signal from a two-dimensional video signal, when depth information of the two-dimensional video signal is given, the two-dimensional video signal and the depth information are stored in a memory for a plurality of frames. Detecting a boundary between at least a subject image and a background image in a two-dimensional video signal, and generating parallax information near the boundary, a pixel for obtaining parallax is positioned before and / or after the two-dimensional video signal. In the case where the interpolated pixel is generated by obtaining an interpolated pixel by searching from a frame and performing pixel operation, and when generating disparity information other than the vicinity of the boundary, the interpolated pixel is generated by obtaining an interpolated pixel by performing a pixel operation in the frame of the two-dimensional video signal A method for generating a three-dimensional image from a two-dimensional image characterized by the following. 2. A method for retrieving pixels for obtaining disparity information from a front and rear frame when disparity is provided in the vicinity of the boundary, comprising: comparing a current frame with a front and / or rear frame to obtain a Estimate the respective motion vectors of the pixels located on the foreground area side and the pixels located on the background area side on both sides, and subtract the motion vector of the pixel located on the background area side from the motion vector of the pixel located on the foreground area side. If the direction of the relative motion vector is directed to the foreground direction, it is determined where the pixels located on both sides of the boundary in the current frame move on the rear frame, and the pixel is determined in the rear frame. And the pixel belonging to the background side is a pixel corresponding to the parallax, and when the direction of the relative motion vector is in the background direction, It is determined where the pixels located on both sides of the boundary in the current frame move on the front frame, and pixels located between the pixels in the front frame and belonging to the background side are regarded as pixels corresponding to parallax. The method for generating a three-dimensional image from a two-dimensional image according to claim 1. 3. A block matching method is used for obtaining a motion vector on the foreground area side and a motion vector on the background area side on both sides of the boundary, and the block shape is determined according to the boundary shape of the subject, and the same block is determined. 3. The method according to claim 2, wherein the block shape is determined so that only the pixels in the same area are included. 4. When giving parallax to a pixel near the boundary, if a plurality of pixels corresponding to the parallax exist in front and / or rear frames, a pixel on a frame which is temporally closest to the current frame is determined. The method for generating a three-dimensional image from a two-dimensional image according to claim 2, wherein the method is used. 5. When a pixel corresponding to the parallax is not present in the front and rear frames when parallax is applied to the pixel near the boundary, the pixel adjacent to the boundary is located on the background side in the depth information direction among the pixels in contact with the boundary. The method according to claim 2, wherein the parallax information is generated by substituting pixels as pixels corresponding to parallax. 6. When storing the two-dimensional video signal and the depth information in a memory for a plurality of frames, a frame interval to be stored depends on a horizontal component of the relative motion vector. The method for generating a three-dimensional image from a two-dimensional image according to claim 2, wherein 7. When storing the two-dimensional image and the depth information in a memory for a plurality of frames, the frame interval to be stored depends on the size of the depth information difference between pixels. 3. The method for generating a three-dimensional image from a two-dimensional image according to claim 2, wherein: 8. A pixel located in a range considered as a boundary of a subject image is not used for an interpolation operation when generating a stereoscopic image.
3. The two-dimensional image according to claim 2, wherein the range considered as the boundary of the subject image follows the spatial frequency component near the boundary of the subject image, and the range in which the boundary of the subject image is considered becomes wider as the spatial frequency component becomes lower. 3D image generation method.