JP5971017B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program for executing processing for converting a 2D image into a 3D image for stereoscopic viewing.

  In recent years, 3D video contents such as 3D movies and 3D broadcasts have become widespread. In order to make an observer stereoscopically view, a right eye image and a left eye image having parallax are required. When displaying a 3D image, the right eye image and the left eye image are displayed in a time-sharing manner, and the right eye image and the left eye image are separated by image separation glasses such as shutter glasses and polarized glasses. Thus, the observer can observe the right eye image only with the right eye and the left eye image only with the left eye and can perform stereoscopic viewing. Note that if the right eye image and the left eye image are divided not by time division but by space division, glasses are not necessary, but resolution is lowered. Both the glasses method and the glassesless method are common in that a right eye image and a left eye image are required.

  There are two main methods for producing 3D video. A method for simultaneously capturing a right eye image and a left eye image using two cameras, and a later editing of a 2D image captured by one camera. There is a method for generating a parallax image. The present invention relates to the latter method, and to 2D3D conversion technology.

  FIG. 1 is a diagram for explaining a basic processing process of 2D3D conversion. First, a depth map (also referred to as depth information) is generated from the 2D input image (step S10). Then, a 3D image is generated using the 2D input image and the depth map (step S30). In FIG. 1, a 2D input image is a right eye image of a 3D output image, and an image obtained by pixel shifting the 2D input image using a depth map is a left eye image of the 3D output image. Hereinafter, a combination of a right eye image and a left eye image having a predetermined parallax is referred to as a 3D image or a parallax image.

  When generating a 3D image, the 2D image is pixel-shifted using a depth map, and a 2D image of another viewpoint having a parallax with respect to the 2D image is generated. Due to this pixel shift, missing pixels are generated in the 2D image of the different viewpoint to be generated. Generally, this missing pixel is interpolated from surrounding pixels.

JP 2009-44722 A

  When the depth difference at the object boundary is large in the screen, the pixel shift amount at the boundary part is also large. Therefore, the number of missing pixels, that is, the area of the missing region also increases. As described above, peripheral pixels are interpolated in the missing pixel, but when the area of the missing region is increased, a portion where the interpolated pixel does not match the interpolation position is likely to occur.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technique for improving the image quality of an object boundary portion when generating a 3D image from a 2D image.

In order to solve the above-described problem, an image processing apparatus (100) according to an aspect of the present invention includes a mask correction unit (80) that corrects an externally set mask pattern, and a depth map of an input image. A depth map processing unit (20) that processes each of a plurality of areas specified by the plurality of mask patterns corrected by (80), the input image, and a depth map processed by the depth map processing unit (20). An image generation unit (30) that shifts pixels of the input image to generate an image of another viewpoint having a parallax with respect to the input image . The mask correction unit (80) compares whether the image of the different viewpoint is a right-eye image or a left-eye image, and compares an inner depth value and an outer depth value of an object boundary portion of the mask pattern. depending on the context the object and the surroundings resulting from the result, the filter shape setting unit for setting the filter shape of the low-pass filter comprises a (85), Kio object boundary before using at least the right and left unsymmetrical said low pass filter A blurring process is performed. The filter shape setting unit (85) sets the edge of the other side of the object to be a gentler filter shape than the edge of the one side when the object is pixel-shifted to the left and right sides.

Another aspect of the present invention is an image processing method. The method includes a step of correcting an externally set mask pattern, a step of processing a depth map of an input image for each of a plurality of regions designated by the corrected plurality of mask patterns, the input image, and the processing based on the depth map, which is, by shifting the pixels of the input image, and a step of generating an image of another viewpoint with parallax with respect to the input image. The step of correcting the mask pattern comprises comparing whether the image of the different viewpoint is a right-eye image or a left-eye image, and comparing an inner depth value and an outer depth value of the boundary in the object boundary portion of the mask pattern. depending on the context the object and the surroundings resulting from the results, including the step of setting the filter shape of the low-pass filter, the Kio object boundary before at least the left and right by using the asymmetric said low pass filter is subjected to blurring processing . In the step of setting the filter shape, when the object is pixel-shifted to the left and right sides, the other edge of the object is set to a gentler filter shape than the one side edge.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, when generating a 3D image from a 2D image, it is possible to generate a high-quality 3D image while reducing the labor of the producer.

It is a figure for demonstrating the basic processing process of 2D3D conversion. It is a figure which shows the structure of the image editing system which concerns on the Example used as the foundation of this invention. It is a figure which shows the structural example of the depth map production | generation part which concerns on the Example of this invention. It is a figure for demonstrating the whole processing process of the image editing system which concerns on the Example used as the foundation of this invention. It is a figure for demonstrating the gain adjustment process of an input depth map. It is a figure for demonstrating the offset adjustment process of an input depth map. It is a figure for demonstrating the synthetic | combination process of a layer depth map. It is a figure for demonstrating the gain adjustment process of the input depth map which does not use a mask. It is a figure for demonstrating the offset adjustment process of the input depth map which does not use a mask. It is a figure for demonstrating pixel shift and pixel interpolation. It is a figure for demonstrating pixel shift and pixel interpolation in case the depth level | step difference of an object boundary is large. It is a figure for demonstrating pixel shift and pixel interpolation which does not generate | occur | produce unnatural even when the level | step difference of the depth of an object boundary is large. 1 is a diagram illustrating a configuration of an image editing system according to Embodiment 1 of the present invention. It is a figure for demonstrating the whole processing process of the image editing system which concerns on Example 1 of this invention. It is a figure for demonstrating the alpha blend of a layer depth map. It is a figure which shows the structural example of a mask correction | amendment part. It is a figure for demonstrating the mask blurring process by the mask correction part of FIG. It is a figure for demonstrating the relationship between the inclination formed in a mask signal by a 1st low-pass filter, and the 1st threshold value set to a binarization part. It is a figure for comparing the inclination added by a 1st low-pass filter, and the inclination added by a 2nd low-pass filter. It is a figure for demonstrating the whole processing process of the image editing system which concerns on Example 2 of this invention. It is a figure which shows the structural example of the mask correction part which concerns on Example 3 of this invention. FIGS. 22A to 22C are diagrams for explaining mask edge processing using a second symmetrical low-pass filter. FIGS. 23A to 23C are diagrams for explaining mask edge processing using a second low-pass asymmetric filter. It is a flowchart for demonstrating the determination process of the filter shape by the filter shape setting part which concerns on Example 3 of this invention. FIGS. 25A to 25C are diagrams for explaining mask edge processing using a first symmetrical low-pass filter. FIGS. 26A to 26C are diagrams for explaining mask edge processing using a first low-pass asymmetric filter.

  FIG. 2 is a diagram showing a configuration of an image editing system 500 according to an embodiment as a basis of the present invention. An image editing system 500 according to the present embodiment includes an image processing device 100 and a console terminal device 200.

  The console terminal device 200 is a terminal device used by an image producer (hereinafter referred to as a user) to produce and edit an image. The console terminal device 200 includes an operation unit 60 and a display unit 70. The operation unit 60 is an input device such as a keyboard and a mouse, and the display unit 70 is an output device such as a display. Note that a touch panel display which is a device in which input and output are integrated may be used. The console terminal device 200 may include a user interface that uses a printed material such as a printer or a scanner as a medium. The operation unit 60 receives a user operation, generates a signal resulting from the user operation, and outputs the signal to the image processing apparatus 100. The display unit 70 displays an image generated by the image processing apparatus 100.

  The image processing apparatus 100 includes a depth map generation unit 10, a depth map processing unit 20, a 3D image generation unit 30, an operation reception unit 40, and a display control unit 50. These configurations can be realized by an arbitrary processor, memory, or other LSI in terms of hardware, and can be realized by a program loaded into the memory in terms of software, but here by their cooperation. Draw functional blocks. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof. For example, all the functions of the depth map generation unit 10, the depth map processing unit 20, and the 3D image generation unit 30 may be realized by software, and the functions of the depth map generation unit 10 and the 3D image generation unit 30 are dedicated logic. It may be configured by a circuit, and the function of the depth map processing unit 20 may be realized by software.

  The depth map generation unit 10 generates a depth map of the 2D image based on the input 2D image and the depth model. The depth map is a grayscale image in which depth values (also referred to as depth values) are represented by luminance values. The depth map generation unit 10 estimates a scene structure and generates a depth map using a depth model suitable for the scene structure. In this embodiment, the depth map generation unit 10 combines a plurality of basic depth models and uses them for depth map generation. At that time, the composition ratio of a plurality of basic depth models is changed according to the scene structure of the 2D image.

  FIG. 3 is a diagram illustrating a configuration example of the depth map generation unit 10 according to the embodiment of the present invention. The depth map generation unit 10 includes an upper screen high-frequency component evaluation unit 11, a lower screen high-frequency component evaluation unit 12, a composition ratio determination unit 13, a first basic depth model frame memory 14, and a second basic depth model frame memory 15. , A third basic depth model frame memory 16, a synthesis unit 17, and an addition unit 18.

  The screen upper high frequency component evaluation unit 11 calculates the ratio of pixels having a high frequency component in the upper screen of the 2D image to be processed. The ratio is used as the high frequency component evaluation value at the top of the screen. The ratio of the upper part of the screen to the entire screen is preferably set to approximately 20%. The lower screen high frequency component evaluation unit 12 calculates the ratio of pixels having a high frequency component in the lower screen of the 2D image. The ratio is used as the high frequency component evaluation value at the bottom of the screen. The ratio of the lower part of the screen to the entire screen is preferably set to approximately 20%.

  The first basic depth model frame memory 14 holds the first basic depth model, the second basic depth model frame memory 15 holds the second basic depth model, and the third basic depth model frame memory 16 stores the third basic depth model. Holds the depth model. The first basic depth model is a model in which the upper part of the screen and the lower part of the screen are concave spherical surfaces. The second basic depth model is a model in which the upper part of the screen is a cylindrical surface having an axis in the vertical direction and the lower part of the screen is a concave spherical surface. The third basic depth model is a model in which the upper part of the screen is a plane and the lower part of the screen is a cylindrical surface having an axis in the horizontal direction.

  The composition ratio determination unit 13 is a first basic depth model based on the high-frequency component evaluation values of the upper screen portion and the lower screen portion calculated by the upper screen region high-frequency component evaluation unit 11 and the lower screen region high-frequency component evaluation unit 12, respectively. The combination ratios k1, k2, and k3 (where k1 + k2 + k3 = 1) of the second basic depth model and the third basic depth model are determined. The synthesizing unit 17 multiplies the synthesis ratios k1, k2, and k3 by the first basic depth model, the second basic depth model, and the third basic depth model, and adds the multiplication results. This calculation result becomes a synthetic basic depth model.

  For example, when the high frequency component evaluation value at the upper part of the screen is small, the composition ratio determining unit 13 recognizes that the scene has an empty or flat wall at the upper part of the screen, and the second basic depth model with a deeper depth at the upper part of the screen. Increase the ratio. If the high-frequency component evaluation value at the bottom of the screen is small, it is recognized as a scene in which the flat ground or water surface continuously spreads at the bottom of the screen, and the ratio of the third basic depth model is increased. In the third basic depth model, the upper part of the screen is approximated as a distant view, and the depth is reduced toward the bottom of the lower part of the screen.

  The adding unit 18 generates a depth map by superimposing the red component (R) signal of the 2D image on the combined basic depth model generated by the combining unit 17. The reason for using the R signal is based on an empirical rule that there is a high probability of matching with the unevenness of the subject in an environment in which the magnitude of the R signal is close to direct light and the brightness of the texture is not significantly different. . Further, red and warm colors are forward colors in chromatics, and the depth is recognized in front of the cold color system, and the stereoscopic effect is emphasized.

  Returning to FIG. The depth map processing unit 20 processes the depth map generated by the depth map generation unit 10. In the present embodiment, the depth map processing unit 20 generates a depth map generated by the depth map generation unit 10 for each of a plurality of areas in the screen specified by a plurality of mask patterns (hereinafter simply referred to as masks) set externally. Are processed individually or independently. For example, processing such as gain adjustment, offset adjustment and gradation processing is performed. Details of the processing by the depth map processing unit 20 will be described later.

  The 3D image generation unit 30 generates a 2D image of another viewpoint based on the above-described 2D image and the depth map processed by the depth map processing unit 20. The 3D image generation unit 30 outputs the 2D image of the original viewpoint and the 2D image of the different viewpoint as the right eye image and the left eye image.

  Hereinafter, a specific example of generating a 2D image of another viewpoint having a parallax with the 2D image using the 2D image of the original viewpoint and the depth map will be described. In this specific example, a 2D image of another viewpoint that is moved to the left is generated based on the viewpoint when the 2D image of the original viewpoint is displayed on the screen. In that case, when displaying the texture as a foreground for the observer, move the texture of the 2D image of the original viewpoint to the left side toward the screen by a predetermined amount, and toward the screen when displaying the texture as a distant view for the observer. Move the texture to the right by a predetermined amount.

  Let Yd be the luminance value of each pixel in the depth map, m be the convergence value representing the pop-out feeling, and n be the depth value representing the stereoscopic effect. The 3D image generation unit 30 sequentially shifts the texture of the 2D image at the original viewpoint corresponding to the luminance value Yd to the left by (Yd−m) / n pixels in order from the smallest luminance value Yd. If the value of (Yd−m) / n is negative, shift to the right by (m−Yd) / n pixels. For the observer, a texture with a small luminance value Yd of the depth map appears on the back side of the screen, and a texture with a large luminance value Yd appears on the front side of the screen. The luminance value Yd, the convergence value m, and the depth value n are values in the range of 0 to 255. For example, the convergence value m = 200 and the depth value n = 20 are set.

  A more detailed description of the depth map generation by the depth map generation unit 10 and the 3D image generation by the 3D image generation unit 30 will be described in Japanese Patent Application Laid-Open Nos. 2005-151534 and 2009-44722 previously filed by the present applicant. It is disclosed in the gazette.

  The operation reception unit 40 receives a signal input from the operation unit 60 of the console terminal device 200. The operation reception unit 40 outputs the input signal to the depth map processing unit 20 or the 3D image generation unit 30 according to the content. The display control unit 50 controls the display unit 70 of the console terminal device 200. Specifically, the display control unit 50 displays the 2D input image, the depth map generated by the depth map generation unit 10, the depth map processed by the depth map processing unit 20, and the 3D image generated by the 3D image generation unit 30. Can be displayed.

  FIG. 4 is a diagram for explaining the entire processing process of the image editing system 500 according to the embodiment as the basis of the present invention. In general, a 2D image includes a plurality of objects. The 2D input image of FIG. 4 includes three objects. Specifically, it includes people, trees, and background objects. First, the depth map generation unit 10 generates a depth map from the 2D input image (step S10). In the depth map, the closer to white, the higher the luminance and the closer to the observer, and the closer to black, the lower the luminance and the farther the observer. When a 3D image is generated, the pop-out amount increases as the area is closer to white in the depth map, and the retraction amount increases as the area is closer to black.

  In this embodiment, in order to individually adjust the sense of depth for a plurality of objects in the image, the effect adjustment is performed independently for each object area in the depth map. Specifically, each object area is specified in the depth map using a plurality of masks representing each object area in the image. Then, effect adjustment is performed individually for each identified object region, and a plurality of depth maps adjusted for effect are obtained. Then, one depth map is generated by combining the plurality of depth maps. This depth map is used when generating a 2D image of another viewpoint from a 2D image of the original viewpoint.

  The depth map generator 10 automatically generates a depth map of the 2D input image (S10). The generated depth map is input to the depth map processing unit 20. The depth map processing unit 20 also receives a plurality of masks respectively representing a plurality of object regions in the 2D input image. These masks are generated based on the outline of the object area traced by the user. For example, the display control unit 50 displays a 2D input image on the display unit 70, and the user traces the outline of the region to be an object region in the 2D input image using the operation unit 60. The operation receiving unit 40 generates contour information of each object region based on a signal from the operation unit 60 and outputs the contour information to the depth map processing unit 20 as a mask. Note that the mask may be read by the image processing apparatus 100 by reading the outline drawn on the printed matter by the scanner.

  In FIG. 4, the effective area of each mask is drawn in white and the ineffective area is drawn in black. The mask of the person is a pattern in which only the person area is valid and the other areas are invalid. The tree mask is a pattern in which only the tree area is valid and the other areas are invalid. The background mask is a pattern in which only the background is valid and other areas are invalid.

  There is no restriction on the number of masks per screen, and the user can arbitrarily set it. The object area can be set to an area determined by the user to be one object area. For example, as shown in FIG. 4, one object area may be set for one person, or an object area may be set for each part of the person and for each part. In particular, when generating a high-quality 3D image, a large number of object regions can be set for one person, and the position in the thickness and depth direction can be adjusted for each part and for each part. is there.

  The depth map processing unit 20 processes the depth map (hereinafter referred to as input depth map) input from the depth map generation unit 10 using a plurality of masks input via the user interface (S20). The depth map processing unit 20 processes the depth map individually for each area specified by each mask. Hereinafter, this depth map processing for each region is referred to as depth map layer processing. A depth map that has been subjected to layer processing is called a layer depth map. In this specification, a layer is used as a concept indicating a unit of processing for an effective area of a mask.

  In FIG. 4, as an example, the depth map processing unit 20 performs layer processing by specifying a person region from the input depth map using a person mask (layer 1 mask) (S <b> 21 a). Similarly, from the input depth map, a tree region is identified using a tree mask (layer 2 mask) and layer processing is performed (S21b). Similarly, a background region is identified from the input depth map using a background mask (layer 3 mask) and layer processing is performed (S21c).

  The depth map processing unit 20 synthesizes the depth maps of the object areas of the layer depth maps of layers 1 to 3 (S22). This combined depth map is referred to as a combined depth map. The 3D image generation unit 30 shifts the pixels of the 2D input image using the composite depth map, and generates an image having a parallax with respect to the 2D input image (S30). The 3D image generation unit 30 outputs the 2D input image as the right eye image (R) of the 3D output image and the generated image as the left eye image (L).

  First, an example of gain adjustment as layer processing by the depth map processing unit 20 will be described. Gain adjustment is a process of adjusting the thickness of the object in the depth direction. Increasing the gain makes the object thicker and lowering the gain makes the object thinner.

  FIG. 5 is a diagram for explaining the gain adjustment process of the input depth map. The depth map processing unit 20 multiplies the input depth map, which is a depth map before processing, by a gain only in the effective area of the person's mask, and increases the amplitude of the depth value only in the portion of the person in the input depth map. (S21a). In FIG. 5, the amplitude of the person portion of the layer depth map, which is a depth map after processing, is large (see symbol a).

  Next, an example in which offset adjustment is performed as layer processing by the depth map processing unit 20 will be described. Offset adjustment is a process of adjusting the position of the object in the depth direction. When a positive offset value is added, the object moves in the pop-out direction, and when a negative offset value is added, the object moves in the retracting direction.

  FIG. 6 is a diagram for explaining the offset adjustment process of the input depth map. The depth map processing unit 20 adds an offset only to the effective area of the tree mask to the input depth map, which is the depth map before processing, and increases the level of the depth value only for the portion of the tree in the input depth map. (S21b). In FIG. 6, the layer depth map, which is the depth map after processing, has a higher level of the tree portion (see symbol b).

  FIG. 7 is a diagram for explaining a process for synthesizing a layer depth map. The depth map processing unit 20 cuts out only the effective area of the layer 1 mask (person's mask) from the layer 1 layer depth map (person's depth map). Similarly, only the effective area of the layer 2 mask (tree mask) is cut out from the layer 2 layer depth map (tree depth map). Similarly, only the effective area of the layer 3 mask (background mask) is cut out from the layer 3 layer depth map (background depth map). The depth map processing unit 20 generates a combined depth map by combining the three extracted depth maps.

  FIG. 8 is a diagram for explaining an input depth map gain adjustment process without using a mask. If the layer depth map is synthesized by using only the effective area of the mask of each layer depth map, the invalid area of the mask of each layer depth map is not reflected in the synthesized depth map. Therefore, the depth map processing unit 20 multiplies the entire input depth map by the gain to increase the amplitude of the depth value of the entire input depth map (S21a). In FIG. 8, the entire amplitude of the layer depth map is large (see symbol c).

  FIG. 9 is a diagram for explaining an offset adjustment process of an input depth map without using a mask. If the layer depth map is synthesized by using only the effective area of the mask of each layer depth map, the invalid area of the mask of each layer depth map is not reflected in the synthesized depth map. Therefore, the depth map processing unit 20 adds an offset to the entire input depth map to increase the level of the depth value of the entire input depth map (S21b). In FIG. 9, the overall level of the layer depth map is high (see symbol d).

  FIG. 10 is a diagram for explaining pixel shift and pixel interpolation. The 3D image generation unit 30 shifts the pixels of the 2D input image based on the combined depth map, and generates an image having a parallax with respect to the 2D input image (S30). FIG. 10 shows an example in which the pixel of the person area in the 2D input image is shifted to the left. In the depth map of FIG. 10, the offset value is added to the depth value of the person area, and the depth value of the person area is increased. When the depth value of the person area increases, the pop-out amount of the person area of the 3D image increases.

  If only the person area is pixel-shifted without shifting the background area around the person area, a missing pixel area in which no pixel exists is generated (see reference symbol e of the pixel-shifted image before correction). The 3D image generation unit 30 corrects the missing pixel region by interpolating the missing pixel region with pixels generated from the surrounding pixels. There are various methods for pixel interpolation. For example, interpolation is performed using pixels at the boundary of the person area (see reference numeral f of the pixel shifted image after correction).

  FIG. 11 is a diagram for explaining pixel shift and pixel interpolation when the depth difference of the object boundary is large. When the depth difference at the object boundary is large, the pixel shift amount is also large, so that the area of the missing pixel region is also large. When this missing pixel area is interpolated with pixels generated from surrounding pixels, the area of the area to be interpolated also increases, so that unnaturalness, inconsistency, and incompatibility of the image at the object boundary become conspicuous.

  In FIG. 11, the pixel shift amount of the person area is larger than that in FIG. The area of the missing pixel region of the pixel shifted image before correction in FIG. 11 is larger than that in FIG. 10 (see symbol g). In the pixel shift image after correction in FIG. 11 as well, the pixels at the boundary of the person area are interpolated in the missing pixel area, as in the pixel shift image after correction in FIG. The missing pixel area is an area that should originally be a background, and when the area becomes larger, the shape of the person collapses (see symbol h).

  Some objects in the image have not only clear boundaries but also unclear edges. The blurring of the object boundary occurs due to, for example, blurring at the time of shooting, camera shake, or motion blur. When the object boundary is unclear, it is difficult to create an appropriate mask that matches the object boundary. When depth map processing, pixel shift, and pixel interpolation are performed using a mask created with an inaccurate contour, the contour of the object of the generated 3D image tends to be unnatural.

  FIG. 12 is a diagram for explaining pixel shift and pixel interpolation in which unnaturalness does not occur even when the depth difference of the object boundary is large. Even when the area of the missing pixel region at the object boundary is large, the unnaturalness of the object boundary can be made inconspicuous by performing interpolation so that the person and the background are switched smoothly.

  The area of the missing pixel region of the pixel shifted image before correction in FIG. 12 is also larger than that in FIG. 10 as in FIG. 11 (see symbol i). In the pixel-shifted image after correction shown in FIG. 12, unlike the pixel-shifted image after correction shown in FIG. 11, unnaturalness at the boundary between the person and the background is less noticeable (see symbol j).

  FIG. 13 is a diagram showing the configuration of the image editing system 500 according to the first embodiment of the present invention. In the image editing system 500 according to the first embodiment, a mask correction unit 80 is added to the image processing apparatus 100 of the image editing system 500 according to the basic embodiment of FIG. Hereinafter, differences between the image editing system 500 according to the first embodiment shown in FIG. 13 and the image editing system 500 according to the basic embodiment shown in FIG. 2 will be described.

  The mask correction unit 80 corrects the mask set from the console terminal device 200 via the operation reception unit 40 and outputs the corrected mask to the depth map processing unit 20. Specifically, the mask correction unit 80 performs a blurring process on the object boundary of the mask. The depth map processing unit 20 alpha-blends each depth map of a plurality of object regions generated based on the mask corrected by the mask correction unit 80. That is, the depth map processing unit 20 combines a plurality of layer depth maps according to the coefficient (α value) defined by each mask.

  FIG. 14 is a diagram for explaining the overall processing process of the image editing system 500 according to the first embodiment of the present invention. The processing process of FIG. 14 has a configuration in which a mask blurring process is added to the processing process of FIG. Hereinafter, differences will be described. In the following description, the pixel value in the effective area of the mask (drawn in white in the figure) is 1, and the pixel value in the invalid area (drawn in black in the drawing) is 0.

  In the first embodiment, the layer 1 mask (person mask), the layer 2 mask (tree mask), and the layer 3 mask (background mask) output from the operation reception unit 40 are input to the depth map processing unit 20. It is input to the mask correction unit 80 before being processed. The mask correction unit 80 performs blurring processing on the object boundary portion of each mask (S15a to S15c). Specifically, the mask correction unit 80 sets the values of the edge of the mask signal (that is, the boundary between 0 and 1) and its neighboring area (hereinafter collectively referred to as edge area) to a value between 0 and 1. (Excluding 0 and 1) (see symbol k).

  The depth map processing unit 20 synthesizes a layer depth map according to the corrected mask level (S22). As a result, a combined depth map in which the depth value changes gently can be generated in the edge region of the mask. The 3D image generation unit 30 performs pixel shift and pixel interpolation using the composite depth map. As a result, the object boundary changes gently, and the unnaturalness of the object boundary becomes inconspicuous.

  FIG. 15 is a diagram for explaining alpha blending of the layer depth map. The depth map processing unit 20 determines a blend ratio of depth values in each layer depth map based on the value of each mask signal corrected by the mask correction unit 80. Each layer depth map is overlaid in the specified order. The order of superimposition is input from the operation unit 60 by the user and is set in the depth map processing unit 20 via the operation reception unit 40. In the example of this specification, the background layer depth map, the tree layer depth map, and the person depth map are overlaid in this order.

  In FIG. 15, the edge region of the mask signal of layer n (n is a natural number) is composed of a vertical rise from 0% to 50% and a gentle slope from 50% to 100%. Each percentage value indicates the composition ratio. Using this layer n mask signal, a layer n depth signal (hereinafter referred to as layer n depth) and a layer Σ (n−1) combined depth signal (hereinafter referred to as layer Σ (n−1) depth) are used. Blend. The layer Σ (n−1) depth is a combined depth signal obtained by blending the depth signals from the layer 1 to the layer (n−1). That is, it is a combined depth signal generated up to the time point when the layer n depth is blended.

  In a period in which the composition ratio of the mask signal of layer n is 0% (see reference numeral 1), layer n depth is blended at 0% and layer Σ (n-1) depth is blended at 100%. That is, the layer Σ (n−1) depth is not overwritten at all by the layer n depth, and the layer Σ (n−1) depth is output as it is (see symbol o). At the time when the composition ratio of the mask signal of layer n is 50% (see symbol p), the layer n depth is blended at a ratio of 50% and the layer Σ (n−1) depth is blended at a ratio of 50%.

  In a period in which the composition ratio of the mask signal of layer n is 50% to 100% (see symbol m), the layer n depth and the layer Σ (n−1) depth are blended while the composition ratio changes. For example, when the composition ratio of the mask signal of layer n is 75% (see symbol q), the layer n depth is blended at a ratio of 75% and the layer Σ (n−1) depth is blended at a ratio of 25%. From the point in time when the composition ratio of the mask signal of layer n reaches 100% (see symbol r), the layer n depth is blended at a ratio of 100% and the layer Σ (n−1) depth is blended at a ratio of 0%. During the period when the composition ratio of the mask signal of layer n is 100% (see symbol n), the layer Σ (n−1) depth is completely overwritten by the layer n depth, and as a result, the layer n depth is output as it is (symbol s). reference). Thereby, the layer Σn depth is generated. The layer Σn depth is a combined depth signal obtained by blending the depth signals from layer 1 to layer n.

  Note that the layer Σ (n−1) depth mask signal is not used during layer depth blending. The combination ratio of the layer Σn depth and the layer Σ (n−1) depth is determined only by the layer Σn depth mask signal. Therefore, in a region where the layer depth overlaps frequently, the depth value of the lower layer depth becomes thinner as the layer depth is overlapped.

  Hereinafter, the process of blurring the edge of the mask signal will be described in detail. This blurring process consists of the following three basic steps. First, the edge of the original mask signal is moved to an appropriate position with respect to the contour of the target object. Thereby, the area of the object region of the mask is enlarged or reduced. Next, the edge of the mask is inclined. Limit the edge area of the mask with the last slope.

  Hereinafter, a method of performing the above blurring process using a low-pass filter will be described. First, a low-pass filter is applied to a mask in which the level of the black part is 0 and the level of the white part is 1. As a result, it is possible to generate a mask in which a slope having an intermediate level between 0 and 1 is added to the edge region without changing the level of the flat portion.

  Further, in some cases, it is desired to incline only the outer side of the effective area of the mask in the edge area of the mask and to maintain the level before processing on the inner side. On the contrary, in some cases, it is desired to incline only the inside of the mask edge area with respect to the effective area of the mask and to maintain the level before processing on the outside. In consideration of these cases, processing for arbitrarily moving the position of the edge of the mask signal by enlarging or reducing the effective area of the mask is inserted before the low-pass filter for generating the blur mask.

  Furthermore, in order to prevent the edge of the mask from expanding more than the intended amount due to the blurring process, a process for limiting the blurred edge region is inserted after the low-pass filter for blur mask generation. Hereinafter, the blurring process using the low-pass filter will be described more specifically.

  FIG. 16 is a diagram illustrating a configuration example of the mask correction unit 80. The mask correction unit 80 includes a first low-pass filter 81, a binarization unit 82, a second low-pass filter 83, and a clipping unit 84. These configurations can also be realized in various forms by hardware only, software only, or a combination thereof.

  The first low-pass filter 81 in the first stage applies a low-pass filter to the original mask signal. The binarization unit 82 binarizes the mask signal output from the first low-pass filter 81 and having an inclined edge using the first threshold value. By the action of the first low-pass filter 81 and the binarization unit 82, the position of the edge of the original mask signal is moved.

  The second low-pass filter 83 at the second stage applies a low-pass filter to the mask signal output from the binarization unit 82 whose edge position has been moved. As a result, the edge of the mask signal is inclined. The clipping unit 84 uses the second threshold value for the mask signal output from the second low-pass filter 83 and having an inclined edge to clip a signal equal to or lower than the second threshold value to zero.

  FIG. 17 is a diagram for explaining mask blurring processing by the mask correction unit 80 of FIG. FIG. 17 shows a processing flow on the left, a mask signal waveform in the center, and a mask image on the right. In the description of FIG. 17, it is assumed that each pixel value of the mask is defined by 8 bits, and black is 0 (0b00000000) and white is 255 (0b11111111) in the mask image. Usually, the original mask signal is a binary signal having only 0 or 255.

 The original mask signal is input to the first low-pass filter 81. The first low-pass filter 81 applies a low-pass filter to the original mask signal as pre-processing for changing the position of the edge of the mask signal (S81). Specifically, the first low-pass filter 81 processes the original mask signal into a mask signal having an inclined edge region. The value of the inclined portion is processed to a value between 0 and 255 (excluding 0 and 255).

  Next, as a preprocessing for changing the position of the edge of the mask signal, the binarization unit 82 compares the value of the processed mask signal with the first threshold value and binarizes the mask signal (S82). Specifically, when the value of the mask signal is larger than the first threshold, the value of the mask signal is set to 255, and when it is smaller, it is set to 0. As a result, the mask signal again becomes a binary signal having only 0 or 255.

  When the first threshold value is set to be smaller than 127, which is an intermediate value between 0 and 255, the edge of the binarized mask signal is moved outside the edge of the original mask signal. In this case, the white area is enlarged in the mask image. On the other hand, when the first threshold value is set to be larger than the intermediate value 127, the edge of the binarized mask signal is moved inward from the edge of the original mask signal. In this case, the white area is reduced in the mask image. When the first threshold is set to 127, which is an intermediate value, the edge of the binarized mask signal is the same as the position of the edge in the original mask signal.

  FIG. 17 shows an example in which the edge position of the mask signal is moved outward. By varying the filter characteristics such as the number of taps and coefficients of the first low-pass filter 81 and the first threshold value of the binarization unit 82, the position of the edge of the mask signal can be arbitrarily adjusted.

  FIG. 18 is a diagram for explaining the relationship between the slope formed in the mask signal by the first low-pass filter 81 and the first threshold value set in the binarization unit 82. When generating mask signals at the same edge position, the first threshold value is high when the slope is slow and long, and the first threshold value is low when the slope is short and steep. When the user inputs the filter characteristics of the first low-pass filter 81 and the first threshold value of the binarization unit 82 from the operation unit 60, the first low-pass filter 81 and the binarization unit 82 are set via the operation reception unit 40. Is done. The user can arbitrarily adjust the position of the edge of the mask signal by adjusting at least one of the filter characteristic of the first low-pass filter 81 and the first threshold value of the binarization unit 82 from the operation unit 60. Since the slope can be shortened by setting the first threshold value low, the number of taps of the first low-pass filter 81 can be reduced, and the first low-pass filter 81 can be simplified.

  Returning to FIG. The mask signal whose edge position has been moved is input to the second low-pass filter 83. The second low-pass filter 83 applies a low-pass filter to the mask signal whose edge position has been moved (S83). As a result, a blur mask in which an inclination is added to the edge region again is generated.

  Next, the clipping unit 84 compares the value of the mask signal having an inclination in the edge region generated by the second low-pass filter 83 with the second threshold, and when the value of the mask signal is equal to or smaller than the second threshold, The value is set to 0 (S84). That is, in the edge region, the white side slope is left and the black side slope is sharply dropped. As a result, a slope that changes from white to gray is provided in an area greater than the second threshold, and a black mask can be generated in an area that is less than or equal to the second threshold. By this clipping process, it is possible to limit the blurred area in the mask and prevent the edge area of the mask from becoming larger than intended.

  Although FIG. 17 shows an example in which the blurring process is performed in the horizontal direction, the blurring process can be performed in both the horizontal direction and the vertical direction by using a two-dimensional low-pass filter. At that time, filters having different coefficients in the horizontal direction and the vertical direction may be used. In that case, the position of the edge of the mask signal, the degree of inclination, and the blur width can be individually adjusted in the horizontal direction and the vertical direction.

  Alternatively, an elliptical two-dimensional low-pass filter having different coefficients in the horizontal direction and the vertical direction and having an intermediate value between the vertical and horizontal coefficients in the oblique direction may be used. If an elliptical two-dimensional low-pass filter is used, the edge position of the mask signal, the degree of inclination, and the blur width can be individually adjusted in the horizontal direction and the vertical direction, and the adjustment can be applied in an oblique direction. For example, a square original mask can be processed into a rectangular shape with arbitrary length and width and rounded corners. In addition, the square original mask has an arbitrary gentle slope in all directions, with the degree of vertical and horizontal inclination being individual and arbitrary, and with the angle gradually changing with the degree of horizontal and vertical inclination and continuity. It can also be processed into a rectangle.

  FIG. 19 is a diagram for comparing the slope added by the first low-pass filter 81 with the slope added by the second low-pass filter 83. The slope added by the first low-pass filter 81 (see symbol t) is a temporary slope used to adjust the binarization boundary position and disappears after binarization. Therefore, the inclination may be an inclination that changes linearly at a constant angle. Since the inclination (see symbol t) added by the second low-pass filter 83 is the remaining inclination, the user may adjust the shape of the inclination. For example, the upper part of the slope may be adjusted to be convex, and the lower part may be adjusted to be convex. By adjusting to this shape, the clipped width can be increased.

  In this way, the mask signal is adjusted by adjusting the filter characteristics such as the number of taps and coefficients of the first low-pass filter 81 and the second low-pass filter 83, the first threshold value of the binarization unit 82, and the second threshold value of the clipping unit 84. The edge position, the effective area of the mask, and the blur width can be freely adjusted. Further, it is not necessary to apply the same blurring process to the mask signals of all layers, and the blurring process may be performed individually for each mask signal of each layer.

  By the blurring process described above, the edge of the mask signal can be moved to an arbitrary position, and the area of the effective area of the mask can be changed. In addition, the edge region can be given an arbitrary inclination. Further, the blur area of the mask can be arbitrarily limited.

  The blur mask is used in the synthesis of the depth map in the subsequent stage. The depth map processing unit 20 alpha blends a plurality of layer depth maps according to the level of the blur mask. At that time, the composition ratio of each layer depth map is determined according to the level of the blur mask.

  As described above, according to the first embodiment, when layer depth maps are combined, by using a blur mask, it is possible to provide continuity to the object boundary portion of the combined depth map. That is, when there is a large step at the object boundary, the step can be made gentle. Therefore, the object boundary portion of the image of another viewpoint generated based on this composite depth map can be finished to a natural boundary.

  In addition, even when a mask created with an inaccurate contour is used for an object with an ambiguous boundary, by adjusting the position of the contour, the contour of the object of the generated 3D image becomes an unnatural contour. Can be avoided.

  Hereinafter, a method for making the unnaturalness of the object boundary inconspicuous without using a blur mask will be described.

  FIG. 20 is a diagram for explaining the overall processing process of the image editing system 500 according to the second embodiment of the present invention. The processing process of FIG. 20 has a configuration in which low-pass filter processing is added to the processing process of FIG. Hereinafter, differences will be described. Unlike the processing process of FIG. 14, the mask blurring process is not added.

  The depth map processing unit 20 applies a low-pass filter to the combined depth map (S85). Thereby, the change of the depth value at the object boundary in the depth map becomes gentle. However, the change in the depth value cannot be arbitrarily adjusted for each object as compared with the method using the blur mask. Further, in the method using the blur mask, only the change of the depth value at the object boundary can be smoothly processed. On the other hand, in the method of applying a low-pass filter to the composite depth map, fine details (unevenness) are lost not only on the object boundary but also on the flat part inside the object.

  The method of using the blur mask and the process of applying the low pass filter to the composite depth map can be used independently of each other, or both processes can be used simultaneously. Moreover, the order of each process can also be replaced. It is also possible to apply a low-pass filter to the depth map before layer synthesis to generate a blurred layer depth map, synthesize the layer depth map using a blur mask, and then apply a low-pass filter to the composite depth map. .

  Next, Example 3 will be described. In Examples 1 and 2, the following processing was performed on a mask serving as a reference when processing a depth map. The area of the effective area of the mask is varied. Slope the edge of the mask (specifically, create an intermediate value such as 0.5 instead of 0 or 1). The depth map is subjected to α blend synthesis according to the value of the inclination (intermediate value) attached to the mask. A low-pass filter is applied to the synthesized depth map to suppress steep changes.

  In the first and second embodiments, by these processes, the boundary portion of the object of the synthesized depth map has continuity, and the boundary portion of the object of the image (3D image) generated based on this synthesized depth map is unnatural. To make it less noticeable.

  In the methods according to the first and second embodiments, there is a case where correction processing is performed up to a portion where correction is not originally required. The image may change unnaturally due to the side effect of the correction. Further, when the effect of the correction is weakened to avoid or reduce the side effect of the correction, the unnaturalness of the boundary portion of the object may not be completely eliminated.

  In the third embodiment, in consideration of the peculiarity of the mask blurring process, means for making the mask correction process locally asymmetrical is provided. As a result, the correction process can be performed exclusively on the target portion, and the above side effects can be avoided or reduced. That is, the processing of the edge portion of the mask is performed asymmetrically, and the mask having the asymmetrical edge is processed. As a result, the unnaturalness of the boundary portion of the object in the generated 3D image can be made inconspicuous while suppressing the side effects of the correction process.

  In view of the peculiarities of the mask blurring process, first, the reason why the target effect can be obtained by performing the mask processing asymmetrically will be described, and then specific means for performing the mask processing asymmetrically will be described.

  First, the reason why the targeted effect can be obtained by performing the mask processing asymmetrically will be described. As described above, in the present specification, a 3D image having a parallax for each object with respect to the input image is obtained by horizontally shifting some pixels of the input image based on the depth value represented by the depth map. Generate. In general, when an input image is a left-eye image and a right-eye image is generated by pixel shift, the pixel is shifted to the left side in order to give parallax in the pop-out direction. In this case, a pixel missing area is generated on the right side of the shifted object due to pixel shift. On the other hand, the shifted pixel covers the background pixel on the left side of the shifted object. There are no missing pixels on the left side of the object.

  That is, pixel omission due to pixel shift occurs only on one side of the object. The direction in which pixel dropout occurs depends on whether the image to be generated is a right-eye image or a left-eye image, and whether the object is given parallax in the pop-out direction or parallax in the depth direction.

  In the above-described example, when uniform mask edge processing is performed on both the left and right sides of the object, the right side of the object can make the unnaturalness of the boundary portion inconspicuous. On the other hand, when some texture exists in the background portion corresponding to the left side of the object, the pixel shift affected by the mask edge processing described above also reaches the background portion. In this case, the background texture may be distorted. For example, when the background texture includes a white line of a road, the white line may be distorted.

  Therefore, the mask edge processing described above is applied only to the right side of the object and not to the left side. This prevents the background texture on the left side of the object from being distorted.

  Next, specific means for performing mask processing asymmetrically will be described. As described in the first embodiment, the following two filters are used for processing the mask edge. One is a filter used for moving the edge position of the mask and changing the area of the effective area of the mask. The other is a filter for inclining the mask edge in order to control the blend ratio of the depth map according to the mask. The former corresponds to the first low-pass filter 81 in FIG. 16, and the latter corresponds to the second low-pass filter 83.

  These filters are generally composed of left-right or up-down symmetrical coefficients. In the third embodiment, a filter in which a coefficient asymmetric with respect to the center is set is used intentionally. As a result, the mask edge can be processed asymmetrically, and side effects due to the correction processing described above can be avoided or reduced.

  FIG. 21 is a diagram illustrating a configuration example of the mask correction unit 80 according to the third embodiment. The mask correction unit 80 according to the third embodiment has a configuration in which a filter shape setting unit 85 is added to the mask correction unit 80 of FIG. Hereinafter, differences from the mask correction unit 80 of FIG. 16 will be described.

  The mask correction unit 80 according to the third embodiment performs a blurring process on the object boundary portion of the mask using the second low-pass filter 83 that is asymmetric at least on the left and right. The filter shape setting unit 85 determines the filter shape of the second low-pass filter 83. The user can set information for specifying the filter shape of the second low-pass filter 83 in the filter shape setting unit 85 from the operation unit 60. The user can set the second low-pass filter 83 having a left-right asymmetric filter shape by setting the number of taps and / or the coefficient value of the second low-pass filter 83 to be left-right asymmetric.

  Further, as described above, since the blurring process is applied not only in the horizontal direction but also in the vertical direction, a two-dimensional low-pass filter can be used. In this case, the user can set the second low-pass filter 83 having an asymmetric filter shape not only on the left and right but also on the top and bottom. If an elliptical two-dimensional low-pass filter is used as described above, natural blurring processing can be performed in an oblique direction.

  In this way, the user can set the second low-pass filter 83 having individual coefficients in the horizontal direction, the vertical direction, and the oblique direction and having coefficients that are asymmetric with respect to the center. That is, it is possible to set the second low-pass filter 83 having an asymmetric shape in all the upper, lower, left, and right oblique directions. In this way, the user can arbitrarily apply the effect of the blurring process to any portion of the target object that is vertically or horizontally oblique.

  Hereinafter, the effect when the second low-pass filter 83 is asymmetrical will be considered. In this consideration, a one-dimensional low-pass filter that performs a blurring process in the horizontal direction is assumed for easy understanding.

  FIGS. 22A to 22C are diagrams for explaining mask edge processing using the second symmetrical low-pass filter 83s. FIGS. 23A to 23C are diagrams for explaining mask edge processing using the second asymmetric low-pass filter 83a. FIG. 22A shows an example of a second low-pass filter 83s that is symmetric. FIG. 23A shows an example of a second asymmetric low-pass filter 83a.

  FIG. 22B shows a process of filtering the mask M1 (dotted line) using the symmetrical second low-pass filter 83s shown in FIG. In the mask M2 (solid line) after filtering, the left and right edges are equally gentle. FIG. 23B shows a process of filtering the mask M4 (dotted line) using the second asymmetric second low-pass filter 83a shown in FIG. The filtered mask M5 (solid line) has different edge shapes on the left and right. The high level area of the left edge is gently rounded. The low level region on the right edge gently spreads outward.

  As shown in FIG. 17, the mask after filtering by the second low-pass filter 83 is subjected to clipping processing using the second threshold. In the clipping process, the mask value at a level below the second threshold is set to zero. The second threshold is set to a level near the middle of the mask level.

  FIG. 22C shows a process of clipping the filtered mask (thin solid line) in FIG. 22B with the second threshold value. In the mask M3 (thick solid line) after clipping, the low level regions of the left and right edges are vertical. FIG. 23C illustrates a process of clipping the filtered mask (thin solid line) in FIG. 23B with the second threshold value. Also in the mask M6 after clipping, the low level regions of the left and right edges are vertical.

  When the mask M3 and the mask M6 are compared, the former has equal inclinations on the left and right edges. The latter has different left and right edge slopes. As shown in FIG. 23A, when the second low-pass filter 83a having a coefficient only on the left side with respect to the center is used, the left edge is inclined, but the right edge is hardly inclined. The right edge maintains a sharp edge substantially similar to the state before processing.

  As shown in FIG. 23B, the low-level region of the right edge of the mask M5 filtered by the asymmetric second low-pass filter 83a has a gentle slope. Thereafter, as shown in FIG. 23C, the mask M5 is compared with the second threshold value, and the mask value at a level lower than the second threshold value is clipped to zero. As a result, the slope of the low level region at the right edge of the mask M6 after clipping is removed.

  If the above-described clipping process is not performed, the mask edge shape may be left-right asymmetric, but the slope of the low-level region at the mask level remains. The purpose of the mask edge processing according to the third embodiment cannot be achieved. That is, the effect of preventing the mask edge processing process from affecting the background portion with the texture by limiting the range of the pixel depth when generating the 3D image, and the asymmetric blending of the layer depth.

  On the other hand, by adding the above-described clipping processing, it is possible to perform processing in which only the edge on one side of the mask is inclined and the other edge is not inclined. Therefore, the clipping processing according to the third embodiment has an advantageous effect that cannot be easily inferred from simple processing that makes the shape of the mask edge asymmetric.

  As described above, the user can arbitrarily set the filter shape of the second low-pass filter 83. Therefore, it is possible to perform a blurring process with a bias in an arbitrary direction rather than in all directions. For example, the bias can be adjusted in accordance with the state of the texture around the object. In the following description, processing for setting the filter shape of the second low-pass filter 83 automatically instead of manually will be described. Thereby, a user's burden can be reduced.

  Returning to FIG. The filter shape setting unit 85 determines whether the image to be generated is a right-eye image or a left-eye image, and an object obtained from a comparison result of the inner depth value and the outer depth value in the object boundary portion of the mask. The filter shape of the second low-pass filter 83 is set in accordance with the anteroposterior relationship with the surroundings.

  When the image to be generated is a left-eye image and the object is positioned in front of the surroundings (that is, in the protruding direction), the 3D image generation unit 30 shifts the object to the right by a pixel. When the image to be generated is a left-eye image and the object is located deeper than the surroundings (that is, in the depth direction), the 3D image generation unit 30 shifts the object to the left by a pixel. When the image to be generated is a right-eye image and the object is positioned in front of the surroundings (that is, the pop-out direction), the 3D image generation unit 30 shifts the object to the left by a pixel. When the image to be generated is a right-eye image and the object is located deeper than the surroundings (that is, in the depth direction), the 3D image generation unit 30 shifts the object to the right by a pixel.

  When the object is pixel-shifted to the right side, the filter shape setting unit 85 sets the left edge of the second low-pass filter 83 to a more gentle filter shape than the right edge. Set the filter shape so that the right edge is not inclined or has a very small inclination. When the object is pixel-shifted to the left side, the filter shape setting unit 85 sets the right edge of the second low-pass filter 83 to a gentler filter shape than the right edge.

  More specific description will be given below. In the image editing system 500 in this specification, an original input image is assigned to one of a right eye image and a left eye image constituting a 3D image, and an image generated by pixel shift is assigned to the other. This assignment is determined by user settings. In the filter shape setting unit 85, the determined assignment is set.

  Next, it is determined whether the object pointed to by the mask exists in a direction that protrudes from the periphery of the object, or exists in the depth direction. When it exists in the pop-out direction, it is necessary to increase the distance between the object in the right eye image and the object in the left eye image. On the other hand, when it exists in the depth direction, it is necessary to shorten the distance. A depth map is used to determine whether an object exists in a direction that protrudes from the surroundings or in a depth direction.

  The filter shape setting unit 85 analyzes the depth map to obtain a relative difference between the depth value of the area of the object indicated by the mask and the surrounding depth values. For example, the difference between the average value of the depth values in the area of the object and the average value of the depth values in the set range around the object is obtained.

  In this specification, the closer to white, the higher the depth value, and the closer to black, the lower the depth value. Therefore, when the depth value of the area of the object is larger than the depth value of the surrounding area, it can be determined that the object is closer to the observer than the surrounding area. On the contrary, when the depth value of the area of the object is smaller than the depth value of the surrounding area, it can be determined that the object is farther from the observer than the surrounding area.

  Consider the case where the depth value of the area of the object pointed to by the mask is larger than the surrounding depth values, that is, when it is determined that the object is located in the pop-out direction. Under this assumption, when an image generated by pixel shift is assigned to the right eye image, the pixel shift direction can be determined to be left. In this case, a pixel omission area occurs on the right side of the object due to the pixel shift. Therefore, a correction process is desired in which the right side of the object is processed to be wider and more inclined, and the left side of the object is not processed. In order to implement such correction processing, the filter shape setting unit 85 sets the second low-pass filter 83 having a filter shape with a gentler right edge.

  As described above, the filter shape setting unit 85 determines whether the image generated by the pixel shift is the left eye image that is the right eye image, and the second parameter based on the two parameters of the relative difference between the object area and the surrounding depth. The direction in which the filter shape of the low-pass filter 83 should be biased is determined.

  FIG. 24 is a flowchart for explaining filter shape determination processing by the filter shape setting unit 85 according to the third embodiment. First, the filter shape setting unit 85 determines whether the generated image is a left eye image or a right eye image (S10). Next, it is determined whether the object protrudes from the periphery or is recessed (S20, S22).

  When the image is a left-eye image (the left eye of S10) and the object is deeper than the periphery (the depth of S20), the filter shape setting unit 85 determines that the pixel shift direction is left, and the second low-pass filter 83 The filter shape is set to a filter shape in which an inclination is added to the right edge (S31). When the image is a left-eye image (the left eye of S10) and the object protrudes from the periphery (the protrusion of S20), the filter shape setting unit 85 determines that the pixel shift direction is right, and the filter of the second low-pass filter 83 The shape is set to a filter shape with a slope added to the left edge (S32).

  When the image is a right eye image (the right eye of S10) and the object protrudes from the periphery (the protrusion of S22), the filter shape setting unit 85 determines that the pixel shift direction is left, and the filter of the second low-pass filter 83 The shape is set to a filter shape in which an inclination is added to the right edge (S31). When the image is a right eye image (the right eye of S10) and the object is deeper than the periphery (the depth of S22), the filter shape setting unit 85 determines that the pixel shift direction is right, and the second low-pass filter 83 The filter shape is set to a filter shape in which an inclination is added to the left edge (S32).

  Further, the filter shape setting unit 85 determines the degree of inclination at the edge on the side that should be made gentler (that is, the side on which the inclination should be added) according to the difference between the depth value inside the object boundary and the depth value outside. Can be determined. The larger the difference, the larger the step of the object boundary, and the larger the pixel shift amount. The filter shape setting unit 85 increases the degree of inclination as the difference increases. That is, as the difference is larger, the slope to be added to the edge is set more gently.

  The example in which the second low-pass filter 83 is left-right asymmetric has been described so far. In the third embodiment, not only the second low-pass filter 83 but also the first low-pass filter 81 can be asymmetrical.

  FIGS. 25A to 25C are diagrams for explaining the mask edge processing using the first low-pass symmetric first low-pass filter 81s. FIGS. 26A to 26C are views for explaining mask edge processing using the first asymmetric low-pass filter 81a. FIG. 25A shows an example of a first symmetrical low-pass filter 81s. FIG. 26A shows an example of a first low-pass filter 81a that is asymmetrical.

  FIG. 25B shows a process of filtering the mask M7 (dotted line) using the symmetrical first low-pass filter 81s shown in FIG. In the mask M8 after filtering (solid line), the left and right edges are equally gentle. FIG. 26B shows a process of filtering the mask M10 (dotted line) using the left-right asymmetric first low-pass filter 81a shown in FIG. The filtered mask M11 (solid line) has different edge shapes on the left and right. The high level area on the left edge is gently rounded. The low level region on the right edge gently spreads outward.

  FIG. 25C shows a processing for widening the edge width of the mask by binarizing the filtered mask (thin solid line) in FIG. 25B with the first threshold (see FIG. 17). When widening the edge width, the first threshold is set near zero. FIG. 26C shows a processing for widening the edge width of the mask by binarizing the filtered mask (thin solid line) in FIG. 26B with the first threshold value.

  As shown in FIG. 25C, the processed mask M9 is spread evenly on the left and right. On the other hand, as shown in FIG. 26 (c), the processed mask M12 differs in the amount of movement of the left and right edges. When the first low-pass filter 81a having a coefficient only on the left side with respect to the center is used, the right edge position moves to the right as shown in FIG. 26C, but the left edge position remains at the original position. is there. In this way, the left and right sides of the mask can be spread unevenly.

  When the edge of the mask is inclined, the boundary portion of the object becomes wider. In this case, if the edge position of the mask is left as it is, the object boundary portion erodes inside the object. Therefore, when the edge is inclined, the edge position is usually moved outward. The greater the inclination, the more the edge position is moved outward. Thus, the degree of inclination and the amount of movement of the edge position are in a proportional relationship.

  As shown in FIGS. 23 (a)-(c) and FIGS. 26 (a)-(c), in a filter-shaped low-pass filter having a coefficient on the left side, the left edge is inclined and the right edge position is outside. Move to. Therefore, the first low-pass filter 81 and the second low-pass filter 83 need to be set to opposite left and right filter shapes. In order to blur the left side of the object, the second low-pass filter 83 having a coefficient on the left side is set, and the first low-pass filter 81 having a coefficient on the right side is set. On the contrary, when it is desired to blur the right side of the object, the second low-pass filter 83 having a coefficient on the right side is set, and the first low-pass filter 81 having a coefficient on the left side is set.

  The example shown in FIGS. 26A to 26C is an example in which the edge position of the mask is moved outward (that is, the width of the mask is widened) using the first low-pass filter 81. When increasing the width of the mask, a filter-shaped first low-pass filter 81 having a coefficient on the side opposite to the edge to be moved outward is used. On the other hand, when the edge position of the mask is moved inward (that is, the width of the mask is narrowed), a filter-shaped first low-pass filter 81 having a coefficient on the same side as the edge to be moved inward is used.

  FIG. 26A shows a filter-shaped first low-pass filter 81a having a coefficient only on the left side. FIG. 26B shows the mask M11 after being filtered by the first low-pass filter 81a. On the left side of the mask M11, the inner side is inclined, and on the right side, the outer side is inclined.

  When the width of the original mask M10 is increased, the first threshold level is set low. When the mask M11 after filtering is binarized with the first threshold value, the right edge having an inclination on the outside of the original mask M10 is moved to the outside, and the width of the original mask M10 extends to the right (FIG. 26C). reference). The left side does not change. Conversely, when the width of the original mask M10 is narrowed, the first threshold level is set high. When the filtered mask M11 is binarized with the first threshold value, the left edge having an inclination inside the original mask M10 is moved inward, and the left side of the original mask M10 is contracted. The right side does not change. When the width of the mask is increased in this way, the side opposite to the side having the coefficient of the first low-pass filter 81 is expanded, and when the width of the mask is decreased, the same side as the side having the coefficient of the first low-pass filter 81 is decreased.

  In the description so far, the method of uniformly processing the entire mask using one filter defined by a unique coefficient when performing edge processing has been shown. In this respect, in order to obtain the effect of performing the asymmetrical processing of the mask edge described so far, the processing is not necessarily limited to processing using one filter. A plurality of filters each having a different filter shape may be used to individually perform filtering for each mask edge area. That is, a filter having a wide shape is used in an area where it is desired to process the mask widely. On the other hand, a filter having a narrow shape is used for an area where it is desired to reduce the degree of mask processing, or no filtering is performed. In this way, a plurality of types of low-pass filters may be switched for each mask edge area.

  As described above, according to the third embodiment, the mask edge can be processed asymmetrically in all the diagonal directions of the object. By correcting the boundary portion of the object using such a mask, it is possible to exclusively correct only the portion where distortion has occurred without affecting the normal background texture. Therefore, a high-quality 3D image can be generated.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

  In the third embodiment, the example in which the filter shape of the first low-pass filter 81 is set to be left-right asymmetric in order to move the edge position of the mask left-right asymmetrically has been described. In this respect, the mask edge position can be moved asymmetrically by setting the first threshold value to be different between the right edge portion and the left edge portion while using the first symmetrical low-pass filter 81. Can do. When moving the edge position of the mask to the outside, the first threshold value is set low, and when moving to the inside, it is set high.

  500 image editing system, 100 image processing apparatus, 10 depth map generation unit, 11 upper screen high frequency component evaluation unit, 12 lower screen high frequency component evaluation unit, 13 synthesis ratio determination unit, 14 frame memory for first basic depth model, 15 Frame memory for the second basic depth model, 16 Frame memory for the third basic depth model, 17 Compositing unit, 18 Adder unit, 20 Depth map processing unit, 30 3D image generation unit, 31 Pixel shift unit, 32 Monitoring region setting unit 33 pixel interpolation unit, 40 operation reception unit, 50 display control unit, 200 console terminal device, 60 operation unit, 70 display unit, 80 mask correction unit, 81 first low pass filter, 82 binarization unit, 83 second low pass Filter, 84 Clipping section, 85 Filter configuration Fixed part.

Claims (8)

A mask correction unit for correcting an externally set mask pattern;
A depth map processing unit that processes a depth map of an input image for each of a plurality of regions specified by a plurality of mask patterns corrected by the mask correction unit;
An image generator that shifts pixels of the input image based on the input image and the depth map processed by the depth map processor, and generates an image of another viewpoint having a parallax with respect to the input image. And comprising
The mask correction unit obtains whether the image of the different viewpoint is a right-eye image or a left-eye image, and a comparison result between an inner depth value and an outer depth value in an object boundary portion of the mask pattern. depending on the context the object and the surroundings to be, comprise filter shape setting unit for setting the filter shape of the low-pass filter, the Kio object boundary before using at least the right and left unsymmetrical said low pass filter, facilities blurring processing And
The filter shape setting unit sets an edge of the other side of the object to a gentler filter shape than an edge of the one side when the object is pixel-shifted to the left and right sides. Processing equipment. A mask correction unit for correcting an externally set mask pattern;
A depth map processing unit that processes a depth map of an input image for each of a plurality of regions specified by a plurality of mask patterns corrected by the mask correction unit;
An image generator that shifts pixels of the input image based on the input image and the depth map processed by the depth map processor, and generates an image of another viewpoint having a parallax with respect to the input image. And comprising
The mask correction unit obtains whether the image of the different viewpoint is a right-eye image or a left-eye image, and a comparison result between an inner depth value and an outer depth value in an object boundary portion of the mask pattern. Including a filter shape setting unit that sets a filter shape of a low-pass filter according to the front-rear relationship between the object and the surroundings, and performing blurring processing on the object boundary using the low-pass filter that is asymmetrical at least to the left and right,
The filter shape setting unit
When the image of the different viewpoint is a left eye image and the object is positioned in front of the surroundings, the left edge of the low-pass filter is set to a gentler filter shape than the right edge,
When the image of the other viewpoint is a left eye image and the object is located behind the surroundings, the right edge of the low-pass filter is set to a gentler filter shape than the left edge,
When the image of the different viewpoint is a right eye image and the object is positioned in front of the surroundings, the right edge of the low-pass filter is set to a gentler filter shape than the left edge,
When the image of the different viewpoint is a right eye image and the object is located behind the surroundings, the left edge of the low-pass filter is set to a gentler filter shape than the right edge,
Images processor you wherein a. The filter shape setting unit determines the degree of inclination at the edge on the side that should be more gentle, according to the difference between the depth value inside the boundary and the depth value outside.
The image processing apparatus according to claim 1 or 2, characterized in that. The mask correction unit
A first low-pass filter for filtering the mask signal;
A binarization unit that binarizes the mask signal output from the first low-pass filter using a first threshold for moving the position of the edge of the mask signal;
A low-pass filter for inclining an edge of the mask signal whose edge position has been moved by binarization by the binarization unit, and a second low-pass filter that is an at least left-right asymmetric low-pass filter;
A clipping unit for clipping a mask signal output by the second low-pass filter to zero using a second threshold value and a signal equal to or lower than the second threshold value;
The image processing apparatus according to any one of claims 1 to 3, characterized in that. The first low-pass filter and the binarization unit move the position of the edge on the side to be inclined in the mask signal to the outside,
The first low-pass filter is a low-pass filter that is asymmetric at least on the left and right, and is set to a filter shape opposite to the left and right of the filter shape of the second low-pass filter.
The image processing apparatus according to claim 4 . The depth map processing unit alpha blends each depth map of the plurality of regions generated based on the mask pattern corrected by the mask correction unit.
The image processing apparatus according to any one of claims 1 to 5, characterized in that. A step of correcting an externally set mask pattern;
Processing the depth map of the input image for each of a plurality of areas specified by the corrected plurality of mask patterns;
The input image, and based on processed depth map, by shifting the pixels of the input image, comprising the steps of: generating an image of another viewpoint with parallax with respect to the input image,
The step of correcting the mask pattern comprises comparing whether the image of the different viewpoint is a right-eye image or a left-eye image, and comparing an inner depth value and an outer depth value of the boundary in the object boundary portion of the mask pattern. depending on the context the object and the surroundings resulting from the results, including the step of setting the filter shape of the low-pass filter, the Kio object boundary before using at least the right and left unsymmetrical said low pass filter, facilities blurring processing And
The step of setting the filter shape is characterized in that, when the object is pixel-shifted to the left and right sides, the other side edge of the object is set to a gentler filter shape than the one side edge. Image processing method. Processing to correct the externally set mask pattern;
Processing the depth map of the input image for each of a plurality of areas specified by a plurality of corrected mask patterns;
Based on the input image and the processed depth map, shift the pixels of the input image to generate an image of another viewpoint having a parallax with respect to the input image ,
The process of correcting the mask pattern is performed by comparing whether the image of the different viewpoint is a right-eye image or a left-eye image, and comparing an inner depth value and an outer depth value of the boundary in the object boundary portion of the mask pattern. depending on the context the object and the surroundings resulting from the results, including the process of setting the filter shape of the low-pass filter, the Kio object boundary before using at least the right and left unsymmetrical said low pass filter, facilities blurring processing And
The process of setting the filter shape is characterized in that, when the object is pixel-shifted to the left and right sides, the other edge of the object is set to a gentler filter shape than the one edge. Image processing program.