KR20190054552A - Method for Determining Causes of Hole Occurrence in Arbitrary View Image Implementation based on Super Multi-View RGB-D Image - Google Patents

Abstract

Provided is a method for determining a hole occurring cause in generating an arbitrary-view image based on a super multi-view RGB-D image. According to an embodiment of the present invention, the method for determining a hole occurring cause comprises the steps of: generating an arbitrary-view image by projecting a multi-view image at an arbitrary view; and determining a type of holes represented in the generated arbitrary-view image. Accordingly, an occurring cause of holes in the arbitrary-view image is determined, thereby applying a hole filling algorithm more suitable and efficient to each occurring cause. Therefore, the accuracy in generating the arbitrary-view image and the speed can be increased.

Description

[0001] The present invention relates to a method for determining a cause of a hole in generating an arbitrary viewpoint image based on an RGB-D image,

More particularly, the present invention relates to a method of generating a random viewpoint image based on a super-multi-view image, and more particularly, to a method of generating a random viewpoint image using a super multi-viewpoint image and a depth map, The method comprising the steps of:

As 3D interactive free-view video is required for realistic AR / VR, it is becoming more important to create a random viewpoint that generates more number of viewpoints accurately with fewer cameras.

Hall filling is one of the most important problems in generating accurate random view images.

In the conventional hole filling method, all the holes generated when an image at a certain time is generated are basically regarded as holes of the same type, and the same hole filling algorithm is applied to all the holes to fill them.

However, the holes of a random viewpoint image generated by using the super high resolution image and the depth map are different from each other, and they appear as holes having patterns of very different characteristics in a random viewpoint image.

It is necessary to improve the accuracy and efficiency of filling the holes by applying the same hole filling algorithm to the holes of the arbitrary view image having different characteristics as described above.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a method for determining a cause of holes generated in a process of generating a random viewpoint image using a super multi-viewpoint image and a depth map, .

According to another aspect of the present invention, there is provided a method of determining a hole generation cause, the method comprising: generating a viewpoint image by projecting a multi-viewpoint image at an arbitrary viewpoint; And discriminating a type of the hole displayed in the generated random viewpoint image.

In the determining step, the hole generated due to the fact that the three-dimensional points to be projected on the arbitrary viewpoint image do not exist in the multi-viewpoint image can be determined as a hole generated outside the angle of view.

In addition, the method for determining a cause of holes according to an embodiment of the present invention may further include filling a hole in a background of a scene in which a hole occurs, with respect to a hole formed outside the angle of view.

In the determining step, the hole generated due to the density of the three-dimensional points projected on the arbitrary viewpoint image is lower than the pixel density of the arbitrary viewpoint image can be discriminated as the resolution hole due to the low density.

Also, in the hole generation cause discrimination method according to the embodiment of the present invention, for a resolution hole due to a low density, a resolution of the arbitrary view image is downsampled until there is no hole, And filling the holes using the image obtained by the sampling.

In the determination step, the hole generated due to the three-dimensional points to be projected on the arbitrary viewpoint image due to the occlusion due to the depth does not exist in the multi-view image can be discriminated as the occlusion hole.

In addition, the method for determining the cause of holes according to an embodiment of the present invention may further include filling holes in the occlusion hole by a depth-based inpainting technique.

According to another aspect of the present invention, there is provided a multi-view image generation system including: a generation unit for generating a multi-view image; And an image processor for generating a viewpoint image by projecting the multi-viewpoint image at an arbitrary viewpoint, and discriminating the type of the hole displayed in the generated viewpoint viewpoint image.

As described above, according to the embodiments of the present invention, it is possible to apply a more suitable and efficient hole filling algorithm to each cause of occurrence by determining the cause of occurrence of holes in a viewpoint image, And the speed can be improved.

1 is a diagram illustrating an RGB-D input image at an 18th time point,
Figure 2 illustrates a random view image including various holes (black)
FIG. 3 is a flowchart illustrating a method of determining a cause of hole occurrence in a random viewpoint image generation according to an exemplary embodiment of the present invention. FIG.
4 is a diagram illustrating a hole mask (black means hole)
5 is a view showing the final labeling result, and FIG.
6 is a block diagram of a super multi-view image generation system according to another embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the drawings.

The second multi-view image means images acquired by dozens of RGB cameras existing at various positions in one space, and the arbitrary view image means a virtual RGB image acquired at an arbitrary virtual position in the space.

Therefore, the arbitrary viewpoint generation based on the second multi-viewpoint image is to generate an image at a virtual position in the space using images acquired by dozens of cameras existing in various locations in one space.

The depth map of the super multi - view images is used for composing the multi view image based on the super multi view image. Specifically, the super-multi-viewpoint images as shown in FIG. 1 are converted into points in a three-dimensional space using given depth maps, and the thus obtained three-dimensional points are projected to a desired arbitrary viewpoint position. In the generated random viewpoint image, a hole occurs as shown in FIG. 2 where the three-dimensional points are empty. Hole means pixels not defined in the image.

In one embodiment of the present invention, in filling the holes generated in the process of generating a virtual arbitrary view image based on the super multi-viewpoint image and the depth map, a hole of a viewpoint image is found, We suggest a method of discrimination.

Specifically, a mask is used to generate a random viewpoint image in order to search for a hole in a viewpoint image, and the cause of each hole from the holes thus determined is determined through holes in the vicinity thereof.

Unlike the conventional technique in which the same hole filling algorithm is applied irrespective of the cause of the holes, different hole filling algorithms are applied to fill the holes according to the generation of each hole.

FIG. 3 is a flowchart provided for explaining a method of determining the cause of holes in the generation of a random viewpoint image according to an embodiment of the present invention.

As shown in FIG. 3, first, a hole mask is generated to find whether a pixel of a certain view image is a hole or not (S110). Specifically, when projecting three-dimensional points onto an arbitrary viewpoint image, pixels that are not projected are regarded as holes.

The hole mask has the same resolution as the arbitrary view image, and each pixel has a value of 0 or 255. [ Initially, every pixel in the mask has a value of 0 and a value of 255 is assigned to the pixel where the three-dimensional point is projected.

Thus, as illustrated in FIG. 4, the pixel value of the final mask indicates whether the pixel is a hole or not.

Next, the cause of the holes is identified, and the hole causes are labeled (S120). There are various causes of occurrence of holes in the arbitrary viewpoint image. In the embodiment of the present invention, the types of causes of holes are classified into the following four types.

(1) Holes generated outside the angle of view

If the random viewpoint image requires a scene that does not exist in the super high viewpoint image, it is a hole generated when the arbitrary viewpoint is out of the full angle of view of the super high viewpoint. In other words, it is a hole caused by the fact that the three-dimensional points to be projected at a certain time point are not present anywhere in the secondary multi-viewpoint image. These holes appear on the edge of the arbitrary-view image, either vertically or horizontally over a wide area. For example, the black portions of the edges of FIG. 4 are holes formed outside the angle of view. Therefore, the hole generated by the angle of view is considered as a hole generated outside the angle of view when the entire hole having continuity between the edge holes is larger than a certain portion by analyzing the edge of the hole mask image. The numerical values above a certain level are confirmed through experiments.

(2) resolution hole due to low density

If the density of the projected three-dimensional points is lower than the pixel density of the arbitrary viewpoint image, a hole occurs in the image at a certain viewpoint. That is, a random viewpoint image is a hole that is created by requiring more dense three-dimensional points, which usually occurs when an arbitrary viewpoint is closer to the scene than a second viewpoint. These holes have a pattern in which holes are generated in a lattice shape in a random view image due to their characteristics. For example, the right grid-shaped black portions in FIG. 4 are holes by resolution. Since the size of the hole is small and has continuity in a specific direction, if the size of the hole is less than a certain portion and the directionality is possessed, the hole is regarded as a hole by resolution. First, small patches are extracted from a random viewpoint image and the shapes of holes are checked in each patch. Determine if the holes are in the shape of a small grid with directionality.

(3) Occlusion Hall

If three-dimensional points to be projected at a certain time point are not present in the second multi-view image due to occlusion due to depth, a hole occurs in the image at a certain time point. In other words, it is a hole that occurs because there is no pixel to be projected anywhere in the image of the second multi-viewpoint resembling the hole by the view angle, but this hole is important because it appears in the middle part of the arbitrary viewpoint image closely related to the scene. These holes appear at the boundary between the object and the object in the scene. For example, the spheres of FIG. 4, the triangular pyramids, and the black portions of the triangular-pyramid and background boundaries are occlusion holes. Therefore, the hole caused by the occlusion finds the boundary based on the image and the depth map at a certain point of time, and considers the black portions of the portion corresponding to the boundary line in the hole mask as the hole caused by the occlusion. First, each hole at a certain time point is projected back to each second multi-viewpoint image of the input, and it is determined whether or not the projected hole reaches the image of the second multi-point point and confirmed.

(4) Holes due to other causes

A hole that does not fall into any of the above three categories shall be considered as a hole due to other causes.

Figure 5 shows the final labeling results. In FIG. 5, the holes indicated by blue are holes generated outside the angle of view, the holes indicated by red are resolution holes due to low density, the holes indicated by green are occlusion holes, One hole is a hole due to other causes.

Then, different hole filling algorithms are applied to each hole to fill the hole.

Specifically, when the hole is generated outside the angle of view (S130-YES), the hole is processed as the background of the scene where the hole occurs (S135). The background is detected from the non-hole part in the random viewpoint image, and the hole is filled with pixels corresponding to the background closest to the hole by the angle of view.

In case of a resolution hole due to low density (S140-YES), an ultra-high resolution technique is used (S145). The resolution is down-sampled until there is no hole due to the resolution of the arbitrary-view image. The low-resolution random view image is up-sampled using the super-resolution technique until the original resolution is obtained, and the pixels of the hall portion according to the resolution are taken from the thus obtained image and filled.

If it is an occlusion hole (S150-YES), a depth-based inpainting technique is used (S155). Find the depth of the pixel closest to the occlusion hole to be filled in the arbitrary-view image, find neighboring pixels of similar depth, and add pixel values of these pixels to the distance weight to fill the hole.

On the other hand, in the case of holes due to other causes (S160-YES): the hole is filled with a joint bilateral filter (S165).

6 is a block diagram of a super multi-view image generation system according to another embodiment of the present invention. 6, the super-multi-view image generation system according to the embodiment of the present invention includes a super multi-view image generation unit 210, an image processor 220, an image output unit 230, and a storage unit 240 ).

The super-multi-view image generating unit 210 generates super multi-view images using a plurality of cameras, the image processor 220 generates arbitrary images from the super multi-view images, and the image output unit 230 generates the multi- To a display, an external device, or a network.

The storage unit 240 stores the generated images and provides a storage space necessary for the image processor 220 to generate the arbitrary view image.

It goes without saying that the technical idea of the present invention can also be applied to a computer-readable recording medium having a computer program for performing the functions of the apparatus and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention may be embodied in computer-readable code form recorded on a computer-readable recording medium. The computer-readable recording medium is any data storage device that can be read by a computer and can store data. For example, the computer-readable recording medium may be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, or the like. In addition, the computer readable code or program stored in the computer readable recording medium may be transmitted through a network connected between the computers.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

210: super multi-view image generation unit
220: image processor
230:
240:

Claims (8)

Generating a viewpoint image by projecting a multi-viewpoint image at an arbitrary viewpoint;
And determining a type of a hole displayed in the generated random viewpoint image.
The method according to claim 1,
In the determining step,
Wherein a hole generated due to the fact that three-dimensional points to be projected on the arbitrary viewpoint image do not exist in the multi-viewpoint image is discriminated as a hole which occurs outside the angle of view.
The method of claim 2,
Further comprising the step of filling a hole in the background of the scene where the hole is generated with respect to the hole occurring outside the angle of view.
The method according to claim 1,
In the determining step,
Wherein a hole caused by a density of three-dimensional points projected on the arbitrary viewpoint image is lower than a density of pixels of the arbitrary viewpoint image is discriminated as a resolution hole due to a low density.
The method of claim 4,
Sampling the resolution of the arbitrary view image with respect to the resolution hole due to the low density, until the hole does not exist, and then filling the hole using the image obtained by upsampling the resolution until the resolution becomes the original resolution Wherein the hole is formed by a plurality of holes.
The method according to claim 1,
In the determining step,
Wherein the hole generated when the three-dimensional points to be projected on the arbitrary viewpoint image is not present in the multi-view image due to occlusion due to depth is discriminated as an occlusion hole.
The method of claim 6,
Further comprising the step of filling the hole with the hole based on the depth-based inpainting technique for the occlusion hole.
A generating unit generating a multi-view image; And
And generating an arbitrary viewpoint image by projecting the multi-viewpoint image at an arbitrary viewpoint, and determining a type of the hole displayed in the generated viewpoint viewpoint image.

Images