KR101993961B1 - Hole Filling Method for Arbitrary View Image - Google Patents

Abstract

A method of hole filling of a random viewpoint image is provided. The method of filling a hole includes: generating a viewpoint image by projecting a multi-viewpoint image at an arbitrary viewpoint; Determining a type of a hole displayed in the generated random view image; And filling holes with different techniques according to the discriminated types.
Accordingly, it is possible to improve the accuracy and speed of random viewpoint image generation by dividing the holes of the arbitrary viewpoint image according to the cause and applying a more suitable and efficient hole filling algorithm for each cause.

Description

{Hole Filling Method for Arbitrary View Image}

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a hole filling technique, and more particularly, to a method of filling holes generated in a process of generating a random view image using a super multi-viewpoint image and a depth map.

As the real-time augmented reality requires interactive three-dimensional free-view video, it is becoming more and more important to create a random view generation technique that accurately generates a larger number of viewpoints with a small number of cameras.

Hall filling is one of the most important problems in generating accurate random view images.

In the conventional hole filling method, all the holes generated when an image at a certain time is generated are basically regarded as holes of the same type, and the same hole filling algorithm is applied to all the holes to fill them.

However, the holes of the random viewpoint image generated by using the second multi-viewpoint image and the depth are different from the cause of the occurrence, and they appear as holes having patterns of very different characteristics in a random viewpoint image.

It is necessary to improve the accuracy and efficiency of filling the holes by applying the same hole filling algorithm to the holes of the arbitrary view image having different characteristics as described above.

SUMMARY OF THE INVENTION The present invention has been conceived in order to solve the above-mentioned problems, and it is an object of the present invention to provide a method and an apparatus for classifying a hole of a viewpoint image according to the cause thereof, Method.

According to another aspect of the present invention, there is provided a hole filling method including: generating a viewpoint image by projecting a multi-viewpoint image at an arbitrary viewpoint; Determining a type of a hole displayed in the generated random view image; And filling holes with different techniques according to the discriminated types.

In filling the holes, the hole generated by the angle of view may fill the hole with the first technique.

The first technique may be a technique of filling a hole with the background of the scene where the hole occurs.

The step of filling the holes may fill the holes with the second technique if the density of projected three-dimensional points at any time is a hole generated because the density is lower than the pixel density of the arbitrary view image.

The second technique may be a technique of down sampling a resolution of a viewpoint image until no hole exists and then filling the hole using an image obtained by upsampling the resolution until the resolution of the viewpoint becomes the original resolution.

The step of filling the holes can fill the holes with a third technique if the three-dimensional points to be projected at any time are holes generated by occlusion by depth.

The third technique may be a depth based inpainting technique.

According to another aspect of the present invention, there is provided a multi-view image generation system including: a generation unit generating a multi-view image; And an image processor for generating a viewpoint image by projecting the multi-viewpoint image at an arbitrary viewpoint, determining a type of a hole displayed in the generated viewpoint image, and filling holes with different techniques according to the determined type.

As described above, according to the embodiments of the present invention, it is possible to improve the accuracy of the random view image generation by dividing the holes of the arbitrary viewpoint images according to their causes and applying a more suitable and efficient hole filling algorithm for each cause The speed can be improved.

1 is a diagram showing three-dimensional spatial matching,
FIG. 2 is a diagram showing a hole occurrence in a temporary viewpoint image,
FIG. 3 is a flowchart illustrating a method of filling a random viewpoint image according to an exemplary embodiment of the present invention.
4 is a diagram illustrating a hole mask image,
Figs. 5 and 6 are diagrams provided for explaining hole filling for a hole by an angle of view, Fig.
Figs. 7 and 8 are diagrams provided for explaining hole filling for holes by resolution, and Figs.
Fig. 9 is a view provided for explaining hole filling for holes caused by occlusion.

Hereinafter, the present invention will be described in detail with reference to the drawings.

The second multi-view image means images acquired by dozens of RGB cameras existing at various positions in one space, and the arbitrary view image means a virtual RGB image acquired at an arbitrary virtual position in the space.

Therefore, the arbitrary viewpoint generation based on the second multi-viewpoint image is to generate an image at a virtual position in the space using images acquired by dozens of cameras existing in various places in one space.

The depth map of the super multi - view images is used for composing the multi view image based on the super multi view image. First, the pixels of the second multi-view images are converted into the points of the three-dimensional space using the given depth information and the camera information, and the thus obtained three-dimensional points are projected to a desired arbitrary view position. At this time, since three-dimensional points are obtained for each RGB image of the super-multi-view image, these three-dimensional points are matched in a three-dimensional space to make single three-dimensional points (for example, FIG. The generated arbitrary-view image generates a hole (for example, a dark part in FIG. 2).

In the method of filling a random viewpoint image according to an embodiment of the present invention, holes in a random viewpoint image generated by using a hyperview image and a depth are generated through holes generated outside the angle of view, resolution holes due to low density, Different holes are applied, and different hole filling algorithms are applied.

To do this, a mask is used to generate a random viewpoint image to search for a hole in a random viewpoint image, and the cause of each hole from the holes thus determined is discriminated through holes in the vicinity thereof.

Unlike the conventional technique in which the same hole filling algorithm is applied irrespective of the cause of the holes, different hole filling algorithms are applied to fill the holes according to the generation of each hole.

FIG. 3 is a flowchart provided for explanation of a method of filling a random viewpoint image according to an embodiment of the present invention.

As shown in FIG. 3, first, a hole mask image is generated to quickly find whether a pixel of a certain view image is a hole or not (S110).

When projecting three-dimensional points onto an arbitrary viewpoint image, all pixels that are not projected are regarded as holes, and it is determined whether they are holes or not. The Hall mask image has the same resolution as the arbitrary view image, and each pixel has one of 0, 80, 160, and 240 values. Initially, every pixel in the mask has a value of 0 and increases by 80 every time a three-dimensional point is projected. Thus, the pixel value of the final mask (e.g., FIG. 4) indicates whether the pixel is a hole or not.

Next, the cause of the holes is identified, and the hole causes are labeled (S120). There are various causes for generating holes in the arbitrary viewpoint image. In the embodiment of the present invention, the types of causes of holes are classified into the following four types.

(1) Hole by angle of view

If the random viewpoint image requires a scene that does not exist in the super high viewpoint image, it is a hole generated when the arbitrary viewpoint is out of the full angle of view of the super high viewpoint. In other words, it is a hole caused by the fact that the three-dimensional points to be projected at a certain time point are not present anywhere in the secondary multi-viewpoint image. These holes appear on the edge of the arbitrary-view image, either vertically or horizontally over a wide area. For example, the black portions at the edges of FIG. 4 are holes by the angle of view. Therefore, the hole generated by the angle of view is regarded as the hole by the angle of view when the edge of the image of the hole mask is analyzed and if the size of the hole (that is, the size of the image in which the holes continuously appear)

(2) Hole according to resolution

If the density of the projected three-dimensional points is lower than the pixel density of the arbitrary viewpoint image, a hole occurs in the image at a certain viewpoint. That is, a random viewpoint image is a hole that is created by requiring more dense three-dimensional points, which usually occurs when an arbitrary viewpoint is closer to the scene than a second viewpoint. These holes have a pattern in which holes are generated in a lattice shape in a random view image due to their characteristics. For example, the right grid-shaped black portions in FIG. 4 are holes by resolution. Since the size of the hole is small and has continuity in a specific direction, if the size of the hole is less than a certain portion and the directionality is possessed, the hole is regarded as a hole by resolution.

(3) Holes by occlusion

If three-dimensional points to be projected at a certain time point are not present in the second multi-view image due to occlusion due to depth, a hole occurs in the image at a certain time point. In other words, it is a hole that occurs because there is no pixel to be projected anywhere in the image of the second multi-viewpoint resembling the hole by the view angle, but this hole is important because it appears in the middle part of the arbitrary viewpoint image closely related to the scene. These holes appear at the boundary between the object and the object in the scene. For example, the spheres of FIG. 4, the triangular pyramids, and the black portions of the triangular-pyramid and background boundaries are occlusion holes. Therefore, the hole caused by the occlusion finds the boundary based on the image and the depth map at a certain point of time, and considers the black portions of the portion corresponding to the boundary line in the hole mask as the hole caused by the occlusion.

(4) Holes due to other causes

A hole that does not fall into any of the above three categories shall be considered as a hole due to other causes.

Then, different hole filling algorithms are applied to each hole to fill the hole.

Specifically, in the case of the hole by the angle of view (S130-YES), the hole is processed as the background of the scene where the hole occurs (S135). A background is detected from a non-hole portion in a random viewpoint image, and holes are filled with pixels corresponding to a background closest to the hole by a view angle (for example, FIGS. 5 and 6).

If it is a hole by resolution (S140-YES), an ultra-high resolution technique is used (S145). The resolution is down-sampled until there is no hole due to the resolution of the arbitrary-view image. The low-resolution random view image is up-sampled using the super-resolution technique until the original resolution is obtained, and pixels of the hole portion according to the resolution are taken from the image thus obtained and filled in (for example, FIG. 7 & 8).

In the case of holes due to occlusion (S150-YES), a depth-based inpainting technique is used (S155). In the arbitrary-view image, the depth of a pixel closest to the occlusion hole to be filled is found, and neighboring pixels having a similar depth are found, and the pixel value of these pixels is added to the distance weight to fill the hole (for example, FIG.

On the other hand, a hole due to other causes (S160-Yes): The hole is filled with a joint bilateral filter (S165).

10 is a block diagram of a super multi-view image generation system according to another embodiment of the present invention. 10, the super-multi-view image generation system according to the embodiment of the present invention includes a super multi-view image generation unit 210, an image processor 220, an image output unit 230, and a storage unit 240 ).

The super-multi-view image generating unit 210 generates super multi-view images using a plurality of cameras, the image processor 220 generates arbitrary images from the super multi-view images, and the image output unit 230 generates the multi- To a display, an external device, or a network.

The storage unit 240 stores the generated images and provides a storage space necessary for the image processor 220 to generate the arbitrary view image.

It goes without saying that the technical idea of the present invention can also be applied to a computer-readable recording medium having a computer program for performing the functions of the apparatus and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention may be embodied in computer-readable code form recorded on a computer-readable recording medium. The computer-readable recording medium is any data storage device that can be read by a computer and can store data. For example, the computer-readable recording medium may be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, or the like. In addition, the computer readable code or program stored in the computer readable recording medium may be transmitted through a network connected between the computers.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

210: super multi-view image generation unit
220: image processor
230:
240:

Claims (8)

Generating a viewpoint image by projecting a multi-viewpoint image at an arbitrary viewpoint;
Determining a type of a hole displayed in the generated random view image; And
And filling the holes with different techniques according to the discriminated types,
The step of filling the holes,
The holes generated due to the arbitrary viewpoint image deviating from the viewpoint of the multi-viewpoint image are filled with holes by the first technique,
When a random viewpoint image is located closer to a scene than a multi-viewpoint image, if the density of projected three-dimensional points at a certain time point is lower than the pixel density of the arbitrary viewpoint image, the hole is filled with the second technique,
If the three-dimensional points to be projected at an arbitrary point are holes generated by occlusion due to the depth, the hole is filled with the third technique,
The second technique,
A technique of down sampling a resolution of a random viewpoint image until no hole exists and filling the hole using an image obtained by upsampling the resolution image until the original resolution is obtained,
The step of discriminating the kind of the hole includes:
When a hole is continuous in a size larger than a predetermined size at the edge of the image, the arbitrary viewpoint image is discriminated as a hole generated due to the out of view angle of the multi-viewpoint image,
When the hole has a size smaller than a certain size and has a directionality, the density of three-dimensional points projected at a certain time point is lower than the pixel density of the arbitrary viewpoint image,
Wherein the three-dimensional points to be projected at an arbitrary point of time are determined as holes generated by occlusion due to the depth when the holes appear at the boundary between the object and the object.
delete The method according to claim 1,
The first technique,
And filling the hole with the background of the scene where the hole is generated.
delete delete delete The method according to claim 1,
The third technique,
Wherein the hole filling method is a deep-based in-painting method.
A generating unit generating a multi-view image; And
A method of generating a random viewpoint image by projecting a multi-viewpoint image at an arbitrary viewpoint, determining a type of a hole displayed in the generated viewpoint image, and filling a hole with a different technique according to the determined type,
The processor,
The holes generated due to the arbitrary viewpoint image deviating from the viewpoint of the multi-viewpoint image are filled with holes by the first technique,
When a random viewpoint image is located closer to a scene than a multi-viewpoint image, if the density of projected three-dimensional points at a certain time point is lower than the pixel density of the arbitrary viewpoint image, the hole is filled with the second technique,
If the three-dimensional points to be projected at an arbitrary point are holes generated by occlusion due to the depth, the hole is filled with the third technique,
The second technique,
A technique of down sampling a resolution of a random viewpoint image until no hole exists and filling the hole using an image obtained by upsampling the resolution image until the original resolution is obtained,
The processor,
When a hole is continuous in a size larger than a predetermined size at the edge of the image, the arbitrary viewpoint image is discriminated as a hole generated due to the out of view angle of the multi-viewpoint image,
When the hole has a size smaller than a certain size and has a directionality, the density of three-dimensional points projected at a certain time point is lower than the pixel density of the arbitrary viewpoint image,
Wherein the three-dimensional points to be projected at a certain time are discriminated as holes generated by occlusion due to the depth when the hole appears at the boundary between the object and the object.

Images