JP2013012045A - Image processing method, image processing system, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To synthesize a high-quality virtual viewpoint image while suppressing a depth estimation error, even if mapping between images is difficult.SOLUTION: An image processing method is for synthesizing an image of a subject viewed from a given virtual viewpoint position, referring to a multi-viewpoint image obtained by imaging the subject from a plurality of different viewpoints. The image processing method comprises: calculating a likelihood to a depth of each pixel with regard to the multi-viewpoint image; estimating the depth of each pixel according to the likelihood; calculating an estimation function for estimating a likelihood to the depth from an image feature using a depth estimation result of a high accuracy estimation pixel; correcting the likelihood to a correction targeting pixel using the estimation function; re-estimating the depth of the whole image using the corrected likelihood; and synthesizing the image of the subject in accordance with the virtual viewpoint position, referring to the re-estimated depth and the multi-viewpoint image.

Description

  The present invention relates to a technique that is effective when a subject has a region with little texture and occlusion and is difficult to associate with a stereo matching method.

  Combining images viewed from a virtual viewpoint position using multi-viewpoint images taken from a plurality of cameras is called virtual viewpoint image composition. FIG. 10 is a diagram showing a flow of processing in the prior art for synthesizing an image at an arbitrary viewpoint position using a multi-viewpoint image. Hereinafter, the flow of processing in the prior art for synthesizing an image at an arbitrary viewpoint position using a multi-viewpoint image will be described. First, multi-viewpoint images and camera parameters are input (step Sa1). Next, three-dimensional information (depth) is estimated from the two-dimensional image group (step Sa2). Then, the virtual viewpoint image is synthesized based on the multi-viewpoint image, the camera parameter, and the depth (step Sa3). At this time, if the depth estimation accuracy is low, the quality of the synthesized image of the virtual viewpoint deteriorates.

  There is a stereo matching method for estimating the depth. In the stereo matching method, pixel correspondence between multi-viewpoint images, camera internal parameters, and external parameters are used. Then, the position of the pixel of interest in real space is obtained by calculation based on the principle of triangulation. FIG. 11 is a diagram showing an outline of processing by the stereo matching method. For example, as shown in FIG. 11, it is assumed that the point of interest A is viewed from the points P1 and P2. In this case, if the distance between the straight lines connecting the points P1 and P2 and the angles of the vertices of the triangle with the point of interest A, the point P1, and the point P2 as vertices are obtained, attention is drawn from the point P1 (or the point P2). The distance to the point A can be obtained.

  However, when there is a region with a small pattern (texture), a region where a periodic texture exists, or a region affected by occlusion, it is difficult to associate pixels in the region. FIG. 12 is a diagram showing a problem in the prior art. For example, as shown in FIG. 12, considering a case where occlusion occurs, such as when a subject C such as a bird crosses, the point of interest A cannot be seen from the point P1, so that the association cannot be made.

  At this time, in the stereo matching method, the point of interest A that can be seen from the point P2 is likely to be erroneously associated with the point B that looks like a similar shape from the point P1. Therefore, an unnatural image (artifact) is generated in the synthesized image due to the influence of the pixel such as the point A where the depth estimation is wrong. This is an important issue that leads to the quality of virtual viewpoint image composition.

  In the conventional virtual viewpoint image synthesis method, there has been an approach for dealing with such a pixel that is difficult to be associated by segmenting the image.

  For example, the image is finely segmented based on pixel color (R, G, B) information, and it is assumed that pixels in the same segment exist on the same subject, that is, on the same plane (curved surface). Based on this assumption, there is a method of correcting the depth so that the depth of the target pixel becomes the depth of the plane of the segment to which the pixel belongs (see, for example, Non-Patent Document 1).

  As another technique for correcting the depth, there is a technique for separating the foreground and the background from the color information of the image. This is based on the premise that there are two types of subjects, foreground and background, and detects a subject (foreground or background) having a color similar to that pixel for a pixel that is difficult to be matched by the stereo matching method. There is a method of correcting the likelihood of the depth of the pixel using the depth information (see, for example, Non-Patent Documents 2 and 3).

A. Klaus, M. Sormann, K. Karner: Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure, in Proc. Of ICPR, pp. 15-18 (2006) V. Kolmogorov, A. Criminisi, A. Blake, G. Cross, C. Rother: Bi-layer segmentation of binocular stereo video, In Proc. Of CVPR, vol. 2, pp. 407-414 (2005) Ishii, Takahashi, Naemura: Combining synthesis and segmentation for free viewpoint images, 3D image conference, 5-1, pp.49-52 (2009)

  In the above-described research on virtual viewpoint image synthesis, when correcting the depth information of a certain pixel, the correction is performed using the depth information of the pixel in the same segment as the pixel. This method is effective in an environment where cameras can be placed densely and an error in association occurs in a narrow area. However, if the correspondence is incorrect in a wide area around the target pixel, that is, if the correspondence of most pixels in the same segment is incorrect, the depth estimation accuracy of most pixels in the segment is low. End up. Therefore, there is a problem that it is difficult to correct the depth of the target pixel correctly even if the depth estimation result of the segment is used.

  Further, in the method of separating the foreground subject and the background subject, it is effective to use the color feature of each subject. However, in the case of the virtual viewpoint image composition, the depth is multivalued, and there is a problem that it is difficult to approximate with the binary of the foreground depth and the background depth.

  The present invention has been made in consideration of such circumstances, and suppresses depth estimation errors even when it is difficult to associate images with each other due to the influence of occlusion due to an area with little texture. The object is to provide a technique capable of synthesizing a quality virtual viewpoint image.

  One aspect of the present invention is an image processing method for synthesizing an image of a subject viewed from an arbitrary virtual viewpoint position based on a multi-viewpoint image obtained by photographing the subject from a plurality of different viewpoints, and by a stereo matching method, A first step of calculating the likelihood of each pixel depth for the multi-viewpoint image, and a second step of estimating the depth of each pixel based on the likelihood obtained in the first step And a third step of calculating an estimation function for estimating the likelihood for the depth from the image feature using the depth estimation result of the high-precision estimation pixel that satisfies the condition for estimating that the depth estimation accuracy is high And a fourth step in which likelihood correction is performed using the estimation function calculated in the third step with respect to the correction target pixel that satisfies the condition for estimating that the depth estimation accuracy is low. And a fifth step of re-estimating the depth of the entire image using the likelihood after correction performed in the fourth step, the depth re-estimated in the fifth step, and the multi-viewpoint And a sixth step of synthesizing the subject image according to the virtual viewpoint position based on the image.

  One aspect of the present invention is an image processing apparatus that synthesizes an image of a subject viewed from an arbitrary virtual viewpoint position based on a multi-viewpoint image obtained by photographing the subject from a plurality of different viewpoints, and by a stereo matching method, A likelihood calculating unit that calculates the likelihood of the depth of each pixel with respect to the multi-viewpoint image; and a depth estimating unit that estimates the depth of each pixel based on the likelihood obtained by the likelihood calculating unit; The likelihood estimation function calculation that calculates the estimation function for estimating the likelihood for the depth from the image feature using the depth estimation result of the high-precision estimation pixel that satisfies the condition for estimating the depth estimation accuracy is high And likelihood that the likelihood correction is performed using the estimation function calculated by the likelihood estimation function calculation unit for the correction target pixel that satisfies the condition for estimating that the depth estimation accuracy is low. Correction part and A depth re-estimation unit that re-estimates the depth of the entire image using the likelihood after correction performed by the likelihood correction unit, the depth re-estimated by the depth re-estimation unit, and the multi-viewpoint image And an image synthesis unit that synthesizes the image of the subject according to the virtual viewpoint position.

  According to one aspect of the present invention, a stereo matching method is used in a computer of an image processing apparatus that synthesizes an image of the subject viewed from an arbitrary virtual viewpoint position based on a multi-viewpoint image obtained by photographing the subject from a plurality of different viewpoints. A likelihood calculating step for calculating a likelihood for the depth of each pixel with respect to the multi-viewpoint image; and a depth estimating step for estimating the depth of each pixel based on the likelihood obtained in the likelihood calculating step; The likelihood estimation function calculation that calculates the estimation function for estimating the likelihood for the depth from the image feature using the depth estimation result of the high-precision estimation pixel that satisfies the condition for estimating the depth estimation accuracy is high The estimation function calculated in the likelihood estimation function calculating step for the correction target pixel that satisfies the step and the condition for estimating that the depth estimation accuracy is low. A likelihood correction step for correcting the likelihood, a depth re-estimation step for re-estimating the depth of the whole image using the likelihood after the correction performed in the likelihood correction step, and the depth re-estimation step. A computer program for executing an image synthesis step of synthesizing an image of the subject according to the virtual viewpoint position based on the depth re-estimated in the estimation step and a multi-viewpoint image.

  According to the present invention, it is possible to suppress a depth estimation error and synthesize a high-quality virtual viewpoint image even in a case where it is difficult to associate images with each other due to an area with little texture or the influence of occlusion.

It is a block diagram which shows the structure of a virtual viewpoint image composition system. It is a flowchart for demonstrating the virtual viewpoint image synthesis method by this embodiment. It is a conceptual diagram which shows the example of arrangement | positioning of the camera used with the virtual viewpoint image synthesis method by this embodiment. It is a conceptual diagram for demonstrating the calculation method of the likelihood with respect to the depth by this embodiment. It is a conceptual diagram for demonstrating the epipolar line (EL1, EL2) between several images. It is a conceptual diagram for demonstrating the calculation method of the likelihood estimation function F with respect to the depth from an image feature. It is a conceptual diagram for demonstrating the calculation method of the likelihood estimation function F with respect to the depth from an image feature. It is a conceptual diagram for demonstrating the image synthesis | combination of a virtual viewpoint position. It is a conceptual diagram for demonstrating 3D warping method. It is a figure which shows the flow of the process of the prior art which synthesize | combines the image of arbitrary viewpoint positions using a multiview image. It is a figure which shows the outline of the process by a stereo matching method. It is a figure which shows the problem in a prior art.

<Outline>
First, an outline of a virtual viewpoint image composition system (hereinafter simply referred to as “virtual viewpoint image composition system”) according to an embodiment of the present invention will be described.
The virtual viewpoint image composition system is a realistic image that allows viewers to feel as if they were in a shooting environment for table tennis, tennis and other sports appreciation, as well as distance learning materials for classes such as universities. Is synthesized with high quality. Therefore, according to this virtual viewpoint image composition system, even if it is not a shooting environment where the cameras are densely arranged like the ray space method or the visual volume intersection method, or a shooting environment where the subject can be shot from all directions, Realize high-quality virtual viewpoint image composition. In other words, the virtual viewpoint image composition system is intended for shooting at actual stadiums and event venues, and for high-quality virtual viewpoint images in sports scenes such as table tennis and tennis, and even in event scenes such as live concerts. Realize synthesis.

  In order to realize the above-described composition, the virtual viewpoint image composition system determines the likelihood of the depth of the pixel or the segmented region for the pixel of the region or the segmented region that is difficult to be matched. The function to be corrected is calculated from Then, correction is performed based on the result of the function. The image feature means color information, texture information, or motion information.

  Specifically, it is as follows. First, the virtual viewpoint image composition system extracts an image feature for each depth value using a pixel with high matching accuracy detected in advance (hereinafter referred to as a high accuracy estimation pixel). Next, the virtual viewpoint image composition system compares the image feature obtained for each depth value with the feature of a pixel in a region that is difficult to associate (hereinafter referred to as a correction target pixel). Then, the virtual viewpoint image composition system corrects the depth of the pixel that is difficult to be associated using the depth value having the most similar image feature. A pixel that is difficult to associate and a small region that is difficult to associate (segmented small region) are not different from each other only in the scale (spatial size). In the following description, only a correction method for pixels that are difficult to associate will be described.

<Details>
Next, details of the virtual viewpoint image composition system will be described.
FIG. 1 is a block diagram illustrating a configuration of a virtual viewpoint image synthesis system. The subject photographing unit 101 is a multi-viewpoint image acquisition system including a plurality of cameras. The subject photographing unit 101 supplies the photographed video signal S1 to the camera image acquisition unit 102. The camera parameter input unit 103 is a device that inputs calibrated camera parameters P1. The virtual viewpoint position input unit 105 is an apparatus that inputs a viewpoint position desired by the user. The camera parameter input unit 103 and the virtual viewpoint position input unit 105 are, for example, a user interface such as a keyboard, a mouse, or a touch input device, or an external storage device such as a DVD (Digital Versatile Disc) or a USB (Universal Serial Bus) memory. It is.

  The virtual viewpoint image composition device 100 includes a camera image acquisition unit 102, a depth estimation unit 104, a virtual viewpoint position determination unit 106, an image data storage unit 107, an image composition unit 108, and a composite image output unit 109. The camera image acquisition unit 102 acquires the video signal S1 from the subject photographing unit 101 and supplies it to the image data storage unit 107 as image data D1. The virtual viewpoint position determination unit 106 determines the camera parameter of the virtual viewpoint position given by the virtual viewpoint position input unit 105 and supplies the camera parameter to the image composition unit 108.

  The image data storage unit 107 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The image data storage unit 107 includes a camera image / camera parameter storage unit 107a, a depth storage unit 107b, and a composite image storage unit 107c. Each storage unit may be configured on the same storage device, or may be configured on different storage devices. The camera image / camera parameter storage unit 107a stores the image data D1 from the camera image acquisition unit 102. The depth storage unit 107b stores estimated depth data D2 output from the depth estimation unit 104 described later. The composite image storage unit 107c stores image data D3 output from the image composition unit 108 described later. An image of a scene captured in advance by subject photographing by the camera of the subject photographing unit 101, a camera parameter P1 obtained by calibration, and an output result D2 of the depth estimation unit 104 are respectively converted into a camera image / camera parameter storage unit 107a and a depth storage unit. 107b, and image synthesis can be executed independently according to the input of the virtual viewpoint position desired by the user.

The depth estimation unit 104 extracts the camera parameter P1 and the image data D1 from the camera image / camera parameter storage unit 107a, outputs a depth estimation result D2, and supplies the result to the depth storage unit 107b.
The image composition unit 108 retrieves the camera parameter P1 and the image data D1 from the camera image / camera parameter storage unit 107a, retrieves the depth estimation result D2 from the depth storage unit 107b, and composes image (image viewed from a virtual viewpoint) data D3. Is output.

  The composite image output unit 109 reads the composite image data D3 stored in the composite image storage unit 107c as output image data, and outputs it as a video signal S2 for display display to the composite image display unit 110. The composite image display unit 110 is a display device such as a CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), or PDP (Plasma Display Panel) connected to the composite image output unit 109 such as a display terminal. The composite image display unit 110 displays a composite image in accordance with the video signal S2 from the composite image output unit 109. The composite image display unit 110 may be, for example, a two-dimensional planar device or a curved display device that surrounds the device user.

(Description of image composition method)
Next, a virtual viewpoint image composition method by the virtual viewpoint image composition apparatus 100 of the present embodiment will be described. FIG. 2 is a flowchart for explaining the virtual viewpoint image synthesis method according to this embodiment. In the virtual viewpoint image composition, the arrangement of the cameras may be originally free. However, in this embodiment, the cameras are arranged in a lattice shape or on a straight line in order to easily secure a common field of view with a plurality of cameras. FIG. 3 is a conceptual diagram illustrating an arrangement example of cameras used in the virtual viewpoint image synthesis method according to the present embodiment. As shown in FIG. 3, the directions of the cameras C _n−2 , C _n−1 , C _n , C _{n + 1} ,... Are arranged in parallel or in a radial pattern so that a specific subject is the gazing point M. , All cameras C _n-2 , C _n−1 , C _n , C _{n + 1} ,... Are synchronized.

[Multi-viewpoint image and camera parameter input]
First, the camera parameter input unit 103 obtains camera parameters of each camera by calibration as preprocessing (step S1). The camera number n (= 1,2,3, ..., N ), the internal parameters of the camera _{A n,} the external parameter _{R n, T} n, when the position of the pixel of the camera _{C n} of an image and _{m n,} camera position on C _n images _{m n = [x n, y} n] position of the coordinate system of the camera _{C n Mc = [X c,} Y c, Z c], the position M = [X world coordinate system, Y , Z] is obtained by the following equations (1) and (2).

  From Equation (1) and Equation (2),

It becomes. Here, s _n is a positive constant that determines the scale in the depth direction, T in the upper right subscript means a transposed matrix, tilde (~) _mn and tilde (~) M are extension vectors, and tilde (~) m _n = [x _n , y _n , 1] ^T , tilde (˜) M = [X, Y, Z, 1] ^T
If the depth of the image is known, it is found position M _c in the coordinate system of the constant s _n is determined camera C _n by Equation (1). Then, the position M in the world coordinate system can be obtained from Equation (2).
Further, when the depth of the pixel _{m n} of the camera _{C n} is Z = d, the pixel _{m n-1} on the camera _{C n-1} of the image can be obtained homography matrix _{H n,} the _n-1.

[Calculation of likelihood to depth]
Next, the depth estimation unit 104, the image _{I n} of camera _{C n,} obtains a likelihood for depth by a stereo matching method (step S2). The depth can be estimated in the same manner for images of all cameras other than the camera C _n . Since multi-viewpoint images are assumed, a stereo matching method using multiple baseline lengths expanded from SSD (Sum of Squared Difference) used in binocular stereo (Reference 1: Okutomi, Kinde: Multiple baseline lengths) The stereo matching method used, SSSD (Sum of SSDs) of the theory of theory, vol. J75-D-II, no. 8, pp. 1317-1327 (1992)) is used for the likelihood calculation.

Below, the calculation of likelihood when NCC (Normalized Cross Correlation) is used is shown. For the target pixel p of the image _{I n} of camera _{C n,} the likelihood _L p for the depth d (d) is expressed by the following equation (6).

Where O is a set of peripheral cameras of the camera C _n , r is the position of the pixel of the image I _o of the camera Co obtained by the homography matrix of Equation (4), and ν _γ is a local region around the pixel r in the image I _o This is a vector in which the luminance values of R, G, and B of the images are arranged. ν _p · ν _γ represents an inner product of vectors, norm represents the magnitude of the vector, and means 1-norm, 2-norm, and the like. Γ _p is a normalization coefficient that makes the sum of the likelihoods L _p (d) become 1 when the depth d is changed.

FIG. 4 is a conceptual diagram for explaining a method of calculating likelihood with respect to depth according to the present embodiment. FIG. 5 is a conceptual diagram for explaining epipolar lines (EL1, EL2) between a plurality of images. As shown in FIG. 4, the peripheral region is a region of 3 × 3, 5 × 5, 7 × 7 pixels, etc. around the pixel of interest p. ν _p and ν _γ can be represented by a raster scan vector of R, G, and B component values. For example, when the size of the local region is 3 × 3, each component is 9-dimensional, so ν _p is a 27 (= 9-dimensional × 3-component) -dimensional vector.

The equation (6), seeking likelihood by computing the correlation information of the local region on the epipolar lines between the plurality of images (see Figure 5) for pixel p of the camera C _n. In addition, with respect to the camera C _n, the choice of peripheral camera C _o is dependent on the shooting environment. Matching is facilitated by selecting cameras with as many common fields of view as possible. Therefore, it is preferable to chose two or more cameras close to the camera C _n.

[Depth estimation]
Next, the depth estimation unit 104 estimates the depth based on the likelihood of each pixel (step S3). In this embodiment, a method of estimating the depth of a multi-viewpoint image by solving the minimization problem of the energy function defined by the likelihood of each pixel and the smoothing term is used. In this method, the energy function is defined by a smoothing term such that the likelihood with respect to the depth of each pixel is close to the estimation result of the depth of neighboring pixels. The subject surface tends to be estimated to have an uneven depth only by the result of the likelihood of the stereo matching method. However, setting smoothing terms has the effect of smoothing the estimation results, and its effectiveness has been reported (Reference 2: Li Hong, George Chen: Segment-based Stereo matching Using Graph Cuts, in Proc of CVPR, vol.1, pp. 74-81 (2004)).
The image _{I n} of camera _{C n,} expressed the pixel of interest p, neighboring pixels in q, the energy function, the following equation (7), (8), is defined as (9).

Where uppercase D (p) is the estimated depth of pixel p, E _Likelihood is a function that outputs the cost when the depth of pixel p is estimated to be D (p), and E _smooth is , Is a smoothing term, and λ is a ratio that emphasizes two functions. The higher the likelihood, the lower the cost. E _smooth is a function that outputs a smaller cost as the difference between the depth estimation results D (p) and D (q) of the pixel p and the neighboring pixels is smaller.

  As for the smoothing term, in addition to the formula (9), the form of the following formula (10) that changes the cost due to the color difference from the neighboring pixel q, and the depth of the pixels of p and q If they are different, the following equation (11) may be used to calculate a certain cost.

Here, I (p) and I (q) is the color information of the pixel p and the pixel q of the camera _{C n,} be a vector obtained by arranging [R, G, B] component of the position of the pixel p and q , || I (p) -I (q) || represents a 2-norm. The smoothing term of Equation (10) provides an effect that the depth of neighboring pixels is likely to change where the color changes.

Finally, a depth that minimizes the energy function E _total of Equation (7) is obtained. For this minimization problem, an approximate solution can be obtained by an algorithm such as the Simulated Annealing method, the Graph Cuts method, or the Belief Propagation method.

[Evaluation of depth estimation results]
Next, the depth estimation unit 104 performs detection of a pixel whose depth is to be corrected (correction target pixel) and detection of a pixel with high depth estimation accuracy (high accuracy estimation pixel) (step S4). The depth estimation unit 104 selects a pixel that is difficult to associate with the stereo matching method as the correction target pixel. Two evaluation methods will be described below.
(1) In a pixel in an area where the evaluation texture using the likelihood of the stereo matching method is small or an occlusion area, the maximum value of the likelihood function value with respect to the depth of the pixel p in Equation (6) tends to be small. In addition, if the depth estimation is incorrect, an artifact occurs when the virtual viewpoint image is synthesized using the depth.

  Accordingly, the depth estimation unit 104 is a pixel whose maximum likelihood value is smaller than the threshold Th_like as the correction target pixel, and the difference between the image synthesized with the estimated depth value and the video of the actual camera is smaller than the threshold Th_diff. Large pixels may be selected. On the other hand, the depth estimation unit 104 calculates a pixel whose maximum likelihood value is larger than the threshold value Th_like and whose difference between the image synthesized with the estimated depth value and the video of the actual camera is smaller than the threshold value Th_diff. Alternatively, it may be selected as a high-precision estimated pixel.

These threshold values are parameters determined by conducting an experiment in advance, for example. In the present embodiment, the average value of the likelihood of the entire image and the average of the differences are set as threshold values Th_like and Th_diff, respectively. Hereinafter, the high-precision estimated pixel is represented by u, and the set of high-precision estimated pixels is represented by U.
(2) Evaluation by comparison with neighboring camera images The evaluation of the estimation accuracy for the pixel p of the image Ii of the camera Cn is described.
The estimated depth of the pixel p is Di (p), the position of the pixel projected to the neighboring camera Co by the homography matrix of the equation (4) based on the depth is q, and the pixel q of the camera Co is estimated. When the depth is expressed as Do (q), the following expression is used for evaluation.

  For the pixel p, the threshold Th_SD; Th_SI is set for SD and SI that compare the depth and color of the pixel q of the neighboring camera Co, and the pixels that are equal to or lower than the threshold are determined to have high estimation accuracy. These thresholds are experimentally determined parameters.

[Calculation of depth estimation function f from image features]
Next, a depth estimation function is calculated for the pixel p to be corrected by the camera C _n (step S5). Hereinafter, calculation of the depth estimation function will be described with reference to FIGS. 6 and 7. 6 and 7 are conceptual diagrams for explaining a method of calculating the depth estimation function f from the image feature. For the calculation of the depth estimation function, a high-precision estimated pixel u (∈U) within a radius R from the correction target pixel p is used (see FIGS. 6 and 7). Here, the pixel set U cameras _C in the vicinity of _n cameras _{C o (o = ... n-} 2, n-1, n, n + 1, ...) also include pixels projected on the camera _{C n.} Expressed camera C _o accurate estimate pixels u _o of _the pixel of coordinates obtained by projecting the pixel u _o to camera C _n by u ^o _n, a set of high-precision estimated pixel u used in the calculation of likelihood estimation function U is obtained by the following equations (12) and (13).

Next, D multi-layer planes are set in the depth direction of the camera C _n , and image features are extracted from high-precision estimated pixels u (∈U) belonging to each layer (d (= 1, 2,..., D)). . The image feature is extracted from an N × N local region including the high-precision estimated pixel u having a depth of d. For example, when N = 1, a three-dimensional vector in which the color (R, G, B) components of the high-precision estimated pixel u are arranged, or R, G, B in a 5 × 5 region as shown in FIG. Vectors including texture information arranged by raster scanning, HOG (Histograms of Oriented Gradients) features, and SURF (Speeded-Up Robust Features) features are used.

A set of pixels in which the depth of the high-precision estimated pixel u is estimated to be _d is represented by Ud, and a feature vector of the pixel u is represented by vd. The depth estimation function estimates the depth from the similarity or distance between the dictionary vector and the image feature vector of the correction pixel. The similarity calculation method is, for example, the Mahalanobis distance between the dictionary vector and the image feature vector of the correction pixel, the distance between the nearest neighbor vector obtained as a result of the nearest neighbor search and the image feature vector of the correction pixel, or the dictionary vector. It is calculated by the inner product angle of the feature vector of the generated partial space and the correction pixel.
In the following, a method for calculating the likelihood F (d) in which the depth estimation function f when using the Mahalanobis distance belongs to the depth d for the correction pixel will be described. When the feature vector of the correction pixel is expressed by xp, it is expressed by the following equations (14), (15), (16), and (17).

Where Γ _F is a normalization coefficient for the sum of likelihoods of depth d (= 1, 2,..., D) to be 1, num (vd) is the number of dictionary vectors vd, and dist (xp, μd) Is a Mahalanobis distance, μd is an average vector of dictionary vectors vd having a depth d, Sd is a covariance matrix, and ε is a minute value for avoiding 0 division. The radius R is a parameter determined experimentally. In this embodiment, R = 10 to 40 and ε is set to 0.1.

[Likelihood correction]
Next, the likelihood of the correction target pixel is corrected based on the depth information of the subject to which the pixel belongs (step S6). For the likelihood Lp (d) obtained by the stereo matching method of the correction target pixel p, the corrected likelihood L′ p (d) is expressed by the following equation (18).

  Here, w (0 <w <1) is a parameter for adjusting the likelihood calculated by the stereo matching method and the ratio indicating which of the outputs of the depth estimation function is important. If w is large, the likelihood of the stereo matching method is emphasized, and is determined experimentally.

[Re-estimation of image depth]
Next, the depth is re-estimated by solving the minimization problem of the energy function defined by the likelihood of each pixel and the smoothing term (step S7). That is, the depth is re-estimated by substituting the corrected likelihood into Equation (7).

[Image composition of virtual viewpoint position]
Next, the image composition unit 108 selects a camera close to the virtual viewpoint position, and composes an image by the 3D warping method from the selected N camera images and the estimated depth information (step S8). When blending colors, a weighted average is performed according to the proximity of the position of each camera and the virtual viewpoint and the strength of the estimated likelihood of depth.

Here, FIG. 8 is a conceptual diagram for explaining the image composition of the virtual viewpoint position. The 3D warping method is a method of determining the color I _v (m _v ) of the pixel m _v of the image of the camera C _{v at} the virtual viewpoint position based on the multi-view image and the depth (depth map) of the image. FIG. 8 shows an example of two cameras. The cameras can be selected by using cameras within a suitable distance from the virtual viewpoint, so two or more cameras can be selected.

The internal parameters and the external parameters of the camera C ₁ and the camera C ₂ are respectively A ₁ , A ₂ , R ₁ , T ₁ , R ₂ , T _2, and the depths of the images of the camera C ₁ and the camera C ₂ are D ₁ , D ₂ . At this time, the color of the point M is projected from the cameras C ₁ and C ₂ to the virtual viewpoint camera C _{v according} to the equation (3). Assuming that the internal parameters of the virtual viewpoint camera are A _v and the external parameters are R _v and T _v ,

It becomes. Here, the tilde (˜) m _v ¹ and the tilde (˜) m _v ² are extended vectors of positions when the pixels m ₁ and m ₂ of the camera C ₁ and the camera C ₂ are projected by Expression (3). is there.

The color I (m _v ) of the pixel m _v of the image of the virtual viewpoint is obtained by a weighted average based on the ratio of the distance between the virtual viewpoint and the camera C ₁ and the camera C ₂ and the likelihood of the depths of the pixels m ₁ and m _2. . When the ratio of the distance between the virtual viewpoint and the camera C ₁ and the camera C ₂ is α: (1-α) (0 <α <1) and the likelihood is L (Dm ₁ ): L (Dm ₂ ),

However, L (Dm ₁ ) and L (Dm ₂ ) are likelihoods calculated at the time of depth estimation for the pixels m ₁ and m ₂ of the images of the cameras C ₁ and C ₂ . Although the ratio and the ratio of the likelihood of the distance to determine the w ₁ and w ₂ by the addition, and to use only one or the other may be determined by multiplying the ratio.

Here, FIG. 9A and FIG. 9B are conceptual diagrams for explaining the 3D warping method according to the present embodiment. When performing projection in accordance with the depth of the pixel by Expression (4), different from P and the point Q as shown in FIG. 9 when viewed from a virtual viewpoint camera C _v, it may be present on a straight line. At this time, P points towards the depth is small in the coordinate system of the camera C _v for points P and Q are visible from the virtual camera C _v. For example, regarding the point P visible from the camera C ₁ and the point Q visible from the camera C ₂ , when the depth in the coordinate system of the camera C _v is D _v (P) and D _v (Q), respectively (D _v ( Q) −D _v (P))> δ,

  It becomes. However, δ is a threshold parameter and is determined in advance by a preliminary experiment. If it is less than or equal to the threshold δ, the colors are mixed according to Equation (21).

  Next, effects of the virtual viewpoint image composition device 100 according to the embodiment of the present invention will be described.

  In the conventional method, for regions (pixels) that are difficult to associate, correction is performed using depth information of pixels in the same segment. The conventional method also uses depth information of a subject (foreground or background) having a color similar to that of a pixel that is difficult to associate, assuming that the subject's depth is binary, such as foreground and background. The correction that was done was also performed.

  However, with the former method, correction cannot be performed correctly unless the depth estimation accuracy of most pixels in the same segment is high. That is, when the area with less texture or the area affected by occlusion is wide, the depth estimation error may increase. In addition, it is assumed that the same subject becomes the same segment, but it is difficult to segment an image with high accuracy.

  In the latter method, it is assumed that the subject exists in the foreground or background, that is, the depth is approximated in two stages. However, in the virtual viewpoint image composition, the depth value is multivalued, so that it is difficult to apply. The latter method separates the background and the foreground based on the color information. However, when there are colors similar to the foreground and background, separation becomes difficult.

  On the other hand, according to the virtual viewpoint image composition device 100 described above, it is possible to suppress a depth estimation error even when it is difficult to associate images. Therefore, even in such a case, a high-quality virtual viewpoint image can be synthesized. As a result, it is possible to prevent artifacts from occurring in the parts (face, foot, hand, etc.) of the subject and improve the quality of the composite image.

  Note that the case where it is difficult to associate images is, for example, a case where a region with little texture is wide or a region affected by occlusion is wide. In addition, when there is another subject having a color similar to the subject near the subject boundary, it is difficult to associate the images. Artifacts generated in a subject part are, for example, an image in which a part of the part is lost or an image in which a part of the part is enlarged or reduced.

<Modification>
The process for selecting the correction target pixel is not necessarily limited to the above-described process. For example, when there are few textures around the target pixel, the target pixel may be selected as a correction target pixel. For example, when there is a repetitive texture around the target pixel, the target pixel may be selected as a correction target pixel. For example, when the periphery of the target pixel is affected by occlusion, the target pixel may be selected as a correction target pixel. For example, whether or not the texture is small can be determined by the following method. First, a Sobel filter (horizontal and vertical luminance value differential filter) is applied to the image of interest. Then, the value after filtering is used as the edge strength for each pixel, and it is possible to determine whether the texture is large or small based on the edge strength.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  DESCRIPTION OF SYMBOLS 100 ... Virtual viewpoint image synthesizer, 101 ... Subject imaging | photography part, 102 ... Camera image acquisition part, 103 ... Camera parameter input part, 104 ... Depth estimation part, 105 ... Virtual viewpoint position input part, 106 ... Virtual viewpoint position determination part, DESCRIPTION OF SYMBOLS 107 ... Image data storage part, 107a ... Camera image / camera parameter storage part, 107b ... Depth storage part, 107c ... Composite image storage part, 108 ... Image composition part, 109 ... Composite image output part, 110 ... Composite image display part

Claims (3)

An image processing method for synthesizing an image of a subject viewed from an arbitrary virtual viewpoint position based on a multi-viewpoint image obtained by photographing the subject from a plurality of different viewpoints,
A first step of calculating a likelihood for the depth of each pixel with respect to the multi-viewpoint image by a stereo matching method;
A second step of estimating the depth of each pixel based on the likelihood obtained in the first step;
A third step of calculating an estimation function for estimating the likelihood for the depth from the image feature using the depth estimation result of the high-precision estimation pixel that satisfies the condition for estimating that the depth estimation accuracy is high;
A fourth step of performing likelihood correction using the estimation function calculated in the third step for a correction target pixel that satisfies a condition for estimating that the depth estimation accuracy is low;
A fifth step of re-estimating the depth of the entire image using the corrected likelihood performed in the fourth step;
An image processing method comprising: a sixth step of synthesizing the subject image corresponding to the virtual viewpoint position based on the depth re-estimated in the fifth step and the multi-viewpoint image. An image processing apparatus that synthesizes an image of a subject viewed from an arbitrary virtual viewpoint position based on a multi-viewpoint image obtained by photographing the subject from a plurality of different viewpoints,
A likelihood calculating unit that calculates a likelihood for the depth of each pixel with respect to the multi-viewpoint image by a stereo matching method;
A depth estimation unit that estimates the depth of each pixel based on the likelihood obtained by the likelihood calculation unit;
A likelihood estimation function calculation unit that calculates an estimation function for estimating the likelihood with respect to the depth from the image feature using the depth estimation result of the high-precision estimation pixel that satisfies the condition for estimating that the depth estimation accuracy is high. When,
A likelihood correction unit that corrects likelihood using the estimation function calculated by the likelihood estimation function calculation unit for a correction target pixel that satisfies a condition for estimating that the depth estimation accuracy is low; ,
A depth re-estimation unit that re-estimates the depth of the entire image using the likelihood after correction performed in the likelihood correction unit;
An image processing device comprising: an image composition unit that composes an image of the subject according to the virtual viewpoint position based on the depth re-estimated by the depth re-estimation unit and the multi-viewpoint image. . A computer of an image processing apparatus that synthesizes an image of the subject viewed from an arbitrary virtual viewpoint position based on a multi-viewpoint image obtained by photographing the subject from a plurality of different viewpoints.
A likelihood calculating step of calculating a likelihood for the depth of each pixel with respect to the multi-viewpoint image by a stereo matching method;
A depth estimation step for estimating the depth of each pixel based on the likelihood obtained in the likelihood calculation step;
A likelihood estimation function calculating step for calculating an estimation function for estimating a likelihood with respect to a depth from an image feature using a depth estimation result of a high-precision estimation pixel that satisfies a condition for estimating that the depth estimation accuracy is high When,
A likelihood correction step of correcting the likelihood using the estimation function calculated in the likelihood estimation function calculation step for a correction target pixel that satisfies the condition for estimating that the depth estimation accuracy is low; ,
A depth re-estimation step of re-estimating the depth of the entire image using the likelihood after correction performed in the likelihood correction step;
A computer program for executing an image synthesis step of synthesizing an image of the subject according to the virtual viewpoint position based on the depth re-estimated in the depth re-estimation step and a multi-viewpoint image.