JP2967086B1 - Estimation of 3D pose of a person by multi-view image processing - Google Patents

Abstract

Abstract: PROBLEM TO BE SOLVED: To provide a method for estimating a three-dimensional posture of a person by multi-view image processing capable of detecting in real time each characteristic portion of a top, a hand, and a toe and performing three-dimensional reconstruction from them. . SOLUTION: A front view image, a side view image, and an elevation image of a person are captured by CCD cameras 1 to 3, a background is separated by a chroma key background separation device 4, and a silhouette is obtained for an image of each viewpoint by an image processing device 5. The extraction, the center of gravity detection, the main axis detection, and the feature point detection are performed, and the posture is estimated based on the three-dimensional position of the feature point.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for estimating a three-dimensional posture of a person by multi-view image processing, and more particularly, to using a priori knowledge of a human body shape for a multi-view silhouette image of a person captured by a plurality of cameras. The present invention relates to a method for estimating a three-dimensional posture of a person by multi-view image processing for detecting a characteristic part and estimating a three-dimensional posture by stereoscopic vision between images.

[0002]

2. Description of the Related Art A method of estimating the posture of an observation object reflected in an image from a single silhouette image by model matching is disclosed in Kameda Yoshinari et al. IEICE Transactions D-II, Vol. J79-D-II, No. 1, p
p. 26-35 (January 1996).

The inventors of the present invention have also disclosed Japanese Patent Application No. 9-8613.
In No.2, a method of estimating human body posture was proposed.

[0004]

The method disclosed in the above-mentioned paper is difficult to apply to an arbitrary person because an accurate shape model of the target person is required, and the calculation cost for fitting the model is relatively high. Unsuitable for real-time processing.

In Japanese Patent Application No. 9-86132, the posture of a non-contact person is estimated using a single-eye image. However, since the single-eye image is used, a specific error due to self-shielding between body parts, The problem that three-dimensional attitude information cannot be obtained has not been solved.

[0006] Therefore, a main object of the present invention is to provide a tertiary tertiary image of a person by multi-view image processing that can detect in real time each characteristic portion such as the top of the head, the tip of the hand, and the tip of the foot. An object of the present invention is to provide an original posture estimation method.

[0007]

The invention according to claim 1 is
A method for estimating a three-dimensional posture of a person by multi-view image processing for estimating a three-dimensional posture of a person from images captured from different directions by at least two cameras, wherein each image captured by at least two cameras Step of obtaining a body center of gravity by performing distance conversion on a person area obtained by performing threshold processing on the arm area, and an arm area from an image above the horizontal position of each body center of gravity obtained in the first step To remove
A second step of calculating a main axis of the upper body distance image to obtain an upper body tilt angle and a rotation angle, and obtaining a maximum value of the distance conversion image subjected to the distance conversion to form a skeleton image, and generating a skeleton image of the skeleton image; A third step of selecting an end point that satisfies a predetermined condition from the end point of the lower body skeleton image below the vertical position of the center of gravity as a toe point; A fourth step, a fifth step of specifying the positions of the elbow and the knee from the characteristic parts of the head vertex, the toe point, and the toe point by using a genetic algorithm, and each characteristic part obtained in the fifth step A sixth step of determining the three-dimensional posture of the person by using the two-dimensional position of.

According to the second aspect of the present invention, the camera of the first aspect captures an image of the front, side, or elevation of a person, and when any one of the images of the front, side, or elevation is blocked, the remaining two images are taken. Three-dimensional pose is estimated using two images.

[0009]

FIG. 1 is a schematic block diagram of an embodiment of the present invention. In FIG. 1, three CCD cameras 1, 2, and 3 capture a front image, a side image, and an elevation image of a person, respectively. In FIG. 1, three CCD cameras 1 are shown.
However, in order to restore a three-dimensional image,
Although the accuracy is somewhat poor, at least two cameras are required.

The image outputs of the CCD cameras 1 to 3 are supplied to a chroma key background separating device 4 and to an image processing device 5 from which a background image is separated by chroma key processing. The image processing device 5 extracts a silhouette, detects a center of gravity, detects a principal axis,
Each process of feature point detection is performed, and three-dimensional restoration is performed.

FIG. 2 is a flowchart for explaining the operation of one embodiment of the present invention. FIG. 3 is a diagram showing processing from the captured image to the center of gravity detection. FIG. FIG. 5 is a diagram showing an example of a restored three-dimensional image.

Next, a specific operation of the embodiment of the present invention will be described with reference to FIGS. In step SP1 shown in FIG. 2 (abbreviated as SP in the figure),
For example, FIG.
3 (b1) and (b).
2), a front image, a side image, and an elevation image as shown in (b3) are captured. The image output of each of the cameras 1 to 3 is provided to a chroma key background separation device 4 to separate a background image. The image processing device 5 performs a threshold process on the front image, the side image, and the elevation image from which the background image is separated in step SP2, and generates a silhouette image as shown in FIGS. 3 (c1), (c2), and (c3). Extract.

Regarding these processes, an infrared camera is used as described in Japanese Patent Application No. 9-86132, and the obtained thermal image is subjected to threshold processing to extract a region corresponding to a person. To obtain a silhouette image.

Next, in step SP3, FIG.
1) From the silhouette images as shown in (c2) and (c3), the barycenters shown in FIGS. 3 (d1), (d2) and (d3) are detected. That is, distance conversion is performed to obtain the center of gravity of the body from the silhouette image f _ij . The center of gravity (I _c , J _c ) of the distance conversion image _dij is obtained using the following equation.

[0015]

(Equation 1) Here, I _c and J _c are the center of gravity of the area weighted by the distance value. When the center of gravity is directly obtained from the binary silhouette image, the position of the center of gravity remarkably shifts depending on the posture of the arm or the foot, which is not stable. However, by using the distance conversion image, it is possible to minimize the influence of the posture of the limb on the position of the center of gravity.

In step SP4, FIG.
The main body main axis shown in (a2) and (a3) is extracted.
For the image above the horizontal position of the center of gravity of the body in the distance-converted image, the Gaussian distribution having the upper body main axis of the previous frame as the axis of symmetry is multiplied by the distance position in order to remove the arm region.
Since the region of the obtained image (upper body distance image) is almost concentrated on the torso and the head, the center of gravity of the upper body distance image g _ij is obtained. The angle θ of the principal axis of inertia of g _ij passing through the center of gravity is expressed by the following equation, and can be regarded as the inclination of the upper body.

[0017]

(Equation 2) It is conceivable that the main axis is not determined in the direction along the spinal cord when the hand is extended in the horizontal direction, but this problem can be solved by the processing for removing the arm region.

Next, front, side, and elevation feature points are detected. That is, in step SP5, FIG.
Contour shapes as shown in (b1), (b2), and (b3) are obtained. The silhouette image is raster-scanned from the center of gravity of the body, and the boundary line is traced in a counterclockwise direction with the contour (boundary) pixel found first as a starting point. A binary image having "1" pixels on the boundary (contour) line and "0" pixels on other boundaries is referred to as a contour image. Here, a contour point (the one located above the center of gravity of the upper body) that is the shortest distance from the main axis of the upper body is selected as the temporary head vertex.

In step SP6, a toe point is detected. A skeleton image is created by obtaining the maximum value of the distance conversion image used for calculating the center of gravity of the body. The condition of “the maximum horizontal deviation from the body center of gravity” is selected from the end points of the skeleton that exists in the lower body (below the vertical position of the body center of gravity) and the left (right) half body (left or right centering on the horizontal position of the body center of gravity) Is selected as the toe point.

At step SP7, a hand point is detected. The number of pixels included in the contour corresponding to the to the toes from the temporary crown on the contour l _h: m _h: dividing the ratio of n _h,
The contour point included in the central portion is set as a candidate section of the hand point.
Here, l _h , m _h , and n _h are constants empirically obtained.

The highest vertical position among the candidates (candidate point A)
And a point having the longest horizontal distance from the center of gravity of the body (hand candidate point B) and a point having the longest horizontal distance from the center of gravity of the body (hand candidate point C). Then, evaluation is sequentially performed in accordance with the following conditions, and candidate points satisfying the above are determined as hand points.

Vertical position of hand candidate point A> Vertical position of temporary head vertex−T ₁ Vertical position of hand candidate point B <Vertical position of body center of gravity + T
Horizontal position of _one hand candidate point B <horizontal position of body center of gravity + T
Here _the end point of _the candidate section, T _1, T ₂ are constants determined by the height of the circumscribed rectangle of each silhouette.

At step SP8, the head vertex is detected. The number of pixels of the contour corresponding to the hand from the temporary top to the hand on the contour is divided into the ratio of l _p : m _p : n _p , and the contour point included in the central portion is set as a hand point candidate. The point at which the distance to the upper body main axis is the shortest is defined as the neck position, and the bisecting point of the contour sandwiched between the left and right neck positions is defined as the head vertex. As in the case of detecting the hand point, l _p , m _p , and n _p are constants empirically obtained.

When trying to reproduce the posture of the whole body as a computer graphic model, the center of gravity,
Information on head, hand, and foot positions alone is not enough. For example, if the position of the elbow is not known, the posture of the arm can take many patterns, cannot be uniquely determined, and the position information of the elbow and the knee is important. However, these parts do not have clear features as contours, and it is difficult to detect them by a heuristic method based on contours such as a fingertip. Therefore, in this embodiment, the positions of the elbow and the knee are estimated by a genetic algorithm.

In the genetic algorithm, the solution candidate of the target problem is one-dimensionally represented by a chromosome as a genotype. Then, an initial population of chromosomes is generated, and while performing genetic operations such as selection, crossover, and mutation according to the evaluation function, a search is advanced in parallel with the solution population as an operation target. In this embodiment, the positions of the elbows and knees are expressed by a linear combination of the feature points of the body center of gravity, the crown, the hands, and the feet.
The coefficients of the connection equation are regarded as genetic information, and a combination of coefficients that reduces the estimation error is searched for by a genetic algorithm. The fitness (evaluation function) f when the position of the knee (elbow) in each frame is x _i and the estimated position is x ＾ _i is the average value / e of the estimation error x _i −x ＾ _i and the error. Is defined as the reciprocal of the sum of the variances of

[0026]

(Equation 3) In step SP9, the elbow position is estimated. That is,
Body center of gravity (x _g , y _g ), head (x _p , y _p ), hand (x
_The elbow position (x _e , y _e ) is expressed as shown in the following equation using the load linear sum of each position of _h , y _h ) and toe (x _f , y _f ). In addition, each part is represented by a relative position from the center of gravity of the body, such as x ~ (where ~ represents a superscript broken line) = x- _xg . As previously described, is estimated by the genetic algorithm the coefficients C _i of the following equation.

[0027]

(Equation 4) Next, in step SP10, the knee position is estimated. In the contour from the hand to the toe, a contour point (x ₁ , y ₁ ) existing at a vertical position obtained by bisecting the difference between the body center of gravity and the vertical position of the toe is searched. Body center of gravity (x _g , y
_g ), the head position (x _p , y _p ), the foot tip (x _f , y _f ), and the knee position as shown in the following equation using the sum of the load linear values of the contour points (x ₁ , y ₁ ). (X _k , y _k ) is expressed, and the coefficient W _i is estimated by a genetic algorithm. FIG. 4 (c1),
(C2) and (c3) show detected images of each feature point.

[0028]

(Equation 5) When the front feature point, the side feature point, and the elevation feature point are detected in this way, in step SP11 of performing triangulation of the feature point group, the posture is estimated based on the three-dimensional position of the feature point group.

FIG. 5 is a diagram showing an example of a restored three-dimensional image. FIG. 5 (a1) shows the reproduction of the posture of a doll character generated by computer graphics (CG) based on the three-dimensional posture estimated as described above.
(A2) and (a3). These are shown in FIG. 3 (b1),
(B2), an actually imaged front image shown in (b3),
It supports side and elevation images. FIG. 5 (b1),
(B2) is a front image and a side image reproduced by another character.

As described above, according to this embodiment, each feature point is obtained by applying a predetermined algorithm using each image of the front, side, and elevation. Even if self-shielding occurs only in
It is easy to obtain a three-dimensional feature point position using images from three viewpoints.

The embodiments disclosed this time are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

[0032]

As described above, according to the present invention, it is possible to detect a characteristic portion of a silhouette image of a person captured by at least two cameras from different directions by using a priori knowledge of a human body shape. By doing so, a tertiary point posture can be estimated by stereo vision between images. In addition, the non-contact method does not place a burden on the human body, and is effective for an arbitrary posture of a person. Moreover, it can be easily realized by software processing of a personal computer without using dedicated hardware. Furthermore, since it is possible to have an accuracy that can be used for gesture recognition and the like, application to fields such as welfare, video games, and communication becomes possible.

[Brief description of the drawings]

FIG. 1 is a schematic block diagram of an embodiment of the present invention.

FIG. 2 is a flowchart for explaining the operation of the embodiment of the present invention;

FIG. 3 is a diagram illustrating a process from a captured image to a center of gravity detection.

FIG. 4 is a diagram showing processing from spindle detection to feature point detection.

FIG. 5 is a diagram illustrating an example of a restored three-dimensional image.

[Explanation of symbols]

 1-3 CCD camera 4 Chroma key background separation device 5 Pixel processing device

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-9-237348 (JP, A) JP-A-10-165394 (JP, A) (58) Fields investigated (Int. Cl. ⁶ , DB name) A61B 5/107 A61B 5/11 G01B 11/24 G06T 7/00

Claims (2)

(57) [Claims] 1. A method for estimating a three-dimensional posture of a person by multi-view image processing for estimating a three-dimensional posture of a person from images captured by different cameras at least in two directions, wherein the at least two cameras A first step of performing a distance conversion on a person region obtained by performing a threshold process on each image captured in step 1 to obtain a center of gravity of the body, from a horizontal position of the center of gravity of each body obtained in the first step A second step of calculating the main axis of the upper body distance image to obtain the upper body tilt angle and the rotation angle by removing the arm region from the upper image, and obtaining the local maximum value of the distance conversion image subjected to the distance conversion. A third step of selecting an end point satisfying a predetermined condition from an end point of the lower body skeleton image below the vertical position of the body center of gravity in the skeleton image as a toe point; A fourth step of determining a tip point and a neck position according to the above, and determining a head vertex; and a fifth step of estimating a position of an elbow and a knee from the characteristic portion of the head vertex, the tip point and the toe point using a genetic algorithm. And a sixth step of determining a three-dimensional posture of the person based on a two-dimensional position of each characteristic part obtained in the fifth step. Posture estimation method. 2. The camera captures an image of the front, side, or elevation of the person, and when any one of the images of the front, side, or elevation is occluded, three-dimensionally uses the remaining two images. The three-dimensional posture estimation method for a person by multi-view image processing according to claim 1, wherein the posture is estimated.