JP2001236505A - Method, device and system for estimating coordinate - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve such a problem that an expensive system or a complicated procedure is generally required for acquiring a three-dimensional (3D) coordinate from a 2D image. SOLUTION: A coordinate estimating device 10 has a correction unit 14, an initial setting unit 16 and an estimation unit 20 when roughly classified. The correction unit 14 specifies the pixel position of a camera picture and the transformation rules of world coordinates. The initial setting unit 16 acquires an absolute distance between respective concerned points on a target such as a human body, for example, to estimate the 3D coordinate. On the basis of information provided by the correction and initial setting, the estimation unit 20 estimates the 3D coordinate of each of concerned points. In this case, solutions are narrowed down on the basis of restraint conditions concerning the existence range of concerned points, which are applied from a restraint condition supply unit 24.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] The present invention relates to a coordinate estimation technique. In particular, the present invention relates to a method and an apparatus for estimating three-dimensional coordinates of an object included in a two-dimensional image, and a system using the same.

[0002]

2. Description of the Related Art With the improvement or stabilization of the standard of living of people, funds and time allotted for leisure such as sports are increasing. For example, it is said that the number of golf lovers in Japan exceeds 10 million, and there is generally a high interest in improving their skills. For this reason, one-point lessons using magazines and videos and individual lessons by professionals are held. However, these methods are not satisfactory for everyone due to cost and unidirectionality of information.

On the other hand, interest in the movement of the human body has recently increased in research in various fields such as CG (computer graphics), robotics, and virtual reality. There have been many studies that apply CG to sports and analyze the movement of the human body. One of such efforts is a three-dimensional coordinate measurement technology using a stereo camera. When this technique is applied to, for example, golf, a user first captures his or her swing with a stereo camera and measures three-dimensional coordinates of various points on the body based on the principle of triangulation. Thereafter, for example, the swing motion can be displayed three-dimensionally.

[0004]

However, this method has the following problems. That is, the entire system is generally expensive, difficult to install and handle, and it takes time to specify corresponding points between a plurality of camera images. Therefore, it is not generally suitable for an application in which an individual wants to easily check a swing outdoors or the like, regardless of a special video studio.

[0005] The present invention has been made in view of such problems, and an object of the present invention is to provide a technique for estimating three-dimensional coordinates of a human body or the like by an inexpensive and easy method. Another object of the present invention is to provide a technique for estimating three-dimensional coordinates from one two-dimensional image, such as an image of one camera.

[0006]

A coordinate estimating apparatus according to the present invention comprises: a unit for acquiring a two-dimensional image including an object; and a coordinate estimating apparatus for acquiring the object based on a positional constraint unique to the object. And a unit for estimating the three-dimensional coordinates of the plurality of points of interest taken in step (a).

Here, the "two-dimensional image" may be an arbitrary image, for example, an artificial image such as a CG image, in addition to a camera image. The “target” may be a subject such as a human body or an object included as a part of the image, a predetermined area included in a CG or manga image, or the entire image may be the target. The “positional constraint condition” is a condition that acts in a direction that limits the range of the three-dimensional coordinates of a certain point of interest, and refers to a condition related to an object. The “three-dimensional coordinates” may be world coordinates or other coordinates, or may be coordinates determined from a camera viewpoint of an image.

Generally, in order to specify three-dimensional coordinates of a point of interest from a two-dimensional image, images from at least two directions are required. However, according to the coordinate estimating device of the present invention, in order to use the knowledge about the target, that is, the constraint condition, the possible range of the three-dimensional coordinates of the point of interest is limited,
Three-dimensional coordinates can be estimated. The apparatus may further include a calibration unit for deriving a conversion rule between the in-screen coordinates of the two-dimensional image and the three-dimensional coordinates. As an example of the coordinates in the screen, there are vertical and horizontal numbers of pixels. However, in this calibration stage, it is not necessary to specify the depth component of the three-dimensional coordinates, that is, the distance from the viewpoint of the image over the entire coordinates in the screen.

The estimating unit sequentially applies the constraint conditions starting from one of the plurality of points of interest whose three-dimensional coordinates are determined based on the result of the processing by the calibration unit, thereby obtaining another point. The three-dimensional coordinates of the point of interest may be sequentially estimated. As described above, it is generally impossible to specify the depth component of the three-dimensional coordinates using only the in-screen coordinates as a clue. However, for example, when a rectangular parallelepiped having a known size is placed on the floor and its three sides are determined in the world coordinate system, at least the equation of the floor in the world coordinate system is determined. Next, if one mark is put on the floor and this is projected on the screen, the three-dimensional coordinates of the point are determined as the intersection of the direction of the line of sight toward the mark and the floor plane. Such a singular point whose three-dimensional coordinates are determined depending on the calibration method is a candidate for the “starting point”.

The unit for estimating may acquire connection information between the plurality of points of interest based on a processing order of the plurality of points of interest. The processing order includes, for example, the order specified by the user, and the order in which the present apparatus instructs the user.

[0011] The apparatus may further comprise an initialization unit for acquiring a distance between the points of interest, that is, a relative distance or an absolute distance, when the depths of the plurality of points of interest are known values. The states include, for example, a state in which the points of interest are on a known reference plane, a state in which the points are kept at a known distance from the plane, and a state in which the points are on a known curved surface. In any case, if the depth is known, the conversion rule between the screen coordinates and the three-dimensional coordinates can be uniquely specified, so that the three-dimensional coordinates of each point of interest are known. As a result, the distance between arbitrary points of interest, in this case, the distance in the three-dimensional coordinate system is determined.

The apparatus may include a control unit for displaying the positions of the plurality of points of interest three-dimensionally based on the result of the estimation. At this time, a graph reflecting the three-dimensional coordinates and the connection information of the plurality of points of interest may be displayed, or the graph may be displayed as an animation.

On the other hand, a coordinate estimating method according to the present invention includes a step of obtaining a two-dimensional image including an object, and a method of estimating a plurality of objects of the object based on positional constraints unique to the object. Estimating the three-dimensional coordinates of the point. here,
This constraint condition may be employed to provide for estimation of the depth relationship between the plurality of points of interest. When the object is a human body, the constraint condition may be determined in consideration of the possible angle of the joint.

On the other hand, the coordinate estimating system of the present invention comprises a client and a server. This server is
A unit for receiving images sent from the client,
A unit for estimating the three-dimensional coordinates of the point of interest of the target included in the image; and a unit for returning information related to the estimation result to the client. This information
It may include data for visualization generated based on the three-dimensional coordinates of the point of interest. The data for the visualization is:
It may include a graph in which the three-dimensional coordinates of the point of interest and the connection information are reflected.

The features of the coordinate estimating apparatus of the present invention may be introduced into a coordinate estimating method or a coordinate estimating system, and vice versa. That is, the features and configurations of these devices, methods, and systems can be freely exchanged by those skilled in the art. When the processing of the present invention is performed in the form of a program, the program may be provided by being stored in a computer-readable recording medium.

[0016]

FIG. 1 shows the configuration of a coordinate estimating apparatus according to an embodiment. FIG. 2 is a flowchart showing the outline of the processing performed by the coordinate estimating apparatus 10. Roughly speaking, the coordinate estimating apparatus 10 takes in an image from the outside and acquires or estimates three-dimensional coordinates at a number of points of interest taken by a subject in the image. Here, a human body will be described as an object, and main joints of the human body will be described as points of interest.

The image input unit 12 is an interface for inputting an image captured by a digital camera or the like, or is a camera itself. Here, as an example, it is assumed that the image input unit 12 is a camera. The image input unit 12 first inputs an image (S10 in FIG. 2).

Next, calibration is performed by the calibration unit 14 (S12). FIG. 3 shows a cubic frame 44 of known size placed in the screen 40 for calibration. Frame 44
Is placed on the floor surface 42. Here, three directions defined by the sides of the frame 44 are defined as a world coordinate system (Xw, Yw, Zw). The final purpose of the calibration is to obtain a conversion rule between the world coordinate system and the coordinate system in the screen 40 of the camera which is the image input unit 12. However, since the derivation method itself is known, only an outline will be given below.

First, a camera coordinate system is defined to specify a conversion rule. In the camera coordinate system, the viewing direction of the camera is Z
c, and two directions perpendicular thereto are determined as Xc and Yc. The coordinates in the screen are defined as (x, y, 1). As the coordinates, for example, vertical and horizontal pixel numbers can be used. Now, the point of the camera coordinates (Xc, Yc, Zc) is the screen coordinates (x, y,
If projected to 1), then x = f.Xc / Zc, y = f.Yc / Zc. Here, f is the focal length of the camera. If this equation is rewritten using a matrix, sm = P ・^M〜c (Equation 1). Here, m = (x, y, 1) ^T , M ^c = (X
c, Yc, Zc, 1) ^T , s is a scalar, and P is

[0020]

(Equation 1)It is. Also, the point Mc in the camera coordinate system and the world coordinate system
Between the coordinates Mw of the point^{~ C}= DM^{~ W}  There is a relationship. Where D is a three-dimensional Euclidean transformation
This is a matrix that represents a permutation, and is a combination of rotation and translation. did
Therefore, from equation 1, sm = PM^{~ C}= P ・ D ・ M^{~ W}  Becomes Here, if the projection matrix Pw is newly set as Pw = PD, sm = Pw · M^{~ W} (Equation 2) This is the conversion rule between on-screen coordinates and world coordinates.
You. Screen for 5 points in theory to determine Pw
It is known that the relationship between interior coordinates and world coordinates should be checked.
Here, five or more of the vertices of the frame 44 in FIG.
Use for With the above, the calibration is completed.
The data is stored in the preparation data storage unit 18.

It should be noted here that the scalar s in equation (2) is not generally determined. This is because the depth of a point in the same line of sight from the camera cannot be determined. For this reason,
In this embodiment, as will be described later, a procedure is introduced in which three-dimensional coordinates are completely determined only from calibration, that is, the coordinates of each point of interest are determined starting from a singular point whose depth is also completely determined.

Subsequently, an image of the human body is input (S14 in FIG. 2), and initialization is performed by the initialization unit 16 (S16). The initial setting refers to a process of acquiring in advance the absolute distance between the target points of interest of the target object for later coordinate estimation. FIG. 4 shows an image input for an initial setting relating to the human body. Here, the person stretches his back, reaches for his hand and stands upright. Although the details of the attention point will be described later, it is assumed here that the attention points other than the thumb and the little finger are on the same plane. In this state, since the toe is on the floor surface 42 and the expression of the floor surface 42 has been determined, the three-dimensional coordinates of the toe are specified from the intersection of the line of sight direction and the expression of the floor surface 42. can do. In other words, the toes are initially set
And in the process of the next estimation, it is used as the starting point of processing.

In the initial setting, first, the three-dimensional coordinates of the toe and the little toe are obtained. Next, the three-dimensional coordinates of the ankle are determined according to the distance from the finger. Hereinafter, since it is assumed that the other points of interest are on the same plane as the ankle, the absolute distance in the three-dimensional coordinate system is determined from the distance on the screen between any two of these points. Thus, the initial setting is completed, and the inter-joint distance data relating to the human body is stored in the preparation data storage unit 18.

Next, the process proceeds to an actual estimation process. Here, during the motion of the golf swing, the three-dimensional coordinates of each point of interest at an important timing are estimated, and a three-dimensional graph having each point of interest as a node is generated and displayed. FIG. 5 shows a detailed procedure of the estimation (S20) of FIG.
First, the user specifies a posture to be processed via the posture specifying unit 22 (S200 in FIG. 5), and inputs the image (S18 in FIG. 2). Here, various postures are hierarchically designated, such as "address" of "golf", "takeback" of "golf", and "impact" of "serve" of "tennis".

Next, the restraint condition supply unit 24 specifies an optimum restraint condition based on the specified posture (S20).
2), provide this to the estimation unit 20. For example, in the case of a golf address, one constraint condition is that "the knee joint is located forward of the ankle". The constraint condition is determined from the posture of the human body, the angle that the joint of the human body can take, and other parameters unique to the human body. Therefore, it is desirable that the constraint condition supply unit 24 previously holds as many such data as possible.

Next, a starting point for specifying three-dimensional coordinates is determined (S204). Here, one of the above-mentioned thumb or little finger is used. One of the features of this embodiment is that three-dimensional coordinates are gradually determined from a point of interest close to the starting point by applying a constraint condition sequentially from the starting point.

FIG. 6 shows the distribution of points of interest whose three-dimensional coordinates are to be estimated in the swing address posture, and FIG. 7 shows the order in which the three-dimensional coordinates of the point of interest are specified in order from the starting point.

As shown in FIG. 6, a number of points of interest are taken on the human body. Now, considering the right half of the human body, the three-dimensional coordinates of the point of interest of the human body are determined in the following order starting from the right thumb 130 or the right little finger 128 as shown in FIG.

Right ankle 132 → right knee joint 124 → right hip joint 120 → belly 114 → right shoulder 106 → right elbow 110 → right hand 11
8 → Club Head 140 Similarly, considering the left half body, the following order is determined starting from the left thumb 138 or the left little finger 136.

Left ankle 134 → Left knee joint 126 → Left hip joint 122 → Belly 114 → Left shoulder 108 → Left elbow 112 → Left hand 11
6 → club head 140 Note that the apex 100 of the head from the neck 104 is determined in the order of the right shoulder 106 and the left shoulder 108 → the neck 104 → the nose 102 → the apex 100. The belly 114 is determined by considering the positions of both the left and right hip joints, the club head 140 is determined by considering the positions of both the left and right hands, and the neck 104 is determined by considering the positions of the right shoulder 106 and the left shoulder 108.

It is now assumed that the three-dimensional coordinates of the starting point are determined and the coordinates are stored in the coordinate data storage unit 28. The user specifies a point of interest to be processed next via the point of interest specifying unit 26 (S206). For example, the attention point designation unit 26 may prompt the user to designate in the order of the arrows in FIG. 7, or the user may designate the attention point while inputting the name of the attention point in his or her favorite order. The designation is performed by clicking the position of the point of interest on the screen. As a result, the clicked coordinates in the screen are obtained.

The coordinate estimating apparatus 10 needs to know which point in FIG. 6 or FIG. 7 the point of interest currently clicked by the user falls on. Thus, for example, if the user seems to have clicked on the wrong attention point, the attention point designation unit 26 may issue a warning to the user. Whether it is wrong may be that the coordinates of the point of interest to be clicked on the screen are significantly different from the assumed values or that it is unnatural. For example, if the coordinates clicked on the right body are further left than the coordinates clicked on the input left body, there is a possibility of an input error.

In any case, when a point of interest after the starting point is designated (S206), the estimation unit 20 applies a constraint condition (S208) and estimates the coordinates of the point of interest (S210).

FIG. 8 shows the principle of estimating three-dimensional coordinates.
As described above, the parameter s in Equation 2 is not generally determined. For example, when obtaining the coordinates of the right knee joint 124 based on the coordinates of the right ankle 132 already obtained, the absolute distance r between the two is known at the stage of initialization. Therefore, a sphere having a radius r centered on the right ankle 132 is assumed, and the sphere and a straight line (Xc, Yc, Z) in the line of sight toward the right knee joint 124 are assumed.
c) = t (l, m, n) (where l, m, n are constants,
When the intersection of (t is a variable) is obtained, two solutions P1 and P2 appear as shown in FIG. Here, since the constraint condition determined for the point of interest currently under consideration is “the knee is before the ankle”, it is clear that the position of the right knee joint 124 is the point P1. That is, by introducing the constraint condition, the depth of the point of interest can be specified from one image.

For generalization, a certain point of interest P_iFrom next
Attention point P_{i + 1}The distance of r_iAnd P _iTo P_{i + 1}Position of
Can be described as having the following two solutions:

[0036] ^{_{P i + 1 = f i N}} (P i, r i) and ^{_{f i F (P i, r}} i) ( Equation 3) Here, the solution closer by subscript N cameras, F is more Shows the farther solution.

In FIG. 7, when estimating the position of the next point of interest from one point of interest, which of N and F is adopted is indicated by "N" or "F" on each route. As long as the point of interest remains (S212Y in FIG. 5), the coordinates of each point of interest can be estimated while employing one of the solutions “N” or “F”. Note that the following cases should be noted in the series of estimation. Coordinates estimated from hip joint and belly right hip joint 120 and left hip joint 12
The midpoint of the coordinates estimated based on 2 is taken. Consider shoulder-to-neck balance and do not use the absolute distance between points of interest. The z coordinate of the neck is the average of those of the shoulders, and x, y
Determine the coordinates.

Thus, the processing by the estimation unit 20 is completed. The three-dimensional coordinates of each point of interest are stored in the coordinate data storage unit 28.

Next, display on the display device 32 is performed by the display control unit 30 based on the estimated three-dimensional coordinates (S22 in FIG. 2). Since the coordinate estimation device 10 obtains connection information between the points of interest from the order of designation in the point-of-interest designation unit 26, the display control unit 30 determines the necessary points of interest so that the skeleton of the human body is displayed in a natural form A three-dimensional graph of the human body is created by connecting the edges with edges. FIG. 9 shows the address attitude generated and displayed in this manner. As can be understood from the figure, a natural three-dimensional graph can be generated from one image by effectively utilizing the constraint conditions.

Although the address posture has been described above, the same coordinate estimation can be actually performed for five postures of takeback, top swing, down swing, impact, and follow-through. In this case, the points to note regarding the constraint conditions are as follows: 1) When defining the shoulder from the belly and the elbow from the shoulder, "F"
And, in other cases, the solution of "N", and 2) when defining the club head from the hand, the solution of "N" in address, takeback, downswing and impact, otherwise That is, the solution of "F" is adopted. By acquiring and displaying a series of movements during a swing in three dimensions in this way, it is very easy to grasp and improve the posture, and the equipment for this is simple and inexpensive.

FIG. 10 shows the configuration of a client-server system 300 according to the embodiment. A server 302 for three-dimensional coordinate estimation and a PC 304 on the user side, which is a client, are connected by a network 306.
The server 302 has the configuration of FIG. 1 or its main part implemented. FIG. 11 shows this system 300.
Is shown.

In the figure, first, the client transmits a photographic image of the golf posture of the client (S30).
0), and this is received on the server side (S302). Assume that the images to be sent at this time are three sets of images for calibration, for initial setting, and images for which coordinates are to be actually estimated.

Subsequently, the server 302 performs calibration, initial setting, and estimation by the methods described above, and acquires three-dimensional coordinates related to the posture of the user (S304). Server 30
2 generates a three-dimensional graph based on this (S306),
This is sent back to the client (S308). The PC 304 on the user side receives the transmitted image, particularly the three-dimensional graph (S310), and uses it by a method such as displaying it on the screen.

The above is the outline of the processing. According to this method, the part of the process of estimating the three-dimensional coordinates can be performed by the server 302, so that the burden on the client can be reduced.

Modification Techniques Modification or improvement techniques of the embodiment described above are listed. Although the embodiment requires a calibration step, this is not always necessary. For example, when the coordinate estimating apparatus 10 is installed in a place such as a driving range, it is only necessary to calibrate the first time only once, and then the initial setting and the estimating process may be performed. If the distance to the target is long and the difference in depth between the parts of the target can be ignored, there is room for skipping calibration at all.

When calibration is impossible, as in the case of a golfer's image displayed on a television, the depth is assumed to be constant, and the three-dimensional coordinates of each point of interest are estimated. The estimated value may be modified. For example, in a golf swing, the depth of the club head is significantly different than in other parts. In such a case, assuming a constant depth as described above, an error occurring in the estimated club head coordinates is relatively large. Therefore, it is possible to correct the coordinates by introducing the knowledge that the club head is just off the ground at the time of impact, or by introducing a constraint condition. If you do n’t have an image of the moment of impact,
Interpolate the image at impact from the images before and after it,
The position of the club head at that time may be estimated. The accuracy of coordinate estimation can be improved using the movement of the target object.

In some cases, the initial setting can be skipped. For example, the coordinate estimation device 10 includes a “parameter input unit” for inputting specific information such as the user's gender and height, so that the distance between joints can be estimated to some extent. The estimation process may be performed using the assumed value.

In the embodiment, the coordinate estimating apparatus 10 automatically determines the constraint condition. However, the coordinate estimating apparatus 10 may include a "user setting unit" for inputting a setting by a user. In this case, the user can specify “N” or “F” even when the first target is input to the coordinate estimating apparatus 10, so that three-dimensional coordinates can be correctly acquired. As a general rule, for a point of interest whose coordinates are difficult to estimate, the user may be allowed to input positional information such as a camera or a target using the user interface of the client.

The attention point designation unit 26 designates an attention point under the initiative of the user or the system. There is considerable freedom in the design of this part. For example, since the arrangement of the attention points as shown in FIG. 6 can be determined only by “the golf address attitude”, the user is allowed to click in an arbitrary order, while the attention point designation unit 26 is clicked with the template of the arrangement of the attention points. A configuration may be adopted in which each point of interest is automatically associated with a joint by matching each point.

In the case where the movement has continuity or is close to a certain degree such that the golf ball moves from the golf address posture to the takeback, there is a certain correlation between the relative positions between the points of interest. Therefore, when the attention point and the joint are associated with one posture, the attention point designation unit 26 may predict the association in the next image based on the information.

At this time, a large weight can be given to the maintenance of the angle between the points of interest. For example, in the case of FIG.
Since the angle between the elbow and the hand is considered to be almost constant throughout the motion, it can be predicted based on the knowledge. In any case, by showing the prediction result to the user, the user only needs to correct only the error part.

In the embodiment, the constraint condition between adjacent points of interest, that is, the local constraint condition is determined, but the constraint condition supply unit 24 may hold the global constraint condition. For example, if the belly is more "F", that is, farther from the ankle, the top of the head is considered to be "N", that is, closer, than the belly to maintain body balance. As described above, by using the concept of the balance or the center of gravity of the body, first, the entire posture can be specified without error.

In the embodiment, the target object is a human body, but it may be an arbitrary three-dimensional object. For example, when the “car” is the target object and an image taken from the front left is processed, the user inputs to the posture specifying unit 22 that the vehicle is “viewed from the front left”. Then, the constraint condition supply unit 24 successively determines constraint conditions such as “the right headlight is farther than the left headlight” and “the upper end points of the windshield are farther than the lower end points”. As a point of interest, a point at which a feature can be easily captured to some extent, for example, a salient point or both end points of a curve may be adopted.

Although the display control unit 30 displays the target object in the form of a graph, there are naturally various modifications. For example, it is possible to create a scene where the user moves freely on the screen by modeling the human body to some extent accurately and pasting the texture. With this technology,
The user can enter into any scene like a role playing game.

When the display control unit 30 is used for practicing the movement, the example movement may be displayed in another window or overlaid on the same window. For example,
If the swings of golf professionals of various body shapes are preliminarily converted into a three-dimensional graph and held, and the professional swing of the body shape closest to the user and the swing of the user are displayed in a superimposed manner, the practice effect is enhanced. Further, various measures are conceivable, such as displaying only the difference between the two, highlighting the difference, and displaying only where the difference exceeds a predetermined allowable value. The difference may be that of the coordinates of each point of interest, may be obtained for a point determined from a plurality of points of interest such as joint angles, or may be displayed in a form converted to a difference in muscle strength.

Instead of the difference, the posture to be taken by the user may be instructed based on the difference. For example, when the upper body of the user is biased to the right, an instruction such as “the upper body is a little more to the left” may be displayed on the screen, or a “←” symbol may be displayed at a position of the upper body on the screen. You can put it on the left. Similarly, the instruction may be an instruction relating to muscle strength or other parameters, such as “please release power at takeback”.

In displaying the difference, in addition to displaying the difference on the actual three-dimensional coordinates as it is, for example, the joint angle of the golfer who is a model and the change during the swing of the joint angle of the user are superimposed on one graph. May be displayed. The same applies to the trajectory of the club head.

Similarly, as an option for displaying the difference, the difference may be corrected in consideration of the difference in the body shape and the difference in muscle strength between the user and the model golfer. In other words, the difference can be obtained after correcting the golfer's swing for the user in order to obtain an ideal movement that meets the user's body type or the like. In addition, when unnatural three-dimensional coordinates are newly obtained by this correction, for example, when the club head is “moving in the ground” which is originally impossible during a swing, the third order of the attention point related thereto is The original coordinates may be modified.

In order to increase the estimation accuracy by the estimation unit, the estimation value may be corrected from the continuous image. For example, when it is assumed that the swing surface is within a predetermined plane, the estimated value can be adjusted based on the knowledge or the constraint condition. Further, according to the constraint condition, it is possible to detect a portion where the assumption regarding “N” and “F” is likely to be incorrect, and call the user's attention. This series of adjustment processing may be a repetition processing between the present apparatus and the user.

Each time the estimation unit estimates the three-dimensional coordinates of a certain point of interest, it may be sequentially added to the graph shown in FIG. In this case, while the user designates the point of interest, a graph obtained by the processing up to that point is drawn, and the processing contents can be easily confirmed.

When it is necessary to estimate the coordinates of a plurality of images, the position of a point of interest of a new image may be automatically detected based on the position of the point of interest of an already processed image. For this purpose, techniques such as optical flow and image matching can be used.

Although the PC 304 is considered as the client terminal in the client server system 300 of the embodiment, it may be an arbitrary device. For example, L
The form is arbitrary, such as a mobile phone having a CD, a digital television equipped with client software, and a lightweight client having only a viewer function.

In the embodiment, the server generates a graph and sends it to the client. However, in addition to such visualized data, a result obtained by analyzing muscle strength, torque, and the like may be sent. Known inverse kinematics can be used to calculate such parameters.

In the embodiment, the client simply transmits the image to the server. However, the client may assist the server in estimating the coordinates to some extent by an operation such as clicking.

By storing and accumulating various data in the server, the estimation of coordinates can be improved. For example, by searching for and using similar motions and images of the object to be processed from the collected motions and image data, the user's setting work and the like can be reduced.

In the embodiment, golf practice is described as an example. However, the present technology can naturally be used for other uses, for example, sports in general, rehabilitation, martial arts, and butoh. Further, the present technology widely generates a three-dimensional image from a two-dimensional image. For example, a three-dimensional image such as a hologram is created from a single photograph taken by a user, or a three-dimensional CG is generated.
Can also be made. In that case, the constraint conditions for the target object that is the subject in the image may be manually determined, or the system side prepares a combination of a large number of "target templates" and their postures or gaze directions looking at them, The three-dimensional coordinates may be determined by a simple click operation of the user.

When the target object is a human body, the movement torque may be decomposed into muscle components, and which muscle is used may be indicated by different colors. The same processing may be performed on professional movements to facilitate the user's comparative study.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a coordinate estimation device according to an embodiment.

FIG. 2 is a flowchart showing a procedure of processing by the apparatus of FIG. 1;

FIG. 3 is a flowchart illustrating a procedure of a calibration process.

FIG. 4 is a diagram illustrating a posture to be taken by a human body for initial setting.

FIG. 5 is a diagram showing a detailed procedure of an estimation process S20 of FIG. 2;

FIG. 6 is a diagram showing the position of a point of interest to be set on a target human body.

FIG. 7 is a diagram showing an order of estimating coordinates of a point of interest in the embodiment.

FIG. 8 is a diagram showing two solutions that can occur when acquiring three-dimensional coordinates from a two-dimensional image.

FIG. 9 is a diagram illustrating a three-dimensional graph of an object generated as a result of coordinate estimation.

FIG. 10 is a configuration diagram of a client server system according to an embodiment.

11 is a flowchart showing a processing procedure of the client server system of FIG.

[Explanation of symbols]

 Reference Signs List 10 Coordinate estimation device 12 Image input unit 14 Calibration unit 16 Initial setting unit 18 Preparation data storage unit 20 Estimation unit 22 Attitude specifying unit 24 Constraint condition supply unit 26 Attention point designation unit 28 Coordinate data storage unit 30 Display control unit 32 Display device 300 Client server system 302 Server 304 PC

Continuation of the front page (72) Inventor Taku Yukimura 1096-3 Yamaguchi, Tokorozawa-shi, Saitama (72) Inventor Masahiko Murai 4-15-3, Mihama, Urayasu-shi, Chiba (72) Inventor Yoshihisa Shinagawa 5-10 Nishikasai, Edogawa-ku −26−204 F term (reference) 5B050 CA08 EA07 EA28 FA02 5B057 BA02 CA12 CB13 CD14 DA07 DB02

Claims (14)

[Claims] A unit for acquiring a two-dimensional image including an object; and three-dimensional coordinates of a plurality of points of interest taken on the object based on positional constraints unique to the object. And a unit for estimating the coordinate. 2. The coordinate estimating apparatus according to claim 1, further comprising a calibration unit for deriving a conversion rule between the in-screen coordinates of the two-dimensional image and the three-dimensional coordinates. 3. The unit for estimating, by starting from the plurality of points of interest whose three-dimensional coordinates are determined based on the result of processing by the calibration unit, sequentially applying the constraint condition, 3. The coordinate estimating device according to claim 2, wherein three-dimensional coordinates of another point of interest are sequentially estimated. 4. The coordinate estimating apparatus according to claim 1, wherein the estimating unit further acquires connection information between the plurality of points of interest based on a processing order of the plurality of points of interest. 5. The coordinate estimating apparatus according to claim 1, further comprising an initialization unit that acquires a distance between the plurality of points of interest in a state where the depth of the plurality of points of interest is a known value. . 6. The coordinate estimating device according to claim 1, further comprising a control unit that three-dimensionally displays the positions of the plurality of points of interest based on a result of the estimation. 7. A step of acquiring a two-dimensional image including an object, and estimating three-dimensional coordinates of a plurality of points of interest on the object based on positional constraints unique to the object. A method for estimating coordinates, comprising the steps of: 8. The system according to claim 7, wherein the constraint condition is used for estimating a depth relationship between the plurality of points of interest.
The coordinate estimation method described in 1. 9. The coordinate estimating method according to claim 8, wherein the object is a human body, and the constraint condition is determined in consideration of a possible angle of a joint of the human body. 10. A coordinate estimating system comprising a client and a server, wherein the server receives an image transmitted from the client, and a three-dimensional image of a target point of interest included in the image. A coordinate estimating system comprising: a unit for estimating coordinates; and a unit for sending information related to the estimation result back to the client. 11. The coordinate estimating system according to claim 10, wherein the estimating unit estimates the three-dimensional coordinates based on positional constraints unique to the object. 12. The coordinate estimation system according to claim 10, wherein the information includes data for visualization generated based on three-dimensional coordinates of the point of interest. 13. The coordinate estimation system according to claim 12, wherein the data for visualization includes a graph in which three-dimensional coordinates of the point of interest and connection information are reflected. 14. A recording medium on which a program readable by a computer is recorded, the program comprising: a function of acquiring a two-dimensional image including an object; and a positional constraint condition unique to the object. A recording medium characterized by causing a computer to execute a function of estimating three-dimensional coordinates of a plurality of points of interest taken on the object.