JP6723061B2 - Information processing apparatus, information processing apparatus control method, and program - Google Patents

Description

The present invention relates to an information processing device, a method of controlling the information processing device, and a program.

Computer graphics (hereinafter referred to as "CG") are superimposed and presented on the background of a live-action scenery to give the experience to the experience person as if a virtual object exists on the spot. There is a Mixed Reality (MR) technology that allows you to do this. In order to realize a highly realistic experience using this MR technology, it is not enough to simply superimpose and display the CG on the live-action scenery that is the background, and the interaction between the experience person and the CG is not sufficient. It becomes important. Specifically, it is important for the user to interact with the virtual object drawn by CG or operate the virtual object (to experience it as if it were doing). Then, in order to realize such an interaction, information on the three-dimensional shape of the finger of the experiencer who operates the virtual object is required.

Patent Document 1 discloses that a three-dimensional shape of a finger is estimated by a matching process of contour lines of the finger extracted from a stereo image. Further, Patent Document 2 discloses that a three-dimensional shape (angle between joints) of a finger is estimated by inputting a feature amount extracted from a biosignal of a finger joint to a neural network that has undergone learning processing in advance. Has been done.

Japanese Patent No. 5574852 Japanese Patent No. 5252432

However, the technique described in Patent Document 1 cannot estimate the detailed shape of the portion other than the three-dimensional position of the contour of the finger. Further, with the technique described in Patent Document 2, it is possible to estimate a detailed three-dimensional shape of a portion other than the contour, but a feature amount that is significantly different from the learning pattern such as an error is included in the biomedical signal of the finger joint is input. Then, a three-dimensional shape that is very different from the actual one may be estimated.

The present invention has been made in view of the above problems, and an object thereof is to provide a technique for improving the estimation accuracy of the three-dimensional shape of a finger.

An information processing apparatus according to the present invention which achieves the above object,
An information processing apparatus for estimating a three-dimensional shape of an articulated object,
Image acquisition means for acquiring image data of the articulated object;
Pre-stage processing means for performing pre-stage processing on the image data,
Reliability calculation means for calculating a reliability for the result of the preceding stage processing,
Estimation condition determining means for determining the estimation condition of the three-dimensional shape based on the reliability,
Shape estimating means for estimating the three-dimensional shape based on the estimation condition,
Equipped with
The estimated condition determining means, characterized that you determine the shape estimation means performs any estimation process estimation processing or schematic three-dimensional shape of the detailed three-dimensional shape as the estimated condition.

According to the present invention, it is possible to improve the estimation accuracy of the three-dimensional shape of a finger.

It is a figure showing an example of composition of an information processing system concerning one embodiment of the present invention. It is a figure showing an example of hardware constitutions of an information processor concerning one embodiment of the present invention. It is a figure showing an example of functional composition of an information processor concerning one embodiment of the present invention. 6 is a flowchart illustrating a procedure of processing performed by the information processing apparatus according to the first embodiment. It is a figure which shows typically the extraction processing of the peripheral region of a finger|toe which concerns on 1st Embodiment. It is a figure which shows typically the example of a calculation process of the three-dimensional position of the joint which concerns on 1st Embodiment. 6 is a flowchart showing an example of reliability calculation processing executed by the information processing apparatus according to the first embodiment. 6 is a flowchart showing a detailed three-dimensional shape estimation process performed by the information processing apparatus according to the first embodiment. It is a figure which shows typically the rough estimation method of a three-dimensional shape which the information processing apparatus which concerns on 1st Embodiment implements. 3 is a flowchart showing a schematic three-dimensional shape estimation process executed by the information processing apparatus according to the first embodiment. It is a flow chart which shows a procedure of processing which an information processor concerning a 2nd embodiment performs. 9 is a flowchart showing an example of reliability calculation processing executed by the information processing apparatus according to the second embodiment. It is a figure which shows typically the example of the calculation process of the three-dimensional position of the joint which the information processing apparatus which concerns on 3rd Embodiment implements. It is a flowchart which shows the procedure of the process which the information processing apparatus which concerns on 4th Embodiment implements. It is a figure which shows the site|part of the finger which concerns on 5th Embodiment typically.

Hereinafter, embodiments will be described with reference to the drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

(First embodiment)
<Structure of information processing system>
FIG. 1 is a diagram showing a configuration example of an information processing system according to the present embodiment. The information processing system includes an information processing device 100, a head mounted display 101 (hereinafter, described as “HMD”), and a distance sensor 102. The HMD 101 includes image capturing units 103 and 104 and display units 105 and 106, and displays captured images of the image capturing units 103 and 104 on the display units 105 and 106. The distance sensor 102 generates a distance image having the distance to the subject as a pixel value. The information processing apparatus 100 acquires the images captured by the imaging units 103 and 104 and the distance image captured by the distance sensor 102, and estimates the three-dimensional shape of a finger (articulated object) included in the subject. The present invention is not limited to the above system configuration. For example, the HMD 101 is not essential, and the image capturing units 103 and 104 may be configured to capture images of fingers.

<Hardware configuration of information processing device>
FIG. 2 is a diagram showing an example of the hardware configuration of the information processing apparatus 100 according to the present embodiment. The information processing device 100 is configured to include a CPU 201, a RAM 202, a ROM 203, a HDD 204, an interface 205, and a system bus 206, and is connected to the HMD 101 and the distance sensor 102.

The CPU 201 uses the RAM 202 as a work memory to read and execute a program stored in the ROM 203, and centrally controls each component described below via the system bus 206. As a result, various processes described below are executed.

The HDD 204 has a role as a secondary storage device. The CPU 201 can read data from the HDD 204 and write data to the HDD 204. Note that the secondary storage device may be a storage device such as an optical disk drive other than the HDD. The interface 205 exchanges data with external devices such as the HMD 101 and the distance sensor 102. It should be noted that although the constituent elements of the information processing apparatus 100 exist in addition to the above, they are not the main subject of the present invention, and thus the description thereof is omitted.

<Functional configuration of information processing device>
FIG. 3 is a diagram showing an example of the functional configuration of the information processing apparatus 100 according to the present embodiment. The information processing device 100 includes an image acquisition unit 301, a pre-stage processing unit 302, a reliability calculation unit, an estimation condition determination unit 304, and a shape estimation unit 305.

The image acquisition unit 301 acquires distance image data from the distance sensor 102, and also acquires captured image data from the imaging units 103 and 104, respectively. The pre-processing unit 302 detects a finger area in the distance image, extracts a peripheral area of the finger, and calculates a three-dimensional position of a joint that constitutes the finger. The reliability calculation unit 303 uses the distance image to calculate the reliability of the three-dimensional position of the joint that constitutes the finger. The estimation condition determining unit 304 determines the estimation condition by the shape estimating unit 305 based on the reliability calculated by the reliability calculating unit 303. The shape estimation unit 305 estimates the three-dimensional shape of the finger based on the estimation condition determined by the estimation condition determination unit 304.

<Processing of information processing device>
FIG. 4 is a flowchart showing a procedure of processing executed by the information processing apparatus 100 according to this embodiment. In step S401, the image acquisition unit 301 acquires captured images from the imaging units 103 and 104, respectively. Hereinafter, the captured images acquired from the capturing units 103 and 104 may be collectively referred to as “stereo image”. In step S402, the image acquisition unit 301 acquires a distance image from the distance sensor 102.

In step S403, the upstream processing unit 302 detects the finger region in the distance image acquired in step S402. Any method can be used to detect the finger region. For example, assuming that the user holds his/her finger in front of the distance sensor 102 and uses it, an area having a small distance value indicated by the distance image can be detected as a finger area. In addition, a finger polygon is estimated from the stereo image acquired in step S401 using a schematic three-dimensional shape estimation method described later, and the finger polygon is projected on a distance image to calculate a finger area in the distance image. You can also do it. Since the polygon projection process is not the main object of the present invention, its explanation is omitted.

In step S404, the pre-processing unit 302 extracts the peripheral area of the finger using the information of the finger area detected in step S403. Details will be described later. In step S405, the pre-processing unit 302 calculates the three-dimensional position of the joint forming the finger using the distance image acquired in step S402 and the information on the peripheral area of the finger extracted in step S404. Details will be described later.

In step S406, the reliability calculation unit 303 uses the distance image acquired in step S402 to calculate the reliability of the three-dimensional position of the joint calculated in step S405. Details will be described later.

In step S407, the estimation condition determining unit 304 determines whether the reliability calculated in step S406 is equal to or higher than a preset threshold value. If the result of this determination is that the reliability is greater than or equal to the threshold value, the flow proceeds to step S408. On the other hand, if it is determined that the reliability is lower than the threshold value, the process proceeds to step S409.

In step S408, the shape estimation unit 305 estimates the detailed three-dimensional shape of the finger from the three-dimensional position of the joint calculated in step S406 using the distance image acquired in step S402. Details will be described later. In step S409, the shape estimation unit 305 estimates a schematic three-dimensional shape from the stereo image acquired in step S401. Details will be described later. With this, the series of processing in FIG. 4 is completed.

<Extraction processing of peripheral area of fingers>
Here, FIG. 5 is a diagram schematically showing an example of the extraction process of the peripheral region of the finger. In the extraction process of the peripheral region of the finger, the peripheral region 502 of the finger region 501 detected in step S403 of FIG. 4 is extracted. As the peripheral area 502, for example, a rectangular area circumscribing the finger area 501 can be extracted. The extraction processing of the peripheral area 502 of the finger in the present invention is not limited to this, and any extraction processing can be performed. For example, a quadrangular area centered on the center of gravity of the finger area 501 may be extracted as the peripheral area 502. Further, the peripheral area of the fingers does not have to be a quadrangle, and may have any shape.

<Calculation method of three-dimensional position of finger joint>
Further, FIG. 6 is a diagram schematically showing a method of calculating the three-dimensional position of the finger joint according to the present embodiment. The pre-processing unit 302 according to the present embodiment calculates the three-dimensional position of the finger joint 601 using a known neural network technique.

The distance value indicated by the distance image in each pixel of the peripheral region 502 is input to each node of the input layer of the neural network. At this time, the peripheral area 502 is resized according to the number of nodes in the input layer. The three-dimensional position of the finger joint 601 is output to the output layer. Specifically, each element of the one-dimensional vector in which the x-coordinate, y-coordinate, and z-coordinate of each joint are arranged is output to each node in the output layer. Here, when the number of joints is n, the number of nodes in the output layer is n×3.

The neural network according to this embodiment is not limited to the above. For example, the neural network may output the angle between the joints and calculate the three-dimensional position of the finger joint 601 again from the angle. Further, the neural network shown in FIG. 6 has one intermediate layer, but a multilayered neural network may be used.

In addition, the neural network in the present embodiment is assumed to be subjected to learning processing in advance so as to have the above-mentioned input/output relationship. Since the learning method of the neural network is not the main purpose of the present invention, its explanation is omitted.

In addition, the input/output data of the neural network may be either absolute coordinate values (coordinate values when a certain point in the real space is the origin) or relative coordinate values (coordinate values when a certain element in the data is the origin). It doesn't matter which one. However, it is necessary to unify whether the input/output data is an absolute value or a relative value by the learning process in advance and the three-dimensional position calculation process of the joints forming the fingers in step S405.

<Calculation process of reliability>
FIG. 7 is a flowchart showing a procedure of reliability calculation processing executed by the information processing apparatus 100 according to the present embodiment. Hereinafter, the procedure of the reliability calculation processing according to the present embodiment will be described with reference to FIG. 7. The process of FIG. 7 is the detail of the process of S406.

In step S701, the reliability calculation unit 303 generates a finger polygon representing the three-dimensional shape of a thick finger from the finger joint 601 whose three-dimensional position is calculated in step S405. A method for generating a finger polygon is arbitrary, but for example, a finger polygon is generated by applying a predetermined three-dimensional figure (a cylinder for a finger, an elliptic cylinder for a palm, etc.) to each joint. be able to.

Here, it is desirable to determine the size of the three-dimensional figure fitted to each joint based on the actual length and thickness of the finger. Therefore, the length and the thickness of the finger may be input from the outside, or the stereo image or the distance image acquired by the image acquisition unit 301 may be used for calibration.

In step S702, the reliability calculation unit 303 projects the finger polygon generated in step S701 on the distance image. In step S703, the reliability calculation unit 303 calculates the difference between the distance image acquired in step S402 and the distance of the finger polygon projected in step S702. Specifically, for each pixel of the distance image, the difference between the distance of the finger polygon projected on the pixel and the pixel value representing the distance of the distance image is calculated. Here, an arbitrary scale can be used as the difference, and for example, the mean square error of the distance can be calculated as the difference.

In step S704, the reliability calculation unit 303 converts the difference calculated in step S703 into reliability. As the reliability, an arbitrary measure having a negative correlation with the difference can be used, and for example, the reciprocal of the difference can be calculated as the reliability. With that, the series of processing in FIG. 7 is completed.

<Detailed three-dimensional shape estimation processing>
FIG. 8 is a flowchart showing the procedure of a detailed three-dimensional shape estimation process executed by the information processing apparatus 100 according to this embodiment. The process of FIG. 8 is a detail of the process of S408. In the detailed three-dimensional shape estimation processing, the precision of the finger polygon is improved by the known model fitting processing. In the model fitting process, first, similarly to the process of the reliability calculation unit 303, a finger polygon is generated and the difference between the distance image and the distance between the finger polygon is calculated.

Then, the polygon parameters described later are changed so that the difference becomes smaller. By repeating these processes, the three-dimensional shape is made highly accurate. Hereinafter, the procedure of a detailed three-dimensional shape estimation process will be described with reference to FIG. Since each processing of steps S701 to S703 is the same as each processing of steps S701 to S703 described with reference to FIG. 7, description thereof will be omitted.

In step S801, the shape estimation unit 305 determines whether the difference calculated in step S703 is less than or equal to a preset threshold value. Here, any value can be used as the threshold value, but in consideration of the fact that an error of about the thickness of the finger occurs in the rough estimation process of the three-dimensional shape described later, the threshold value is set to It is desirable to set the degree (for example, 1 cm). By setting such a threshold value, the detailed three-dimensional shape of the finger calculated in step S408 can be adjusted to be more accurate than the rough three-dimensional shape of the finger calculated in step S409. .. If the result of this determination is that the difference is less than or equal to the threshold value, the detailed three-dimensional shape estimation process ends. On the other hand, if the difference is not less than or equal to the threshold value, the process proceeds to step S802, and the processes of steps S701 to S802 are repeated until the difference becomes less than or equal to the threshold value.

In step S802, the shape estimation unit 305 changes the polygon parameter so that the difference calculated in step S703 becomes smaller. Here, the polygon parameter refers to the three-dimensional position of the joint, the length of the finger, the thickness of the finger, the size of the palm, and the like. A known optimization method can be used as a method for changing the polygon parameters. The present embodiment is not limited to the method of using the model fitting process for the detailed estimation of the three-dimensional shape. It is possible to estimate a detailed three-dimensional shape by using an arbitrary method having a higher accuracy (detailedness) regarding thickness, as compared with a schematic three-dimensional shape estimation process described later.

Thus, the series of processing shown in FIG. 8 is completed.

<Schematic three-dimensional shape estimation process>
FIG. 9 is a diagram schematically showing a schematic three-dimensional shape estimation method implemented by the information processing apparatus 100 according to the present embodiment. Further, FIG. 10 is a flowchart showing a procedure of a schematic three-dimensional shape estimation process executed by the information processing apparatus 100 according to the present embodiment. The process of FIG. 10 is the detail of the process of S409. Hereinafter, a schematic three-dimensional shape estimation process according to the present embodiment will be described with reference to FIGS. 9 and 10.

In step S1001, the shape estimation unit 305 detects the finger regions 903 and 904 in the captured images 901 and 902 acquired in step S401. Although any method can be used for the extraction processing, for example, the color of the finger can be registered in advance and the pixel of that color can be detected as the finger area.

In step S1002, the shape estimation unit 305 extracts points (hereinafter, referred to as “contour points”) that form the contours of the finger regions 903 and 904 detected in step S1001. Although any method can be used as a method for extracting contour points, for example, intersection points between the finger regions 903 and 904 and horizontal lines (or epipolar lines) drawn at regular intervals can be extracted as contour points. it can.

In step S1003, the shape estimation unit 305 calculates the corresponding points of the stereo image by the matching process of the contour points extracted in step S1002. Although any method can be used as the matching process, for example, as described in Patent Document 1, a matching process based on the features of the contour points can be used.

In step S1004, the shape estimation unit 305 calculates the distance between the contour points by applying a known triangulation technique to the corresponding points calculated in step S1003. Since the triangulation technique is not the main object of the present invention, the description is omitted.

In step S1005, the shape estimation unit 305 connects the contour points detected in step S1002 to form a polygon. In polygonization, the distance between the contour points calculated in step S1004 is used to generate a three-dimensional polygon (not two-dimensional). This three-dimensional polygon is a polygon that connects contour points and has no thickness. Therefore, the detailed shape of the actual finger is not reflected. However, the distance near the contour is highly accurate, and the area other than the contour also has an approximate distance. In addition, it is more robust than the detailed three-dimensional shape calculation process described above. Thus, the series of processing in FIG. 10 is completed.

The method for calculating a schematic three-dimensional shape in the present invention is not limited to the above method, and a schematic three-dimensional shape can be calculated by any method. For example, the distance value represented by the distance image may be polygonized to be used as a rough three-dimensional shape. Further, in the above, a polygon having no thickness is generated as a schematic three-dimensional shape, but the present invention is not limited to this. It is possible to generate a schematic three-dimensional shape by an arbitrary method that is lower in accuracy (detailedness) with respect to the thickness than the fitting processing described above.

As described above, in the present embodiment, first, the neural network is used to calculate the three-dimensional position of the finger joint 601 and then the reliability of the obtained three-dimensional position is calculated. Then, the estimation condition regarding the thickness (detail) is switched according to the reliability. Specifically, when the reliability is high, a detailed (thickness is considered) three-dimensional shape is estimated by the model fitting process, and when the reliability is low, a rough (thickness is determined by the contour matching process). Estimate the three-dimensional shape.

Here, with the combination of the neural network and the model fitting process, it is possible to calculate the three-dimensional shape in consideration of the thickness of the finger. However, when the range image includes an error, a three-dimensional shape that is significantly different from the actual one may be calculated (low robustness). On the other hand, in the polygon processing of the contour points, the thickness of the fingers cannot be taken into consideration, but it is unlikely that a three-dimensional shape that is significantly different from the actual one will be calculated (high robustness). In the present embodiment, by properly using the two three-dimensional shape estimation methods described above according to the reliability, it is possible to prevent the estimation of a three-dimensional shape that is significantly different from the actual one, and to make the three-dimensional shape of the finger as detailed as possible. Can be estimated.

In addition, although the example which switches two estimation conditions according to reliability is demonstrated above, this invention is not limited to it. An arbitrary number of estimation conditions of two or more can be switched according to the reliability. Further, in the above, an example in which the reliability is calculated first and then the estimation condition is switched has been described, but the present invention is not limited thereto. It is also possible to first perform the estimation under both estimation conditions and then decide which estimation result to use depending on the reliability. Moreover, although the example in which the reliability is calculated from the three-dimensional position of the joint and the estimation condition is determined according to the reliability has been described above, the present invention is not limited to this. The estimation condition may be determined directly from the three-dimensional position of the joint.

According to the present embodiment, the estimation accuracy of the three-dimensional shape of the finger is improved because the three-dimensional shape of the finger can be estimated as detailed as possible while suppressing the estimation of the three-dimensional shape that is significantly different from the actual one. It becomes possible.

(Second embodiment)
Next, a second embodiment will be described. In the first embodiment, the case where the three-dimensional position of the finger joint is calculated by inputting the distance information into the neural network has been described as an example. On the other hand, in the present embodiment, a case will be described as an example in which a three-dimensional position of a finger joint is calculated by inputting a silhouette image representing a finger region to the neural network. As described above, the present embodiment and the first embodiment are mainly different in that a silhouette image is input to the neural network. Therefore, in the description of the present embodiment, the same parts as those in the first embodiment will be denoted by the same reference numerals as those in FIGS. 1 to 10, and detailed description thereof will be omitted.

<Processing of information processing device>
FIG. 11 is a flowchart showing a procedure of processing executed by the information processing apparatus 100 according to this embodiment. Below, the procedure of the process of the present embodiment will be described with reference to FIG. 11. Note that the processes of steps S401, S404, and S409 are the same as the processes described with reference to FIG. 4 in the first embodiment, so description thereof will be omitted.

In step S1101, the pre-processing unit 302 detects the finger regions 903 and 904 (multi-joint regions) in the captured images 901 and 902 acquired in step S401. The detection method is the same as the above-described detection method by the shape estimation unit 305, and thus description thereof will be omitted.

In step S1102, the pre-stage processing unit 302 generates detected silhouette images from the finger regions 903 and 904 detected in step S1101. The detected silhouette image is an image in which the pixel value of the pixel detected as the finger region is 1 and the pixel value of the pixel not detected is 0.

In step S1103, the pre-processing unit 302 uses the detected silhouette image generated in step S1102 and the information of the peripheral region 502 of the finger extracted in step S404 to calculate the approximate three-dimensional position (relative coordinates) of the finger joint 601. Value) is calculated. Details will be described later.

In step S1104, the pre-stage processing unit 302 converts the schematic three-dimensional position of the finger joint 601 generated in step S1103 into absolute coordinate values. Although the conversion method is arbitrary, for example, the distance at the center of gravity of the finger regions 903 and 904 can be acquired from the distance sensor 102, and the distance can be added as an offset to be converted into an absolute coordinate value. Alternatively, a finger polygon having no thickness may be generated by the above-described schematic three-dimensional shape estimation method, and the distance between the finger polygons at the centers of gravity of the finger regions 903 and 904 may be added as an offset.

In step S1105, the reliability calculation unit 303 calculates the reliability of the three-dimensional position of the joint calculated in step S1104, using the detected silhouette image generated in step S1102. Details will be described later.

In step S1106, the estimation condition determining unit 304 determines whether the reliability calculated in step S1105 is equal to or higher than a preset threshold value. If the result of this determination is that the reliability is greater than or equal to the threshold value, the flow proceeds to step S1107. On the other hand, if it is determined that the reliability is lower than the threshold value, the process proceeds to step S409.

In step S1107, the shape estimation unit 305 estimates the detailed three-dimensional shape of the finger from the three-dimensional position of the joint calculated in step S1104, using the detected silhouette image generated in step S1102. Details will be described later.

In the calculation processing of the three-dimensional position of the finger joint 601 in the present embodiment, the detected silhouette image is input to the neural network. Specifically, the pixel value of each pixel of the detected silhouette image is input to each node of the input layer. The output layer is the same as that in the first embodiment, and thus the description is omitted. It is assumed that the neural network according to the present embodiment performs learning processing in advance so that when the detected silhouette image is input, the three-dimensional position of the finger joint is output. Further, it is not always necessary to input both the finger areas 903 and 904 to the neural network in the present embodiment, and only one of them may be input.

<Calculation process of reliability>
FIG. 12 is a flowchart showing a procedure of reliability calculation processing executed by the information processing apparatus 100 according to the present embodiment. Hereinafter, the procedure of the reliability calculation processing according to the present embodiment will be described with reference to FIG. The process of FIG. 12 is the detail of the process of S1105. Note that the processing of step S701 is the same as that of the first embodiment, so description thereof will be omitted.

In step S1201, the reliability calculation unit 303 projects the finger polygons generated in step S701 on the captured images 901 and 902. In step S1202, the reliability calculation unit 303 generates a polygon silhouette image from the projection result of step S1201. The polygon silhouette image is an image in which the pixel value of the pixel on which the finger polygon is projected is 1 and the pixel value of the pixel which is not projected is 0.

In step S1203, the reliability calculation unit 303 calculates the difference between the polygon silhouette image generated in step S1202 and the detected silhouette image generated in step S1102. The method of calculating the difference is the same as that in the first embodiment, and thus the description is omitted.

In step S1204, the reliability calculation unit 303 converts the difference calculated in step S1203 into reliability. Since the method of converting to reliability is the same as that of the first embodiment, the description is omitted.

As described above, in the present embodiment, the three-dimensional position of the finger joint 601 is calculated by inputting the detected silhouette image to the neural network, and the processing is switched according to the reliability of the three-dimensional position. Therefore, it is possible to estimate the detailed three-dimensional shape of the finger as much as possible while suppressing the estimation of a three-dimensional shape that is significantly different from the actual one without using the distance sensor. Therefore, it is possible to improve the estimation accuracy of the three-dimensional shape of the finger.

(Third Embodiment)
In the first and second embodiments, the case where each node in the output layer of the neural network outputs the coordinate value of the finger joint has been described as an example. In the third embodiment, an example will be described in which each node in the output layer of the neural network outputs the likelihood of a preset finger pattern. As described above, the present embodiment is different from the first and second embodiments mainly in that each node of the neural network outputs the likelihood of a preset finger pattern. Therefore, in the description of the present embodiment, the same parts as those in the first and second embodiments will be denoted by the same reference numerals as those in FIGS. 1 to 12, and the detailed description thereof will be omitted.

<Calculation method of three-dimensional position of finger joint>
FIG. 13 is a diagram schematically showing a method for calculating the three-dimensional position of the finger joint 601 according to this embodiment. The pre-processing unit 302 in the present embodiment calculates the most likely finger pattern from the preset finger pattern group 1301 by using the neural network technology. Specifically, each node in the output layer corresponds to each finger pattern, and the likelihood of each finger pattern is output.

Then, the three-dimensional position of the finger joint 601 is calculated by selecting the finger pattern corresponding to the node having the highest likelihood. It should be noted that the neural network in the present embodiment performs learning processing in advance so that when a distance image or a detected silhouette image is input, the output value of the node corresponding to the finger pattern closest to the actual shape of the finger becomes the largest. It is assumed that

The reliability calculation unit 303 in this embodiment calculates the reliability of the three-dimensional position of the finger joint 601 from the likelihood of each finger pattern output by the pre-processing unit 302. Here, the larger the difference between the likelihood of the finger pattern selected by the neural network and the likelihood of another finger pattern, the more reliable the finger pattern selected by the neural network.

Therefore, the difference between the average likelihood of the finger pattern group 1301 and the likelihood of the finger pattern selected by the neural network is calculated as the reliability. The reliability in this embodiment is not limited to this. For example, the difference between the likelihood of the finger pattern selected by the neural network (that is, the likelihood of the largest value) and the likelihood of the second largest value may be calculated as the reliability.

As described above, in the present embodiment, the most probable finger pattern is calculated from the preset finger pattern group by the neural network, and the process is switched according to the reliability thereof. Therefore, in cases such as gesture recognition where the finger pattern to be estimated is limited, it is possible to estimate the most detailed three-dimensional shape of the finger while suppressing the estimation of a three-dimensional shape that is significantly different from the actual one. it can. Therefore, it is possible to improve the estimation accuracy of the three-dimensional shape of the finger.

(Fourth Embodiment)
In the first to third embodiments, the case where the three-dimensional position of the finger joint is calculated as the pre-stage process and the process is switched according to the reliability of the three-dimensional position has been described as an example. In the fourth embodiment, a case will be described as an example in which a finger joint is detected from a captured image or a distance image as preprocessing and the processing is switched according to the reliability of the detection result of the finger joint.

As described above, the present embodiment and the first to third embodiments are mainly different in that a finger joint is detected from a captured image or a distance image and that processing is switched depending on the reliability of the detection result of the finger joint. .. Therefore, in the description of the present embodiment, the same parts as those in the first to third embodiments will be denoted by the same reference numerals as those in FIGS. 1 to 13, and the detailed description thereof will be omitted.

<Processing of information processing device>
FIG. 14 is a flowchart showing a procedure of processing executed by the information processing apparatus 100 according to this embodiment. Below, the procedure of the process of this embodiment will be described with reference to FIG. Note that the processes of steps S408 and S409 are the same as those in the first to third embodiments, and thus the description thereof will be omitted.

In step S1401, the image acquisition unit 301 acquires an image. In the present embodiment, at least one of the captured image and the distance image is acquired and used for subsequent processing. Hereinafter, an example of acquiring and using a captured image will be described, but the present invention is not limited to this.

In step S1402, the upstream processing unit 302 detects the finger joint 601 from the image acquired in step S1401. Although any method can be used to detect the finger joint, for example, the finger joint can be detected using a learning type discriminator.

In step S1403, the reliability calculation unit 303 calculates the reliability of the finger joint 601 detected in step S1402. For example, a difference between the average value of the number of joints of a finger pattern set in advance and the number of detected joints can be calculated as the reliability. Note that the present invention is not limited to this, and any value having a correlation with the number of detected joints may be used as the reliability. Further, the likelihood of the positional relationship of joints may be calculated as the reliability. Hereinafter, an example in which the number of detected joints is used as the reliability will be described, but the present invention is not limited to this.

In step S1404, the estimation condition determining unit 304 determines whether the reliability calculated in step S1403 is equal to or higher than a preset threshold value. If the result of this determination is that the reliability is greater than or equal to the threshold value, the flow proceeds to step S1405. On the other hand, if it is determined that the reliability is lower than the threshold value, the process proceeds to step S409.

In step S1405, the shape estimation unit 305 calculates the three-dimensional position of the joint from the finger joint 601 detected in step S1402. In this embodiment, the detected two-dimensional coordinates of the finger joint 601 are input to the input layer of the neural network. The other processes of the neural network are the same as those of the first to third embodiments, and thus the description thereof will be omitted.

As described above, in the present embodiment, the finger joint 601 is detected from the captured image or the distance image as the pre-stage processing, and the three-dimensional position of the joint is detected by the neural network according to the reliability of the detection result of the finger joint 601. Switches whether to perform calculation. That is, when the reliability of the detection result of the finger joint 601 is low, it is possible to switch to the estimation of the approximate three-dimensional shape without calculating the three-dimensional position of the joint.

Therefore, it is possible to estimate the detailed three-dimensional shape of the finger as much as possible while suppressing the estimation of a three-dimensional shape that is significantly different from the actual one, and it is also possible to reduce the processing. Therefore, it is possible to improve the estimation accuracy of the three-dimensional shape of the finger.

(Fifth Embodiment)
Next, a fifth embodiment will be described. In the first to fourth embodiments, the case has been described as an example in which the estimation condition regarding the thickness of the entire finger is switched according to the reliability of the former process. In the fifth embodiment, an example will be described in which the reliability is calculated for each part of the finger and the estimation condition regarding the thickness (detail) is switched for each part of the finger according to the reliability.

As described above, the present embodiment is different from the first to fourth embodiments mainly in that the reliability is calculated for each finger part and that the estimation condition regarding the thickness (detail) is switched for each finger part. Therefore, in the description of the present embodiment, the same parts as those in the first to fourth embodiments will be denoted by the same reference numerals as those in FIGS. 1 to 14, and the detailed description thereof will be omitted.

<Finger part>
FIG. 15: is a figure which shows the site|part of a finger typically. In the present embodiment, as shown in FIG. 15, the finger is divided into parts 1501 to 1515. Then, the reliability is calculated for each part. The method of calculating the reliability is the same as in the first to fourth embodiments, and therefore the description thereof will be omitted. After that, a detailed three-dimensional shape is estimated for a portion whose reliability is equal to or higher than a preset threshold value, and a rough three-dimensional shape is estimated for a portion that is not so.

Alternatively, both the detailed three-dimensional shape and the rough three-dimensional shape are estimated in advance, and the detailed three-dimensional shape is applied to the portion whose reliability is equal to or higher than the threshold, and the rough three-dimensional shape is applied to the other portions. You may make it fit a three-dimensional shape. The method of dividing the parts is arbitrary and is not limited to the division as shown in FIG.

As described above, in the present embodiment, the reliability is calculated for each part of the finger, and the estimation condition regarding the thickness (detail) is switched for each part of the finger according to the reliability. Therefore, even when the reliability differs depending on the part, it is possible to estimate the three-dimensional shape of the finger as detailed as possible while suppressing the estimation of the three-dimensional shape that is significantly different from the actual one. Therefore, it is possible to improve the estimation accuracy of the three-dimensional shape of the finger.

[Modification]
In the above embodiments, the case where the three-dimensional position of the joint is calculated using the neural network has been described as an example. However, the calculation method is not limited to the neural network, and the present invention can calculate the three-dimensional position of the joint by any method. For example, the most likely finger pattern may be calculated from a preset finger pattern group by performing a principal component analysis on the distance image. In that case, the distance between the coordinate obtained by the principal component analysis of the distance image acquired by the image obtaining unit 301 and the coordinate obtained by the principal component analysis of the distance image formed by projecting each finger pattern is defined as the likelihood, and the reliability is calculated. To do.

Moreover, in each of the above-described embodiments, the case where the estimation target is a finger has been described as an example. However, the estimation target is not limited to the fingers, and the present invention can estimate the three-dimensional shape of any articulated object. For example, the estimation target may be the whole body of an articulated organism. Further, in each of the above embodiments, the captured image is described as a visible image, but the present invention is not limited to this. For example, the captured image may be an infrared image.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: information processing device, 301: image acquisition unit, 302: pre-stage processing unit, 303: reliability calculation unit, 304: estimation condition determination unit, 305: shape estimation unit

Claims (16)

An information processing apparatus for estimating a three-dimensional shape of an articulated object,
Image acquisition means for acquiring image data of the articulated object;
Pre-stage processing means for performing pre-stage processing on the image data,
Reliability calculation means for calculating a reliability for the result of the preceding stage processing,
Estimation condition determining means for determining the estimation condition of the three-dimensional shape based on the reliability,
Shape estimating means for estimating the three-dimensional shape based on the estimation condition,
Equipped with
The estimated condition determining means, information processing, characterized that you determine the shape estimation means performs any estimation process estimation processing or schematic three-dimensional shape of the detailed three-dimensional shape as the estimated condition apparatus. The pre-stage process is a process of calculating a three-dimensional position of a joint of the multi-joint object using the image data,
The information processing apparatus according to claim 1, wherein the reliability calculation means calculates reliability of a three-dimensional position of the joint. The information processing apparatus according to claim 2, wherein the pre-stage process is a process of calculating a three-dimensional position of a joint of the multi-joint object using the image data and a neural network technique. The image data is range image data,
4. The information processing according to claim 3, wherein the pre-stage process is a process of calculating a three-dimensional position of a joint of the multi-joint object by inputting distance information of the data of the distance image into a neural network. apparatus. The pre-stage processing detects a multi-joint area from the image data, generates a silhouette image of the multi-joint area, and inputs pixel values of the silhouette image to a neural network to thereby generate a three-dimensional joint of the multi-joint object. The information processing apparatus according to claim 3, which is a process of calculating a position. The pre-stage process is a process of calculating a three-dimensional position of a joint of the articulated object based on the likelihood of the pattern of the articulated object corresponding to each node of the output layer of the neural network. The information processing device according to claim 3. The pre-stage process is a process of detecting a joint of the multi-joint object from the image data,
The information processing apparatus according to claim 1, wherein the reliability calculation means calculates reliability of the detected result. The information processing apparatus according to claim 7, wherein the reliability calculation means calculates the reliability based on the detected number of joints. The information processing apparatus according to claim 7, wherein the reliability calculation means calculates the likelihood of the detected positional relationship of the joint as the reliability. The reliability calculation means calculates reliability for each part of the multi-joint object,
The information processing apparatus according to claim 1, wherein the estimation condition determining unit determines an estimation condition for each of the parts. The information processing apparatus according to any one of claims 1 to 10 , wherein the image data includes at least one of data of a distance image, a visible image, and an infrared image. The information processing apparatus according to any one of claims 1 to 11 , wherein the multi-joint object is a finger. An information processing apparatus for estimating a three-dimensional shape of an articulated object,
Image acquisition means for acquiring image data of the articulated object;
Pre-processing means for performing pre-processing on the image data, which is processing for detecting a joint of the multi-joint object from the image data;
Reliability calculation means for calculating the reliability of the result of the pre-processing based on the number of joints detected by the pre-processing,
Estimation condition determining means for determining the estimation condition of the three-dimensional shape based on the reliability,
Shape estimating means for estimating the three-dimensional shape based on the estimation condition,
An information processing apparatus comprising: A control method of an information processing device for estimating a three-dimensional shape of an articulated object, comprising:
An image acquisition means, an image acquisition step of acquiring image data of the articulated object;
A pre-stage processing means for performing pre-stage processing on the image data,
A reliability calculation step in which the reliability calculation means calculates a reliability with respect to the result of the pre-processing,
Estimating condition determining means, an estimating condition determining step of determining an estimating condition of the three-dimensional shape based on the reliability,
A shape estimating step, in which the shape estimating means estimates the three-dimensional shape based on the estimation condition,
Have a,
In the estimation condition determining step, which of the detailed three-dimensional shape estimation processing and the rough three-dimensional shape estimation processing is to be performed in the shape estimation step is determined as the estimation condition. Control method. A control method of an information processing device for estimating a three-dimensional shape of an articulated object, comprising:
An image acquisition means, an image acquisition step of acquiring image data of the articulated object;
A pre-stage processing step in which the pre-stage processing means performs a pre-stage process for the image data, which is a process of detecting a joint of the multi-joint object from the image data;
A reliability calculation step in which the reliability calculation means calculates the reliability of the result of the pre-processing based on the number of joints detected by the pre-processing;
Estimating condition determining means, an estimating condition determining step of determining an estimating condition of the three-dimensional shape based on the reliability,
A shape estimating step, in which the shape estimating means estimates the three-dimensional shape based on the estimation condition,
A method for controlling an information processing device, comprising: A program for causing a computer to function as the information processing device according to any one of claims 1 to 13.

Images