CN109816724B - Method and device for 3D feature extraction based on machine vision - Google Patents

Abstract

The invention belongs to the field of machine vision, and specifically provides a method and device for extracting three-dimensional features based on machine vision. The present invention aims to solve the problems in the prior art that the reconstruction process of the three-dimensional model is complicated, time-consuming, and difficult to popularize. To this end, the steps of the three-dimensional feature extraction method based on machine vision of the present invention include: acquiring a multi-angle image containing preset feature points to be measured of the target; extracting the feature points to be measured in each of the images. position information; obtain the spatial position information of the feature points to be tested according to the position information of the feature points to be tested in each of the images; The first distance information and/or the second distance information corresponding to the measured feature points. Through machine vision, images of different angles containing the feature points to be measured are obtained, and then the spatial position information of the feature points to be measured is obtained so as to calculate the distance information of the target object.

Description

Method and device for 3D feature extraction based on machine vision

technical field

The invention belongs to the field of machine vision, and in particular relates to a three-dimensional feature extraction method and device based on machine vision.

Background technique

With the development of cloud manufacturing and cloud computing and the approach of "Industry 4.0", the social manufacturing mode, that is, the mode of customized production for customers, came into being. The characteristic of social manufacturing is that it can directly convert consumers' needs into products. Based on social computing theory, based on mobile Internet technology, social media and 3D printing technology, the public can fully participate in the full-life manufacturing of products through crowdsourcing and other forms. process to achieve personalized, real-time, and economical production and consumption patterns. That is to say, in social manufacturing, every consumer can participate in all stages of the whole life cycle of product production, including product design, manufacture and consumption. Taking shoemaking as an example, the application of social manufacturing in the shoemaking process is reflected in the fact that users can customize and select according to their own needs, which requires the ability to simply, quickly and accurately obtain the three-dimensional characteristics of the user's foot shape.

However, the original manual measurement can obtain fewer foot parameters, and cannot accurately describe the foot shape. Only professional tools in the shoe industry can obtain accurate measurement results. In order to enable non-professionals to obtain more accurate foot shape parameters so as to realize personalized customization of shoes, the present invention proposes a method for obtaining foot shape parameters by establishing a model and calculating. Since the height of the arch of the foot and the angle between the toes and the sole of the foot are different for each person, if only the two characteristic dimensions of foot length and foot width are obtained, it cannot accurately reflect the difference between different individual foot shapes belonging to the same model. The foot shape is reconstructed by 3D model to obtain accurate foot shape parameters. At present, the 3D model of the foot can be reconstructed by equipment such as laser 3D scanning, but this method is complicated and time-consuming, the hardware cost is high, and the popularization is difficult. In this way, a simpler three-dimensional model method is required to accurately obtain foot shape parameters.

Accordingly, there is a need in the art for a new three-dimensional model reconstruction method to solve the above problems.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned problems in the prior art, that is, in order to solve the problems of the existing three-dimensional model reconstruction process being complicated, time-consuming, and difficult to popularize, the first aspect of the present invention discloses a three-dimensional feature extraction method based on machine vision. The feature extraction method includes the following steps: acquiring a multi-angle image of a preset feature point to be measured including a reference object and a target set relative to the reference object; extracting the position of the feature point to be measured in each of the images obtain the spatial position information of the feature point to be tested according to the position information of the feature point to be tested in each of the images; based on the spatial position information and the preset three-dimensional feature category, calculate a certain The first distance information and/or the second distance information corresponding to the feature point; wherein, the first distance information is the distance information between the certain feature point to be measured and other feature points to be measured, and the second distance information The information is vertical distance information between the certain feature point to be measured and a preset plane; the certain feature point to be measured, the other feature points to be measured and the plane all depend on the three-dimensional feature category.

In the preferred technical solution of the above-mentioned machine vision-based three-dimensional feature extraction method, the step of "extracting the position information of the feature point to be measured in each of the images" includes: using a manual marking method to obtain a certain image in the image. The pixel positions of the feature points to be tested are extracted; the pixel positions corresponding to the feature points to be tested in other images are extracted by using a preset feature point matching method and according to the acquired pixel positions.

In the preferred technical solution of the above-mentioned machine vision-based three-dimensional feature extraction method, the step of "extracting the position information of the feature point to be measured in each of the images" includes: acquiring the feature to be measured in the target The area shape corresponding to the area where the point is located; the area to be measured corresponding to each image is obtained according to the area shape; the relative position between the characteristic point to be measured and the area shape and each area to be measured , and obtain the position information of the feature point to be measured in each image.

In the preferred technical solution of the above three-dimensional feature extraction method based on machine vision, the step of "extracting the position information of the feature points to be measured in each of the images" includes: using a pre-built neural network to obtain the Position information of feature points in each of the images; wherein, the neural network is a deep neural network trained based on a preset training set and using a deep learning related algorithm.

In the preferred technical solution of the above three-dimensional feature extraction method based on machine vision, the step of "acquiring the spatial position information of the feature point to be tested according to the position information of the feature point to be tested in each of the images" includes: The Euclidean position of the feature point to be tested is obtained by using the triangulation method and according to the position information of the feature point to be tested in each of the images and the internal and external parameters of the camera.

In the preferred technical solution of the above three-dimensional feature extraction method based on machine vision, the step of "acquiring the spatial position information of the feature point to be tested according to the position information of the feature point to be tested in each of the images" includes: A sparse model is constructed by using the incremental SFM method and the position information of the feature points to be measured in each of the images, and the triangulation method is used to calculate the spatial position information of the feature points to be measured in the world coordinate system; The obtained scale coefficient restores the spatial position information of the feature point to be measured in the world coordinate system obtained in the above steps, so as to obtain the real position of the feature point to be measured.

In the preferred technical solution of the above three-dimensional feature extraction method based on machine vision, in "using the pre-acquired scale coefficient to restore the spatial position information of the feature point in the world coordinate system obtained in the above step, the feature to be measured is obtained. Before the actual position of the point", the three-dimensional feature extraction method based on machine vision further includes: using the sparse model and according to the pixel position of the reference object vertex in the camera coordinate system, obtain the reference object vertex in the world coordinate system. It should be noted that the coordinate of the vertex in the world coordinate system differs from the real position in space by the scale coefficient λ; according to the coordinates of the vertex of the reference object in the world coordinate system and the real position of the vertex of the reference object, the scale coefficient is calculated. λ.

In a preferred technical solution of the above three-dimensional feature extraction method based on machine vision, the triangulation method includes: obtaining the The projective space position of the feature point to be measured, and the Euclidean space position of the feature point to be measured is obtained by homogenizing the projective space position.

It can be understood by those skilled in the art that, in the technical solution of the present invention, by acquiring images of the target object at different angles and extracting the positions of the feature points to be measured in the images, and then using the triangulation method or solving the sparse reconstruction problem to calculate the The spatial position of the feature point in the world coordinate system is measured, and the first distance information and/or the second distance information between the feature points is calculated according to the calculated spatial position information of the feature point to be measured. The three-dimensional feature extraction method of the present invention can quickly determine the three-dimensional feature points of the target object only through the multi-angle images obtained by the photographing device, and then calculate the distance information of the target object without using high-cost and complicated hardware equipment such as laser three-dimensional scanning. , which simplifies the 3D reconstruction process.

In a preferred technical solution of the present invention, the pixel positions of the feature points to be measured in each image are determined by manual marking or an automatic method, wherein the automatic method includes reusing each image according to the shape of the region corresponding to the region where the feature points to be measured are located or use a pre-built neural network to obtain the location information of the feature points to be measured in each image. Then use the reference object to automatically calibrate the camera parameters and then triangulate or solve the sparse reconstruction problem to obtain the real spatial position of the feature point to be measured. It does not need to reconstruct the model of the entire target object, which can reduce the amount of calculation and simplify the model establishment process. Finally, based on the real space position of the feature point to be measured and the preset three-dimensional feature category, the distance information corresponding to the feature point to be measured is calculated.

A second aspect of the present invention provides a storage device, the storage device stores a plurality of programs, and the programs are adapted to be loaded by a processor to execute any one of the aforementioned methods for extracting three-dimensional features based on machine vision.

It should be noted that the storage device has all the technical effects of the aforementioned three-dimensional feature extraction method based on machine vision, which will not be repeated here.

A third aspect of the present invention further provides a control device, the control device includes a processor and a storage device, the storage device is adapted to store a plurality of programs, the programs are adapted to be loaded by the processor to execute the foregoing Any one of the three-dimensional feature extraction methods based on machine vision.

It should be noted that the control device has all the technical effects of the aforementioned three-dimensional feature extraction method based on machine vision, which will not be repeated here.

Description of drawings

The three-dimensional feature extraction method based on machine vision of the present invention will be described below with reference to the accompanying drawings and in conjunction with foot shapes. In the attached picture:

1 is a flow chart of the main steps of a machine vision-based three-dimensional feature extraction method for feet in an embodiment of the present invention;

2 is a schematic diagram of a machine vision-based three-dimensional feature extraction method for feet in an embodiment of the present invention using a circle as a template to detect feature points using generalized Hough transform;

3 is a schematic diagram of a machine vision-based foot shape three-dimensional feature extraction method in an embodiment of the present invention using a circle as a template to detect feature points using generalized Hough transform;

4 is a schematic diagram of a machine vision-based three-dimensional feature extraction method for feet in an embodiment of the present invention using a circle as a template to detect feature points using generalized Hough transform;

5 is a schematic diagram of a machine vision-based three-dimensional feature extraction method for feet in an embodiment of the present invention using a straight line as a template to detect a reference object using generalized Hough transform;

6 is a schematic diagram of the process of solving the spatial position information of feature points by a triangulation process of a machine vision-based three-dimensional feature extraction method for feet in an embodiment of the present invention;

7 is a schematic diagram of a process of obtaining spatial position information of feature points in a sparse reconstruction process of a machine vision-based three-dimensional feature extraction method for feet in an embodiment of the present invention.

Detailed ways

Preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principle of the present invention, and are not intended to limit the protection scope of the present invention. For example, although the present invention is described by taking a foot shape as an example, it can also be other objects that can be converted into products by building a model, such as clothes. In addition, the present invention is described with reference to A4 paper, but other objects of known size (eg floor tiles) are also possible. Those skilled in the art can adjust it as needed to adapt to specific applications.

It should be noted that, in the description of the present invention, the terms "first", "second" and "third" are only used for description purposes, and cannot be understood as indicating or implying relative importance.

The method for extracting three-dimensional features of a foot shape based on machine vision provided by the present invention will be described below with reference to the accompanying drawings.

In a specific embodiment of the present invention, the extraction and calculation process of the three-dimensional feature parameters of the foot shape is transformed into determining the spatial position of the corresponding feature point, and then the Euclidean distance formula is used to calculate the feature parameters of the foot shape to be measured. Among them, the basic parameters of the foot shape that can be obtained include: foot length, foot circumference, instep circumference height, arch bending point height, foot width, thumb height, heel convexity point height, foot outer ankle bone center point height, etc. Required foot parameter information. A possible implementation of the method for extracting three-dimensional features of a foot shape based on machine vision of the present invention is described below by taking the acquisition of three parameters of foot length, foot width and ankle point height as examples.

Referring first to FIG. 1 , FIG. 1 exemplarily shows the main steps of a machine vision-based three-dimensional feature extraction method for feet in an embodiment of the present invention. The machine vision-based three-dimensional feature extraction method for feet in the present invention may include the following steps :

Step S100, acquiring a multi-angle image including preset feature points to be measured of the target.

Specifically, put the foot on the A4 paper, and use a mobile photographing device, such as a camera, to take images of the foot shape from multiple angles, so as to fully express the foot shape characteristics and obtain enough feature points to be measured, such as the longest Toe apex and heel convex point are used to calculate the length of the foot to be measured. Another example is the outer point of the ball of the thumb and the outer point of the base of the tail toe to calculate the width of the foot to be measured. Another example is the ankle point to calculate the height of the ankle. It should be noted that the number of images of the foot shape taken should be at least three or more, and the more images containing the feature points to be measured, the more accurate the foot shape parameters calculated according to the feature points to be measured.

Step S200, extracting the position information of the feature point to be measured in each image.

Specifically, in a preferred implementation of this embodiment, the three-dimensional feature extraction method shown in FIG. 1 can obtain the pixel position (x, y) of the feature point to be measured in each image according to the following steps, specifically:

First, manually mark the pixel positions of the feature points to be measured in an image, and then use feature point matching methods, such as Scale Invariant Feature Transform (SIFT) or Iterative Closest Point (Iterative Closest Point, ICP), etc. , find the corresponding pixel position of the feature point to be measured in other images. Taking measuring the height of the ankle as an example, select an image containing the ankle point, manually mark the pixel position of the ankle point in the image, and then use the feature point matching method such as SIFT or ICP to find the image of the ankle point at other angles containing the ankle point. The corresponding pixel position in . Through this method, the corresponding pixel positions of the feature points to be detected in all images can be quickly found without the need to manually mark the feature points for each image, which improves the efficiency of obtaining the pixel positions of the feature points.

Optionally, in another preferred implementation of this embodiment, the three-dimensional feature extraction method shown in FIG. 1 can also obtain the pixel position (x, y) of the feature point to be measured in each image according to the following steps, specifically: :

According to the uniqueness of the shape of the region where the feature points to be detected are located, a feature detection method, such as generalized Hough transform, is used to detect a specific shape and then determine the position information of the feature points to be detected in each image. Specifically, first determine the area shape corresponding to the area where the feature points to be measured are located, then automatically find the area to be measured corresponding to the feature points to be measured in each image according to the shape of the area and use the generalized Hough transform, and then use the generalized Hough transform to automatically find the area to be measured corresponding to the feature points to be measured in each image, and then The relative position between the point and the shape of the area and the area to be measured in each image obtain the location information of the feature point to be measured in each image. A possible implementation is described below by taking a circle as a template and finding feature points by using the generalized Hough transform as an example.

Referring to Fig. 2, Fig. 3 and Fig. 4, Fig. 2 is a schematic diagram of using a generalized Hough transform to detect feature points with a circle as a template of a machine vision-based three-dimensional feature extraction method for feet in an embodiment of the present invention; Fig. 3 is a A schematic diagram of a machine vision-based three-dimensional feature extraction method for a foot shape in an embodiment of the present invention using a circle as a template to detect feature points using generalized Hough transform; FIG. 4 is a machine vision-based foot shape in an embodiment of the present invention. The schematic diagram of the three-dimensional feature extraction method using a circle as a template to detect feature points using the generalized Hough transform. Figure 2, Figure 3 and Figure 4 show the use of the generalized Hough transform at different angles. The specific implementation of Hough transform to find feature points. As shown in Figure 2, Figure 3 and Figure 4, the ankle where the center of the ankle is located is circular. It can be seen from the figure that this circular outline is unique in the foot shape. In this way, using the generalized Hough When transforming, take the circle as the template, and automatically find the position of the circle in the image (the circle template shown by the dotted line in Figure 2-4). This position is the position of the ankle, and the center of the circle position is searched. Point G is the position of the ankle point of the feature point to be measured in the image.

It can be understood that when determining the position information of the vertex of the longest toe, the contour of the longest toe can be used as the template of the generalized Hough transform to search in the image. to determine the pixel position of the feature point.

Optionally, in another preferred implementation of this embodiment, the three-dimensional feature extraction method shown in FIG. 1 can also obtain the pixel position (x, y) of the feature point to be measured in each image according to the following steps, specifically: :

A deep neural network is constructed based on a sufficient number of data samples of marked feature points of the foot shape and a deep learning algorithm is used, and then the position information of the feature points to be measured in each image is obtained by using the neural network. Specifically, when training the neural network, the input is image data containing the feature points to be tested, and the output is the pixel position (x, y) of the feature points to be tested in the image, wherein the output includes the actual output and the expected output, The actual output of the last fully connected layer of the network is the pixel position (x, y) corresponding to the feature point to be tested in the image, and the expected output of the network is the actual pixel position marked in the image by the feature point to be tested. Then use the error between the actual output of the network and the expected output to reversely train the entire network, and iteratively train until the network converges. After the neural network is trained, input an image to be tested that contains the feature points to be tested, and the neural network automatically outputs the neural network. pixel location in this image. Take obtaining the pixel position of the ankle point as an example, select a sufficient number of image samples marked with the ankle point as the training set, and build a deep neural network, and then use the training set to train the deep neural network. The image to be tested, the neural network automatically outputs the pixel position of the ankle point in the image. It can be understood that when determining the pixel positions of other feature points, use the image data samples corresponding to the feature points to train the pre-built deep neural network, and then input the image to be tested containing the feature points to obtain the feature points in the image. pixel location.

Step S300, obtaining spatial position information of the feature point to be measured according to the position information of the feature point to be measured in each image.

Specifically, in a preferred implementation of this embodiment, the three-dimensional feature extraction method shown in FIG. 1 can obtain the spatial position information of the feature points to be measured according to the following steps, specifically:

Firstly, the camera parameters are calibrated by the reference object, and then the spatial position information of the feature points to be measured is calculated by the triangulation method. Specifically, taking A4 paper as an example, the foot shape is placed on the A4 paper, and imaging equipment, such as a camera, is used to obtain multiple images at different angles, and these images at different angles contain the outline of the A4 paper. Use these images of different angles to calibrate the camera, determine the camera's internal parameter matrix K, the external parameters relative to the world coordinate system rotation matrix R, translation matrix t. Then, according to the pixel position (x, y) of the feature point to be measured in the image obtained in step S200, and using the triangulation method and homogenization to solve the spatial position information (X, Y, Z). A possible implementation manner of obtaining the real spatial position of the feature point by the triangulation method will be described below with reference to FIG. 5 and FIG. 6 .

Referring to FIG. 5 , FIG. 5 is a schematic diagram of a method for extracting three-dimensional features of a foot shape based on machine vision according to an embodiment of the present invention, using a straight line as a template and using generalized Hough transform to detect a reference object. As shown in Fig. 5, the straight line template and the random Hough transform are used to detect the edge straight line of the A4 paper in the image. It can be seen that four edge straight lines are detected, and each straight line intersects two by two, and the intersection point is the pixel position (x _i , y _i ) of the four vertices (A, B, C, D) of the A4 paper, i=1, 2 , 3, 4. Continue to refer to Figure 2, Figure 3 and Figure 4, from the knowledge of spatial geometric transformation, the following relationship between point A in Euclidean space and projective space can be obtained:

The parameters K, R and t in formula (1) are the camera internal parameter matrix, the rotation matrix and translation matrix of the camera relative to the world coordinate system ([R|t] are collectively referred to as the camera external parameter matrix). Among them, the symbol "|" represents the augmented matrix, and r ₁ , r ₂ , and r ₃ are respectively the expanded form of the rotation matrix R of the camera relative to the world coordinate system. It can be seen from the matrix multiplication that r ₃ and 0 elements are multiplied and eliminated.

是A4纸顶点A的像素位置，(X^A，Y^A，Z^A)^T是其在世界坐标系下的真实位置，K[R|t]是相机的内外参数。单应矩阵H＝K[r₁r₂|t]有8个自由度，将世界坐标系建立在A4纸的顶点A上，则A4纸的四个顶点的世界坐标系为(0，0，0)，(X，0，0)，(0，Y，0)，(X，Y，0)，其中，X＝210mm，Y＝297mm。每个顶点都能写成式(1)形式以构造两组线性方程。因此，四组顶点可以构建8组线性方程，通过直接线性变换(Direct Linear Transform，DLT)方式来求解H。in,

is the pixel position of vertex A of the A4 paper, (X ^A , Y ^A , Z ^A ) ^T is its real position in the world coordinate system, and K[R|t] is the internal and external parameters of the camera. The homography matrix H=K[r ₁ r ₂ |t] has 8 degrees of freedom. If the world coordinate system is established on the vertex A of the A4 paper, the world coordinate system of the four vertices of the A4 paper is (0, 0, 0), (X, 0, 0), (0, Y, 0), (X, Y, 0), where X=210mm, Y=297mm. Each vertex can be written in the form of equation (1) to construct two sets of linear equations. Therefore, four sets of vertices can construct 8 sets of linear equations, and H can be solved by means of Direct Linear Transform (DLT).

Since the angles of the three photos are different, the camera poses of the three photos are different. According to the same method above, three sets of homography matrices H ₁ , H ₂ , and H ₃ of the world coordinate system in the camera can be obtained.

K can be obtained from the homography matrix H. Since H=[h ₁ h ₂ h ₃ ]=K[r ₁ r ₂ |t], we can obtain:

K ^-1 [h ₁ h ₂ h ₃ ]=[r ₁ r ₂ |t] (2)

The parameters K ^-1 , R, t and H in formula (2) are the inverse matrix of the parameter matrix in the camera, the rotation matrix of the camera relative to the world coordinate system, the translation matrix and the homography matrix, respectively. Among them, r ₁ , r ₂ are the rotation matrices of the external camera parameters relative to the world coordinate system obtained through two images of different angles, respectively, h ₁ , h ₂ , h ₃ are the relative world obtained through three images of different angles respectively. The three sets of homography matrices for the coordinate system in the camera.

Wherein, R=[r ₁ r ₂ r ₃ ] is a rotation matrix with orthogonal properties, namely: r ₁ ^T r ₂ =0 and ‖r ₁ ‖=‖r ₂ ‖=1. Therefore, we can get: h ₁ ^T K ^-T K ^-1 h ₂ =0, and then we can get:

h ₁ ^T K ^-T K ^-1 h ₁ =h ₂ ^T K ^-T K ^-1 h ₂ (3)

The parameters K- ^T and K ^-1 in formula (3) are the orthogonal matrix and the inverse matrix of the transposed matrix of the parameter matrix in the camera, respectively, and h ₁ and h ₂ are the relative world obtained by two images of different angles. The two sets of homography matrices of the coordinate system in the camera, h ₁ ^T , h ₂ ^T are the transpose matrices of the homography matrices h ₁ , h ₂ , which can be obtained from the above, and the internal parameters of two cameras can be obtained for each two images constraint equation.

The camera internal parameter matrix K is an upper triangular matrix, and w=K- ^T K ^-1 is a symmetric matrix. According to the images of three different angles in Figure 2, Figure 3 and Figure 4, w is linearly solved by DLT, and then through the orthogonal The decomposition can be solved to obtain K. According to formula (1), [r ₁ r ₂ |t]=K ^-1 [h ₁ h ₂ h ₃ ], combined with h ₁ , h ₂ , h ₃ and K obtained by the above solution, we can obtain r ₁ , r ₂ , t. From the orthogonality of the rotation matrix, r ₃ =r ₁ ×r ₂ , so R=[r ₁ r ₂ r ₃ ]. By this method, the intrinsic and extrinsic parameters K[R ₁ |t ₁ ], K[R ₂ |t ₂ ], and K[R ₃ |t ₃ ] of the camera at the time of shooting FIG. 2 , FIG. 3 and FIG. 4 are obtained.

Referring to FIG. 6 , FIG. 6 is a schematic diagram of a process of solving the spatial position information of feature points by a triangulation process of a machine vision-based three-dimensional feature extraction method for feet in an embodiment of the present invention. As shown in FIG. 6, FIG. 6 shows the triangulation process taking the ankle point G in FIG. 3 and FIG. 4 (ie, Image1 and Image2) as an example, according to the ankle point G obtained in step S200, in Image1 and Image2 The pixel positions x ₁ and x ₂ of , and the internal and external parameters of the camera obtained in the above steps P ₁ =K ₁ [R ₁ |t ₁ ], P ₂ =K ₂ [R ₂ |t ₂ ], reprojection is performed in sequence Minimize the sum of squares of errors min∑ _i ‖P _i Xx _i ‖, so as to obtain the position of the feature point to be measured in the projective space X=(M, N, O, w), where P ₁ and P ₂ are respectively based on the calibration method The obtained internal and external parameters when the camera shoots Image1 and Image2, K ₁ and K ₂ are the camera's internal parameter matrix when the camera shoots Image1 and Image2 respectively, R ₁ and R ₂ are the camera shooting Image1 and Image2 respectively. The rotation matrix relative to the world coordinate system when an image is taken, t ₁ and t ₂ are translation matrices respectively. Finally, by homogenizing the projective space coordinates, the Euclidean space position of the feature point G to be measured can be obtained X=(M/w, N/w, O/w)=(X, Y, Z), where M , N, O, and w are the position coordinates of the feature point G in the projective space, respectively.

Optionally, in another preferred implementation of this embodiment, the three-dimensional feature extraction method shown in FIG. 1 can also obtain the real spatial position of the feature point to be measured according to the following steps, specifically:

Convert the 3D reconstruction problem into a sparse reconstruction problem of the feature points to be measured, such as using the incremental SFM method to build a sparse model and using the triangulation method to solve the sparse reconstruction problem. Specifically, according to the pixel positions (x, y) of the feature points to be measured in the multiple images obtained in step S200, the difference from the previous embodiment is that the incremental SFM method is used to directly solve the camera internal parameter matrix K, The camera rotation matrix R, the translation t relative to the world coordinate, the coordinates λ (X, Y, Z) of the feature point to be measured in the world coordinate system, omit the process of calibrating the camera with the reference object, and then use the known specifications The reference object determines the scale coefficient λ, and then obtains the real space position coordinates (X, Y, Z) of the feature point. A possible implementation manner of using the incremental SFM method to solve the sparse reconstruction problem will be described below with reference to FIG. 7 , taking three images from different angles as an example.

Referring to FIG. 7 , FIG. 7 is a schematic diagram of a process of obtaining spatial position information of feature points in a sparse reconstruction process of a machine vision-based three-dimensional feature extraction method for feet in an embodiment of the present invention. As shown in Figure 7, the steps of using the incremental SFM method to solve the sparse reconstruction problem include:

Step 1: Randomly select two images Image1 and Image2 from three different angles to determine the initial image pair, and use the incremental SFM method to calculate the initial values [R|t] of the internal and external parameters of the camera that captured the images Image1 and Image2 Matrix: Using the 5 sets of feature point pairs in the images Image1 and Image2 (the vertex of the longest toe and the convex point of the heel, the lateral point of the thumb ball and the lateral point of the root of the tail toe, and the ankle point), the images Image1 and Image2 are calculated using the 5-point method respectively. The corresponding essential matrices E ₁ and E ₂ , where E=[R|t], can be decomposed from the essential matrix E to decompose the camera rotation matrices R ₁ , R ₂ and the translations t ₁ , t ₂ matrices relative to the world coordinates. Then, in combination with the pixel positions of the feature points to be measured in the images Image1 and Image2 obtained in the camera coordinate system obtained in step S200, an initial sparse model is constructed;

Step 2: According to the initial sparse model constructed in Step 1, and use the triangulation method to calculate the position coordinates λ (X ₁ , Y ₁ , Z ₁ ) and λ(X ₂ , Y ₂ , Z ₂ );

Step 3: Input the pixel positions of the feature points to be measured in the camera coordinate system of the image Image3 obtained in step S200 into the initial sparse model obtained in step 2, and the camera internal and external parameters [R|t] matrix can be re-obtained, that is, the camera rotation matrix R ₃ and the translation t ₃ relative to the world coordinate, and use the camera intrinsic and extrinsic parameters to correct the initial sparse model;

Step 4: According to the sparse model corrected in Step 3, and use the triangulation method to calculate the spatial position coordinates λ (X ₃ , Y ₃ , Z ₃ ) of the feature points to be measured under the world coordinate system in the image Image3;

Step 5: Use a bundle adjustment (BA) method to correct the position coordinates of the feature points obtained in steps 2 and 4 to obtain an optimized sparse model.

Among them, in step 5, different coordinate positions of the feature points to be measured are obtained in the remaining other images, and the binding adjustment is repeatedly performed until the error of the coordinates λ (X, Y, Z) of the feature points to be measured obtained by the two calculations before and after is less than or equal to the predetermined value. set threshold.

Although the present invention only provides a specific implementation of using the incremental SFM method to solve the spatial position information of the feature points to be measured in the three images, those skilled in the art can understand that the incremental SFM method provided by the present invention The SFM method can also be used to solve multiple images of different angles. In the process of building a sparse model using the incremental SFM method, the pixel position information of the feature points to be measured in the new image in the camera coordinate system is repeatedly substituted, and the camera is re-obtained. extrinsic and extrinsic parameters and use the camera extrinsic parameters to correct the sparse model until all images are added to the sparse model. It can be understood that the more different angles of the acquired images, the more iterative calculations are performed, and the more accurate the internal and external parameters of the camera are obtained. The feature points to be measured calculated according to the sparse model constructed are in the world coordinate system. The more accurate the spatial location information.

Step 6: Taking point A in FIG. 4 as the coordinate origin, according to the pixel position information of the vertex D of the A4 paper in the camera coordinate system obtained in step S200, and then using the sparse model obtained in step 5 to calculate the space of the vertex D The coordinates are (M, N, 0), and the real spatial position of the vertex D is (210mm, 297mm, 0), therefore, the scale coefficient λ=210mm/M=297mm/N can be obtained. Combined with the spatial coordinates λ (X, Y, Z) of the feature point to be measured obtained in step 5 in the world coordinate system, divide by the scale coefficient λ to obtain the real spatial position (X, Y, Z) of the feature point to be measured.

Step S400: Calculate the first distance information and/or the second distance information corresponding to a certain feature point to be measured based on the spatial position information and the preset three-dimensional feature category.

It should be noted that the first distance information is the distance information between a certain feature point to be measured and other feature points to be measured, such as the length, and the second distance information is the vertical distance between a certain feature point to be measured and the preset plane. Distance information, such as altitude.

可以得到如下的计算公式：Specifically, taking the foot shape as an example, according to the spatial position information of the five feature points to be measured calculated in step S300, for example, the vertex of the longest toe is (X ₁ , Y ₁ , Z ₁ ), and the convex point of the heel is ( X ₂ , Y ₂ , Z ₂ ), the outside point of the thumb ball is (X ₃ , Y ₃ , Z ₃ ), the outside point of the base of the tail toe is (X ₄ , Y ₄ , Z ₄ ), the ankle point is (X ₅ ) , Y ₅ , Z ₅ ), using the distance formula, such as the Euclidean distance formula

The following calculation formula can be obtained:

The parameters L, W and H in formula (4) are the foot length, foot width and ankle height, respectively.

In this way, the three parameters of foot length, foot width and ankle height can be calculated. Although the present invention only provides a specific embodiment of calculating the three parameters of foot length, foot width and ankle point height by extracting three-dimensional feature points, those skilled in the art can understand that the three-dimensional feature extraction method provided by the present invention Other foot shape parameters can also be calculated, such as calculating the height of the instep. At this time, the images of different angles must include the feature points and the instep points, and then follow the steps of the three-dimensional feature extraction method of the present invention described in the above embodiment in turn. Calculate the instep height.

To sum up, in the preferred technical solution of the present invention, the camera equipment is used to obtain five features to be measured including the vertex of the longest toe, the convex point of the heel, the outer point of the thumb ball, the outer point of the root of the tail and the ankle point. Images of five different angles of the point, the pixel position information of each feature point to be measured in each image is determined by manual marking or automatic method, and then the camera parameters are manually calibrated and then triangulated or solved by sparse reconstruction problem. Taking the real spatial position of the feature point to be measured does not need to reconstruct the model of the entire target object, which can reduce the amount of calculation and simplify the model building process. Finally, based on the spatial positions of the five feature points to be measured, the Euclidean distance formula can be used to calculate the three foot shape parameters of foot length, foot width and ankle point height. By analogy, images of different angles of different feature points can be obtained, and the foot shape parameters corresponding to the feature points can also be obtained by calculation. Thus, the parameter of the instep height is calculated.

Further, based on the above method embodiments, the present invention also provides a storage device, where a plurality of programs are stored in the storage device, and the programs can be loaded by a processor to execute the machine vision based machine vision described in the above method embodiments. 3D feature extraction method.

Further, based on the above method embodiments, the present invention also provides a control device, the control device includes a processor and a storage device, wherein the storage device can be adapted to store a plurality of programs, and the programs can be adapted to The processor is loaded to execute the three-dimensional feature extraction method based on machine vision described in the above method embodiments.

So far, the technical solutions of the present invention have been described with reference to the preferred embodiments shown in the accompanying drawings, however, those skilled in the art can easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims (7)

1. A three-dimensional feature extraction method based on machine vision is characterized by comprising the following steps:

the method comprises the steps that a multi-angle image containing a reference object and preset feature points to be detected of a target object arranged relative to the reference object is obtained through a mobile camera, and the number of the multi-angle images is at least three;

extracting the position information of the feature point to be detected in each image, wherein the feature point to be detected is on a target object;

acquiring spatial position information of the feature points to be detected according to the position information of the feature points to be detected in each image, specifically, calibrating camera parameters by using a reference object, and calculating the spatial position information of the feature points to be detected by using a triangulation method;

calculating first distance information and/or second distance information corresponding to a certain feature point to be detected based on the spatial position information and a preset three-dimensional feature category;

the first distance information is distance information between the certain characteristic point to be measured and other characteristic points to be measured, and the second distance information is vertical distance information between the certain characteristic point to be measured and a preset plane; the certain feature point to be measured, the other feature points to be measured and the plane all depend on the three-dimensional feature category;

wherein the reference object is an object of known dimensions;

wherein the multi-angle image simultaneously comprises the reference object and the target object arranged relative to the reference object;

the step of "extracting the position information of the feature point to be detected in each image" is to obtain the pixel position (x, y) of the feature point to be detected in each image;

the model reconstruction is not performed on the whole target object, and the step of acquiring the spatial position information of the feature point to be detected according to the position information of the feature point to be detected in each image is to acquire the real spatial position (X, Y, Z) of the feature point to be detected.

2. The machine-vision-based three-dimensional feature extraction method according to claim 1, wherein the step of "extracting the position information of the feature point to be measured in each image" includes:

acquiring the pixel position of the characteristic point to be detected in a certain image by using a manual marking method;

and extracting the corresponding pixel positions of the feature points to be detected in other images by using a preset feature point matching method and according to the acquired pixel positions.

3. The machine-vision-based three-dimensional feature extraction method according to claim 1, wherein the step of "extracting the position information of the feature point to be measured in each image" includes:

acquiring the area shape corresponding to the area where the characteristic point to be detected in the target object is located;

acquiring a region to be detected corresponding to each image according to the region shape;

and acquiring the position information of the feature point to be detected in each image according to the relative position between the feature point to be detected and the shape of the region and each region to be detected.

4. The machine-vision-based three-dimensional feature extraction method according to claim 1, wherein the step of "extracting the position information of the feature point to be measured in each image" includes:

acquiring the position information of the feature point to be detected in each image by utilizing a pre-constructed neural network;

the neural network is a deep neural network which is based on a preset training set and trained by using a deep learning correlation algorithm.

5. The machine-vision-based three-dimensional feature extraction method according to any one of claims 1 to 4, wherein the step of acquiring spatial position information of the feature point to be measured from position information of the feature point to be measured in each of the images comprises:

and acquiring the Euclidean position of the feature point to be detected by using a triangulation method according to the position information of the feature point to be detected in each image and the internal and external parameters of the camera.

6. A storage device having stored therein a plurality of programs, characterized in that said programs are adapted to be loaded by a processor for performing the method of machine vision based three-dimensional feature extraction according to any of claims 1-5.

7. A control apparatus comprising a processor and a storage device adapted to store a plurality of programs, characterized in that the programs are adapted to be loaded by the processor to perform the machine vision based three-dimensional feature extraction method of any one of claims 1-5.

Images