Disclosure of Invention
To solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a three-dimensional detection system and a three-dimensional detection method.
The present disclosure provides a three-dimensional inspection system, comprising:
the two image acquisition sensors are used for acquiring surface images of the object;
a two-dimensional hole feature calculator for determining hole two-dimensional feature parameters based on the object surface image;
the three-dimensional hole characteristic calculator is used for carrying out three-dimensional reconstruction on the hole based on the hole two-dimensional characteristic parameters and determining the hole three-dimensional characteristic parameters;
and the auxiliary function calculator is used for optimizing the hole three-dimensional characteristic parameters based on the hole two-dimensional characteristic parameters and the hole three-dimensional characteristic parameters.
In some embodiments, at least two image capturing sensors are provided, the image capturing sensors being provided in the same handheld scanning device.
In some embodiments, the handheld scanning device further comprises a fill light device and a pattern projector;
the pattern projector is used for projecting a structured light pattern to the surface of an object;
the light supplementing device is used for illuminating the surface of an object.
In some embodiments, the pattern projector employs a laser light source, an LED light source, or a halogen lamp light source;
the light supplementing device is a global LED lamp.
In some embodiments, the two-dimensional hole feature calculator comprises:
the hole feature classifier is used for identifying the outline of the hole based on the object surface image and dividing the hole into different categories according to the shape of the outline;
a hole feature extractor for extracting the hole two-dimensional feature parameter based on a category of a contour of the hole.
In some embodiments, the hole feature classifier comprises:
a circular hole identifier for fitting an ellipse based on the outline shape of the hole and classifying the hole as an elliptical hole when the fitting error is less than a first threshold;
the square hole recognizer is used for detecting the angular points of the closed contour which is not classified into the round hole, and classifying the hole which has only four angular points in the contour shape of the hole and has the contour between every two adjacent angular points on a straight line as the square hole;
the long round hole recognizer is used for calculating the minimum circumscribed rectangle of the closed contour which is not classified into the round hole and the square hole, determining the length, the width and the circular arc diameter of the reference long round hole, determining the reference circumscribed rectangle, aligning the reference circumscribed rectangle with the minimum circumscribed rectangle, and classifying the hole into the long round hole when the alignment error of the reference long round hole and the corresponding closed contour is smaller than a second threshold value.
In some embodiments, the hole feature extractor comprises:
the circular hole feature extractor is used for extracting hole two-dimensional feature parameters of the elliptical hole, and the hole two-dimensional feature parameters of the elliptical hole comprise an ellipse equation, a long axis direction, a long half axis and a short half axis;
the square hole feature extractor is used for extracting hole two-dimensional feature parameters of the square hole, and the hole two-dimensional feature parameters of the square hole comprise a center point coordinate and coordinates of four vertexes;
the long circular hole feature extractor is used for extracting hole two-dimensional feature parameters of the long circular hole, and the hole two-dimensional feature parameters of the long circular hole comprise a center point coordinate and four vertex coordinates of a minimum circumscribed rectangle.
In some embodiments, the three-dimensional hole feature calculator comprises:
the hole three-dimensional reconstruction calculator is used for performing three-dimensional reconstruction on the hole based on the hole two-dimensional characteristic parameters and the binocular vision principle to obtain hole three-dimensional characteristic parameters;
and the verification calculator is used for verifying the hole three-dimensional characteristic parameters by utilizing the space geometric shape characteristics and eliminating wrong hole three-dimensional characteristic parameters.
In some embodiments, the bore three-dimensional reconstruction calculator is specifically configured to:
and carrying out epipolar matching on the surface image of the object acquired by the image acquisition sensor based on an epipolar geometry principle, and obtaining the hole three-dimensional characteristic parameters of the square hole, the round hole or the long round hole by utilizing triangular calculation when the matching relation is met.
In some embodiments, the auxiliary function calculator comprises:
a parameter optimization calculator for optimizing the hole three-dimensional characteristic parameters by using the hole two-dimensional characteristic parameters, the hole three-dimensional characteristic parameters and the weight coefficients based on a minimized reprojection error optimization method;
wherein the weight coefficient represents the weight corresponding to different object surface images.
In some embodiments, the accessibility calculator comprises:
the spatial positioning calculator is used for determining an included angle between a vector of origin points of the coordinate systems of the at least two image acquisition sensors pointing to the center of the hole and a normal direction of the hole, and registering spatial parameters of the hole obtained at different moments to the same coordinate system; the spatial parameters comprise the hole two-dimensional characteristic parameters and/or the hole three-dimensional characteristic parameters;
the data optimization calculator is used for determining the weight coefficient of the object surface image at the moment when the object surface image participates in calculation based on the included angle;
wherein the larger the included angle is, the smaller the weight coefficient is.
In some embodiments, the parameter optimization calculator is specifically configured to:
based on the hole two-dimensional characteristic parameter, the hole three-dimensional characteristic parameter, the weight coefficient and an objective function E = ∑ W i (P i (S)-F i ) 2 Optimizing the three-dimensional characteristic parameters of the holes;
determining a three-dimensional contour of the hole based on the three-dimensional characteristic parameters of the hole, projecting the three-dimensional contour onto each object surface image at each moment to obtain a contour reference value set { P of the three-dimensional contour on the object surface image i (S) }, in which P i For the projected contour reference value, P, of the hole on the ith image of the object surface i Is a function of S, S is a hole three-dimensional characteristic parameter of the spatial hole; f i Representing the two-dimensional feature parameters of the hole determined by the two-dimensional feature calculator; w i Represents the weight coefficient; and performing optimization calculation on the target function by adopting a nonlinear least square method to obtain the optimized hole three-dimensional characteristic parameters, wherein i is a positive integer.
The present disclosure also provides a three-dimensional detection method implemented by any one of the above three-dimensional detection systems, the three-dimensional detection method including:
an image acquisition sensor acquires an object surface image;
a two-dimensional hole feature calculator determines hole two-dimensional feature parameters based on the object surface image;
the three-dimensional hole characteristic calculator carries out three-dimensional reconstruction on the hole based on the hole two-dimensional characteristic parameters to determine the hole three-dimensional characteristic parameters;
and the auxiliary function calculator optimizes the hole three-dimensional characteristic parameters based on the hole two-dimensional characteristic parameters and the hole three-dimensional characteristic parameters.
In some embodiments, the two-dimensional hole feature calculator comprises a hole feature classifier and a hole feature extractor; the two-dimensional hole feature calculator determines hole two-dimensional feature parameters based on the object surface image, including:
the hole feature classifier identifies the outline of the hole based on the object surface image and divides the hole into different categories according to the shape of the outline;
the hole feature extractor extracts the hole two-dimensional feature parameter based on a category of a contour of a hole.
In some embodiments, the three-dimensional bore feature calculator comprises a bore three-dimensional reconstruction calculator and a verification calculator; the three-dimensional hole feature calculator performs three-dimensional reconstruction on the hole based on the hole two-dimensional feature parameters, and determines the hole three-dimensional feature parameters, and the method comprises the following steps:
the hole three-dimensional reconstruction calculator carries out three-dimensional reconstruction on the hole based on the hole two-dimensional characteristic parameters and the binocular vision principle to obtain hole three-dimensional characteristic parameters;
and the verification calculator verifies the hole three-dimensional characteristic parameters by utilizing the space geometric shape characteristics to eliminate wrong hole three-dimensional characteristic parameters.
In some embodiments, the auxiliary function calculator spatial location calculator, data preference calculator, spatial location calculator, and parameter optimization calculator; the auxiliary function calculator optimizes the hole three-dimensional characteristic parameter based on the hole two-dimensional characteristic parameter and the hole three-dimensional characteristic parameter, and includes:
the space positioning calculator determines an included angle between a vector of the origin points of the coordinate systems of the at least two image acquisition sensors pointing to the center of the hole and the normal direction of the hole, and registers space parameters of the hole obtained at different moments to the same coordinate system; the spatial parameters comprise the hole two-dimensional characteristic parameters and/or the hole three-dimensional characteristic parameters;
the data optimization calculator determines a weight coefficient of the object surface image at the moment when the object surface image participates in calculation based on the included angle; the larger the included angle is, the smaller the weight coefficient is;
the parameter optimization calculator optimizes the hole three-dimensional characteristic parameters by adopting the hole two-dimensional characteristic parameters, the hole three-dimensional characteristic parameters and the weight coefficients based on a minimized reprojection error optimization method;
wherein the weight coefficient represents the weight corresponding to different object surface images.
Specifically, the parameter optimization calculator is based on the hole two-dimensional feature parameter, the hole three-dimensional feature parameter, the weight coefficient, and an objective function E = ∑ W i (P i (S)-F i ) 2 Optimizing the three-dimensional characteristic parameters of the holes;
determining a three-dimensional contour of the hole based on the three-dimensional characteristic parameters of the hole, projecting the three-dimensional contour onto each object surface image at each moment to obtain a contour reference value set { P of the three-dimensional contour on the object surface image i (S) }, in which, P i For the projected contour reference, P, of the hole on the ith image of the object surface i Is a function of S, S is a hole three-dimensional characteristic parameter of the spatial hole; f i Representing the two-dimensional feature parameters of the hole determined by the two-dimensional feature calculator; w is a group of i Represents the weight coefficient; and performing optimization calculation on the target function by adopting a nonlinear least square method to obtain the optimized hole three-dimensional characteristic parameters, wherein i is a positive integer.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:
the embodiment of the present disclosure provides a three-dimensional detection system, in which: the image acquisition sensor is used for acquiring an object surface image; a two-dimensional hole feature calculator for determining hole two-dimensional feature parameters based on the object surface image; the three-dimensional hole characteristic calculator is used for carrying out three-dimensional reconstruction on the hole based on the hole two-dimensional characteristic parameters and determining hole three-dimensional characteristic parameters; and the auxiliary function calculator is used for optimizing the hole three-dimensional characteristic parameters based on the hole two-dimensional characteristic parameters and the hole three-dimensional characteristic parameters. Therefore, the hole two-dimensional characteristic parameters and the hole three-dimensional characteristic parameters are determined by identifying the surface image of the object, and the hole three-dimensional characteristic parameters are optimized by combining the hole two-dimensional characteristic parameters and the hole three-dimensional characteristic parameters by using the auxiliary function calculator, so that the effective measurement of hole positions can be realized.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.
In the related art, the method of measuring the hole site may include a conventional measuring method, a three-coordinate measuring method, and an auxiliary three-dimensional scanning measuring method.
The traditional measuring method adopts traditional tools, such as a ruler, a pen and the like, to carry out manual measurement, and has low measuring efficiency and poor precision.
In the three-coordinate measuring method, auxiliary measurement is performed by combining three-coordinate equipment, wherein the three-coordinate equipment can comprise a traditional three-coordinate machine and a three-coordinate measuring instrument, and can also comprise an optical (such as a light pen) three-coordinate measuring instrument; the principle is to perform dotting measurement on the surface of the feature, and then to use the sampling point to fit the feature to perform the required size detection. Thus, the equipment cost is high; the method belongs to a contact type measuring mode, and is poor in universality; and it is single point type measurement, and measurement efficiency is lower.
In the method for measuring the three-dimensional scanning of the auxiliary object, the auxiliary object such as a standard ball is placed on a hole, and information such as the size and the center coordinate of the hole is obtained through methods such as fitting and intersection operation of a point cloud of a sphere obtained by scanning and a point cloud of a local surface of a workpiece (hereinafter also referred to as an object), so that a measuring result required by hole position measurement is realized. However, various different standard auxiliary parts need to be prepared for hole sites with different specifications and sizes, and the preparation work is complicated; the method is usually used for measuring round holes, and holes with other shapes are difficult to apply; under the influence of gravity, the auxiliary part is limited in use mode and can only be placed on the upper surface of a workpiece, and other angles are difficult to use or the auxiliary structure is very complex to avoid the problems of loosening, falling and the like; the method is an indirect measurement method, and the uncertainty of system measurement is increased; and the process is complicated, the efficiency is low and the operator is required to have higher use experience.
In order to solve the above problems, embodiments of the present disclosure provide a three-dimensional detection system and a three-dimensional detection method, so as to implement high-efficiency hole site detection; the three-dimensional detection system solves the problems that a traditional optical three-dimensional detection system is complex in scheme, high in hardware and use cost and low in efficiency when measuring hole sites, enables a real-time high-precision hole site measurement technology based on a multi-view stereoscopic vision principle to be possible, enables traditional three-dimensional scanning equipment to have the capacity of efficiently detecting hole sites, and greatly improves the practicability of the system.
The three-dimensional inspection system and method provided by the embodiments of the present disclosure are exemplified below with reference to fig. 1 to 4.
Fig. 1 is a schematic structural diagram of a three-dimensional inspection system according to an embodiment of the present disclosure. Referring to fig. 1, the three-dimensional inspection system may include: an image capturing sensor 12 for capturing an image of a surface of an object; a two-dimensional hole feature calculator 13 for determining a hole two-dimensional feature parameter based on the object surface image; a three-dimensional hole feature calculator 14, configured to perform three-dimensional reconstruction on the hole based on the hole two-dimensional feature parameter, and determine a hole three-dimensional feature parameter; and the auxiliary function calculator 15 is used for optimizing the hole three-dimensional characteristic parameters based on the hole two-dimensional characteristic parameters and the hole three-dimensional characteristic parameters.
Optionally, the three-dimensional inspection system may further comprise at least one pattern projector 11 for projecting a structured light pattern onto the surface of the object; and the number of the image pickup sensors 12 may be one, two, or more.
The structured light pattern is a projection pattern with a geometric figure and is used for carrying out feature identification on the surface of the object to be detected so as to realize feature matching of the surface of the object to be detected. Illustratively, the shape of the projected pattern may be a fringe and/or speckle, for example: the stripes may include sinusoidal stripes or binary stripes, the speckles may include black and white snowflake patterns or patterns similar to the light and dark gaps of the two-dimensional code, and may also include other optical patterns with specific structures, which is not limited by the embodiments of the present disclosure.
Based on this, the image projector 11 can project the structured light pattern to the surface of the object, and the image capturing sensor 12 can implement three-dimensional measurement of the shape of the object by using the structured light three-dimensional scanning principle.
The number of the image capturing sensors 12 may be at least two, and specifically may be two, three or more, so as to capture the surface image of the object at different angles of the object, and form data that can be used for performing three-dimensional reconstruction and two-dimensional feature extraction. Illustratively, when the number of the image capturing sensors 12 is more than two, the object surface image can be captured using any at least two thereof.
Illustratively, the image capturing sensors 12 are rigidly connected to each other, so that they are not easily deformed and have high data reliability.
The two-dimensional hole feature calculator 13 may extract two-dimensional hole feature parameters based on the surface image of the object acquired by the image acquisition sensor 12, and is an important basis for three-dimensional hole measurement.
The three-dimensional hole feature calculator 14 can perform three-dimensional reconstruction on the hole based on the two-dimensional hole feature calculator 13 to determine the three-dimensional hole feature parameters, and is favorable for realizing real-time high-precision hole position measurement.
The auxiliary function calculator 15 can optimize the hole three-dimensional characteristic parameters based on the data of dynamic multi-frame sampling by combining the hole two-dimensional characteristic parameters and the hole three-dimensional characteristic parameters, thereby improving the precision and robustness of hole position measurement.
The three-dimensional detection system provided by the embodiment of the disclosure can be used for hole site measurement, and in the three-dimensional detection system, the image acquisition sensor 12 can acquire an image of the surface of an object; the two-dimensional hole feature calculator 13 may determine a hole two-dimensional feature parameter based on the object surface image; the three-dimensional hole feature calculator 14 may perform three-dimensional reconstruction of the hole based on the hole two-dimensional feature parameters to determine hole three-dimensional feature parameters; the auxiliary function calculator 15 may optimize the hole three-dimensional characteristic parameter based on the hole two-dimensional characteristic parameter and the hole three-dimensional characteristic parameter; because the dotting measurement is not needed, and the auxiliary objects such as standard balls are not needed, when the system is used for hole position measurement, the universality is higher, the measurement efficiency is higher, the scheme is simple, the hardware and the use cost are lower, the measurement precision is higher, and the robustness is better.
Alternatively, when the three-dimensional detection system comprises the pattern projector 11, a three-dimensional reconstruction may be achieved based on the structured light pattern.
In some embodiments, fig. 2 is a schematic structural diagram of another three-dimensional inspection system according to an embodiment of the disclosure. On the basis of fig. 1, with reference to fig. 2, at least two image acquisition sensors 12 and at least one pattern projector 11 are provided in the same handheld scanning device 01.
Wherein, the basic hardware that realizes hole site measurement function is two image acquisition sensors 12, and it can adopt hand-held type scanning equipment 01 to realize promptly and realize sharing basic hardware with hand-held type three-dimensional scanning equipment, so can need not additionally to increase hardware structure, has promoted hand-held type scanning equipment 01 simultaneously by a wide margin and has been used for the hole site measuring practicality.
In addition, in this embodiment, the spatial hole position measurement can be realized only by using at least two image acquisition sensors 12, and the optical scanning system has a simple structure, high portability and low cost.
In some embodiments, with continued reference to fig. 2, the handheld scanning device 01 further comprises at least one light supplementing apparatus 011; the light supplement device 011 is used for illuminating the surface of an object.
The light supplement device 011 is used for illuminating the surface of an object, and can improve the contrast of patterns on two sides of the outline of the hole or assist in identifying other features on the patterns, such as mark points.
It should be noted that fig. 1 and fig. 2 only show 2 image capture sensors 12 and 1 pattern projector 11, and fig. 2 only shows 1 fill light device 011, but neither of them constitutes a limitation on the three-dimensional detection system provided in the embodiment of the present disclosure. In other embodiments, the number of the image capturing sensor 12, the number of the pattern projector 11, and the number of the light supplement device 011 can be set according to the requirement of the three-dimensional detection system, which is not limited in the embodiments of the present disclosure.
In some embodiments, the pattern projector 11 employs a laser light source, an LED light source, or a halogen lamp light source; the light supplement device 011 is a global LED lamp.
The light source of the pattern projector 11 can be flexibly set according to the three-dimensional scanning requirement, and can also be structured light source of other types; similarly, the light supplement device 011 can also be other types of light sources, which is not limited in the embodiments of the present disclosure.
In some embodiments, fig. 3 is a schematic structural diagram of another three-dimensional inspection system according to an embodiment of the present disclosure. On the basis of fig. 1, referring to fig. 3, the two-dimensional hole feature calculator 13 includes: a hole feature classifier 131 for recognizing the outline of the hole based on the object surface image and classifying the hole into different categories according to the shape of the outline; an aperture feature extractor 132 for extracting aperture two-dimensional feature parameters based on the class of the outline of the aperture.
The hole feature classifier 131 may perform image processing on the object surface image, identify or extract a hole contour (herein, a closed contour) in the object surface image, and classify the holes into different categories according to the contour shape, such as round holes, square holes, oblong holes, or other categories of holes.
In some embodiments, with continued reference to fig. 3, the pore feature classifier 131 includes: a circular hole identifier 1311 for fitting an ellipse based on the outline shape of the hole, and classifying the hole as an elliptical hole when a fitting error is smaller than a first threshold; a square hole identifier 1312 for detecting the corner points of the closed contour not classified into circular holes, and classifying the holes with four corner points in the contour shape of the holes and the contour on a straight line between every two adjacent corner points as square holes; the long round hole identifier 1313 is configured to calculate a minimum circumscribed rectangle that is not classified into the closed contours of the round hole and the square hole, determine the length, the width, and the arc diameter of the reference long round hole, determine the reference circumscribed rectangle, align the reference circumscribed rectangle with the minimum circumscribed rectangle, and classify the hole as a long round hole when an alignment error between the reference long round hole and the corresponding closed contour is smaller than a second threshold.
First, the circular hole identifier 1311 performs ellipse fitting, and if the fitting error is smaller than the first threshold, the corresponding hole is classified as a circular hole.
The rear hole identifier 1312 performs corner detection on the closed contour that is not classified by the circular hole identifier 1311, and if the closed contour has only four corners and the contour between two adjacent corners is on a straight line, the corresponding hole is classified as a square hole.
Finally, the oblong hole identifier 1313 is used to calculate holes that are not identified by the square hole identifier 1312. The method specifically comprises the following steps: calculating a minimum circumscribed rectangle of the unclassified closed contour Pa, wherein the length of the minimum circumscribed rectangle corresponds to the length L of the long round hole, if only two sections of parallel straight edges with approximate equal lengths coincide with one group of opposite edges of the minimum circumscribed rectangle, counting the length of the straight edges of the long round hole, and marking the length as L0, the width of the long round hole, namely the diameter of the circular arc is D = L-L0 (a subsequent data optimization calculator ensures that the plane of the hole involved in calculation is approximately vertical to the direction of the optical axis of the camera, namely approximately orthographic projection is carried out on an imaging plane, and at the moment, the shearing distortion in the length/width direction is negligible). Based on the length L and the arc diameter D, generating an oblong hole template P0 with the length L and the width and the arc diameters at two sides D, namely generating a reference oblong hole P0; and aligning the minimum circumscribed rectangle (namely, the reference circumscribed rectangle) of the long circular hole template P0 with the minimum circumscribed rectangle of the contour to be classified through affine transformation, and calculating the alignment error of the aligned long circular hole template P0 and the corresponding closed contour Pa. For example, n sampling points may be generated on the long circular hole template P0, n closest points on the corresponding closed contour Pa are searched, n sets of corresponding point pairs are generated, an average value of distances between the corresponding points may be used as an alignment error between the long circular hole template P0 and the corresponding closed contour Pa, and n is a positive integer. When the alignment error of the reference oblong hole P0 with its corresponding closed profile Pa is smaller than a second threshold, then the closed profile Pa is classified as an oblong hole.
The first threshold and the second threshold may be set according to requirements of the three-dimensional detection system, and may be related to measurement accuracy requirements and a specific fitting algorithm adopted, which is not limited in the embodiment of the present disclosure.
Of course, the order of the holes is not limited.
In other embodiments, the hole feature classifier 131 may further include other types of hole identifiers, that is, multiple different types of identifiers (classifiers) may be set according to the hole location detection requirement, which is neither described nor limited in this embodiment of the present disclosure.
The hole feature extractor 132 may extract features of the classified holes, that is, extract feature shape parameters corresponding to the classified holes according to the shape categories to which the holes belong, thereby determining two-dimensional feature parameters of the holes.
In some embodiments, with continued reference to fig. 3, the aperture feature extractor 132 includes: the circular hole feature extractor 1321 is configured to extract hole two-dimensional feature parameters of the elliptical holes, where the hole two-dimensional feature parameters of the elliptical holes include an ellipse equation, a major axis direction, a major semi-axis and a minor semi-axis; the square hole feature extractor 1322 is used for extracting hole two-dimensional feature parameters of the square hole, and the hole two-dimensional feature parameters of the square hole comprise a center point coordinate and coordinates of four vertexes; and the slotted hole feature extractor 1323 is used for extracting hole two-dimensional feature parameters of the slotted hole, wherein the hole two-dimensional feature parameters of the slotted hole comprise a center point coordinate and four vertex coordinates of a minimum circumscribed rectangle.
When the hole is a circular hole, the circular hole feature extraction 1321 extracts elliptical shape parameters, for example, parameters including center coordinates, major/minor semi-axes, major axis direction, an elliptical equation, and the like. When the hole is a quadrangle, that is, the projection shape of the square hole in the image of the object surface is a square, the square hole feature extractor 1322 extracts the shape parameters thereof, which may include parameters such as coordinates of a center point, coordinates of four vertices, and the like. When the hole is a long circular hole, the long circular hole feature extractor 1323 extracts shape parameters of the hole, which may include parameters such as coordinates of a central point, coordinates of four vertexes of a minimum circumscribed rectangle, and the like.
Therefore, the extraction of the two-dimensional characteristic parameters of the holes is realized based on the hole types.
In other embodiments, when the holes are holes of other types, the corresponding characteristic parameters may also be extracted, which is not limited in the embodiments of the present disclosure.
In some embodiments, with continued reference to fig. 3, the three-dimensional hole feature calculator 14 may include: a hole three-dimensional reconstruction calculator 141 which performs three-dimensional reconstruction of holes based on a binocular vision (multi-eye stereoscopic vision) principle to obtain hole three-dimensional characteristic parameters; and the verification calculator 142 is used for verifying the hole three-dimensional characteristic parameters by utilizing the space geometric shape characteristics to eliminate the wrong hole three-dimensional characteristic parameters.
Thus, the three-dimensional hole feature calculator 14 can realize three-dimensional reconstruction and verification of the hole, and can improve the accuracy and robustness of hole location measurement.
In some embodiments, the bore three-dimensional reconstruction calculator 141 is specifically configured to: based on an epipolar geometry principle, the object surface images acquired by the image acquisition sensor are subjected to epipolar matching, and when a matching relation is met, hole three-dimensional characteristic parameters of square holes, round holes or long round holes are obtained through triangular calculation.
Aiming at the circular holes, the center coordinates of ellipses in the surface images (hereinafter, the images can be simply referred to as 'cameras') of the object acquired by two image acquisition sensors (hereinafter, the images can be simply referred to as 'images') are subjected to epipolar matching by utilizing an epipolar geometry principle, an ellipse A and an ellipse B are set as a group of matched hole pairs in the two images obtained by the epipolar matching, m points are sampled on the outline of the ellipse A, m is a positive integer, an epipolar equation is calculated for each sampling point a0, the intersection point of the epipolar line and the outline of the ellipse B is a candidate matching point of a0, and the number of the intersection points is at most 2. The condition that the local contour section is overlapped with the epipolar line is not calculated, and the subsequent overall shape fitting is not influenced. Because the imaging planes of the two image acquisition sensors are approximately consistent in the space direction, the relative orientation relation (up-down, left-right) between the corresponding points and the located outline in the two images has invariance; based on the method, the unique corresponding point B0 of the a0 on the outline of the ellipse B can be obtained by screening according to the direction of the a0 on the outline of the ellipse A, and the three-dimensional space point p0 is obtained by utilizing triangular calculation, namely, one point on the space outline of the round hole is obtained. And similarly, obtaining the spatial coordinates of all m sampling points, fitting a circular equation, namely a three-dimensional profile equation, and obtaining the three-dimensional characteristic parameters of the holes of the round holes to obtain the initial values of the spatial shape parameters of the round holes.
Aiming at the square hole, the center coordinates of the square hole in the images collected by the two cameras are subjected to antipodal matching by utilizing an antipodal geometry principle, then four vertexes of the matching hole pair are subjected to antipodal matching respectively, if the four vertexes can meet the one-to-one corresponding matching relation, the space coordinates of the four vertexes of the square hole are reconstructed by utilizing triangular calculation, and the initial values of the space shape parameters (center coordinates, side length, direction and the like) of the square hole can be obtained, namely the hole three-dimensional characteristic parameters of the square hole are obtained.
Aiming at the oblong holes, similar to the round holes, matched oblong hole pairs are obtained through center point-to-pole matching, then contour three-dimensional reconstruction is carried out, a minimum circumscribed rectangle of the contour is calculated, the length and the width of the circumscribed rectangle are the length and the width (the diameter of an arc) of the oblong hole, and it can be seen that the shape parameters of the oblong hole are equivalent to the space shape parameters of the minimum circumscribed rectangle, including parameters such as a center coordinate, four-vertex coordinates, side length and direction. Therefore, the hole three-dimensional characteristic parameters of the oblong hole can be obtained.
In other embodiments, when the holes are holes of other types, the hollow three-dimensional characteristic parameters may also be obtained by using the epipolar geometry principle, corresponding to the characteristic shape parameters, which is not limited in the embodiments of the present disclosure.
On the basis, the verification calculator 142 may verify the three-dimensional reconstruction result of the hole by using the spatial geometry characteristics (the spatial geometry characteristics of the hole are obtained based on the category of the outline of the hole), and exclude an error result.
Illustratively, for a round hole, when the fitting error of the circle is greater than a third threshold, the round hole is excluded from the result; for the square holes, when four sides are not equal (the maximum deviation of the side length is greater than a fourth threshold), the corresponding square holes are excluded from the result; and fitting straight lines to the straight edges and fitting circles to the circular edges for the long circular holes, and when the fitting error is larger than a fifth threshold value, excluding the corresponding long circular holes from the result. The values of the third threshold, the fourth threshold, and the fifth threshold may be set according to requirements of the three-dimensional detection system, which is not limited in the embodiment of the present disclosure.
Therefore, the hole site measurement accuracy can be further improved by verifying the three-dimensional reconstruction result.
In some embodiments, with continued reference to fig. 3, the auxiliary function calculator 15 includes: a spatial orientation calculator 151, configured to determine an included angle between a vector in which the origin points of the coordinate systems of the at least two image capturing sensors point to the center of the hole and a normal direction of the hole (a spatial pose of the image capturing sensors can be obtained), and to register spatial parameters of the hole obtained at different times to the same coordinate system; the included angle can be determined by combining the result of three-dimensional reconstruction, and the space parameters comprise hole two-dimensional characteristic parameters and/or hole three-dimensional characteristic parameters; a data preference calculator 152 for determining a weight coefficient of the object surface image at the time point when the object surface image participates in the calculation based on the included angle; the larger the angle, the smaller the weight factor.
The spatial location calculator 151 may use any location method known to those skilled in the art and known in the three-dimensional scanning technology to register the spatial information (center coordinates, contour, etc.) of the hole in the device coordinate system obtained at different times into the same coordinate system.
For example, three-dimensional reconstruction calculation is performed on feature points (including mark points, geometric features, texture features and the like) on the surface of the workpiece at each moment, and a coordinate transformation matrix is calculated by using three or more feature points with the same name at different moments, that is, the transformation matrix can be used for registering spatial information of holes at different moments to the same coordinate system, which may be an object coordinate system for example. Therefore, the splicing of data at different moments can be realized, and a relatively complete data set of the surface of the station can be obtained.
The data preference calculator 152 is used for determining the weighting coefficients of the data at different time points in the splicing process. In the three-dimensional scanning process, when the included angle between the sight line direction from a camera to a hole (namely a vector of the origin of a camera coordinate system pointing to the center of the hole) and the normal direction of a spatial hole is large, the imaging of the inner wall of the hole may influence the detection precision of the hole outline during image processing, and the smaller the included angle is, the smaller the influence is, the more accurate the measurement result tends to; the larger the included angle, the larger the influence, and the measurement result tends to be inaccurate.
Based on this, the spatial pose (i.e. positioning information) of the image capturing sensor given by the spatial positioning calculator 151 is combined, the included angle between the normal direction of the hole and the sight line direction can be calculated in real time, the measurement data at this moment is evaluated by using the function of the included angle, and the estimated measurement data is used as the weight coefficient participating in the final optimization calculation, so that the larger the included angle, the smaller the weight of the sample data is. The function of the included angle may be an inverse distance weight function, a gaussian weight function, or other functions known to those skilled in the art, which is not limited in the embodiments of the present disclosure. It should be noted that, in this paragraph, the same weighting factor is given to the measurement data at a certain time, and individual measurement points in the measurement data are not distinguished, which is beneficial to simplifying the data processing process while ensuring higher measurement accuracy.
In some embodiments, with continued reference to fig. 3, the auxiliary function calculator 15 further comprises: a parameter optimization calculator 153, wherein the parameter optimization calculator 153 is configured to optimize the hole three-dimensional feature parameters by using the hole two-dimensional feature parameters, the hole three-dimensional feature parameters, and the weight coefficients based on a minimized reprojection error optimization method; wherein the weight coefficient represents the weight corresponding to different object surface images.
The method for optimizing the hole three-dimensional characteristic parameters based on the minimized reprojection error optimization method comprises the following steps: and constructing a minimized re-projection error optimization function based on the hole two-dimensional characteristic parameters, the hole three-dimensional characteristic parameters and the weight coefficients, and performing optimization calculation to obtain optimized hole three-dimensional characteristic parameters.
Wherein, the optimization function for minimizing the reprojection error is an objective function E = ∑ W i (P i (S)-F i ) 2 Therefore, the hole three-dimensional characteristic parameters can be optimized based on the hole two-dimensional characteristic parameters, the hole three-dimensional characteristic parameters, the weight coefficients and the objective function; determining a three-dimensional profile of the hole based on the three-dimensional characteristic parameters of the hole, namely determining a hole profile equation; combining the spatial pose of the image acquisition sensor, projecting the three-dimensional contour to each object surface image at each moment to acquire a contour reference value set { P } of the three-dimensional contour on the object surface image i (S) }, in which P i For the projected contour reference, P, of the hole on the ith object surface image i Is a function of S, S being a hole three-dimensional characteristic parameter of the spatial hole; f i Representing a two-dimensional feature parameter of the hole determined by the two-dimensional feature calculator; w i Represents a weight coefficient; and (3) performing optimization calculation on the target function by adopting a nonlinear least square method to obtain optimized hole three-dimensional characteristic parameters, wherein i is a positive integer.
The hole site information (i.e., the hole three-dimensional characteristic parameters) obtained by the three-dimensional hole characteristic calculator 14 is affected by errors caused by single measurement data noise, measurement angles, and the like, and has a problem of insufficient precision and robustness.
In view of the above, by combining the dynamic scanning characteristic of the handheld scanning device and performing optimal calculation on the hole location information by using a large number of sampling results of the time sequence, the measurement accuracy and robustness can be significantly improved.
Exemplarily, in combination with the above data stitching process, determining a three-dimensional profile of the hole based on the three-dimensional characteristic parameters of the hole at a plurality of different times; projecting the three-dimensional contour to each camera image at each moment to obtain an estimated value { P ] of the hole contour on the image i (S) }, also called the profile reference value set { P } i (S) }; wherein, P i Also referred to as the estimate of the projected profile of the hole on the ith image, P i Is a function of S, S being a shape parameter (e.g. a profile equation) of the spatial hole, the observed value of the hole profile on the image having been obtained by a two-dimensional feature calculator, i.e. F i . Based on this, an objective function E = ∑ W is set i (P i (S)-F i ) 2; wherein, W i The weight coefficients determined by the data preference calculator 152 are determined using a non-linear least squares method(for example, an L-M algorithm) performs optimization calculation on the objective function M to obtain the final S, namely the shape parameters of the optimized space hole, so that more accurate measurement information such as three-dimensional coordinates, aperture and the like of the space hole can be obtained through calculation.
The contour projection is to project a three-dimensional contour of the calculated hole in space onto an image, wherein the three-dimensional contour is a parameterized result fitted by the cooperative calculation of the calculators according to the acquired surface image of the object, and the projection of the three-dimensional contour is deviated from a two-dimensional contour actually detected on the image; on the other hand, the optimization calculation is to project the three-dimensional contour onto all effective images for optimization, and is not to project the reconstruction result of a certain frame back onto two images of the frame, and is not to simply perform the inverse calculation in the reconstruction process.
In the three-dimensional detection system provided by the embodiment of the disclosure, an image acquisition sensor in the handheld scanning device can acquire an image of the surface of an object, which is substantially a 2D image; a hole feature classifier in the two-dimensional hole feature calculator may determine a shape class to which the hole belongs based on the 2D image; a hollow feature extractor in the two-dimensional hole feature calculator can determine the feature parameters of the holes based on the shape types of the holes, namely determine the two-dimensional feature parameters of the holes; the hole three-dimensional reconstruction calculator in the three-dimensional hole feature calculator can perform three-dimensional reconstruction based on the object surface image and the shape feature parameters, determine the hole three-dimensional feature parameters, for example, can determine coordinates of a space central point, an angular point and the like of a hole, and determine a hole contour equation, shape parameters and the like; the checking calculator in the three-dimensional hole characteristic calculator can eliminate the special-shaped result, namely the wrong hole three-dimensional characteristic parameter, based on the shape class and the three-dimensional reconstruction result of the hole; the spatial positioning calculator in the auxiliary function calculator can determine the spatial position of the handheld scanning equipment (the position relation between the handheld scanning equipment and the hole plane in a certain image acquisition process) based on the three-dimensional reconstruction result and register the spatial parameters of the holes under an object coordinate system; the data optimization calculator in the auxiliary function calculator can determine the weight coefficient of the measurement data at the moment to participate in final calculation based on the spatial pose of the handheld scanning device; and the parameter optimization calculator in the auxiliary function calculator is used for optimizing the hole three-dimensional characteristic parameters based on the hole two-dimensional characteristic parameters (namely characteristic shape parameters), the space pose of the handheld scanning equipment and the three-dimensional reconstruction result registered to the object coordinate system, and obtaining the optimized hole three-dimensional special diagnosis parameters.
The three-dimensional detection system provided by the embodiment of the disclosure can be used for hole site measurement, realizes a real-time high-precision hole site measurement method based on a multi-eye stereo vision principle, also enables the traditional handheld three-dimensional scanning equipment to have the capacity of efficiently detecting hole sites, greatly improves the practicability of the system, and solves the problems of complex scheme, high hardware and use cost, low efficiency and the like when the traditional three-dimensional detection system is used for hole site measurement.
The three-dimensional detection system provided by the embodiment of the disclosure has at least the following beneficial effects:
firstly, a hole site detection scheme based on a multi-eye stereoscopic vision principle is realized. The embodiment provides the space hole site detection system which can be realized by using at least two optical acquisition sensors, and the device has the advantages of simple structure, portability, low cost, capability of realizing real-time measurement, high measurement efficiency and high precision.
Second, the system can be integrated with three-dimensional scanning equipment, including handheld three-dimensional scanning equipment, so that the three-dimensional scanning equipment has the capability of rapidly measuring hole sites. Specifically, in the system provided in this embodiment, the basic hardware for implementing the hole site measurement function is at least two optical acquisition sensors (i.e., image acquisition sensors), which can be shared by the three-dimensional scanning device, so that the cost is not increased, but the practicability of the three-dimensional scanning device is greatly improved.
Thirdly, a two-dimensional hole characteristic calculator is arranged, the shape characteristics of the holes can be used for image recognition, the holes are classified, and characteristic property parameters are extracted, so that the two-dimensional hole characteristic calculator is an important basis for hole site three-dimensional measurement.
And fourthly, a three-dimensional hole characteristic calculator is arranged, so that three-dimensional reconstruction and verification of hole positions can be realized, and the accuracy and robustness of hole position measurement are improved.
And fifthly, an auxiliary function calculator is arranged, optimization and statistical optimization calculation can be carried out based on the data of dynamic multi-frame sampling, and the precision and the robustness of hole site measurement are improved.
On the basis of the above embodiments, the embodiments of the present disclosure further provide a three-dimensional detection method, which can be performed by any one of the three-dimensional detection systems in the above embodiments. Therefore, the three-dimensional detection method also has the beneficial effects of the three-dimensional detection system. The same can be understood by referring to the explanation of the three-dimensional detection system above, and the explanation is not repeated below.
In some embodiments, fig. 4 is a schematic flow chart of a three-dimensional detection method according to an embodiment of the disclosure. Referring to fig. 4, the three-dimensional detection method includes:
s201, an image acquisition sensor acquires an object surface image.
S202, the two-dimensional hole feature calculator determines hole two-dimensional feature parameters based on the object surface image.
S203, the three-dimensional hole feature calculator carries out three-dimensional reconstruction on the hole based on the hole two-dimensional feature parameters to determine the hole three-dimensional feature parameters.
And S204, optimizing the hole three-dimensional characteristic parameters by the aid of the auxiliary function calculator based on the hole two-dimensional characteristic parameters and the hole three-dimensional characteristic parameters.
The three-dimensional detection method provided by the embodiment of the disclosure can be used for hole site measurement, and in the three-dimensional detection method, an image acquisition sensor acquires an object surface image; a two-dimensional hole feature calculator determines hole two-dimensional feature parameters based on the object surface image; the three-dimensional hole feature calculator carries out three-dimensional reconstruction on the hole based on the hole two-dimensional feature parameters to determine the hole three-dimensional feature parameters; the auxiliary function calculator optimizes the hole three-dimensional characteristic parameters based on the hole two-dimensional characteristic parameters and the hole three-dimensional characteristic parameters; because the dotting measurement is not needed, and the auxiliary objects such as standard balls are not needed, when the system is used for hole position measurement, the universality is higher, the measurement efficiency is higher, the scheme is simple, the hardware and the use cost are lower, the measurement precision is higher, and the robustness is better.
In some embodiments, which may be combined with FIG. 3, the two-dimensional hole feature calculator includes a hole feature classifier and a hole feature extractor. Based on this, S202 in fig. 4 may include:
the hole feature classifier identifies the outline of the hole based on the object surface image and divides the hole into different categories according to the shape of the outline;
the hole feature extractor extracts a hole two-dimensional feature parameter based on a category of a contour of a hole.
In this way, the object surface image can be identified through the two-dimensional hole feature calculator, hole outlines (herein, closed outlines) in the object surface image are extracted, and the holes are divided into different categories according to the outline shapes; and extracting the hole two-dimensional characteristic parameters of the classified holes, and providing data support for subsequent hole three-dimensional reconstruction.
In some embodiments, which may be combined with FIG. 3, the three-dimensional bore feature calculator includes a bore three-dimensional reconstruction calculator and a verification calculator. Based on this, S203 in fig. 4 may include:
the hole three-dimensional reconstruction calculator carries out three-dimensional reconstruction on the holes based on the hole two-dimensional characteristic parameters and the binocular vision principle to obtain the hole three-dimensional characteristic parameters;
the checking calculator checks the hole three-dimensional characteristic parameters by using the space geometric shape characteristics, and eliminates wrong hole three-dimensional characteristic parameters.
Therefore, the three-dimensional reconstruction and verification of the hole can be realized through the three-dimensional hole characteristic calculator, and the accuracy and the robustness of hole position measurement can be improved.
In some embodiments, an auxiliary function calculator, a spatial location calculator, a data preference calculator, a spatial location calculator, and a parameter optimization calculator may be combined with FIG. 3. Based on this, S204 in fig. 4 may include:
the spatial positioning calculator determines an included angle between a vector of the origin points of the coordinate systems of the at least two image acquisition sensors pointing to the center of the hole and the normal direction of the hole, and registers spatial parameters of the hole obtained at different moments to the same coordinate system; the spatial parameters comprise hole two-dimensional characteristic parameters and/or hole three-dimensional characteristic parameters;
the data optimization calculator determines a weight coefficient of the object surface image at the moment when the object surface image participates in calculation based on the included angle; the larger the included angle is, the smaller the weight coefficient is;
the parameter optimization calculator is based on the hole two-dimensional characteristic parameters, the hole three-dimensional characteristic parameters, the weight coefficients and the objective function E =sigmaW i (P i (S)-F i ) 2 Optimizing the three-dimensional characteristic parameters of the hole;
determining a three-dimensional contour of the hole based on the three-dimensional characteristic parameters of the hole, projecting the three-dimensional contour onto each object surface image at each moment to obtain a contour reference value set { P ] of the three-dimensional contour on the object surface image i (S) }, in which P i For the projected contour reference value, P, of the hole on the ith object surface image i Is a function of S, S is a hole three-dimensional characteristic parameter of the spatial hole; f i Representing the two-dimensional characteristic parameters of the hole determined by the two-dimensional characteristic calculator; w i Represents a weight coefficient; and (4) performing optimization calculation on the target function by adopting a nonlinear least square method to obtain an optimized hole three-dimensional characteristic parameter, wherein i is a positive integer.
Therefore, the auxiliary function calculator can be used for carrying out optimization and statistical optimization calculation based on the data of dynamic multi-frame sampling, and the precision and the robustness of hole site measurement are improved.
In the three-dimensional detection method provided by the embodiment of the disclosure, feature extraction is performed based on an object surface image, holes are classified according to shapes, and hole two-dimensional shape parameters are obtained, namely, a 2D contour is obtained; then, performing three-dimensional reconstruction based on the sampling points on the 2D contour to obtain a single-frame (sheet) three-dimensional point cloud at the moment, and fitting and calculating space shape parameters by using the three-dimensional point cloud to obtain hole three-dimensional shape parameters including the three-dimensional contour of the hole; the fitted space shape parameters can be verified by utilizing the space geometric shapes, and error results in the space shape parameters are eliminated; on the basis, splicing a plurality of pieces of three-dimensional point cloud; determining the weight coefficient of each frame of three-dimensional point cloud in the corresponding multiple frames of different three-dimensional point clouds based on the sight line direction from the camera to the hole and the normal included angle of the space hole during image acquisition; projecting the three-dimensional outline of the hole to each camera image at each moment to obtain an estimation value of the outline of the hole on the image, and establishing a function between the estimation value and a two-dimensional hole characteristic calculation value to perform optimization calculation to obtain an optimized three-dimensional shape parameter of the hole; thus, the measurement accuracy and robustness can be improved.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of other like elements in a process, method, article, or system comprising the element.
The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.