CN114012731B - Hand-eye calibration method and device, computer equipment and storage medium - Google Patents
Hand-eye calibration method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN114012731B CN114012731B CN202111394621.9A CN202111394621A CN114012731B CN 114012731 B CN114012731 B CN 114012731B CN 202111394621 A CN202111394621 A CN 202111394621A CN 114012731 B CN114012731 B CN 114012731B
- Authority
- CN
- China
- Prior art keywords
- coordinate system
- mechanical arm
- marker
- hand
- coordinate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000003860 storage Methods 0.000 title claims abstract description 14
- 239000003550 marker Substances 0.000 claims abstract description 215
- 238000005070 sampling Methods 0.000 claims abstract description 191
- 238000005457 optimization Methods 0.000 claims description 39
- 230000009466 transformation Effects 0.000 claims description 35
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 25
- 238000006243 chemical reaction Methods 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 13
- 238000013519 translation Methods 0.000 claims description 8
- 239000011159 matrix material Substances 0.000 description 12
- 238000004364 calculation method Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 238000013507 mapping Methods 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000010187 selection method Methods 0.000 description 2
- OAICVXFJPJFONN-UHFFFAOYSA-N Phosphorus Chemical compound [P] OAICVXFJPJFONN-UHFFFAOYSA-N 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000004745 nonwoven fabric Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1679—Programme controls characterised by the tasks executed
- B25J9/1692—Calibration of manipulator
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
- B25J19/0095—Means or methods for testing manipulators
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Manipulator (AREA)
Abstract
The application relates to a hand-eye calibration method, a hand-eye calibration device, computer equipment and a storage medium. The method comprises the following steps: establishing a hand-eye calibration system, wherein the hand-eye calibration system comprises a mechanical arm, a marker and a camera; controlling the tail end of the mechanical arm to move to a first sampling point from an initial position, and determining an initial calibration relation based on the coordinates of the first sampling point in a mechanical arm base coordinate system and the coordinates of the marker in a pixel coordinate system; selecting a second sampling point from the pixel coordinate system, and determining a corresponding base coordinate of the second sampling point in a mechanical arm base coordinate system based on the initial calibration relation; controlling the tail end of the mechanical arm to move to a target position corresponding to the base coordinate, and acquiring a tail end pose of the mechanical arm and a coordinate of the marker corresponding to the camera coordinate system at the target position; and determining the hand-eye calibration relation and the coordinates of the marker in the mechanical arm coordinate system based on the terminal pose of the mechanical arm and the coordinates of the marker in the camera coordinate system. By adopting the method, the accuracy and the efficiency of calibrating the hands and the eyes can be improved.
Description
Technical Field
The present application relates to the field of mechanical arm calibration technologies, and in particular, to a hand-eye calibration method and apparatus, a computer device, and a storage medium.
Background
With the rapid development of computer technology and digital image processing, machine vision has wide application in the fields of scientific research, industrial production and the like. In robot vision applications, hand-eye calibration is a very fundamental and critical issue. The aim of hand-eye calibration is simply to acquire the relationship between the coordinate system of the mechanical arm and the coordinate system of the camera, so as to transfer the result of visual recognition to the coordinate system of the mechanical arm.
At present, the hand-eye calibration is usually realized by controlling the translation and rotation of a mechanical arm in a manual teaching mode, the hand-eye calibration scheme has complex operation and long time consumption, and the calibration precision can be achieved is low, so that the hand-eye calibration effect is poor; in addition, in the existing automatic hand-eye calibration method, during calculation, coordinate information of a marker in a mechanical arm coordinate needs to be given, and when the information is difficult to obtain, the applicability of the method is limited, and unnecessary external errors are easily introduced into a hand-eye system, so that the hand-eye calibration precision is difficult to meet the requirement.
Disclosure of Invention
In view of the above, it is necessary to provide an automatic hand-eye calibration method, an automatic hand-eye calibration device, a computer device, and a storage medium with higher applicability and higher precision.
A hand-eye calibration method, the method comprising:
establishing a hand-eye calibration system, wherein the hand-eye calibration system comprises a mechanical arm, a marker and a camera;
controlling the tail end of the mechanical arm to move to a first sampling point from an initial position, and determining an initial calibration relation based on the coordinates of the first sampling point in a mechanical arm base coordinate system and the coordinates of the marker in a pixel coordinate system; the initial calibration relation is an approximate conversion relation between a pixel coordinate system and the mechanical arm base coordinate system;
selecting a second sampling point from the pixel coordinate system, and determining a corresponding base coordinate of the second sampling point in a mechanical arm base coordinate system based on the initial calibration relation;
controlling the tail end of the mechanical arm to move to a target position corresponding to the base coordinate, and acquiring the tail end pose of the mechanical arm and the coordinates of the marker in a camera coordinate system at the target position;
and determining the hand-eye calibration relation and the coordinates of the markers in the coordinate system of the mechanical arm based on the end pose of the mechanical arm and the coordinates of the markers in the coordinate system of the camera so as to realize the hand-eye calibration from the camera to the mechanical arm.
A hand-eye calibration apparatus, the apparatus comprising:
the hand-eye calibration system establishing module is used for establishing a hand-eye calibration system, and the hand-eye calibration system comprises a mechanical arm, a marker and a camera;
the initial calibration relation determining module is used for controlling the tail end of the mechanical arm to move from an initial position to a first sampling point, and determining an initial calibration relation based on the coordinates of the first sampling point in a mechanical arm base coordinate system and the coordinates of the marker in a pixel coordinate system; the initial calibration relation is an approximate conversion relation between a pixel coordinate system and the mechanical arm base coordinate system;
the second sampling point determining module is used for selecting a second sampling point from the pixel coordinate system and determining the corresponding base coordinate of the second sampling point in the mechanical arm base coordinate system based on the initial calibration relation;
the pose determining module is used for controlling the tail end of the mechanical arm to move to a target position corresponding to the base coordinate, and acquiring the tail end pose of the mechanical arm and the coordinates of the marker in a camera coordinate system at the target position;
and the calibration module is used for determining the hand-eye calibration relation and the coordinates of the markers in the coordinate system of the mechanical arm based on the terminal pose of the mechanical arm and the coordinates of the markers in the coordinate system of the camera so as to realize the hand-eye calibration from the camera to the mechanical arm.
A computer device comprising a memory storing a computer program and a processor implementing the steps of the method described above when executing the computer program.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method.
After a hand-eye calibration system comprising a mechanical arm, a marker and a camera is established, an initial calibration relation is determined by controlling the tail end of the mechanical arm to move from an initial position to a first sampling point, based on the coordinate of the first sampling point in a mechanical arm base coordinate system and the coordinate of the marker in a pixel coordinate system, wherein the initial calibration relation is an approximate conversion relation between the pixel coordinate system and the mechanical arm base coordinate system, selecting a second sampling point from the pixel coordinate system, determining the corresponding base coordinate of the second sampling point in the mechanical arm base coordinate system based on the initial calibration relation, controlling the tail end of the mechanical arm to move to a target position corresponding to the base coordinate, acquiring the tail end pose and the marker of the mechanical arm in the target position corresponding to the coordinate in the camera coordinate system, and determining the hand-eye calibration relation and the coordinate of the marker in the mechanical arm coordinate system based on the tail end pose and the coordinate of the marker of the mechanical arm, so as to realize the hand-eye calibration of the camera to the mechanical arm. According to the hand-eye calibration method, on the basis of avoiding manual complicated operation, the calibration precision is improved, and a good hand-eye calibration effect is achieved.
Drawings
FIG. 1 is a diagram of an exemplary embodiment of a hand-eye calibration method;
FIG. 2 is a schematic diagram of a coordinate system in the hand-eye calibration method according to an embodiment;
FIG. 3 is a schematic flow chart of a hand-eye calibration method according to an embodiment;
FIG. 4 is a schematic diagram of sample point selection in one embodiment;
FIG. 5 is a schematic flow chart of a hand-eye calibration method in another embodiment;
FIG. 6 is a block diagram showing the structure of a hand-eye calibration apparatus according to an embodiment;
FIG. 7 is a diagram of the internal structure of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The hand-eye calibration method provided by the application can be applied to the application environment shown in fig. 1. Wherein the computer device 106 communicates with the robotic arm 102 and the camera 104, respectively, over a network to effect control of the robotic arm 102 and the camera 104, and processing of related data. There are two kinds of position relationships between the camera 102 and the mechanical arm 104, specifically, eye-to-hand (eye-to-hand) or eye-in-hand (eye-to-hand), where the eye-to-hand (eye-to-hand) means that, as shown in fig. 1, the camera 102 is disposed at a fixed position outside the mechanical arm 104, and a marker 108 is disposed at an end of the mechanical arm, and the marker 108 can move along with the mechanical arm 104; eye-in-hand (eye-in-hand) means that the camera 102 is disposed on the mechanical arm 104, and a fixed point outside the mechanical arm is selected to set a marker, and the camera 102 can move along with the mechanical arm 104.
Taking an example that a hand-eye calibration method is executed on the computer device 106, a hand-eye calibration system is established in advance, the hand-eye calibration system comprises a mechanical arm, a marker and a camera, the computer device controls the tail end of the mechanical arm to move from an initial position to a first sampling point, and an initial calibration relation is determined based on the coordinates of the first sampling point in a mechanical arm base coordinate system and the coordinates of the marker in a pixel coordinate system; the initial calibration relation is an approximate conversion relation between a pixel coordinate system and the mechanical arm base coordinate system; selecting a second sampling point from the pixel coordinate system, and determining a corresponding base coordinate of the second sampling point in a mechanical arm base coordinate system based on the initial calibration relation; controlling the tail end of the mechanical arm to move to a target position corresponding to the base coordinate, and acquiring the tail end pose of the mechanical arm and the coordinates of the marker in a camera coordinate system at the target position; and determining the hand-eye calibration relation and the coordinates of the markers in the coordinate system of the mechanical arm based on the end pose of the mechanical arm and the coordinates of the markers in the coordinate system of the camera so as to realize the hand-eye calibration from the camera to the mechanical arm.
The computer device may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The computer equipment can send a control signal to the mechanical arm driver on one hand, the mechanical arm driver is utilized to drive the multi-degree-of-freedom mechanical arm to move, and the camera can be controlled by the other hand to shoot the marker to obtain a shot image, and the shot image is subjected to image processing and recognition.
For the convenience of describing the hand-eye calibration method, as shown in fig. 2, the following coordinate systems are defined:
O R the mechanical arm base coordinate system is a coordinate system established by taking the position of a mechanical arm base as an original point; o is r A coordinate system of the tail end of the mechanical arm refers to a coordinate system established by taking the position of the tail end of the mechanical arm as an origin; o is J Camera coordinate systemA coordinate system established with the position of the camera as an origin; o is I The pixel coordinate system refers to a plane coordinate system established by taking one corner of the image shot by the camera as an origin. It will be understood that the above-described coordinate system is established in its usual meaning.
In the case of the eye-on-hand system, since the camera moves together with the end of the robot arm, the relationship of the camera coordinate system with respect to the robot arm end coordinate system is fixed.
In one embodiment, as shown in fig. 3, a hand-eye calibration method is provided, which is described by taking an example of the method applied to a computer device, and includes the following steps:
s302, establishing a hand-eye calibration system, wherein the hand-eye calibration system comprises a mechanical arm, a marker and a camera.
The hand-eye calibration system at least comprises a mechanical arm, a marker and a camera. The hand-eye calibration system can be specifically an eye-on-hand system or an eye-off-hand system. The mechanical arm can be a four-axis mechanical arm or a six-axis mechanical arm.
The eye-on-hand system is characterized in that a camera is fixed at the tail end of a mechanical arm, a marker is arranged on a working plane of the hand-eye calibration system, and the camera moves along with the movement of the tail end of the mechanical arm. Generally, by controlling the movement of the tail end of the mechanical arm, the shooting plane of the camera is approximately parallel to the working plane of the marker, and the marker is in the visual field range of the camera, so that the camera can shoot the marker to obtain a shot image. The marker can be artificially placed on the working plane, or can be a feature of the working plane, and the position of the marker is generally represented by the position of a specific marking point in the marker.
The eye-outside-hand system refers to a fixed position where a camera is fixedly arranged outside a mechanical arm, the fixed position can be on a working plane of the hand-eye calibration system or other positions capable of meeting operation requirements, and a marker is arranged at the tail end of the mechanical arm, so that the marker moves along with the movement of the tail end of the mechanical arm. In general, by controlling the motion of the mechanical arm, the camera shooting plane is approximately parallel to the plane of the marker, and the marker is in the visual field range of the camera, so that the camera can shoot the marker to obtain a shot image. The marker may be a calibration sticker manually set at the end of the robot arm, such as one pasted on a planar area at the end, or a calibration plate fixed at the end, or may be a tool itself or a local area of the tool fixed at the end, where the specific marker point is generally a center of the tool or another angular point or vertex on the tool, such as a vertex of a conical tool, and generally represents the position of the marker by the position of the specific marker point in the marker.
It should be noted that, when the camera is a 3D camera, the marker may include only one marker point for hand-eye calibration; when the camera is a 2D camera, three marker points with known information can be included in the marker for hand-eye calibration.
S304, the tail end of the mechanical arm is controlled to move to a first sampling point from an initial position, and an initial calibration relation is determined based on the coordinate of the first sampling point in a mechanical arm base coordinate system and the coordinate of the marker in a pixel coordinate system.
The first sampling points are effective sampling points, and the effective sampling points refer to the fact that the camera can shoot the marker and can identify specific marker points in the marker from an image obtained through shooting so as to obtain coordinates of the marker in a pixel coordinate system. It can be understood that for an eye-out-of-hand system, the marker can move to a preset position along with the end of the mechanical arm, so that the camera can shoot the marker; for the eye-on-hand system, the position of the marker is fixed, and the camera can move to a preset position along with the tail end of the mechanical arm, so that the camera can shoot the marker.
The initial calibration relation is an approximate conversion relation between a pixel coordinate system and the mechanical arm base coordinate system. The initial calibration refers to rough calibration of the coordinate conversion relation between the pixel coordinate system and the mechanical arm base coordinate system.
In the initial position, the marker is located in the field of view of the camera, preferably, the marker can be located at the center of the field of view of the camera, and the shooting plane of the camera is approximately parallel to the plane of the marker.
In one embodiment, S304 specifically includes the following steps: selecting a first sampling point from the mechanical arm base coordinate system, and determining the coordinate of the first sampling point in the mechanical arm base coordinate system; controlling the tail end of the mechanical arm to move from an initial position to a first sampling point, and shooting the marker at the first sampling point through a camera to obtain a first image; determining coordinates of the marker in a pixel coordinate system based on the first image; and establishing an affine transformation relation according to the coordinates of the first sampling point in the mechanical arm base coordinate system and the coordinates of the marker in the pixel coordinate system, and determining an initial calibration relation based on the affine transformation relation.
The number of the first sampling points is at least 4, and the first sampling points are required to be in the same plane and not collinear. For convenience of sampling, the first sampling point may be a point located in a base plane in the base coordinate system of the robot arm. The plane in which the two coordinate axes of the base coordinate system lie is the base plane, e.g. the base plane of the base coordinate system of the robot arm in fig. 2 includes the plane O R XY, plane O R -XZ and plane O R YZ, it is understood that the coordinate systems corresponding to the two coordinate axes in the base plane are base plane coordinate systems, and the plane coordinates corresponding to the base plane are the coordinates in the corresponding base plane coordinate system, for example, P1 (X) R1 ,Y R1 ,Z R0 ) In the plane O R The corresponding planar coordinates in XY are P1 (X) R1 ,Y R1 ) In the plane O R The corresponding planar coordinates in XZ are P1 (X) R1 ,Z R0 ) In the plane O R The corresponding plane coordinate in-YZ is P1 (Y) R1 ,Z R0 ) The same applies to points P2, P3 and P4. Furthermore, the sampling can also be performed in a plane parallel to the plane of the markers.
Specifically, the moving direction and the initial step length of the tail end of the mechanical arm are set, and candidate sampling points are selected from the base coordinate system based on the moving direction and the initial step length to obtain coordinates of the candidate sampling points in the base coordinate system. Controlling the tail end of the mechanical arm to move to a first candidate sampling point from an initial position, controlling a camera to shoot the marker at the candidate sampling point to obtain a candidate image, judging whether a specific marker point in the marker can be identified according to the candidate image, if so, determining the candidate sampling point to be a first sampling point, determining the obtained candidate image to be a first image, if not, controlling the tail end of the mechanical arm to move to a next candidate sampling point, executing the steps of controlling the camera to shoot the marker at the candidate sampling point to obtain a candidate image, judging whether the specific marker point in the marker can be identified according to the candidate image, until a first image corresponding to at least four first sampling points which are not coplanar and collinear is obtained, identifying the marker based on the first image to obtain the coordinate of the marker in a pixel coordinate system, establishing an affine transformation relation according to the coordinate of the first sampling point in a mechanical arm base coordinate system and the coordinate of the marker in the pixel coordinate system, and determining an initial calibration relation based on the determined affine transformation relation.
For example, the first sampling point is a point located in a base plane of the mechanical arm, and specifically, the first sampling point is centered on the initial position, the tail end is controlled to translate a set distance L along one of four directions, namely positive and negative directions of two coordinate axes of a certain base plane coordinate system, a picture is taken, whether a specific mark point in a marker can be identified in an image or not is judged, if the specific mark point can be identified in the image, the specific mark point is an effective sampling point, the tail end pose and the pixel coordinate of the marker at the moment are recorded, if the marker does not exist or the specific mark point cannot be identified, relevant information is not recorded, and the tail end is controlled to translate the set distance L back to the initial position 1 ，L 1 Typically one tenth of L (other values may be taken) until a particular marker point in the marker can be identified in the image. And returning to the initial position, controlling the tail end to translate for a set distance L along one of other three directions, and repeating the steps until effective sampling points, namely the first sampling points, are obtained in the four directions. L is set according to empirical values and can be typically 10cm, so that four sets of first sampling points can be obtained in four different directions.
The above-described embodiment will be explained as an example. Suppose that in the base plane Z = Z R0 The first sampling points are respectively P1, P2, P3 and P4, and the coordinate value of the initial position end is (X) R0 ,Y R0 ,Z R0 ) Wherein, a firstThe coordinates of a sampling point in the base coordinate system of the mechanical arm are respectively P1 (X) R1 ,Y R1 ,Z R0 )、P2(X R2 ,Y R2 ,Z R0 )、P3(X R3 ,Y R3 ,Z R0 ) And P4 (X) R4 ,Y R4 ,Z R0 ) The pixel coordinates of the markers in the corresponding captured images in the pixel coordinate system are respectively P1' (U) I1 ,V I1 )、P2’(U I2 ,V I2 )、P3’(U I3 ,V I3 ) And P4' (U) I4 ,V I4 ) I.e. four coordinate pairs P1 (X) are formed R1 ,Y R1 ,Z R0 ) And P1' (U) I1 ,V I1 )、P2(X R2 ,Y R2 ,Z R0 ) And P2' (U) I2 ,V I2 )、P3(X R3 ,Y R3 ,Z R0 ) And P3' (U) I3 ,V I3 )、P4(X R4 ,Y R4 ,Z R0 ) And P4' (U) I4 ,V I4 ) And determining the coordinate conversion relation between the pixel coordinate system and the mechanical arm base coordinate system based on the four coordinate pairs, so as to realize the initial calibration from the pixel coordinate system to the mechanical arm base coordinate system and obtain the initial calibration relation.
In one embodiment, an affine transformation relationship is established according to coordinates of the first sampling point in the robot arm base coordinate system and coordinates of the marker in the pixel coordinate system, and the process of determining the initial calibration relationship based on the affine transformation relationship includes the following steps: establishing an affine transformation model; and establishing an affine transformation relation according to the coordinates of the marker in the pixel coordinate system and the coordinates of the first sampling point in the mechanical arm base coordinate system, and determining an initial calibration relation based on the determined affine transformation relation.
If the calibration system is an eye-out-of-hand system, approximating the coordinates of the first sampling point in the base coordinate system of the mechanical arm as the coordinates of the marker in the base coordinate system of the mechanical arm, establishing an affine transformation relationship according to the coordinates of the marker in the base coordinate system of the mechanical arm and the coordinates of the marker in the pixel coordinate system, and determining an initial calibration relationship based on the affine transformation relationship; if the calibration system is an eye-on-hand system, approximating the coordinates of the first sampling point in the mechanical arm base coordinate system to the coordinates of the camera in the mechanical arm base coordinate system, establishing an affine transformation relation according to the coordinates of the camera in the mechanical arm base coordinate system and the coordinates of the marker in the pixel coordinate system, and determining an initial calibration relation based on the affine transformation relation.
To determine the pixel coordinate system O in fig. 2 I To the base plane coordinate system O R The above description is made by taking the affine transformation relationship of-XY as an example. Pixel coordinate system O I To the base plane coordinate system O R The XY mapping relationship uses the following formula (1):
wherein w is a homogeneous coefficient, and the following formula (2) can be obtained by developing the formula (1):
dividing the right numerator and denominator of formula (3) by a 33 It is possible to obtain:
wherein,
order to
Equation (3) establishes the pixel coordinate system O I To the base plane coordinate system O R -XY mapping, with reference to equations (1) to (3), and similarly, the pixel coordinate system to base plane coordinate system O can be determined R -a mapping of the XZ and the pixel coordinate system to the base plane coordinate system O R -mapping of YZ.
It will be appreciated that the computer device is obtaining the base plane coordinate system O R After the plane coordinates of XY and the coordinates in the corresponding pixel coordinate system, the affine transformation matrix K can be obtained based on the base plane coordinates, the coordinates in the pixel coordinate system, and the formula (3). The base plane O obtained in one example is shown in Table 1 below R -the planar coordinates of XY with the corresponding pixel coordinates:
TABLE 1
| Serial number | [X R ,Y R ] | [u,v] |
| P1(P1’) | 0.98899716,0.02066533 | 893.706,311.864 |
| P2(P2’) | 0.78905356,0.02065816 | 889.619,589.633 |
| P3(P3’) | 0.8889875,0.12065843 | 753.122,448.347 |
| P4(P4’) | 0.8890044,-0.07924276 | 1027.475,453.502 |
Based on the coordinate data of table 1 and equation (3), K can be found:
after the affine transformation matrix is obtained, a coordinate transformation relation, namely an initial calibration relation, between the pixel coordinate system and the mechanical arm base plane coordinate system can be determined based on the mapping relation and the affine transformation matrix shown in the formula (3).
S306, selecting a second sampling point from the pixel coordinate system, and determining the corresponding base coordinate of the second sampling point in the mechanical arm base coordinate system based on the initial calibration relation.
The second sampling points are randomly selected points in the pixel coordinate system, the number of the second sampling points is at least 9, the at least 9 sampling points are not collinear in the pixel plane, and the corresponding space points in the base coordinate system are not coplanar. As shown in FIG. 4, the point corresponding to "X" in the figure is the randomly selected second sampling point. In addition to the pixel plane point selection method shown in fig. four, other random point selection methods such as generating random coordinates by random numbers may be adopted.
In one embodiment, S304 specifically includes the following steps: determining the corresponding base plane coordinate of the second sampling point in the base coordinate system of the mechanical arm based on the initial calibration relation; and randomly selecting a target coordinate value from a preset coordinate value range, and determining the coordinate of the second sampling point in the mechanical arm base coordinate system based on the base plane coordinate and the target coordinate value.
The preset coordinate range is a coordinate value range of a third base coordinate axis, and the third base coordinate axis is perpendicular to the base plane where the base plane coordinates are located.
Specifically, after the initial calibration relationship is obtained, a plurality of second sampling points are randomly selected in the pixel coordinate system, coordinates of the plurality of second sampling points in the pixel coordinate system are obtained, base plane coordinates corresponding to the second sampling points are determined according to the coordinates of the second sampling points in the pixel coordinate system and the initial calibration relationship, coordinate values are randomly selected in the preset coordinate range, and coordinates of the second sampling points in the mechanical arm base coordinate system are determined according to the base plane coordinates corresponding to the second sampling points and the randomly selected coordinate values.
With the base plane O R Taking XY as an example, randomly selecting N second sampling points in the pixel coordinate system, wherein the coordinate of the ith second sampling point in the pixel coordinate system is (U) i ，V i ) The coordinate in the pixel coordinate system according to the second sampling point is (U) i ，V i ) A second sampling point determined by the initial calibration relation is in a mechanical arm base plane coordinate system O R -XY coordinates (X) i ，Y i ) Randomly selecting a coordinate value Z on a Z coordinate axis perpendicular to the base plane based on a preset Z coordinate value range i So as to obtain the coordinate value (X) of the second sampling point in the base coordinate system of the mechanical arm i ，Y i ，Z i ). Wherein the initial calibration relationship may be the above equation (3).
In one example, the predetermined coordinate range may be determined based on a depth of field of the camera, requiring the marker to be within the depth of field of the camera. For example, sitting on the base planeSystem of symbols O R The third coordinate axis perpendicular to XY is the Z axis, and the Z coordinate value usually ranges from (Z) R0 -z 0 ，Z R0 +z 0 ) Wherein z is 0 Typically 5-10cm.
And S308, controlling the tail end of the mechanical arm to move to the target position corresponding to the base coordinate, and acquiring the tail end pose of the mechanical arm and the coordinate of the marker in the camera coordinate system at the target position.
The movement comprises translation and rotation, and the target position is the position of a corresponding base coordinate of the second sampling point in the base coordinate system of the mechanical arm in the space; the pose refers to the position and the posture of the tail end of the mechanical arm in a mechanical arm base coordinate system.
In one embodiment, S308 specifically includes the following steps: acquiring a preset tail end posture; controlling the tail end of the mechanical arm to move from the initial position to a target position corresponding to the base coordinate, and enabling the tail end of the mechanical arm to be in a preset tail end posture at the target position to obtain a tail end posture of the mechanical arm; shooting the marker through a camera at a target position to obtain a second image; coordinates of the marker in the camera coordinate system are determined based on the second image.
Specifically, the gesture of the tail end of the mechanical arm at the target position is obtained in advance, and the preset tail end gesture of the tail end of the mechanical arm at the target position is obtained; and then controlling the tail end of the mechanical arm to move from the initial position to a target position corresponding to the base coordinate, adjusting the posture of the tail end of the mechanical arm in the moving process, so that the tail end of the mechanical arm is in a preset tail end posture at the target position to obtain a tail end posture of the mechanical arm, shooting the marker at the target position through a camera to obtain a second image, wherein the second image is an effective image, and determining the coordinate of the marker in a camera coordinate system according to the second image.
The preset tail end posture is obtained by adjusting a preset angle on the basis of the tail end posture of the mechanical arm at an initial position, and the tail end posture of the initial position is r 0 ＝(RX 0 ，RY 0 ，RZ 0 ) The preset angle is an angle r randomly selected from a preset angle value range i ＝(RX i ，RY i ，RZ i ) Wherein Rxi = (RX) 0 -w 1 ，RX 0 +w 1 )，RYi＝(RY 0 -w 2 ，RY 0 +w 2 )，RZi＝(RZ 0 -w 3 ，RZ 0 +w 3 ) In general w 1 ，w 2 ，w 3 The value is 5-15 degrees. Specifically, the selected angles may be uniformly distributed within the value range, or may be generated by a random number generator. For example, the second sampling point includes a second sampling point 1, a second sampling point 2, a second sampling point 3, a second sampling point 4, a second sampling point 5, a second sampling point 6, a second sampling point 7, a second sampling point 8, and a second sampling point 9, and then the preset angle 1 corresponding to the second sampling point 1, the preset angle 2 corresponding to the second sampling point 2, the preset angle 3 corresponding to the second sampling point 3, the preset angle 4 corresponding to the second sampling point 4, the preset angle 5 corresponding to the second sampling point 5, the preset angle 6 corresponding to the second sampling point 6, the preset angle 7 corresponding to the second sampling point 7, the preset angle 8 corresponding to the second sampling point 8, and the preset angle 9 corresponding to the second sampling point 9 may be different and uniformly distributed within the above range.
Assume that the base coordinates are P5 (X), respectively R5 ,Y R5 ,Z R1 )、P6(X R6 ,Y R6 ,Z R1 )、P7(X R7 ,Y R7 ,Z R1 ) And P8 (X) R8 ,Y R8 ,Z R1 ) Then the end of the robot arm is controlled to move to the determined target position P5 (X) R5 ,Y R5 ,Z R1 ) And at P5 (X) R5 ,Y R5 ,Z R1 ) Controlling the tail end of the mechanical arm to rotate by a preset angle r 5 Obtaining the pose P5 (X) of the tail end in the base coordinate system of the mechanical arm R5 ,Y R5 ,Z R1 ,r 5 ) (ii) a Similarly controlling the movement of the tail end of the mechanical arm to the determined positionTarget position P6 (X) R6 ,Y R6 ,Z R1 )、P7(X R7 ,Y R7 ,Z R1 ) And P8 (X) R8 ,Y R8 ,Z R1 ) And are each at P6 (X) R6 ,Y R6 ,Z R1 )、P7(X R7 ,Y R7 ,Z R1 ) And P8 (X) R8 ,Y R8 ,Z R1 ) The tail end of the control mechanical arm rotates by a preset angle to obtain a position P6 (X) of the tail end in a mechanical arm base coordinate system R6 ,Y R6 ,Z R1 ,r 6 )、P7(X R7 ,Y R7 ,Z R1 ,r 7 ) And P8 (X) R8 ,Y R8 ,Z R1 ,r 8 )。
In one embodiment, the base coordinates corresponding to the second sampling points lie in at least two parallel planes, and the base coordinates lying in the first plane are assumed to be P5 (X) respectively R5 ,Y R5 ,Z R1 )、P6(X R6 ,Y R6 ,Z R1 )、P7(X R7 ,Y R7 ,Z R1 ) And P8 (X) R8 ,Y R8 ,Z R1 ) And controlling the tail end of the mechanical arm to move to a target position P5 (X) corresponding to the determined base coordinate in the first plane R5 ,Y R5 ,Z R1 ) And at P5 (X) R5 ,Y R5 ,Z R1 ) Controlling the tail end of the mechanical arm to rotate by a preset angle to obtain a pose P5 (X) of the tail end in a base coordinate system of the mechanical arm R5 ,Y R5 ,Z R1 ,r 5 ) Similarly, the end of the mechanical arm is controlled to move to the determined target position P6 (X) R6 ,Y R6 ,Z R1 )、P7(X R7 ,Y R7 ,Z R1 ) And P8 (X) R8 ,Y R8 ,Z R1 ) And are each at P6 (X) R6 ,Y R6 ,Z R1 )、P7(X R7 ,Y R7 ,Z R1 ) And P8 (X) R8 ,Y R8 ,Z R1 ) The tail end of the control mechanical arm rotates by a preset angle to obtain a pose P6 (X) of the tail end in a base coordinate system of the mechanical arm R6 ,Y R6 ,Z R1 ,r 6 )、P7(X R7 ,Y R7 ,Z R1 ,r 7 ) And P8 (X) R8 ,Y R8 ,Z R1 ,r 8 ) (ii) a In the second planeThe target position corresponding to the base coordinate of (2) is P9 (X) R9 ,Y R9 ,Z R2 )、P10(X R10 ,Y R10 ,Z R2 )、P11(X R11 ,Y R11 ,Z R2 ) And P12 (X) R12 ,Y R12 ,Z R2 ) Then the end of the robot arm is controlled to move to the determined target position P9 (X) R9 ,Y R9 ,Z R2 ) And at P9 (X) R9 ,Y R9 ,Z R2 ) The tail end of the mechanical arm is controlled to rotate by a preset angle to obtain a pose P9 (X) of the tail end in a base coordinate system of the mechanical arm R9 ,Y R9 ,Z R2 ,r 9 ) Similarly, the end of the mechanical arm is controlled to move to the determined target position P10 (X) R10 ,Y R10 ,Z R2 )、P11(X R11 ,Y R11 ,Z R2 ) And P12 (X) R12 ,Y R12 ,Z R2 ) And are each at P10 (X) R10 ,Y R10 ,Z R2 )、P11(X R11 ,Y R11 ,Z R2 ) And P12 (X) R12 ,Y R12 ,Z R2 ) The tail end of the control mechanical arm rotates by a preset angle to obtain a position P10 (X) of the tail end in a base coordinate system of the mechanical arm R10 ,Y R10 ,Z R2 ,r 10 )、P11(X R11 ,Y R11 ,Z R2 ,r 11 ) And P12 (X) R12 ,Y R12 ,Z R2 ,r 12 ). If the base coordinates in one plane are a group, the end poses corresponding to at least two groups of base coordinates can be obtained. In some embodiments, at least 9 sets of base coordinates and the end pose corresponding to each set of base coordinates may be obtained, and generally, in order to ensure the accuracy of the calculation, the correlation data of 20 to 30 second sampling points may be obtained.
In one embodiment, the process of controlling the rotation of the tip by the preset angle by the computer device includes: based on the attitude of the tail end of the mechanical arm at the initial position, the tail end rotates by a preset angle around at least one of three axes of a coordinate system of the tail end of the mechanical arm.
For example, after the end of the robot arm reaches the target position, the computer device may control the end of the robot arm to rotate at the target position by a predetermined angle about a first coordinate axis of a coordinate system of the end of the robot arm, then control the end to rotate by a predetermined angle about a second coordinate axis of the coordinate system of the end of the robot arm, and then control the end to rotate by a predetermined angle about a third coordinate axis of a coordinate system of the base of the robot arm. The angles of rotation around the three coordinate axes can be the same or different.
In one embodiment, the process of the computer device determining the coordinates of the marker in the camera coordinate system based on the second image comprises: identifying the second image to obtain the coordinates of the marker in a pixel coordinate system; the coordinates of the marker in the camera coordinate system are determined based on the camera intrinsic parameters and the coordinates of the marker in the pixel coordinate system.
Specifically, the coordinate conversion relationship between the pixel coordinate system and the camera coordinate system may be determined based on camera internal parameters, and the coordinates of the marker in the camera coordinate system may be determined based on the coordinate conversion relationship between the pixel coordinate system and the camera coordinate system and the coordinates of the marker in the pixel coordinate system. Wherein the camera may be a 2D camera or a 3D camera. The 2D camera requires that at least 3 marker points with known information are included in the marker and converted into a PNP problem to be solved.
In one embodiment, the camera is a monocular camera, which may be simplified to a pinhole camera model, and the coordinate transformation relationship between the pixel coordinate system and the camera coordinate system is the coordinate transformation relationship between the pixel coordinate system corresponding to the pinhole camera model and the camera coordinate system.
And S310, determining a hand-eye calibration relation and coordinates of the markers in a coordinate system of the mechanical arm based on the terminal pose of the mechanical arm and the coordinates of the markers in the coordinate system of the camera, so as to realize the hand-eye calibration from the camera to the mechanical arm.
For a system with eyes outside the hand, the hand-eye calibration relation refers to a coordinate conversion relation between a camera coordinate system and a mechanical arm base coordinate system; for the eye-on-hand system, the hand-eye calibration relation refers to the coordinate conversion relation from a camera coordinate system to a mechanical arm tail end coordinate system. The robot arm coordinate system can be a robot arm base coordinate system or a robot arm end coordinate system, and the coordinates of the marker in the robot arm coordinate system can be the coordinates of the marker in the robot arm base coordinate system or the coordinates of the marker in the robot arm end coordinate system.
In one embodiment, S310 specifically includes the following steps: establishing a hand-eye calibration model as follows: AX = ZB; establishing a nonlinear optimization model by taking the minimum Euclidean distance | | | AX-ZB | | as an optimization target based on a hand-eye calibration model; and iteratively determining the hand-eye calibration relation Z and the coordinate X of the marker in the base coordinate system of the mechanical arm by using a nonlinear optimization method based on the terminal pose of the mechanical arm, the coordinate of the marker in the camera coordinate system and the nonlinear optimization model.
Wherein A, B is a known quantity, a is the terminal pose, and B is the coordinates of the marker in the camera coordinate system. The pose can be understood to refer to the position and the attitude of the tail end of the mechanical arm in the mechanical arm base coordinate system. Therefore, the coordinate transformation relation between the mechanical arm tail end coordinate system and the mechanical arm base coordinate system can be obtained through the pose of the mechanical arm tail end in the mechanical arm base coordinate system.
For an eye-outside-hand system, Z is a coordinate conversion relation from a camera coordinate system to be determined to a mechanical arm base coordinate system, and X is a coordinate of a marker to be determined in a mechanical arm tail end coordinate system; for an eye-on-hand system, X is the coordinate of the marker to be determined under a base coordinate system of the mechanical arm, and Z is the coordinate conversion relation from a camera coordinate system to be determined to a terminal coordinate system of the mechanical arm.
In the embodiment of the application, a is represented by a 7-dimensional vector, wherein the former three dimensions represent the spatial position, i.e. the translation transformation relation between the robot arm end coordinate system and the robot arm base coordinate system, and the latter 4 dimensions represent the attitude, i.e. the rotation transformation relation between the robot arm end coordinate system and the robot arm base coordinate system, by using a 4-element number.
The above solving process is illustrated by way of example: let X = (a) 1 ,a 2 ,a 3 ,1) T B may also be represented by (X) C ,Y C ,Z C ,1) T In the form of (a);
is a 4 × 4 matrix, a can also be represented in a similar form, where T = (T =) 1 ,t 2 ,t 3 ) T Represents a translation matrix, R = { m = } ij J ∈ {1,2,3} is a rotation matrix, whose corresponding quaternion q = q 0 +q 1 i+q 2 j+q 3 k can be calculated according to the existing formula, q 0 、q 1 、q 2 And q is 3 Is real and q 0 2 +q 1 2 +q 2 2 +q 3 2 ＝1，i 2 ＝j 2 ＝k 2 ＝ijk＝-1。
To facilitate data recording, Z is recorded as (t) 1 ,t 2 ,t 3 ,q 0 ,q 1 ,q 2 ,q 3 ) X is recorded as (a) 1 ,a 2 ,a 3 )。
Setting the objective function as
There is a constraint q 0 2 +q 1 2 +q 2 2 +q 3 2 ＝1；
f i ＝||A i X-ZB i | | is euclidean distance, which is taken as a cost function, and for convenience of expression, Q = (a) 1 ,a 2 ,a 3 ,t 1 ,t 2 ,t 3 ,q 0 ,q 1 ,q 2 ,q 3 ) Q is an optimization variable, and i represents the ith group of sampling data.
Generally given an initial value X 0 And Z 0 ，X 0 And Z 0 Can be set arbitrarily, i.e. the initial value Q of the variable Q is optimized 0 ＝(a 1 ,a 2 ,a 3 ,t 1 ,t 2 ,t 3 ,q 0 ,q 1 ,q 2 ,q 3 ) 0 For any given purpose
For the k-th iteration, an optimization is foundIncrement of variable
So that->
Reaches a minimum value if>
If the threshold value is smaller than the set threshold value, stopping iteration, otherwise, repeating the steps. In addition, for increments>
The determination of (2) can be solved by using a Newton method, a Gaussian Newton method, an LM algorithm and the like.
Table 2 below shows a and B input for solving X and Z in one embodiment, specifically, solving X and Z by using the above-mentioned nonlinear optimization method, where initial values of X and Z are randomly set, and then iteratively updating X and Z until convergence, so as to obtain the solution results shown in table 3.
TABLE 2
| Serial number | A | B |
| 0 | 0.981 0.189 0.538 0.634 0.773 -0.005 -0.015 | -0.118 -0.059 0.939 |
| 1 | 0.903 0.19 0.538 0.634 0.773 0.01 0.008 | -0.117 0.019 0.93 |
| 2 | 0.824 0.191 0.538 0.572 0.82 0.015 0.002 | -0.138 0.095 0.926 |
| 3 | 0.981 0.062 0.538 0.667 0.743 0.048 0.03 | 0.022 -0.057 0.944 |
| 4 | 0.901 0.063 0.538 0.627 0.779 0.021 -0.005 | 0.006 0.019 0.952 |
| 5 | 0.821 0.064 0.538 0.634 0.773 0.015 0.001 | 0.008 0.100 0.95 |
| 6 | 0.981 -0.071 0.538 0.631 0.771 0.065 -0.059 | 0.137 -0.062 0.982 |
| 7 | 0.9 -0.07 0.538 0.621 0.784 0.015 0.002 | 0.135 0.019 0.971 |
| 8 | 0.818 -0.069 0.538 0.66 0.749 0.042 0.024 | 0.150 0.104 0.964 |
| 9 | 0.942 0.127 0.638 0.648 0.762 0.004 0.014 | -0.034 -0.02 0.842 |
| 10 | 0.862 0.127 0.638 0.634 0.772 0.042 0.023 | -0.037 0.058 0.833 |
| 11 | 0.941 -0.003 0.638 0.629 0.764 -0.079 0.116 | 0.090 -0.021 0.847 |
| 12 | 0.86 -0.002 0.638 0.627 0.779 0.015 0.001 | 0.087 0.059 0.862 |
TABLE 3
| Z | X | |
| Output of | 1.045 0.192 1.479 0.702 -0.708 0.06 0.052 | -0.055 0.127 -0.000 |
In the above embodiment, after the hand-eye calibration system including the mechanical arm, the marker and the camera is established, the end of the mechanical arm is controlled to move from the initial position to the first sampling point, the initial calibration relationship is determined based on the coordinates of the first sampling point in the mechanical arm base coordinate system and the coordinates of the marker in the pixel coordinate system, wherein the initial calibration relationship is an approximate conversion relationship between the pixel coordinate system and the mechanical arm base coordinate system, the second sampling point is selected from the pixel coordinate system, the base coordinate corresponding to the second sampling point in the mechanical arm base coordinate system is determined based on the initial calibration relationship, the end of the mechanical arm is controlled to move to the target position corresponding to the base coordinate system, the end of the mechanical arm and the marker correspond to the coordinates in the camera coordinate system are obtained at the target position, and the hand-eye calibration relationship and the coordinates of the marker in the mechanical arm base coordinate system are determined based on the end position of the mechanical arm and the coordinates of the marker in the camera coordinate system, so as to achieve the hand-eye calibration from the camera to the mechanical arm. According to the hand-eye calibration method, on the basis of avoiding manual tedious operations, the calibration precision is improved, and a good hand-eye calibration effect is achieved.
In one embodiment, the hand-eye calibration method further includes the following steps: inputting the hand-eye calibration relation Z, the coordinate X of the marker in the mechanical arm coordinate system, the end pose of the mechanical arm and the coordinate of the target position in the camera coordinate system into a hand-eye calibration model, and determining the calibration error corresponding to each second sampling point; selecting a target second sampling point from the second sampling points based on the calibration error; and re-determining the hand-eye calibration relation and the coordinates of the marker in the coordinate system of the mechanical arm based on the terminal pose of the mechanical arm corresponding to the target second sampling point and the coordinates of the marker in the coordinate system of the camera so as to realize the hand-eye calibration from the camera to the mechanical arm.
For example, the solved positions of the X and Z and the end of the mechanical arm corresponding to the second sampling points and the coordinates of the marker in the camera coordinate system are input into the hand-eye calibration model AX = ZB, and the calibration error Δ D = AX-ZB corresponding to the second sampling points is determined, that is, the calculation error Δ D of each sampling point can be solved i And removing corresponding sampling point data for the second sampling point with the calculation error exceeding the error threshold D, thereby obtaining the end pose of the mechanical arm and the mark of the mechanical arm corresponding to the target second sampling point and the target second sampling point on the camera seatAnd determining the hand-eye calibration relation Z and the coordinate X of the marker in the coordinate system of the mechanical arm again in an iterative manner by utilizing a nonlinear optimization method in combination with a nonlinear optimization model based on the terminal pose of the mechanical arm corresponding to the target second sampling point and the coordinate of the marker in the coordinate system of the camera.
In the above embodiment, the hand-eye calibration relation Z and the coordinate X of the marker in the base coordinate system of the mechanical arm, and the end pose of the mechanical arm and the coordinate of the marker in the camera coordinate system are input into the hand-eye calibration model, so as to determine the calibration error corresponding to each second sampling point; selecting a target second sampling point from the second sampling points based on the calibration error; and re-determining the hand-eye calibration relation and the coordinates of the markers in the coordinate system of the mechanical arm based on the terminal pose of the mechanical arm corresponding to the target second sampling point and the coordinates of the markers in the coordinate system of the mechanical arm, so that the calibration precision of the hand-eye calibration can be improved.
In one embodiment, the end of the mechanical arm is further fixedly provided with a tool, and the hand-eye calibration method further comprises the following steps: and determining the coordinate of the tool center point in the terminal coordinate system based on the coordinate of the marker in the mechanical arm coordinate system so as to realize tool calibration.
For the eye-outside-the-hand system, a tool is fixedly arranged at the tail end of the mechanical arm, the tool itself or a part of the tool can be used as a marker, the center point of the tool can be used as a specific marker point, and the specific marker can also be arranged at the center point of the tool. . It can be understood that, at this time, the coordinates of the marker in the coordinate system of the end of the robot arm are the coordinates of the tool center point in the coordinate system of the end of the robot arm, and at this time, the tool calibration is realized. The markers may be located at other positions on the tool or tip, the position of which relative to the center point of the tool is known.
For an eye-on-hand system, a tool arranged at the tail end of a mechanical arm is controlled to be in contact with a marker, a tool central point or other points with known position relation relative to the tool central point are generally enabled to be superposed with a specific marker point in the marker, the pose of the tail end of the mechanical arm at the moment is acquired, and then the coordinate of the tool central point in a mechanical arm tail end coordinate system can be determined according to the pose of the tail end of the mechanical arm at the moment and the determined coordinate of the marker in the mechanical arm base coordinate system, so that the calibration of the tool is realized.
In the above embodiment, when the tool is further fixedly mounted at the tail end of the arm, the coordinate of the center point of the tool in the tail end coordinate system can be determined through the coordinate of the marker in the arm coordinate system, so that the calibration of the tool can be realized while the calibration of the hand and the eye is realized.
In one embodiment, as shown in fig. 5, a hand-eye calibration method is provided, which is described by taking the application of the method to a computer device as an example, and includes the following steps:
and S502, establishing a hand-eye calibration system, wherein the hand-eye calibration system comprises a mechanical arm, a marker and a camera.
S504, selecting a first sampling point from the mechanical arm base coordinate system, and determining the coordinate of the first sampling point in the mechanical arm base coordinate system.
S506, the tail end of the mechanical arm is controlled to move to a first sampling point from the initial position, the marker is shot through a camera at the first sampling point, and a first image is obtained.
And S508, determining the coordinates of the marker in the pixel coordinate system based on the first image.
S510, an affine transformation relation is established according to the coordinates of the first sampling point in the mechanical arm base coordinate system and the coordinates of the marker in the pixel coordinate system, and an initial calibration relation is determined based on the affine transformation relation.
And S512, determining the corresponding base plane coordinate of the second sampling point in the base coordinate system of the mechanical arm based on the initial calibration relation.
And S514, randomly selecting a target coordinate value from a preset coordinate value range, and determining the coordinate of the second sampling point in the mechanical arm base coordinate system based on the base plane coordinate and the target coordinate value.
And S516, acquiring a preset tail end posture.
The preset tail end posture is obtained by adjusting a preset angle on the basis of the tail end posture of the mechanical arm at the initial position.
And S518, controlling the tail end of the mechanical arm to move from the initial position to a target position corresponding to the base coordinate, and enabling the tail end of the mechanical arm to be in a preset tail end posture at the target position to obtain a tail end posture of the mechanical arm.
Wherein the movement comprises translation and rotation.
And S520, shooting the marker through the camera at the target position to obtain a second image.
S522, the coordinates of the marker in the camera coordinate system are determined based on the second image.
S524, establishing a hand-eye calibration model as follows: AX = ZB.
The method comprises the following steps of determining a coordinate of a marker in a camera coordinate system, wherein A, B is a known quantity, A is a terminal pose, B is a coordinate of the marker in the camera coordinate system, X is a coordinate of the marker to be determined under a mechanical arm base coordinate system, and Z is a hand-eye calibration relation to be determined.
S526, based on the hand-eye calibration model, establishing a nonlinear optimization model by taking the minimized Euclidean distance | | | AX-ZB | | as an optimization target.
And S528, iteratively determining the hand-eye calibration relation Z and the coordinate X of the marker in the mechanical arm coordinate system by using a nonlinear optimization method based on the end pose of the mechanical arm, the coordinate of the marker in the camera coordinate system and the nonlinear optimization model.
The application also provides an application scene, and the hand-eye calibration method is applied to the application scene. Specifically, the application of the hand-eye calibration method in the application scene is as follows:
step 1, establishing a hand-eye calibration system.
Fixing a tool on the flange at the tail end of the mechanical arm, and pasting a marker on the tool or directly pasting the marker on the flange at the tail end of the mechanical arm. The marker is moved into the camera working field of view and this position is recorded as the initial position of the end of the arm.
The plane of the marker is approximately parallel to the shooting plane of the camera, and the sampling effect is good under the general condition.
And 2, selecting a first sampling point for sampling.
For the scene with the camera installed above or below, taking the initial position of the end of the mechanical arm in the step 1 as a reference, along X R Moving the positive direction of the moving object by a preset distance to reach a first preset position; along X R Moving the negative direction of the moving object by a preset distance to reach a second preset position; along Y R Moving the positive direction of the moving object by a preset distance to reach a third preset position; along Y R The positive direction of the sampling point is moved by a preset distance to reach a fourth preset position, the marker is shot at each preset position through a camera to obtain each shot image, the pixel coordinate of the marker in a pixel coordinate system is determined based on the shot images, the mechanical arm base coordinate of the tail end at each preset position is determined, and therefore four coordinate pairs are obtained, each coordinate pair comprises a preset position coordinate and a corresponding pixel coordinate, namely the data of the first sampling point obtained through sampling.
Similarly, for a scene with cameras mounted on the sides, in Z, according to the actual arrangement R 、Y R In the directions of the positive and negative coordinate axes, or Z R 、X R The data of the first sampling point is collected in the directions of the positive and negative coordinate axes.
Wherein the preset distance can be any value in the range of 1-10 cm. Because the visual fields of the cameras are greatly different in scenes of different work and camera installation modes, the tail end of the mechanical arm moves by 10cm, then the marker is identified according to the shot image, if the marker cannot be identified, the feedback marker exceeds the visual field of the camera, the mechanical arm moves back by 1 cm, the marker is repeatedly moved in sequence until the marker is identified in the shot image, and relevant data are recorded at the moment.
And 3, calculating an initial calibration relation.
And (3) calculating to obtain an affine transformation matrix K from the pixel coordinate system to the mechanical arm base coordinate system according to the data of the first sampling point obtained in the step (2) and the formula (3).
It is understood that only four coordinate pairs are required to solve for K in this embodiment. Generally, the more coordinate pairs, the less sensitive the obtained affine transformation matrix K is to noise, but in the present application, the final hand-eye calibration is not directly performed based on K, and K determined in this step is only used to obtain a rough correspondence between the pixel coordinate system and the robot arm base coordinate system, so as to serve for subsequent visual guidance of the robot arm movement.
Generally, a 6-axis mechanical arm can move in a space with 6 degrees of freedom, namely, a mechanical arm base coordinate system is three-dimensional, a pixel coordinate system is two-dimensional, so that the conversion relation between the pixel coordinate system and the mechanical arm base coordinate system cannot be directly determined, and by referring to formula (3), the mapping relation between each base plane of the pixel coordinate system and each base plane of the mechanical arm base coordinate system can be respectively obtained, so that the conversion relation between the pixel coordinate system and the mechanical arm base coordinate system is obtained.
The terminal pose of the mechanical arm can be directly obtained according to the motion state of the mechanical arm.
And 4, selecting a second sampling point for sampling.
Uniformly sampling 9-12 points in a pixel coordinate system, and then obtaining a coordinate system O of the tail end of the mechanical arm in a base plane of the mechanical arm according to a formula (3) R Coordinates in XY, given a constant value Z of the Z-axis coordinate 1 And presetting the end poses corresponding to each point, guiding the mechanical arm to autonomously plan the path to move to each designated position, triggering a camera to shoot, recording the end poses of the mechanical arm, and determining the coordinates of the marker in a pixel coordinate system based on the shot image.
Uniformly sampling 4-6 points in a pixel coordinate system, and obtaining a mechanical arm end-on-mechanical arm base plane coordinate system O according to a formula (3) without repeating the sampling points in the step 4 R Coordinates in XY, given a constant value Z of the Z-axis coordinate 2 And presetting the corresponding terminal poses of each point, guiding the mechanical arm to autonomously plan the path to move to each specified position, triggering a camera to shoot, recording the poses of the tail end of the mechanical arm, and determining the coordinates of the marker in a pixel coordinate system based on the shot image.
Wherein, the pose is directly obtained according to the motion state of the mechanical arm; the preset end position and pose corresponding to each point are randomly generated in the pose value range based on the initial position end position and pose, and the initial positionSet the tail end at the posture of r 0 ＝(RX 0 ，RY 0 ，RZ 0 ) The preset terminal pose is a randomly selected attitude value r in a preset attitude value range i ＝(RX i ，RY i ，RZ i ) Wherein RX i ＝(RX 0 -w 1 ，RX 0 +w 1 )，RY i ＝(RY 0 -w 2 ，RY 0 +w 2 )，RZ i ＝(RZ 0 -w 3 ，RZ 0 +w 3 ) In general w 1 ，w 2 ，w 3 The value is 5-15 degrees.
And 4, data acquisition is carried out at different positions in space, so that the calibration result is more accurate, and if data are acquired only on a certain fixed plane, the hand-eye calibration result is possibly over-fitted, and the accurate calibration result cannot be obtained. Generally, it is ensured that a sufficient number of spatial points corresponding to the second sampling point are obtained in the working space of the system, and the requirement can be met if there is a certain difference between each point.
And 5, automatically calculating a hand-eye calibration result according to the model with AX = ZB.
Specifically, the optimal hand-eye calibration result can be iteratively calculated according to a nonlinear optimization method. The optimization goal of the nonlinear optimization method is to minimize the euclidean distance, and the optimization can be specifically performed by a numerical optimization method, such as convex optimization, including an LM algorithm and the like.
For the situation that eyes are out of hand, A is a transformation matrix between a mechanical arm tail end coordinate system and a mechanical arm base coordinate system and can be obtained through calculation according to tail end poses, X is a coordinate of a calibration object in the mechanical arm tail end coordinate system, Z is a transformation matrix between a camera coordinate system and the mechanical arm base coordinate system, and B is a coordinate of the calibration object in a camera coordinate system.
If the eyes are outside the hands and the tool calibration needs to be performed simultaneously, the marker can be set at the tool center point or other positions associated with the tool center point, and X in the calculation result is the tool calibration result.
And 6, carrying out quantitative and qualitative verification on the hand-eye calibration precision.
Taking the example that the eyes are out of the hands, assuming that N groups of effective data are collected, each group of effective data comprises a transformation matrix between a mechanical arm tail end coordinate system and a mechanical arm base coordinate system i A, and coordinates of the markers in the camera coordinate system i B, where i ∈ [0,N-1]Representing the ith set of acquired data. In step 7, a transformation matrix Z between the camera coordinate system to the robot arm base coordinate system and coordinates X of the marker in the robot end coordinate system can be obtained.
According to the calibration data, the coordinates of the marker in the mechanical arm base coordinate system can be obtained from the two paths and are respectively marked as i P base1 And i P base2 wherein, i P base1 ＝ i AX， i P base2 ＝Z i B
in the ideal situation, the temperature of the air conditioner, i P base1 ＝ i P base2 however, because of the error of the hand-eye system, the error of the calculation of the calibration result, etc., these two values are not constant, we can use i P base1 And i P base2 the Euclidean distance | non-woven fabrics between i P base1 - i P base2 | | i | non-conducting phosphor i AX-Z i And B | | measures the calibration error of each group of calibration data, and the calibration total error is measured by the error mean value of all calibration data. Table 4 below shows the calibration number error in one embodiment:
TABLE 4
| Serial number | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | Mean value |
| Error/mm | 0.236 | 0.114 | 0.109 | 0.053 | 0.096 | 0.221 | 0.153 | 0.366 | 0.224 | 0.325 | 0.062 | 0.290 | 0.187 |
For the application scene of eyes on the hand, except that the operation modes of establishing the hand-eye calibration system in step 1 and calibrating the tool in step 5 are different, the other steps are not different, and for the specific setting of the system outside the hand and the operation of calibrating the tool, reference may be made to the above definition of the hand-eye calibration method, and details are not repeated here.
It should be understood that although the steps in the flowcharts of fig. 3 and 5 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 3 and 5 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.
In one embodiment, as shown in fig. 6, there is provided a hand-eye calibration device, comprising: a hand-eye calibration system establishing module 602, an initial calibration relationship determining module 604, a second sampling point determining module 606, a pose determining module 608, and a calibration module 610, wherein:
a hand-eye calibration system establishing module 602, configured to establish a hand-eye calibration system, where the hand-eye calibration system includes a mechanical arm, a marker, and a camera;
an initial calibration relationship determining module 604, configured to control the end of the mechanical arm to move from an initial position to a first sampling point, and determine an initial calibration relationship based on coordinates of the first sampling point in a mechanical arm base coordinate system and coordinates of the marker in a pixel coordinate system; the initial calibration relation is an approximate conversion relation between a pixel coordinate system and a mechanical arm base coordinate system;
a second sampling point determining module 606, configured to select a second sampling point from the pixel coordinate system, and determine, based on the initial calibration relationship, a corresponding base coordinate of the second sampling point in the mechanical arm base coordinate system;
a pose determining module 608, configured to control the end of the mechanical arm to move to a target position corresponding to the base coordinate, and obtain, at the target position, the pose of the end of the mechanical arm and the coordinates of the marker in the camera coordinate system;
the calibration module 610 is configured to determine a hand-eye calibration relationship and coordinates of the markers in the coordinate system of the robot arm based on the end pose of the robot arm and the coordinates of the markers in the coordinate system of the camera, so as to implement hand-eye calibration from the camera to the robot arm.
In the above embodiment, after the hand-eye calibration system including the mechanical arm, the marker and the camera is established, the end of the mechanical arm is controlled to move from the initial position to the first sampling point, the initial calibration relationship is determined based on the coordinates of the first sampling point in the mechanical arm base coordinate system and the coordinates of the marker in the pixel coordinate system, wherein the initial calibration relationship is an approximate conversion relationship between the pixel coordinate system and the mechanical arm base coordinate system, the second sampling point is selected from the pixel coordinate system, the base coordinate corresponding to the second sampling point in the mechanical arm base coordinate system is determined based on the initial calibration relationship, the end of the mechanical arm is controlled to move to the target position corresponding to the base coordinate, the end of the mechanical arm and the marker correspond to the coordinates in the camera coordinate system are obtained at the target position, and the coordinates of the hand-eye calibration relationship and the marker in the mechanical arm coordinate system are determined based on the end position of the mechanical arm and the coordinates of the marker in the camera coordinate system, so as to implement the hand-eye calibration from the camera to the mechanical arm. According to the hand-eye calibration method, on the basis of avoiding manual complicated operation, the calibration precision is improved, and a good hand-eye calibration effect is achieved.
In one embodiment, the initial calibration relationship determination module 604 is further configured to:
selecting a first sampling point from the mechanical arm base coordinate system, and determining the coordinate of the first sampling point in the mechanical arm base coordinate system;
controlling the tail end of the mechanical arm to move from an initial position to a first sampling point, and shooting the marker at the first sampling point through a camera to obtain a first image;
determining coordinates of the marker in a pixel coordinate system based on the first image;
and establishing an affine transformation relation according to the coordinates of the first sampling point in the mechanical arm base coordinate system and the coordinates of the marker in the pixel coordinate system, and determining an initial calibration relation based on the affine transformation relation.
In one embodiment, the second sampling point determining module 606 is further configured to:
determining the corresponding base plane coordinate of the second sampling point in the base coordinate system of the mechanical arm based on the initial calibration relation;
randomly selecting a target coordinate value from a preset coordinate value range, and determining the coordinate of the second sampling point in a mechanical arm base coordinate system based on the base plane coordinate and the target coordinate value; the preset coordinate range is a coordinate value range of a third base coordinate axis, and the third base coordinate axis is perpendicular to the base plane where the base plane coordinates are located.
In one embodiment, the pose determination module 608 is further configured to:
acquiring a preset tail end posture; the preset tail end posture is obtained by adjusting a preset angle on the basis of the tail end posture of the mechanical arm at the initial position;
controlling the tail end of the mechanical arm to move from the initial position to a target position corresponding to the base coordinate, and controlling the tail end of the mechanical arm to be in a preset tail end posture at the target position to obtain a tail end posture of the mechanical arm; the movement comprises translation and rotation;
shooting the marker through a camera at a target position to obtain a second image;
coordinates of the marker in the camera coordinate system are determined based on the second image.
In one embodiment, the calibration module 610 is configured to:
establishing a hand-eye calibration model as follows: AX = ZB, wherein A, B is a known quantity, A is a terminal pose, B is a coordinate of a marker in a camera coordinate system, X is a coordinate of the marker to be determined under a mechanical arm base coordinate system, and Z is a hand-eye calibration relation to be determined;
based on the hand-eye calibration model, establishing a nonlinear optimization model by taking the minimized Euclidean distance (I) AX-ZB (I) as an optimization target;
and iteratively determining the hand-eye calibration relation Z and the coordinate X of the marker in the base coordinate system of the mechanical arm by using a nonlinear optimization method based on the terminal pose of the mechanical arm, the coordinate of the marker in the camera coordinate system and the nonlinear optimization model.
In one embodiment, the apparatus further comprises:
the error determination module is used for inputting the hand-eye calibration relation Z, the coordinate X of the marker in the mechanical arm base coordinate system, the tail end pose of the mechanical arm and the coordinate of the target position in the camera coordinate system into the hand-eye calibration model and determining the calibration error corresponding to each second sampling point;
the target second sampling point selecting module is used for selecting a target second sampling point from the second sampling points based on the calibration error;
and the calibration module is used for re-determining the hand-eye calibration relation and the coordinates of the marker in the mechanical arm coordinate system based on the terminal pose of the mechanical arm corresponding to the target second sampling point and the coordinates of the target position corresponding to the camera coordinate system so as to realize the hand-eye calibration from the camera to the mechanical arm.
In one embodiment, the end of the mechanical arm is also fixedly provided with a tool, and the device further comprises:
and the tool calibration module 610 is used for determining the coordinates of the central point of the tool in the terminal coordinate system based on the coordinates of the marker in the base coordinate system of the mechanical arm so as to realize tool calibration.
For specific definition of the hand-eye calibration device, reference may be made to the above definition of the hand-eye calibration method, which is not described herein again. The various modules in the hand-eye calibration device described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent of a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, the internal structure of which may be as shown in FIG. 7. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program to run on the non-volatile storage medium. The communication interface of the computer device is used for communicating with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a hand-eye calibration method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In an embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, in which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.
All possible combinations of the technical features in the above embodiments may not be described for the sake of brevity, but should be considered as being within the scope of the present disclosure as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, and these are all within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A hand-eye calibration method, the method comprising:
establishing a hand-eye calibration system, wherein the hand-eye calibration system comprises a mechanical arm, a marker and a camera;
controlling the tail end of the mechanical arm to move to a first sampling point from an initial position, and determining an initial calibration relation based on the coordinates of the first sampling point in a mechanical arm base coordinate system and the coordinates of the marker in a pixel coordinate system; the initial calibration relation is an approximate conversion relation between a pixel coordinate system and the mechanical arm base coordinate system;
selecting a second sampling point from the pixel coordinate system, and determining a corresponding base coordinate of the second sampling point in a mechanical arm base coordinate system based on the initial calibration relation;
controlling the tail end of the mechanical arm to move to a target position corresponding to the base coordinate, and acquiring the tail end pose of the mechanical arm and the coordinates of the marker in a camera coordinate system at the target position;
establishing a hand-eye calibration model as follows: AX = ZB, wherein A, B is a known quantity, A is the terminal pose, B is the coordinate of the marker in a camera coordinate system, X is the coordinate of the marker to be determined in a mechanical arm coordinate system, and Z is the hand-eye calibration relation to be determined;
establishing a nonlinear optimization model by taking the minimum Euclidean distance as an optimization target based on the hand-eye calibration model;
based on the terminal pose of the mechanical arm, the coordinates of the marker in a camera coordinate system and the nonlinear optimization model, a non-established hand-eye calibration model is utilized as follows: AX = ZB, wherein A, B is a known quantity, A is the terminal pose, B is the coordinate of the marker in a camera coordinate system, X is the coordinate of the marker to be determined in a mechanical arm coordinate system, and Z is the hand-eye calibration relation to be determined;
based on the hand-eye calibration model, establishing a nonlinear optimization model by taking the minimum Euclidean distance AX-ZB as an optimization target;
based on the terminal pose of the mechanical arm, the coordinates of the marker in a camera coordinate system and the nonlinear optimization model, iteratively determining a hand-eye calibration relation Z by using a nonlinear optimization method and iteratively determining a hand-eye calibration relation Z and a coordinate X of the marker in a mechanical arm coordinate system by using a linear optimization method;
inputting the hand-eye calibration relation Z and the coordinate X of the marker in a mechanical arm coordinate system, the end pose of the mechanical arm and the coordinate of the marker in a camera coordinate system into the hand-eye calibration model, and determining the calibration error corresponding to each second sampling point;
selecting a target second sampling point from the second sampling points based on the calibration error;
and re-determining the hand-eye calibration relation and the coordinates of the marker in the coordinate system of the mechanical arm based on the terminal pose of the mechanical arm corresponding to the target second sampling point and the coordinates of the marker in the coordinate system of the camera so as to realize the hand-eye calibration from the camera to the mechanical arm.
2. The method of claim 1, wherein controlling the end of the robotic arm to move from the initial position to a first sampling point, determining an initial calibration relationship based on coordinates of the first sampling point in a robot arm base coordinate system and coordinates of the marker in a pixel coordinate system, comprises:
selecting a first sampling point from a mechanical arm base coordinate system, and determining the coordinate of the first sampling point in the mechanical arm base coordinate system;
controlling the tail end of the mechanical arm to move to the first sampling point from the initial position, and shooting the marker through the camera at the first sampling point to obtain a first image;
determining coordinates of the marker in a pixel coordinate system based on the first image;
and establishing an affine transformation relation according to the coordinates of the first sampling point in the mechanical arm base coordinate system and the coordinates of the marker in the pixel coordinate system, and determining an initial calibration relation based on the affine transformation relation.
3. The method of claim 2, wherein said determining the corresponding base coordinate of the second sample point in the arm base coordinate system based on the initial calibration relationship comprises:
determining the corresponding base plane coordinate of the second sampling point in a mechanical arm base coordinate system based on the initial calibration relation;
randomly selecting a target coordinate value from a preset coordinate value range, and determining the coordinate of the second sampling point in a mechanical arm base coordinate system based on the base plane coordinate and the target coordinate value; the preset coordinate range is a coordinate value range of a third base coordinate axis, and the third base coordinate axis is perpendicular to the base plane where the base plane coordinate is located.
4. The method according to claim 1, wherein the controlling the tip end of the robot arm to move to a target position corresponding to the base coordinate, and acquiring the tip end pose of the robot arm and the coordinates of the marker in a camera coordinate system at the target position comprises:
acquiring a preset tail end posture; the preset tail end posture is obtained by adjusting a preset angle on the basis of the tail end posture of the mechanical arm at the initial position;
controlling the tail end of the mechanical arm to move from an initial position to a target position corresponding to the base coordinate, and enabling the tail end of the mechanical arm to be in the preset tail end posture at the target position to obtain a tail end posture of the mechanical arm; the movement comprises translation and rotation;
shooting the marker at the target position through the camera to obtain a second image;
determining coordinates of the marker in a camera coordinate system based on the second image.
5. The method of claim 4, wherein said determining coordinates of said marker in a camera coordinate system based on said second image comprises:
identifying the second image to obtain the coordinates of the marker in a pixel coordinate system;
the coordinates of the marker in the camera coordinate system are determined based on camera intrinsic parameters and the coordinates of the marker in the pixel coordinate system.
6. The method of claim 1, wherein the movement comprises translation and rotation.
7. The method of claim 1, wherein the robotic arm tip is further fixedly mounted with a tool, the method further comprising:
and determining the coordinates of the central point of the tool in the terminal coordinate system based on the coordinates of the marker in the mechanical arm coordinate system so as to realize tool calibration.
8. A hand-eye calibration apparatus, the apparatus comprising:
the hand-eye calibration system establishing module is used for establishing a hand-eye calibration system, and the hand-eye calibration system comprises a mechanical arm, a marker and a camera;
the initial calibration relation determining module is used for controlling the tail end of the mechanical arm to move from an initial position to a first sampling point, and determining an initial calibration relation based on the coordinates of the first sampling point in a mechanical arm base coordinate system and the coordinates of the marker in a pixel coordinate system; the initial calibration relation is an approximate conversion relation between a pixel coordinate system and the mechanical arm base coordinate system;
the second sampling point determining module is used for selecting a second sampling point from the pixel coordinate system and determining corresponding base coordinates of the second sampling point in a mechanical arm base coordinate system based on the initial calibration relation;
the pose determining module is used for controlling the tail end of the mechanical arm to move to a target position corresponding to the base coordinate, and acquiring the tail end pose of the mechanical arm and the coordinates of the marker in a camera coordinate system at the target position;
the calibration module is used for establishing a hand-eye calibration model as follows: AX = ZB, wherein A, B is a known quantity, A is the terminal pose, B is the coordinate of the marker in a camera coordinate system, X is the coordinate of the marker to be determined in a mechanical arm coordinate system, and Z is the hand-eye calibration relation to be determined; establishing a nonlinear optimization model by taking the minimum Euclidean distance as an optimization target based on the hand-eye calibration model; based on the terminal pose of the mechanical arm, the coordinates of the marker in a camera coordinate system and the nonlinear optimization model, the non-established hand-eye calibration model is used as follows: AX = ZB, wherein A, B is a known quantity, A is the terminal pose, B is the coordinate of the marker in a camera coordinate system, X is the coordinate of the marker to be determined in a mechanical arm coordinate system, and Z is the hand-eye calibration relation to be determined; based on the hand-eye calibration model, establishing a nonlinear optimization model by taking the minimum Euclidean distance AX-ZB as an optimization target; iteratively determining a hand-eye calibration relation Z and a coordinate X of the marker in a mechanical arm coordinate system by using a nonlinear optimization method based on the tail end pose of the mechanical arm, the coordinate of the marker in a camera coordinate system and the nonlinear optimization model, and iteratively determining the hand-eye calibration relation Z and the coordinate X of the marker in the mechanical arm coordinate system by using a nonlinear optimization method;
the error determination module is used for inputting the hand-eye calibration relation Z and the coordinate X of the marker in a mechanical arm coordinate system, the tail end pose of the mechanical arm and the coordinate of the marker in a camera coordinate system into the hand-eye calibration model and determining the calibration error corresponding to each second sampling point;
the target second sampling point selecting module is used for selecting a target second sampling point from the second sampling points based on the calibration error;
and the calibration module is used for re-determining the hand-eye calibration relation and the coordinates of the marker in the coordinate system of the mechanical arm based on the terminal pose of the mechanical arm corresponding to the target second sampling point and the coordinates of the marker in the coordinate system of the camera so as to realize the hand-eye calibration from the camera to the mechanical arm.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111394621.9A CN114012731B (en) | 2021-11-23 | 2021-11-23 | Hand-eye calibration method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111394621.9A CN114012731B (en) | 2021-11-23 | 2021-11-23 | Hand-eye calibration method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114012731A CN114012731A (en) | 2022-02-08 |
| CN114012731B true CN114012731B (en) | 2023-03-31 |
Family
ID=80065836
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111394621.9A Active CN114012731B (en) | 2021-11-23 | 2021-11-23 | Hand-eye calibration method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114012731B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114505864B (en) * | 2022-03-11 | 2024-02-09 | 上海柏楚电子科技股份有限公司 | Hand-eye calibration method, device, equipment and storage medium |
| CN114794666A (en) * | 2022-03-31 | 2022-07-29 | 深圳市如本科技有限公司 | Vamp gluing method, system, computer equipment and computer-readable storage medium |
| CN114794667B (en) * | 2022-03-31 | 2023-04-14 | 深圳市如本科技有限公司 | Tool calibration method, system, device, electronic equipment and readable storage medium |
| CN114536347A (en) * | 2022-04-08 | 2022-05-27 | 上海电气集团股份有限公司 | Mechanical arm calibration position determining method, mechanical arm calibration system and electronic equipment |
| CN114734444B (en) * | 2022-04-27 | 2023-06-27 | 博众精工科技股份有限公司 | Target positioning method and device, electronic equipment and storage medium |
| CN114677429B (en) * | 2022-05-27 | 2022-08-30 | 深圳广成创新技术有限公司 | Positioning method and device of manipulator, computer equipment and storage medium |
| CN114998446A (en) * | 2022-05-30 | 2022-09-02 | 深圳棱镜空间智能科技有限公司 | Calibration method between camera and manipulator, controller, device and storage medium |
| CN115284289A (en) * | 2022-08-13 | 2022-11-04 | 上海由格医疗技术有限公司 | Full-automatic hand-eye calibration system and method for mechanical arm, computer equipment and medium |
| CN115556109B (en) * | 2022-10-24 | 2024-06-11 | 深圳市通用测试系统有限公司 | Positioning method and device for mechanical arm in test system |
| CN115969418B (en) * | 2022-11-07 | 2023-08-15 | 珠海市人民医院 | Nucleic acid detection oral cavity collection point identification method |
| CN117817671B (en) * | 2024-02-21 | 2024-06-21 | 北京迁移科技有限公司 | Robot system based on visual guidance and robot system calibration method |
| CN118342520B (en) * | 2024-06-17 | 2024-10-29 | 深圳思谋信息科技有限公司 | Mechanical arm calibration method and device, intelligent equipment and storage medium |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6897396B2 (en) * | 2017-07-28 | 2021-06-30 | セイコーエプソン株式会社 | Control devices, robot systems and control methods |
| CN111070199A (en) * | 2018-10-18 | 2020-04-28 | 杭州海康威视数字技术股份有限公司 | Hand-eye calibration assessment method and robot |
| CN110238849B (en) * | 2019-06-13 | 2021-11-12 | 深圳市汇川技术股份有限公司 | Robot hand-eye calibration method and device |
| CN113084808B (en) * | 2021-04-02 | 2023-09-22 | 上海智能制造功能平台有限公司 | Monocular vision-based 2D plane grabbing method for mobile mechanical arm |
| CN113160326B (en) * | 2021-04-08 | 2024-07-09 | 福建(泉州)先进制造技术研究院 | Hand-eye calibration method and device based on reconstruction coordinate system |
| CN113021358B (en) * | 2021-05-21 | 2021-08-10 | 季华实验室 | Method and device for calibrating origin of coordinate system of mechanical arm tool and electronic equipment |
| CN113172636B (en) * | 2021-06-29 | 2021-11-02 | 深圳市越疆科技有限公司 | Automatic hand-eye calibration method and device and storage medium |
-
2021
- 2021-11-23 CN CN202111394621.9A patent/CN114012731B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN114012731A (en) | 2022-02-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114012731B (en) | Hand-eye calibration method and device, computer equipment and storage medium | |
| CN109285190B (en) | Object positioning method and device, electronic equipment and storage medium | |
| CN107428009B (en) | Method for commissioning an industrial robot, industrial robot system and control system using the method | |
| US20200282575A1 (en) | Automatic calibration for a robot optical sensor | |
| US20200298411A1 (en) | Method for the orientation of an industrial robot, and industrial robot | |
| Zhan et al. | Hand–eye calibration and positioning for a robot drilling system | |
| CN111801198A (en) | Hand-eye calibration method, system and computer storage medium | |
| Motai et al. | Hand–eye calibration applied to viewpoint selection for robotic vision | |
| TWI677413B (en) | Calibration method and device for robotic arm system | |
| CN104842352A (en) | Robot system using visual feedback | |
| JP2012141962A (en) | Position and orientation measurement device and position and orientation measurement method | |
| CN114474056B (en) | Monocular vision high-precision target positioning method for grabbing operation | |
| CN113379849A (en) | Robot autonomous recognition intelligent grabbing method and system based on depth camera | |
| CN111590593B (en) | Calibration method, device and system of mechanical arm and storage medium | |
| CN114260908B (en) | Robot teaching method, apparatus, computer device and computer program product | |
| CN114310901B (en) | Coordinate system calibration method, device, system and medium for robot | |
| CN111445521A (en) | Target workpiece position determining method, device, terminal and readable storage medium | |
| CN114147728A (en) | Universal robot eye on-hand calibration method and system | |
| Mu et al. | Calibration of a flexible measurement system based on industrial articulated robot and structured light sensor | |
| CN109760107B (en) | Monocular vision-based robot positioning accuracy evaluation method | |
| Lu et al. | Calibration of five-axis motion platform based on monocular vision | |
| CN115042184A (en) | Robot hand-eye coordinate conversion method and device, computer equipment and storage medium | |
| CN114677429B (en) | Positioning method and device of manipulator, computer equipment and storage medium | |
| CN114833825B (en) | Collaborative robot control method, device, computer equipment and storage medium | |
| CN115619877A (en) | Method for calibrating position relation between monocular laser sensor and two-axis machine tool system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |