CN115880372A - Unified calibration method and system for external hub positioning camera of automatic crane - Google Patents
Unified calibration method and system for external hub positioning camera of automatic crane Download PDFInfo
- Publication number
- CN115880372A CN115880372A CN202211600555.0A CN202211600555A CN115880372A CN 115880372 A CN115880372 A CN 115880372A CN 202211600555 A CN202211600555 A CN 202211600555A CN 115880372 A CN115880372 A CN 115880372A
- Authority
- CN
- China
- Prior art keywords
- lock
- anw2
- camera
- positioning
- detection frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000001514 detection method Methods 0.000 claims abstract description 94
- 230000000007 visual effect Effects 0.000 claims description 14
- 101100162703 Caenorhabditis elegans ani-1 gene Proteins 0.000 claims description 12
- 101100162705 Caenorhabditis elegans ani-2 gene Proteins 0.000 claims description 12
- 101100162706 Caenorhabditis elegans ani-3 gene Proteins 0.000 claims description 12
- 238000013135 deep learning Methods 0.000 claims description 8
- 238000013519 translation Methods 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 5
- 238000004364 calculation method Methods 0.000 abstract description 2
- 238000004891 communication Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Landscapes
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention discloses a unified calibration method and a system for an external hub card positioning camera of an automatic crane, wherein the method comprises the following steps: according to the position of a lock head on an outer truck collecting plate under the standard operation condition of an outer truck collecting, four positioning cameras which are calibrated by internal reference are correspondingly arranged on a crane gate leg; respectively collecting video image data of an outer collection truck road through a positioning camera; respectively extracting the tapered end characteristics in the video image data of the outer collection truck road by a target detection method, and acquiring the pixel coordinates of a tapered end detection frame corner point; calculating world coordinates of corner points of the lock detection frame based on internal parameters of the positioning camera and actual sizes of the lock; and respectively calculating the external parameters of each positioning camera according to the pixel coordinates of the corner point of the lock detection frame, the world coordinates of the corner point of the lock detection frame and the internal parameters of each positioning camera. The invention can unify the camera coordinate systems of a plurality of cameras into the world coordinate system without additionally setting ground mark points, thereby reducing the calculation error, simplifying the calibration process and having higher efficiency.
Description
Technical Field
The invention belongs to the technical field of port intelligent hoisting, and particularly relates to a unified calibration method and a system for an external truck positioning camera of an automatic crane.
Background
With the gradual progress of port automation, the requirement of port on the full-automatic operation of external collection cards is gradually increased in order to improve the operation efficiency. At present, due to the problems of multiple types and large characteristic difference of the outer container trucks, the position of the outer container truck relative to a crane is still difficult to obtain by using the traditional three-dimensional point cloud to extract the characteristic information of the outer container truck, and the operation efficiency is low. The video positioning and guiding function of the outer container truck depends on unified calibration of multiple cameras, the relative pose of the cameras is calculated, and then the relative position of the outer container truck is solved. In order to improve the precision of the automatic operation, the camera needs to be calibrated quickly and precisely. According to the traditional camera external parameter calibration method, a calibration plate needs to be placed to solve external parameters of a camera, a container truck positioning camera is installed on a crane door leg, the angle between the optical axis of the camera and the ground is large, under the condition of hardware installation, calibration external parameters need to be manually held by hands or in other fixing modes, the external parameter calibration error is large, and the process is complex. The invention patent with publication number CN113012235A discloses a port crane lifting tool attitude control system and a control method thereof, wherein four industrial cameras are installed on a lifting tool, a plurality of ground marks are required to be matched to finish calibration among the cameras, relative positions among different ground marks need to be strictly controlled, and the calibration process is complicated and is easy to bring calibration errors.
In summary, it is necessary to design a multi-phase external parameter calibration method reasonably for the application scenario of the external hub positioning camera, so as to achieve stable external parameter calibration efficiency and reduce calibration errors.
Disclosure of Invention
In view of the above, the invention provides a unified calibration method and system for an external hub positioning camera of an automatic crane, which are used for solving the problem of large external parameter calibration error of the external hub positioning camera.
The invention discloses a unified calibration method for an external hub positioning camera of an automatic crane, which comprises the following steps:
according to the position of the lock head on the outer truck collecting plate under the standard operation condition of the outer truck collecting, four positioning cameras with internal reference calibration are correspondingly arranged on the crane gate leg, so that the lock head on the outer truck collecting plate is in the visual field range of the corresponding positioning cameras;
respectively collecting video image data of the outer collection truck road through a positioning camera;
respectively extracting the tapered end characteristics in the video image data of the outer collection truck road by a target detection method, and acquiring the pixel coordinates of a tapered end detection frame corner point;
calculating the world coordinate of the corner point of the lock detection frame based on the actual size of the lock;
and respectively calculating the external parameters of each positioning camera according to the pixel coordinates of the corner point of the lock detection frame, the world coordinates of the corner point of the lock detection frame and the internal parameters of each positioning camera.
On the basis of the above technical solution, preferably, four positioning cameras calibrated by internal references are correspondingly installed on the crane gate leg according to the position of the lock head on the outer truck collecting plate under the standard operation condition of the outer truck collecting, so that the lock head on the outer truck collecting plate specifically includes within the visual field range of the corresponding positioning camera:
the central planes of half of the long side sizes of 20-foot containers and 40-foot containers are respectively used as symmetrical planes, four positioning cameras are symmetrically arranged on the door legs of the sides of the truck collecting operation lanes outside the crane at positions 1.5-2 meters away from the ground, and the positioning cameras can capture a pair of lock head features oppositely arranged on the truck collecting boards outside the visual field range according to different containers and operation types under the standard operation condition of the truck collecting operation outside the crane.
On the basis of the technical scheme, preferably, the target detection method is realized by adopting a deep learning algorithm, and four corner coordinates of the lock detection frame are solved after the lock is detected and identified.
On the basis of the above technical solution, preferably, the calculating the world coordinate of the corner point of the tapered end detection frame based on the actual size of the tapered end specifically includes:
establishing a world coordinate system by taking the center of a side door leg of the loading lane as an origin, and respectively establishing a camera coordinate system for each positioning camera;
measuring the world coordinate of one angular point according to the overlapping relation of the lock head and the angular point of the detection frame;
and calculating the world coordinates of other detection frame angular points by combining the actual size of the lock and the actual distance of the two locks in the same image in the direction of the trolley on the basis of the world coordinates of one detection frame angular point to obtain the world coordinates of 8 detection frame angular points of the two locks in the same image.
On the basis of the above technical solution, preferably, the calculating the world coordinates of other detection corner points by using the world coordinates of one detection corner point as a basis and combining the actual size of the lock and the actual distance between two locks in the same image in the direction of the trolley specifically includes:
the actual size of the lock standard part is set as the height h s Thickness t s Width w s And the distance d in the thickness direction of the angular point s In the same image, the actual distance between the two lock heads in the direction of the trolley is D s ；
In a unified image collected by a certain positioning camera A, the pixel coordinates of the corner points of the two lock detection frames at the near end and the far end are as follows: { P Ani1 ,P Ani2 ,P Ani3 ,P Ani4 }、{P Afi1 ,P Afi2 ,P Afi3 ,P Afi4 And the world coordinates corresponding to each corner point represent: { P Anw1 ,P Anw2 ,P Anw3 ,P Anw4 }、{P Afw1 ,P Afw2 ,P Afw3 ,P Afw4 In which P Anw1 ,P Anw2 ,P Anw3 ,P Anw4 The left lower corner, the right upper corner and the left upper corner of the near-end lock detection frame, P Afw1 ,P Afw2 ,P Afw3 ,P Afw4 Detecting the left lower corner, the right upper corner and the left upper corner of a frame corresponding to the remote lock head;
according to the overlapping of the lock and the corner of the detection frameMeasuring the world coordinate P of a corner point Anw2 ：{X Anw2 ,Y Anw2 ,Z Anw2 According to { X } Anw2 ,Y Anw2 ,Z Anw2 Calculate the world coordinates of the remaining 7 corner points:
P Anw1 ：{X Anw2 ,Y Anw2 +w s ,Z Anw2 }；
P Anw3 ：{X Anw2 +d s ,Y Anw2 ,Z Anw2 +h s }；
P Anw4 ：{X Anw2 +d s ,Y Anw2 +w s ,Z Anw2 +h s }；
P Afw1 ：{X Anw2 +D s ,Y Anw2 +w s ,Z Anw2 }；
P Afw2 ：{X Anw2 +D s ,Y Anw2 ,Z Anw2 }
P Afw3 ：{X Anw2 +d s +D s ,Y Anw2 ,Z Anw2 +h s }；
P Afw4 ：{X Anw2 +d s +D s ,Y Anw2 +w s ,Z Anw2 +h s }。
on the basis of the above technical solution, preferably, the calculating the external parameters of each positioning camera according to the pixel coordinates of the lock detection frame corner point, the world coordinates of the lock detection frame corner point, and the internal parameters of each positioning camera respectively includes:
according to the camera pinhole model, establishing a corresponding relation equation between the pixel coordinates and world coordinates of each lock head detection frame corner point based on internal parameters of a positioning camera;
establishing an equation set by simultaneously establishing each corresponding relation equation, and solving the equation set through a PnP algorithm in an image processing algorithm to obtain a rotation vector and a translation vector of a camera coordinate system of each positioning camera relative to a world coordinate system.
On the basis of the above technical solution, preferably, the establishing, according to the camera pinhole model, a correspondence equation between the pixel coordinates of each lock detection frame corner point and the world coordinates based on the positioning camera internal parameters specifically includes:
for any one of the positioning cameras a, the following correspondence equation exists:
Z c to detect the conversion factor between the camera and the world coordinate system, (u) Ani1 ,v Ani1 )、(u Ani2 ,v Ani2 )、(u Ani3 ,v Ani3 )、(u Ani4 ,v Ani4 ) Respectively detecting angular points P of the lock Ani1 、P Ani2 、P Ani3 、P Ani4 Corresponding toPixel coordinate, (u) Afi1 ,v Afi1 )、(u Afi2 ,v Afi2 )、(u Afi3 ,v Afi3 )、(u Afi4 ,v Afi4 ) Respectively detecting angular points P of the lock Afi1 、P Afi2 、P Afi3 、P Afi4 Corresponding pixel coordinate, K A As a result of calibration of camera internal parameters, R AWC And T AWC The rotation vector and translation vector of the camera coordinate system of the positioning camera a relative to the world coordinate system, respectively.
In a second aspect of the present invention, a unified calibration system for an external truck-mounted positioning camera of an automated crane is disclosed, the system comprising:
positioning a camera: the lock heads on the outer truck collecting plates are arranged on the crane door legs and are positioned in the visual field range of the corresponding positioning cameras, and the lock heads are used for collecting video image data of the outer truck collecting lanes;
a feature extraction module: the method is used for respectively extracting the tapered end characteristics in the video image data of the outer collection truck road by a target detection method and acquiring the pixel coordinates of the tapered end detection frame corner points;
a coordinate conversion module: the system is used for calculating the world coordinates of the corner points of the lock detection frame based on the actual size of the lock;
the external parameter calibration module: and the method is used for respectively calculating the external parameters of each positioning camera according to the pixel coordinates of the corner point of the lock detection frame, the world coordinates of the corner point of the lock detection frame and the internal parameters of each positioning camera.
Compared with the prior art, the invention has the following beneficial effects:
1) Four positioning cameras which are subjected to internal reference calibration are correspondingly arranged on a crane door leg and are respectively used for acquiring characteristic data of two lock heads before 40 feet, before 20 feet, after 20 feet and after 40 feet, the information of the external truck lock heads in a visual field range is positioned by a deep learning target detection method, the external reference calibration of the cameras is carried out by taking the external truck lock heads as uniform characteristic marks, the camera coordinate systems of a plurality of cameras can be unified into a world coordinate system without additionally setting ground mark points, the resolving errors caused by additionally setting the mark points are reduced, the calibration can be completed in a standard operation process, the calibration process is simplified, and the efficiency is higher;
2) According to the invention, the overlapping relation of the lock heads and the detection frame corner points is utilized, the world coordinate systems of other detection frame corner points are calculated by combining the actual size of the lock heads and the actual distance of two lock heads in the same image in the direction of the trolley with the world coordinate system of one detection frame corner point obtained through measurement, and finally the external parameters of each positioning camera are respectively calculated according to the pixel coordinates of the lock head detection frame corner points, the world coordinates of the lock head detection frame corner points and the internal parameters of each positioning camera, so that the complexity of external parameter calibration is reduced, the stability is higher, the pose of the external truck positioning camera relative to the truck lock head is more reasonably solved, the accurate relative position of the external truck on the crane can be further directly and quickly calculated by utilizing the pose information, and the functions of video truck guide and the like required by the external truck automation are favorably realized.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a top view of the mounting positions of positioning cameras A, B, C, D;
FIG. 2 is a schematic view of a world coordinate system and a camera mounting location;
FIG. 3 is a schematic view of a camera coordinate system for positioning cameras A, B, C, D;
FIG. 4 is a top view of the distribution of the outer hub locks;
FIG. 5 is a top view of the distribution of 40-foot and 20-foot outer hub lock heads during calibration;
FIG. 6 is a schematic view of the imaging effect of the locking heads of the positioning cameras A, B, C, D;
FIG. 7 is a schematic view of a lock detection box;
fig. 8 is a schematic diagram of a lock cylinder three-view and a distribution of corner points of a lock cylinder detection frame.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments of the present invention, belong to the protection scope of the present invention.
The invention provides a unified calibration method for an external hub card positioning camera of an automatic crane, which comprises the following steps:
s1, according to the position of a lock head on an outer truck collecting plate under the standard operation condition of an outer truck, four positioning cameras with internal reference calibration are correspondingly installed on a crane door leg, so that the lock head on the outer truck collecting plate is in the visual field range of the corresponding positioning cameras.
Specifically, a central plane where half of the long side of each of 20-foot containers and 40-foot containers is located is taken as a symmetry plane, four positioning cameras are symmetrically arranged on a door leg on the side of an outer truck collecting operation lane of the crane at a position 1.5-2 meters away from the ground, and it is ensured that under the standard operation condition of an outer truck, each positioning camera can capture a pair of lock head features oppositely arranged on an outer truck collecting plate in a visual field range according to different containers and operation types, four positioning cameras are respectively taken as positioning cameras a, B, C and D, a mounting position top view is shown in fig. 1, and the positioning cameras a, B, C and D are respectively used for acquiring feature data of two lock heads of the outer truck collecting plate before 40 feet, before 20 feet, after 20 feet and after 40 feet.
Because 4 positioning cameras are used in total, the installation positions of the positioning cameras are different, an independent camera coordinate system is provided, a unified world coordinate system needs to be set, and the relative pose of each camera is solved in a camera external reference calibration mode, so that unified calibration of the multiple positioning cameras is realized. In order to facilitate the positioning of the external container truck, the center of the side door leg of the loading road is set as a world coordinate origin O w With O w The forward side of the vertical door leg is the positive direction of an X axis, the left side of the parallel door leg is the positive direction of a Y axis, and the door leg passes through an O w The point is vertically upward to the ground and is the positive direction of the Z axis, and a world coordinate system C is established w As shown in fig. 2.
The four positioning cameras are cameras subjected to internal reference calibration, the internal reference calibration is realized by a Zhang Yongyou calibration method, and a camera internal reference matrix determined by camera hardware setting can be obtained through the internal reference calibration:
where f represents the focal length of the camera in millimeters; dx and dy respectively represent the widths of the image pixels in the x direction and the y direction and are in unit of millimeter; 1/dx and 1/dy can be understood as the x direction and the y direction, and how many pixels are in 1 millimeter; f/dx represents the length of the focal length in the x-axis direction using the pixel, and f/dy represents the length of the focal length in the y-axis direction using the pixel. u. u 0 、v 0 Respectively, which represent coordinates of the center of the camera plate in the pixel coordinate system. To distinguish the internal reference coefficients of the four positioning cameras, the names of the positioning cameras are taken as subscripts of the internal reference coefficients of the cameras, and the positioning camera a is taken as an example: f. of A 、dx A 、dy A 、u 0A And v 0A Respectively obtaining internal reference data of the positioning camera A by a Zhang Dingyou calibration method.
The internal reference calibration results of all the positioning cameras obtained by the calculation in the steps are used for unified calibration of the external reference of the cameras. When the camera is subjected to unified calibration, except that a unified world coordinate system C needs to be established w And respective camera coordinate systems of the four cameras, and image coordinate systems and pixel coordinate systems corresponding to the cameras are also required to be established.
The camera coordinate systems of the four positioning cameras are established as shown in FIG. 3, which are respectively C A ，C B ，C C And C D Wherein, O A 、O B 、O C And O D Respectively as the origin of each camera coordinate system, and the positive directions of XYZ axes of each camera are all equal to C w X of world coordinate system w Y w Z w And (5) the consistency is achieved. Each positioning camera corresponds to a pixel coordinate system and an image coordinate system, the image coordinate system depends on the camera coordinate system and is a relative coordinate system, X i Axis and Y i Axes respectively parallel to X of the camera coordinate system c And Z c A shaft; pixel coordinate system relies on imagesCoordinate system, also relative coordinate system, X p Axis and Y p Axes parallel to X of the image coordinate system i Axis and Y i And a shaft.
And S2, respectively acquiring video image data of the outer collection truck road through a positioning camera.
After a camera coordinate system, an image coordinate system, a pixel coordinate system and a world coordinate system are established, the characteristic point images in the visual field can be acquired by utilizing each positioning camera.
The invention provides a unified feature for unified external parameter calibration work, which is a fixed lock on an external truck collection plate. The distribution of the lock heads on the outer truck collecting plate is shown in fig. 4, the common outer truck collecting plate has 12 lock heads, the lock heads are uniformly distributed on two sides of the truck collecting plate, and the positions and the number of the lock heads lifted on the truck collecting plate are inconsistent under different operating states. When the unified calibration of the positioning camera is performed, a raised lock head state is required, for example, a 40-size lock head is raised, and a middle 20-size lock head is required, as shown in fig. 5.
When unified calibration, in each location camera field of vision scope, tapered end distribution and formation of image position are as shown in figure 6, and two tapered ends near the distal end department can all be gathered to each location camera to every tapered end becomes the image effect clear in the image.
And S3, respectively extracting the tapered end features in the video image data of the outer collection truck road through a target detection method, and acquiring the pixel coordinates of the tapered end detection frame corner points.
The target detection method is realized by adopting a deep learning algorithm in deep learning, the image data of the lock head under partial actual scenes are collected in advance, a deep learning algorithm model is trained after marking, and after the lock head to be detected is detected and identified through the deep learning algorithm model, the four corner point coordinates of a lock head detection frame are solved.
When the lock detection is executed, the frame selection state of the lock detection frame is shown in fig. 7, the LU, LD, RU, and RD are 4 corner points of the lock detection frame, respectively, and in a state where the model prediction is accurate, the lower edge of the lock detection frame coincides with the lower edge of the lock imaged in the image, and the upper edge of the lock detection frame coincides with the upper edge of the lock imaged in the image. The distribution of four corner points of the lock detection box in the lock three-view is shown in fig. 8, wherein the dots represent the corner points of the detection box.
And S4, calculating the world coordinates of the corner points of the lock detection frame based on the actual size of the lock.
Firstly, a world coordinate system of a detection frame angular point is measured, then, the world coordinate systems of other detection frame angular points are calculated by combining the actual sizes of the lock heads and the actual distances of the two lock heads in the same image in the direction of the trolley on the basis of the world coordinate system of the detection frame angular point, and the world coordinate systems of 8 detection frame angular points of the two lock heads in the same image are obtained.
Specifically, the actual size of the lock standard part is set as the height h s Thickness t s Width w s And the distance d in the thickness direction of the angular point s In the same image, the actual distance between the two lock heads in the direction of the trolley is D s . In a unified image collected by a certain positioning camera A, the pixel coordinates of the corner points of the two lock detection frames at the near end and the far end are as follows: { P Ani1 ,P Ani2 ,P Ani3 ,P Ani4 }、{P Afi1 ,P Afi2 ,P Afi3 ,P Afi4 And expressing the world coordinates corresponding to each corner point as: { P Anw1 ,P Anw2 ,P Anw3 ,P Anw4 }、{P Afw1 ,P Afw2 ,P Afw3 ,P Afw4 In which P is Anw1 ,P Anw2 ,P Anw3 ,P Anw4 World coordinates P of the lower left corner, the lower right corner, the upper right corner and the upper left corner of the detection frame of the near-end lock head respectively Afw1 ,P Afw2 ,P Afw3 ,P Afw4 World coordinates of a lower left corner, a lower right corner, an upper right corner and an upper left corner of the lock detection frame are detected correspondingly to the remote locks.
Measuring the world coordinate of one corner point according to the overlapping relation of the lock and the corner point of the detection frame and the relative relation of the lock of the outer container and the origin of the world coordinate system, and supposing that the world coordinate P of one corner point is obtained by measurement Anw2 ：{X Anw2 ,Y Anw2 ,Z Anw2 Is then according to { X } Anw2 ,Y Anw2 ,Z Anw2 Calculate the world coordinates of the remaining 7 corner points:
P Anw1 ：{X Anw2 ,Y Anw2 +w s ,Z Anw2 }；
P Anw3 ：{X Anw2 +d s ,Y Anw2 ,Z Anw2 +h s }；
P Anw4 ：{X Anw2 +d s ,Y Anw2 +w s ,Z Anw2 +h s }；
P Afw1 ：{X Anw2 +D s ,Y Anw2 +w s ,Z Anw2 }；
P Afw2 ：{X Anw2 +D s ,Y Anw2 ,Z Anw2 }
P Afw3 ：{X Anw2 +d s +D s ,Y Anw2 ,Z Anw2 +h s }；
P Afw4 ：{X Anw2 +d s +D s ,Y Anw2 +w s ,Z Anw2 +h s }。
and for each positioning camera, obtaining pixel coordinates and world coordinates of 8 corner points in total of 2 lock head detection frames in the corresponding image.
And S5, respectively calculating the external parameters of each positioning camera according to the pixel coordinates of the lock detection frame corner point, the world coordinates of the lock detection frame corner point and the internal parameters of each positioning camera.
According to the camera pinhole model and the external reference calibration principle, the invention is based on the pixel coordinates and world coordinates of 8 corner points corresponding to each set of calibration cameras and the internal reference calibration result K of each camera A 、K B 、K C 、K D Calculating the relative world coordinate origin C of each calibration camera coordinate system w Of (2) a rotation vector R AWC 、R BWC 、R CWC 、R DWC And translation vector T AWC 、T BWC 、T CWC 、T DWC 。
Specifically, according to a camera pinhole model, a corresponding relation equation between the pixel coordinates of each lock detection frame corner point and world coordinates is established based on internal parameters of a positioning camera. For example, for the positioning camera a, the following correspondence equation exists between the pixel coordinates of 8 corner points and world coordinates:
Z c to locate the conversion factor between the camera and the world coordinate system, (u) Ani1 ,v Ani1 )、(u Ani2 ,v Ani2 )、(u Ani3 ,v Ani3 )、(u Ani4 ,v Ani4 ) Respectively detecting angular points P of the lock Ani1 、P Ani2 、P Ani3 、P Ani4 Corresponding pixel coordinate, (u) Afi1 ,v Afi1 )、(u Afi2 ,v Afi2 )、(u Afi3 ,v Afi3 )、(u Afi4 ,v Afi4 ) Detect the angular point P for the lock Afi1 、P Afi2 、P Afi3 、P Afi4 Corresponding pixel coordinate, K A To locate the internal reference calibration result of camera A, R AWC And T AWC The rotation vector and translation vector of the camera coordinate system of the positioning camera a relative to the world coordinate system, respectively.
Establishing an equation set by simultaneously establishing the above 8 corresponding relation equations, solving the equation set through a PnP algorithm in an image processing algorithm to obtain a rotation vector R of a camera coordinate system of the positioning camera A relative to a world coordinate system AWC And translation vector T AWC And obtaining the external parameters of the camera.
And other positioning cameras respectively adopt the same mode to establish corresponding relation equations between the pixel coordinates and the world coordinates of the corresponding 8 corner points, the 8 corresponding relation equations are combined to form an equation set, and the equation set is solved to obtain the rotation vector and the translation vector of the camera coordinate system of the corresponding positioning camera relative to the world coordinate system. And calibrating the four positioning cameras into the same world coordinate system, so as to realize the unified calibration of the outer truck positioning cameras of the automatic crane.
According to the invention, four positioning cameras are arranged on the crane door legs according to the positions of the lock heads on the truck boards of 20-foot and 40-foot outer trucks under the standard operation condition of the outer trucks, the fixed lock heads on the truck boards of the outer trucks are uniform, 2 lock heads in respective visual field ranges are respectively positioned by a deep learning method, and the camera coordinate systems of a plurality of positioning cameras are unified into a world coordinate system according to the conversion relation between the pixel coordinates of the lock heads and the world coordinates and by combining an outer parameter solving algorithm, so that the uniform calibration of the outer truck positioning cameras is realized.
Compared with the existing multi-camera pose solving method, the method provided by the invention has the advantages of stronger stability, higher efficiency, smaller solving error and capability of solving the pose of the outer hub positioning camera relative to the hub lock head more reasonably. Therefore, on the basis of realizing unified calibration of the outer hub positioning cameras of the automatic crane, the lock characteristics in the picture can be identified through the four positioning cameras, the pixel coordinates of the lock in the picture are converted into world coordinates of the crane by utilizing the internal and external reference calibration results of the cameras, the position of the outer hub relative to the crane is calculated, and functions such as hub video guidance are realized.
Corresponding to the embodiment of the method, the invention also provides a unified calibration system for the outer truck positioning camera of the automatic crane, which comprises the following steps:
positioning a camera: the lock heads on the outer truck collecting plates are arranged in the visual field range of the corresponding positioning cameras and are used for collecting video image data of the outer truck collecting rail;
a feature extraction module: the method is used for respectively extracting the tapered end characteristics in the video image data of the outer collection truck road by a target detection method and acquiring the pixel coordinates of the tapered end detection frame corner points;
a coordinate conversion module: the system is used for calculating the world coordinates of the corner points of the lock detection frame based on the actual size of the lock;
the external reference calibration module: and the method is used for respectively calculating the external parameters of each positioning camera according to the pixel coordinates of the corner point of the lock detection frame, the world coordinates of the corner point of the lock detection frame and the internal parameters of each positioning camera.
The above system embodiments and method embodiments are in one-to-one correspondence, and please refer to the method embodiments for brief description of the system embodiments.
The invention also discloses an electronic device, comprising: at least one processor, at least one memory, a communication interface, and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the memory stores program instructions executable by the processor which invokes the method of the invention as described above.
The invention also discloses a computer readable storage medium which stores computer instructions for causing the computer to implement all or part of the steps of the method of the embodiment of the invention. The storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a read-only memory ROM, a random access memory RAM, a magnetic disk, or an optical disk.
The above-described system embodiments are merely illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts shown as units may or may not be physical units, i.e. may be distributed over a plurality of network units. Without creative labor, a person skilled in the art can select some or all of the modules according to actual needs to achieve the purpose of the solution of the embodiment.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and should not be taken as limiting the scope of the present invention, which is intended to cover any modifications, equivalents, improvements, etc. within the spirit and scope of the present invention.
Claims (8)
1. A unified calibration method for an external hub positioning camera of an automatic crane is characterized by comprising the following steps:
according to the position of the lock head on the outer truck collecting plate under the standard operation condition of the outer truck collecting, four positioning cameras with internal reference calibration are correspondingly arranged on the crane gate leg, so that the lock head on the outer truck collecting plate is in the visual field range of the corresponding positioning cameras;
respectively collecting video image data of an outer collection truck road through a positioning camera;
respectively extracting the tapered end characteristics in the video image data of the outer collection truck road by a target detection method, and acquiring the pixel coordinates of a tapered end detection frame corner point;
calculating the world coordinate of the corner point of the lock detection frame based on the actual size of the lock;
and respectively calculating the external parameters of each positioning camera according to the pixel coordinates of the corner point of the lock detection frame, the world coordinates of the corner point of the lock detection frame and the internal parameters of each positioning camera.
2. The unified calibration method for the outer truck-mounted positioning cameras of the automatic crane according to claim 1, wherein four positioning cameras calibrated by internal parameters are correspondingly mounted on the crane gate leg according to the position of the lock head on the outer truck-mounted plate under the standard operation condition of the outer truck-mounted positioning cameras, so that the lock head on the outer truck-mounted plate is specifically included in the visual field range of the corresponding positioning camera:
the central plane of half of the long side size of each of the 20-40-foot containers is taken as a symmetry plane, four positioning cameras are symmetrically arranged on a door leg on the side of an outer truck collecting operation lane of the crane at a position 1.5-2 meters away from the ground, and the positioning cameras can capture a pair of lock head features oppositely arranged on an outer truck collecting board in a visual field range under different containers and operation types under the standard operation condition of the outer truck collecting.
3. The unified calibration method for the external truck positioning camera of the automatic crane according to claim 1, wherein the target detection method is implemented by adopting a deep learning algorithm, and after the lock head is detected and identified, the four corner coordinates of the lock head detection frame are solved.
4. The unified calibration method for the outer hub positioning camera of the automatic crane according to claim 1, wherein the step of calculating the world coordinates of the corner points of the lock detection frame based on the actual size of the lock specifically comprises:
establishing a world coordinate system by taking the center of a side door leg of the loading lane as an original point, and respectively establishing a camera coordinate system for each positioning camera;
measuring the world coordinate of one corner point according to the overlapping relation between the lock head and the corner point of the detection frame;
and calculating the world coordinates of other detection frame angular points by combining the actual size of the lock and the actual distance of the two locks in the same image in the direction of the trolley on the basis of the world coordinates of one detection frame angular point to obtain the world coordinates of 8 detection frame angular points of the two locks in the same image.
5. The unified calibration method for the external truck positioning camera of the automatic crane according to claim 4, wherein the step of calculating the world coordinates of other detection corner points by combining the actual sizes of the lock heads and the actual distances of the two lock heads in the same image in the direction of the trolley based on the world coordinates of one detection corner point specifically comprises the following steps:
the actual size of the standard lock part is set as the height h s Thickness t s Width w s And the distance d in the thickness direction of the angular point s In the same image, the actual distance between the two lock heads in the direction of the trolley is D s ；
In a unified image collected by a certain positioning camera A, the pixel coordinates of the corner points of the two lock detection frames at the near end and the far end are as follows: { P Ani1 ,P Ani2 ,P Ani3 ,P Ani4 }、{P Afi1 ,P Afi2 ,P Afi3 ,P Afi4 And the world coordinates corresponding to each corner point represent: { P Anw1 ,P Anw2 ,P Anw3 ,P Anw4 }、{P Afw1 ,P Afw2 ,P Afw3 ,P Afw4 In which P is Anw1 ,P Anw2 ,P Anw3 ,P Anw4 The left lower corner, the right upper corner and the left upper corner of the near-end lock detection frame, P Afw1 ,P Afw2 ,P Afw3 ,P Afw4 Detecting the left lower corner, the right upper corner and the left upper corner of a frame corresponding to the remote lock head;
according to the overlapping relation of the lock and the corner point of the detection frame, measuring the world coordinate P of one corner point Anw2 ：{X Anw2 ,Y Anw2 ,Z Anw2 According to { X } Anw2 ,Y Anw2 ,Z Anw2 Calculate the world coordinates of the remaining 7 corner points:
P Anw1 ：{X Anw2 ,Y Anw2 +w s ,Z Anw2 }；
P Anw3 ：{X Anw2 +d s ,Y Anw2 ,Z Anw2 +h s }；
P Anw4 ：{X Anw2 +d s ,Y Anw2 +w s ,Z Anw2 +h s }；
P Afw1 ：{X Anw2 +D s ,Y Anw2 +w s ,Z Anw2 }；
P Afw2 ：{X Anw2 +D s ,Y Anw2 ,Z Anw2 }
P Afw3 ：{X Anw2 +d s +D s ,Y Anw2 ,Z Anw2 +h s }；
P Afw4 ：{X Anw2 +d s +D s ,Y Anw2 +w s ,Z Anw2 +h s }。
6. the unified calibration method for the external truck-mounted positioning cameras of the automatic crane according to claim 5, wherein the step of respectively calculating the external parameters of each positioning camera according to the pixel coordinates of the lock detection frame corner point, the world coordinates of the lock detection frame corner point and the internal parameters of each positioning camera comprises the steps of:
according to the camera pinhole model, establishing a corresponding relation equation between the pixel coordinates and world coordinates of each lock detection frame corner point based on internal parameters of a positioning camera;
establishing an equation set by simultaneously establishing each corresponding relation equation, and solving the equation set through a PnP algorithm in an image processing algorithm to obtain a rotation vector and a translation vector of a camera coordinate system of each positioning camera relative to a world coordinate system.
7. The unified calibration method for the external truck positioning camera of the automatic crane according to claim 6, wherein the establishing of the corresponding relation equation between the pixel coordinates and the world coordinates of each lock detection frame corner point based on the internal parameters of the positioning camera according to the camera pinhole model specifically comprises:
for any one of the positioning cameras a, the following correspondence equation exists:
Z c to detect the conversion factor between the camera and the world coordinate system, (u) Ani1 ,v Ani1 )、(u Ani2 ,v Ani2 )、(u Ani3 ,v Ani3 )、(u Ani4 ,v Ani4 ) Detect the angular point P for the lock Ani1 、P Ani2 、P Ani3 、P Ani4 Corresponding pixel coordinate, (u) Afi1 ,v Afi1 )、(u Afi2 ,v Afi2 )、(u Afi3 ,v Afi3 )、(u Afi4 ,v Afi4 ) Respectively detecting angular points P of the lock Afi1 、P Afi2 、P Afi3 、P Afi4 Corresponding pixel coordinate, K A As a result of calibration of camera internal parameters, R AWC And T AWC Respectively, a rotation vector and a translation vector of the camera coordinate system of the positioning camera a with respect to the world coordinate system.
8. An external hub positioning camera unified calibration system of an automated crane, the system comprising:
positioning a camera: the lock heads on the outer truck collecting plates are arranged in the visual field range of the corresponding positioning cameras and are used for collecting video image data of the outer truck collecting rail;
a feature extraction module: the method is used for respectively extracting the tapered end characteristics in the video image data of the outer collection truck road through a target detection method and acquiring the pixel coordinates of the tapered end detection frame corner points;
a coordinate conversion module: the system is used for calculating the world coordinates of the corner points of the lock detection frame based on the actual size of the lock;
the external parameter calibration module: and the method is used for respectively calculating the external parameters of each positioning camera according to the pixel coordinates of the lock detection frame corner point, the world coordinates of the lock detection frame corner point and the internal parameters of each positioning camera.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211600555.0A CN115880372A (en) | 2022-12-12 | 2022-12-12 | Unified calibration method and system for external hub positioning camera of automatic crane |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211600555.0A CN115880372A (en) | 2022-12-12 | 2022-12-12 | Unified calibration method and system for external hub positioning camera of automatic crane |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115880372A true CN115880372A (en) | 2023-03-31 |
Family
ID=85767327
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211600555.0A Pending CN115880372A (en) | 2022-12-12 | 2022-12-12 | Unified calibration method and system for external hub positioning camera of automatic crane |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115880372A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117249764A (en) * | 2023-11-17 | 2023-12-19 | 菲特（天津）检测技术有限公司 | Vehicle body positioning method and device and electronic equipment |
| CN117523010A (en) * | 2024-01-05 | 2024-02-06 | 深圳市欧冶半导体有限公司 | Method and device for determining camera pose of vehicle, computer equipment and storage medium |
-
2022
- 2022-12-12 CN CN202211600555.0A patent/CN115880372A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117249764A (en) * | 2023-11-17 | 2023-12-19 | 菲特（天津）检测技术有限公司 | Vehicle body positioning method and device and electronic equipment |
| CN117249764B (en) * | 2023-11-17 | 2024-02-13 | 菲特（天津）检测技术有限公司 | Vehicle body positioning method and device and electronic equipment |
| CN117523010A (en) * | 2024-01-05 | 2024-02-06 | 深圳市欧冶半导体有限公司 | Method and device for determining camera pose of vehicle, computer equipment and storage medium |
| CN117523010B (en) * | 2024-01-05 | 2024-04-09 | 深圳市欧冶半导体有限公司 | Method and device for determining camera pose of vehicle, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110689579B (en) | Rapid monocular vision pose measurement method and measurement system based on cooperative target | |
| CN115880372A (en) | Unified calibration method and system for external hub positioning camera of automatic crane | |
| CN112132523B (en) | Method, system and device for determining quantity of goods | |
| CN112101389B (en) | Method and device for measuring warehoused goods | |
| CN109455619B (en) | Container attitude positioning method and device and lifting appliance controller | |
| CN112648976B (en) | Live-action image measuring method and device, electronic equipment and storage medium | |
| CN109918977A (en) | Determine the method, device and equipment of free time parking stall | |
| CN108897246B (en) | Stack box control method, device, system and medium | |
| CN113658279B (en) | Camera internal reference and external reference estimation method, device, computer equipment and storage medium | |
| CN114219842A (en) | Visual identification, distance measurement and positioning method in port container automatic loading and unloading operation | |
| KR102010823B1 (en) | Method and apparatus for measuring speed of vehicle by using fixed single camera | |
| CN111508027A (en) | Method and device for calibrating external parameters of camera | |
| CN112254633A (en) | Object size measuring method, device and equipment | |
| CN112378333B (en) | Method and device for measuring warehoused goods | |
| CN110458951B (en) | Modeling data acquisition method and related device for power grid pole tower | |
| CN115849191A (en) | Vision-based automatic container grabbing method and system | |
| WO2024021803A1 (en) | Mark hole positioning method and apparatus, assembly device, and storage medium | |
| CN117723563A (en) | Chip pin defect detection method based on point cloud registration | |
| CN111854678B (en) | Pose measurement method based on semantic segmentation and Kalman filtering under monocular vision | |
| CN115760811A (en) | Monocular vision and feature marker based method for measuring 6D pose of workpiece | |
| CN116125489A (en) | Indoor object three-dimensional detection method, computer equipment and storage medium | |
| CN115526989A (en) | Vehicle modeling method, storage medium, device and system | |
| CN117446669A (en) | External set card holding box control method, device, computer equipment and readable storage medium | |
| CN114348879A (en) | Method for determining pose of lifting appliance of intelligent gantry crane | |
| CN113205561B (en) | Visual-based pivot pose detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |