CN116912238B - Weld joint pipeline identification method and system based on multidimensional identification network cascade fusion - Google Patents
Weld joint pipeline identification method and system based on multidimensional identification network cascade fusion Download PDFInfo
- Publication number
- CN116912238B CN116912238B CN202311159689.8A CN202311159689A CN116912238B CN 116912238 B CN116912238 B CN 116912238B CN 202311159689 A CN202311159689 A CN 202311159689A CN 116912238 B CN116912238 B CN 116912238B
- Authority
- CN
- China
- Prior art keywords
- pipeline
- network
- point cloud
- loss function
- identification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 68
- 238000000034 method Methods 0.000 title claims abstract description 67
- 230000006870 function Effects 0.000 claims abstract description 100
- 238000012549 training Methods 0.000 claims abstract description 34
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 25
- 230000011218 segmentation Effects 0.000 claims abstract description 24
- 238000005457 optimization Methods 0.000 claims abstract description 17
- 238000002372 labelling Methods 0.000 claims abstract description 12
- 238000012545 processing Methods 0.000 claims description 18
- 239000011159 matrix material Substances 0.000 claims description 14
- 239000013598 vector Substances 0.000 claims description 13
- 238000001514 detection method Methods 0.000 claims description 12
- 238000012163 sequencing technique Methods 0.000 claims description 10
- 230000001629 suppression Effects 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 238000003860 storage Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 claims description 6
- 230000003313 weakening effect Effects 0.000 claims description 4
- 238000003466 welding Methods 0.000 abstract description 16
- 238000013135 deep learning Methods 0.000 abstract description 7
- 238000007781 pre-processing Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 238000005520 cutting process Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 239000000428 dust Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/66—Analysis of geometric attributes of image moments or centre of gravity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30152—Solder
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Biophysics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Geometry (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a weld joint pipeline identification method and a system based on multi-dimensional identification network cascade fusion, which belong to the technical field of pipeline identification and comprise the following steps: labeling the welded seam pipeline data to obtain a pipeline point cloud data set; adding an improved Soft-NMS algorithm to a YOLOv5 network, constructing an improved F-Pointnet multidimensional identification fusion network, training the improved F-Pointnet multidimensional identification fusion network based on a pipeline point cloud data set, and adding an elastic network regularization optimization loss function to obtain a weld pipeline identification model; and inputting the weld joint pipeline data to be identified into a weld joint pipeline identification model to obtain a pipeline type identification result and a pipeline size identification result. The multi-dimensional recognition network cascade fusion weld joint pipeline recognition and segmentation method provided by the invention fully combines the advantages of deep learning and point cloud preprocessing, so that the method can cope with complex welding scenes, and has higher recognition precision and stronger robustness.
Description
Technical Field
The invention relates to the technical field of pipeline identification, in particular to a weld joint pipeline identification method and system based on multi-dimensional identification network cascade fusion.
Background
In the age of high-speed development of manufacturing industry, various high-efficiency welding and cutting equipment are important fields of engineering machinery enterprises, and the equipment not only can make due contribution to the production efficiency and the product quality of the enterprises, but also can become an important driving force for technological innovation of the enterprises.
A large number of links needing manual assistance exist in the engineering flow of pipeline cutting and welding. For example, a cutter requires manual indexing of cutting start points, setting of cut types, etc. parameters when cutting different types of pipes. In the pipeline welding engineering, manual teaching and manual selection of welding points also need manual intervention.
The current pipeline recognition method mainly comprises three methods of traditional image recognition, 2D deep learning and 3D visual deep learning. The conventional image recognition method mainly recognizes pipelines and welds by extracting and classifying features of images. Although the operation is simple, the method is sensitive to the influence of environmental factors such as image illumination, angles, noise and the like, so that the recognition accuracy is low. The deep learning method based on the two-dimensional image is an image recognition method based on a convolutional neural network, and the recognition of pipelines and welding seams is realized by processing the 2D image. Although the method has certain application in industry, the problems of welding light interference, industrial dust interference and the like mainly exist, and the method is difficult to adapt to complex welding production environments.
Compared with the traditional image recognition and the two-dimensional image-based deep learning method, the 3D neural network-based visual deep learning method is more focused on processing of key cloud data, and the three-dimensional point cloud is processed to realize the recognition of pipelines and welding seams, so that the method has the advantages of high precision, high efficiency, high robustness and the like, and can be suitable for complex welding environments and the influence of factors such as different light rays, angles, noise and the like. There are a number of limitations associated with 2D image processing.
Disclosure of Invention
The invention provides a weld joint pipeline identification method and system based on multi-dimensional identification network cascading fusion, which are used for solving the defects existing in the weld joint pipeline identification process in the prior art.
In a first aspect, the invention provides a weld pipe identification method based on multi-dimensional identification network cascade fusion, comprising the following steps:
acquiring weld joint pipeline data, and marking the weld joint pipeline data to obtain a pipeline point cloud data set;
adding an improved Soft-NMS algorithm to a YOLOv5 network, constructing the improved F-Pointnet multidimensional identification fusion network, training the improved F-Pointnet multidimensional identification fusion network based on a pipeline point cloud data set, and adding an elastic network regularization optimization loss function to obtain a weld pipeline identification model;
And inputting the weld joint pipeline data to be identified into the weld joint pipeline identification model to obtain a pipeline type identification result and a pipeline size identification result.
According to the weld joint pipeline identification method based on multi-dimensional identification network cascade fusion, which is provided by the invention, pipeline data are collected, and the pipeline data are marked to obtain a pipeline type point cloud data set, and the method comprises the following steps:
acquiring an RGB image and a DEPTH image of a welded joint pipeline by using a preset three-dimensional camera, and calculating the RGB image and the DEPTH image by using an internal reference matrix of the preset three-dimensional camera to generate a visual point cloud file;
and marking the visualized point cloud file by adopting preset point cloud registration software to obtain the pipeline type point cloud data set.
According to the weld joint pipeline identification method based on multi-dimensional identification network cascade fusion, the improved Soft-NMS algorithm is added to the YOLOv5 network, and the method comprises the following steps:
step 101: calculating a predicted candidate box of the pipe point cloud dataset by the modified Soft-NMS algorithm, the Soft-NMS scoring function being:
wherein,for arbitrary prediction candidate box->Score of->For the highest scoring prediction candidate box, +.>Overlap threshold value->For Gaussian penalty function, +. >Is a super parameter selected empirically;
step 102: initializing a prediction candidate frame list, and sequencing all the prediction candidate frames according to the confidence level from high to low;
step 103: taking the prediction candidate frame with highest score asPutting the final prediction candidate frame list, and keeping the prediction candidate frame with the highest score as an output detection result;
step 104: taking the rest of the prediction candidate frames except the prediction candidate frame with the highest score as the arbitrary prediction candidatesFrame selectorCalculating the arbitrary prediction candidate box +.>A value of a union ratio IoU with the prediction candidate frame with the highest score, and if the IoU value is determined to be greater than the overlap threshold, weakening the arbitrary prediction candidate frame by using the Gaussian penalty function>Is a score of (2);
step 105: sequencing the processed prediction candidate frames from high to low according to the score;
step 106: and repeating the steps 103 to 105 until all the prediction candidate frames are processed, and obtaining the preset high-score 2D image frame.
According to the weld joint pipeline identification method based on multi-dimensional identification network cascade fusion, which is provided by the invention, the improved F-Pointnet multi-dimensional identification fusion network is constructed, and the method comprises the following steps:
Projecting the preset high-score 2D image frame by taking a preset three-dimensional camera as an origin through a camera rotation matrix, and outputting a cone point cloud and a pipeline category vector;
based on a Pointet network, carrying out semantic segmentation on the cone-viewing point cloud by adopting the pipeline category vector, and outputting each point category score of the three-dimensional point cloud;
removing non-target point clouds from the cone-looking point clouds according to the point category scores to obtain target instance point clouds, calculating by the target instance point clouds to obtain point cloud centroids, and moving an origin of a cone-looking coordinate system to the point cloud centroids;
subtracting the point cloud centroid coordinates from the coordinates of the point cloud of the target instance, and outputting mask coordinate system point cloud data;
calculating the estimated pipeline real center of the point cloud data of the mask coordinate system by adopting a T-Net network, carrying out coordinate conversion on the estimated pipeline real center, and converting the estimated pipeline real center into a coordinate origin;
based on the coordinate origin, performing 3D non-modal frame estimation on the mask coordinate system point cloud data to obtain a pipeline 3D regression frame;
and processing the pipeline 3D regression frame by using a multi-layer perceptron MLP of the T-Net network, and outputting a pipeline 3D regression frame parameter information set by using a full connection layer.
According to the weld joint pipeline identification method based on the multi-dimensional identification network cascade fusion, which is provided by the invention, the improved F-Pointet multi-dimensional identification fusion network is trained based on a pipeline point cloud data set, and the method comprises the following steps:
and determining a learning rate, a learning rate momentum, a batch size, a training total round size, weight attenuation and maximum iteration times, and training the improved F-Pointnet multidimensional recognition fusion network.
According to the weld joint pipeline identification method based on multi-dimensional identification network cascade fusion, an elastic network regularization optimization loss function is added to obtain a weld joint pipeline identification model, and the method comprises the following steps:
obtaining an elastic network regularization function:
wherein,to improve the network loss function->For the original network loss function, +.>Is regularization coefficient, +.>Regularized weights for elastic network, +.>Training sample number, +.>Is a regularization parameter;
constructing an overall loss function by using the YOLOv5 network loss function, the Pointernet network loss function and the T-Net network loss function;
and forming the elastic network regularization optimization loss function by using the integral loss function and the elastic network regularization function.
According to the weld joint pipeline identification method based on multi-dimensional identification network cascade fusion, provided by the invention, an integral loss function is constructed by a YOLOv5 network loss function, a Pointnet network loss function and a T-Net network loss function, and the method comprises the following steps:
Wherein the method comprises the steps ofFor the whole loss function->Is a YOLOv5 network loss function, < ->Is weight(s)>For the T-Net network loss function, +.>For Pointernet network loss function, +.>Predicting a loss function for heading angle,>estimating a loss function for the 3D regression frame, +.>Weight value for corner loss function, +.>As an angular loss function;
correspondingly, forming the elastic network regularization optimization loss function by using the overall loss function and the elastic network regularization function, including:
optimizing a loss function for regularization of an elastic network, +.>To improve the network loss function.
In a second aspect, the present invention further provides a weld pipe identification system based on multi-dimensional identification network cascade fusion, including:
the acquisition and labeling module is used for acquiring weld joint pipeline data and labeling the weld joint pipeline data to obtain a pipeline point cloud data set;
the improved training module is used for adding an improved non-maximum suppression Soft-NMS algorithm to the YOLOv5 network, constructing the improved F-Pointnet multidimensional recognition fusion network, training the improved F-Pointnet multidimensional recognition fusion network based on a pipeline point cloud data set, and adding an elastic network regularization optimization loss function to obtain a weld pipeline recognition model;
And the recognition processing module is used for inputting the weld joint pipeline data to be recognized into the weld joint pipeline recognition model to obtain a pipeline type recognition result and a pipeline size recognition result.
In a third aspect, the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the method for identifying a welded seam pipe based on multi-dimensional identification network cascade fusion as described in any one of the above when executing the program.
In a fourth aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a weld pipe identification method based on multi-dimensional identification network cascade fusion as described in any one of the above.
According to the weld joint pipeline recognition method and system based on multi-dimensional recognition network cascade fusion, the 2D image processing network is combined with the 3D image recognition and segmentation network, so that the problems that three-dimensional recognition calculation amount is large, large-scale scenes are difficult to adapt and the like are solved by using the mature 2D image recognition network, a Soft-NMS algorithm is added into an image recognition YOLOv5 network to replace the original network to divide recognition objects into two categories, view cone point clouds containing recognition objects are generated, and the data calculation amount of three-dimensional recognition is reduced; on the other hand, the three-dimensional recognition partially solves the problems of low recognition rate and the like under the conditions of image shielding and defect caused by welding light and dust in a factory by two-dimensional recognition, has higher learning weight on the geometric structure of a recognition object, adds an elastic network regularization into an F-Pointnet multidimensional recognition fusion network, reduces model overfitting, and ensures that the generalization capability of the model is stronger. Meanwhile, the feature quantity is reduced, and the efficiency and the speed of the model are improved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow diagram of a weld pipe identification method based on multi-dimensional identification network cascade fusion provided by the invention;
FIG. 2 is a second flow chart of the method for identifying welded seam pipes based on multi-dimensional identification network cascade fusion provided by the invention;
FIG. 3 is a flow chart of a Soft-NMS algorithm provided by the present invention;
FIG. 4 is a diagram of a Pointernet network structure provided by the present invention;
FIG. 5 is a diagram of a F-Pointnet multidimensional identification fusion network provided by the invention;
FIG. 6 is a schematic structural diagram of a welded seam pipeline recognition system based on multi-dimensional recognition network cascade fusion provided by the invention;
fig. 7 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Aiming at the problems that a plurality of interference factors exist in the traditional welding seam pipeline recognition technology based on image recognition processing, so that the related recognition technology is difficult to break through, the invention aims to solve the problems existing in the visual recognition direction of an automatic welding and cutting robot, and adopts the YOLOv5 recognition network and the F-PointNet network technology, so that high robustness and high recognition precision are realized, the intelligent welding technology is pushed to a higher level, a welding solution with high efficiency, high accuracy and safety and reliability is provided, and the development process of industrial intelligent manufacturing is further promoted.
Fig. 1 is one of flow diagrams of a method for identifying a welded seam pipeline based on multi-dimensional identification network cascade fusion according to an embodiment of the present invention, as shown in fig. 1, including:
step S1: acquiring weld joint pipeline data, and marking the weld joint pipeline data to obtain a pipeline point cloud data set;
step S2: adding an improved Soft-NMS algorithm to a YOLOv5 network, constructing the improved F-Pointnet multidimensional identification fusion network, training the improved F-Pointnet multidimensional identification fusion network based on a pipeline point cloud data set, and adding an elastic network regularization optimization loss function to obtain a weld pipeline identification model;
Step S3: and inputting the weld joint pipeline data to be identified into the weld joint pipeline identification model to obtain a pipeline type identification result and a pipeline size identification result.
The overall recognition flow of the embodiment of the invention is shown in fig. 2, pipeline data are firstly collected, pipeline data are marked and a pipeline type point cloud data set is manufactured, the marked data set is input into a constructed network model for training, wherein the model training process comprises the steps of taking a pipeline image sample obtained by the data set as a frame, taking an F-Pointernet network as a frame, adopting a YOLOv5 network to replace a 2D recognition network in a 2D pipeline recognition module, completing pipeline image recognition, and scoring to generate a 2D image frame; generating a view cone point cloud only comprising the identification pipeline through a camera rotation matrix based on the obtained high-score 2D image frame; in the three-dimensional instance segmentation module, completing three-dimensional instance segmentation in the view cone point cloud by using a PointNet-based network to obtain a PointNet network 3D instance segmentation result; in a pipeline frame estimation module, performing 3D non-modal frame estimation on the pipeline point cloud obtained by segmentation, and calculating an oriented 3D boundary frame of the pipeline; further adopting an optimized loss function, improving the original loss function, adding an elastic network regularization into the original loss function, and training a model to obtain a weld joint pipeline identification model; and finally, inputting the weld joint pipeline image which is actually required to be identified into a weld joint pipeline identification model to obtain the type and the size of the pipeline in an actual scene, namely, the three-dimensional regression frame of the three-dimensional point cloud of the segmented weld joint pipeline and the size of the actual pipeline.
The multi-dimensional recognition network cascade fusion weld joint pipeline recognition and segmentation method provided by the invention fully combines the advantages of deep learning and point cloud preprocessing, so that the method can cope with complex welding scenes, and has higher recognition precision and stronger robustness.
Based on the above embodiment, collecting pipeline data, labeling the pipeline data to obtain a pipeline type point cloud data set, including:
acquiring an RGB image and a DEPTH image of a welded joint pipeline by using a preset three-dimensional camera, and calculating the RGB image and the DEPTH image by using an internal reference matrix of the preset three-dimensional camera to generate a visual point cloud file;
and marking the visualized point cloud file by adopting preset point cloud registration software to obtain the pipeline type point cloud data set.
Specifically, the embodiment of the invention adopts the Azure Kinect DK three-dimensional camera to collect the RGB image and the DEPTH image of the weld joint pipeline at the same time, and outputs an internal reference matrix and an image distortion coefficient of the camera. And simultaneously calculating the RGB image and the DEPTH image through an internal reference matrix of the camera to generate a visualized point cloud file.
And labeling pipeline data by using a cloudcompare, wherein labeling types are pipelines, establishing different types of pipeline data sets, and dividing the data sets into training sets and test sets.
Based on the above embodiments, adding a modified Soft-NMS algorithm to the YOLOv5 network, comprises:
step 102: initializing a prediction candidate frame list, and sequencing all the prediction candidate frames according to the confidence level from high to low;
step 103: taking the prediction candidate frame with highest score asPutting the final prediction candidate frame list, and keeping the prediction candidate frame with the highest score as an output detection result;
step 104: taking the rest of the prediction candidate frames except the prediction candidate frame with the highest score as the arbitrary prediction candidate frameCalculating the arbitrary prediction candidate box +.>A value of a union ratio IoU with the prediction candidate frame with the highest score, and if the IoU value is determined to be greater than the overlap threshold, weakening the arbitrary prediction candidate frame by using the Gaussian penalty function>Is a score of (2);
step 105: sequencing the processed prediction candidate frames from high to low according to the score;
step 106: and repeating the steps 103 to 105 until all the prediction candidate frames are processed, and obtaining the preset high-score 2D image frame.
Specifically, the embodiment of the invention adds a Soft-NMS algorithm to the original YOLOv5 network, wherein the Soft-NMS algorithm is a non-maximum suppression (Non Maximum Suppression, NMS) algorithm and is used for suppressing redundant frames generated by a detector in target detection. Unlike standard NMS algorithms, soft-NMS allows for retaining non-optimal bounding boxes with lower detection scores while retaining optimal detection bounding boxes to obtain more information.
The Soft-NMS uses an decay function to calculate the score for each border based on the overlap ratio and confidence. The higher the overlap ratio, the lower the confidence the greater the probability that the border will be suppressed. The function is to reduce the weight of the detected border at the highest resolution while increasing the score of the adjacent borders. Finally, the border that is retained is the border that has the highest weight after calculation.
The Soft-NMS may improve accuracy of object detection over conventional NMS algorithms, particularly when there are multiple highly overlapping object bounding boxes, as opposed to non-maximal suppression forced retention of one, the Soft-NMS may retain multiple bounding boxes rather than just select one, thus better capturing subtle differences between objects. In addition, soft-NMS can also be considered a regularization technique that avoids reducing the information in the original data by allowing low scoring areas to be preserved. The Soft-NMS algorithm flow is shown in FIG. 3.
The scoring function expression of Soft-NMS is:
wherein,for prediction frame->Score of->For the highest scoring prediction box, +.>The overlap threshold value is set to be the overlap threshold value,for Gaussian penalty function, +.>Is an empirically selected hyper-parameter.
The detection flow of the Soft-NMS obtaining the pipeline identification regression frame is as follows:
1) Initializing a detection box list, and sequencing all candidate boxes from high to low according to the confidence.
2) For the bounding box with the highest score, the bounding box is taken asPut into the final detection list D, the bounding box is reserved as the output detection result.
3) For the remaining bounding boxes, they are assigned to the same identityThe IoU values of their bounding boxes with highest scores are calculated. If the IoU value is greater than the threshold, the score for the bounding box is decremented.
4) And (5) sequencing the processed bounding boxes from high score to low score.
5) Repeating the steps 2-4 until all the bounding boxes are processed.
Through the steps, a plurality of article marking frames with different scores can be obtained, the possible area frames of the articles with extremely low scores are screened out through a threshold value, other marking frames are all generated into view cone point clouds, and then three-dimensional article identification is carried out, so that the problem of low YOLOv5 network identification rate caused by pipeline stacking and shielding can be effectively reduced.
Based on the above embodiment, the improved F-Pointernet multidimensional recognition fusion network is constructed, which comprises:
projecting the preset high-score 2D image frame by taking a preset three-dimensional camera as an origin through a camera rotation matrix, and outputting a cone point cloud and a pipeline category vector;
Based on a Pointet network, carrying out semantic segmentation on the cone-viewing point cloud by adopting the pipeline category vector, and outputting each point category score of the three-dimensional point cloud;
removing non-target point clouds from the cone-looking point clouds according to the point category scores to obtain target instance point clouds, calculating by the target instance point clouds to obtain point cloud centroids, and moving an origin of a cone-looking coordinate system to the point cloud centroids;
subtracting the point cloud centroid coordinates from the coordinates of the point cloud of the target instance, and outputting mask coordinate system point cloud data;
calculating the estimated pipeline real center of the point cloud data of the mask coordinate system by adopting a T-Net network, carrying out coordinate conversion on the estimated pipeline real center, and converting the estimated pipeline real center into a coordinate origin;
based on the coordinate origin, performing 3D non-modal frame estimation on the mask coordinate system point cloud data to obtain a pipeline 3D regression frame;
and processing the pipeline 3D regression frame by using a multi-layer perceptron MLP of the T-Net network, and outputting a pipeline 3D regression frame parameter information set by using a full connection layer.
Specifically, the embodiment of the invention generates the cone point cloud only comprising the identification pipeline through the camera rotation matrix based on the obtained high-score 2D image frame.
And the obtained 2D image frame area is projected into a view cone by taking the camera as an origin through a camera internal reference rotation matrix in a projection mode. The pixels in the image are distributed in the view cone according to different distances from the camera to form a view cone point cloud. The view cone point clouds projected from the 2D image frame region and the camera origin have different directions in the camera coordinate system. In order to facilitate data processing, the coordinate system of the view cone point cloud data needs to be converted from the camera coordinate system to a unified view cone coordinate system. The center line of the view cone point cloud is rotated to a position orthogonal to the image plane, then the view cone tip is positioned at the origin of a coordinate system, the center line of the view cone is an X axis, a Y axis is parallel to the Y axis of the camera, and a Z axis is perpendicular to the X axis and the Y axis.
In the three-dimensional instance segmentation module, three-dimensional instance segmentation is completed in the view cone point cloud by using a Pointnet-based network, and the Pointnet network structure is as shown in FIG. 4 and comprises:
the set of all point cloud data input as one frame is expressed as a 2d tensor of n×3, where n represents the number of point clouds and 3 corresponds to xyz coordinates.
Input data is aligned by multiplying the input data with a T-Net learned conversion matrix, so that invariance of the model to specific space conversion is ensured.
After feature extraction is performed on the cloud data of each point for a plurality of times mlp, the features are aligned by using a T-Net.
A maxpooling operation is performed on each dimension of the feature to arrive at a final global feature.
Predicting the final classification score of the global feature through mlp for the classification task; and (3) carrying out series connection on the global features and the local features of each point cloud learned before on the segmentation task, and obtaining a classification result of each data point through mlp.
The three-dimensional instance segmentation module receives the view cone point cloud data extracted by the previous module, performs semantic segmentation on the view cone point cloud data by using a Pointet network according to the pipeline category vector generated by the 2D pipeline recognition module, and outputs each point category score of the three-dimensional point cloud. The output score is a binary score for detecting the target point cloud and other non-target point clouds (background point clouds or other clutter point clouds). In the module, non-target point clouds are removed from the view cone point cloud data in combination with scores of semantic segmentation of the Pointernet network, and point clouds of target instances are extracted.
And after the target point cloud is obtained by segmentation from the view cone point cloud, calculating to obtain the point cloud centroid. And moving the origin of the view cone coordinate system to the point cloud centroid, and subtracting the centroid point coordinates from all the point coordinates of the target point cloud to form the point cloud data in the mask coordinate system. Converting the cloud coordinates of the target point from the view cone coordinate system to the mask coordinate system may make rotation and translation of the target in the model space insignificant. In this way, the point cloud data may be represented using a fixed coordinate system relative to the target object, regardless of the actual position and orientation of the target object in space. This makes data processing and modeling simpler and can improve the accuracy and robustness of the neural network model. In addition, the centroid of the target point cloud is used as an origin, so that calculation is further simplified, difficulty in model learning can be reduced, and training speed and performance of the model are improved.
And in the pipeline frame estimation module, 3D non-modal frame estimation is carried out on the pipeline point cloud obtained by segmentation, and an oriented 3D boundary frame of the pipeline is calculated.
Wherein the pipeline bounding box estimation module predicts a 3D bounding box of the target in the 3D point cloud based on the target point cloud data in the mask coordinate system. The target centroid in the mask coordinate system obtained by the three-dimensional instance segmentation module is not the centroid of the real conduit. This is because, when a three-dimensional camera photographs a pipeline, the obtained point cloud is only a part of pipeline point cloud data facing the camera direction. Thus, a lightweight T-Net network is employed to estimate the true center of a complete object. And after obtaining the estimated real center of the pipeline, performing coordinate transformation again to enable the estimated pipeline center point to be an origin, and then processing by the modeless 3D boundary box evaluation module. After passing through the T-Net like MLP processing, the fully connected layer ultimately outputs all parameter information evaluated by the module, including centroid coordinates, length, width and height of bounding box, residual and heading angle, etc.
Based on the above embodiment, training the improved F-pointe multidimensional identification fusion network based on the pipeline point cloud data set includes:
And determining a learning rate, a learning rate momentum, a batch size, a training total round size, weight attenuation and maximum iteration times, and training the improved F-Pointnet multidimensional recognition fusion network.
According to the embodiment of the invention, the improved network is trained by using the training set obtained through shooting; the learning rate, the momentum of the learning rate, the batch size, the total training rounds, the weight attenuation and the maximum iteration number are set as training parameters, and the improved F-Pointnet multidimensional identification fusion network is trained.
Based on the above embodiment, adding an elastic network regularization optimization loss function to obtain a weld pipe identification model includes:
acquiring an elastic network regularization function;
constructing an overall loss function by using the YOLOv5 network loss function, the Pointernet network loss function and the T-Net network loss function;
and forming the elastic network regularization optimization loss function by using the integral loss function and the elastic network regularization function.
Specifically, according to the embodiment of the invention, the elastic network regularization is added in the loss function of the original network, the three-dimensional data model is usually too complex, the feature quantity is more, and the fitting phenomenon is easy to occur. The elastic network regularization suppresses the overfitting phenomenon of the model by introducing additional constraint into the loss function, thereby improving the generalization capability of the model.
In addition, the embodiment of the invention compares the regularization of the elastic network with the L1 (or Lasso) regularization and the ridge regression, and the L1 regularization defines the regularization term as the sum of absolute values of all weights in the weight vector, so that the method is a thinning method, and certain weights can be reduced or even zeroed, thereby realizing feature selection. It is generally applicable to datasets with a large number of redundant features and may reduce the risk of overfitting. However, it introduces non-minutiae in the model, making the model more difficult to optimize. When there is redundancy in the input features, elastic network regularization can use the feature selection capability of L1 regularization to reduce feature dimensions while avoiding the loss of fine features. The reason for this is better than L1 regularization is that L1 regularization can only find more discriminative features for a given case, while the L1 term of elastic network regularization can correct indiscriminate features by the L2 term. Whereas ridge regression defines the regularization term as the sum of squares of the weight vectors, which is a way to smooth the weights, preventing the weights from becoming too large and avoiding overfitting. It is generally applicable to situations where the correlation between features in a dataset is strong. When there is redundancy in the input features, elastic network regularization can use the feature selection capability of L1 regularization to reduce feature dimensions while avoiding the loss of fine features. The reason for this is better than L1 regularization is that L1 regularization can only find more discriminative features for a given case, while the L1 term of elastic network regularization can correct indiscriminate features by the L2 term. Whereas ridge regression defines the regularization term as the sum of squares of the weight vectors, which is a way to smooth the weights, can prevent the weights from becoming too large, avoid overfitting, and is generally applicable to situations where the correlation between features in the dataset is strong.
Elastic network regularization can better control the complexity and parameters of the model when there is co-linearity of the input features (i.e., high correlation between features). L1 regularization (or Lasso) tends to select the feature with the highest discrimination capability and set its coefficient to 0. This is good in data with less co-linearity, whereas in data with more co-linearity it tends to select feature sets with errors and deviations, and over-fitting may occur in the case of all possible structures. In contrast, L2 regularization (or ridge regression) tends to use very small feature coefficients, which can lead to lack of stability and difficulty in using on the problem of stronger classification. This is why elastic network regularization is proposed, which can control the characteristic coefficients within a certain range while guaranteeing a good predictive performance lower limit of the model.
Therefore, the embodiment of the invention uses the elastic network regularization, so that the network can realize variable selection and control of model complexity when processing a high-dimensional data set, and the network is more flexible and robust.
Elastic network regularization is defined as:
wherein the method comprises the steps ofTo improve the network loss function- >For the original network loss function, +.>Is regularization coefficient, +.>Regularized weights for elastic network, +.>Training sample number, +.>Is a regularization parameter.
Meanwhile, three related networks, namely a YOLOv5 network, a Pointernet network and a T-Net network, are optimized, and have multitasking loss, and the loss functions of the whole three networks are as follows:
wherein the method comprises the steps ofFor the whole loss function->Is a YOLOv5 network loss function, < ->Is weight(s)>Is a T-Net network loss function, +.>For Pointernet network loss function, +.>Is a heading angle prediction loss function,/->Is a 3D regression frame estimation loss function, +.>Is the angle loss function weight value, +.>Is the angular loss function.
Loss function of final networkThe method comprises the following steps:
it should be noted that, the F-Pointernet multidimensional identification fusion network structure provided in the embodiment of the present invention is shown in fig. 5, and is specifically as follows:
and the 2D region scoring module part takes an RGB image as input of a YOLOv5 network, acquires a scoring frame of the 2D region through a Soft-NMS algorithm, screens out possible object region frames with extremely low scores through a threshold value, and combines other scoring frames with the DEPTH image according to a camera rotation matrix to generate a view cone point cloud. At this time, the view cone point cloud includes N points, and the feature dimension is C.
The 3D instance segmentation module part comprises a 3D instance segmentation network and mask region coordinate conversion, and a target object point cloud is obtained, wherein M is the number of object points, and C is the feature vector dimension.
And the modeless 3D frame estimation part comprises a real center prediction network T-Net and modeless frame estimation.
And finally, regularizing and optimizing a loss function through an elastic network, and performing model training to obtain the identified 3D frame of the pipeline.
The invention combines two-dimensional recognition with three-dimensional segmentation, solves the problems that the accuracy of two-dimensional image recognition on shielding pipeline recognition is not high, the three-dimensional recognition calculation amount is optimized, and the speed is low, improves the precision of pipeline recognition segmentation, and is suitable for industrial production.
The welded seam pipeline recognition system based on the multi-dimensional recognition network cascade fusion provided by the invention is described below, and the welded seam pipeline recognition system based on the multi-dimensional recognition network cascade fusion described below and the welded seam pipeline recognition method based on the multi-dimensional recognition network cascade fusion described above can be correspondingly referred to each other.
Fig. 6 is a schematic structural diagram of a welded seam pipeline recognition system based on multi-dimensional recognition network cascade fusion, which is provided by an embodiment of the present invention, and as shown in fig. 6, includes: the system comprises an acquisition labeling module 61, a training improvement module 62 and an identification processing module 63, wherein:
The acquisition and labeling module 61 is used for acquiring weld joint pipeline data and labeling the weld joint pipeline data to obtain a pipeline point cloud data set; the improved training module 62 is configured to add an improved non-maximum suppression Soft-NMS algorithm to the YOLOv5 network, construct the improved F-Pointnet multidimensional identification fusion network, train the improved F-Pointnet multidimensional identification fusion network based on a pipeline point cloud data set, and add an elastic network regularization optimization loss function to obtain a weld pipeline identification model; the recognition processing module 63 is configured to input the weld pipe data to be recognized into the weld pipe recognition model, and obtain a pipe type recognition result and a pipe size recognition result.
Fig. 7 illustrates a physical schematic diagram of an electronic device, as shown in fig. 7, which may include: processor 710, communication interface (Communications Interface) 720, memory 730, and communication bus 740, wherein processor 710, communication interface 720, memory 730 communicate with each other via communication bus 740. Processor 710 may invoke logic instructions in memory 730 to perform a weld pipe identification method based on multi-dimensional identification network cascade fusion, the method comprising: acquiring weld joint pipeline data, and marking the weld joint pipeline data to obtain a pipeline point cloud data set; adding an improved Soft-NMS algorithm to a YOLOv5 network, constructing the improved F-Pointnet multidimensional identification fusion network, training the improved F-Pointnet multidimensional identification fusion network based on a pipeline point cloud data set, and adding an elastic network regularization optimization loss function to obtain a weld pipeline identification model; and inputting the weld joint pipeline data to be identified into the weld joint pipeline identification model to obtain a pipeline type identification result and a pipeline size identification result.
Further, the logic instructions in the memory 730 described above may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a stand alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the method for identifying a welded pipe based on multi-dimensional identification network cascade fusion provided by the above methods, the method comprising: acquiring weld joint pipeline data, and marking the weld joint pipeline data to obtain a pipeline point cloud data set; adding an improved Soft-NMS algorithm to a YOLOv5 network, constructing the improved F-Pointnet multidimensional identification fusion network, training the improved F-Pointnet multidimensional identification fusion network based on a pipeline point cloud data set, and adding an elastic network regularization optimization loss function to obtain a weld pipeline identification model; and inputting the weld joint pipeline data to be identified into the weld joint pipeline identification model to obtain a pipeline type identification result and a pipeline size identification result.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. A weld joint pipeline identification method based on multi-dimensional identification network cascade fusion is characterized by comprising the following steps:
acquiring weld joint pipeline data, and marking the weld joint pipeline data to obtain a pipeline point cloud data set;
adding an improved non-maximum suppression Soft-NMS algorithm to a YOLOv5 network, constructing an improved F-Pointnet multidimensional recognition fusion network, training the improved F-Pointnet multidimensional recognition fusion network based on a pipeline point cloud data set, and adding an elastic network regularization optimization loss function to obtain a weld pipeline recognition model;
inputting weld joint pipeline data to be identified into the weld joint pipeline identification model to obtain a pipeline type identification result and a pipeline size identification result;
Wherein adding the modified non-maximum suppression Soft-NMS algorithm to the YOLOv5 network comprises:
step 101: calculating a predicted candidate box of the pipe point cloud dataset by the modified Soft-NMS algorithm, the Soft-NMS scoring function being:
wherein,for arbitrary prediction candidate box->Score of->For the highest scoring prediction candidate box, +.>The overlap threshold value is set to be the overlap threshold value,for Gaussian penalty function, +.>Is a super parameter selected empirically;
step 102: initializing a prediction candidate frame list, and sequencing all the prediction candidate frames according to the confidence level from high to low;
step 103: taking the prediction candidate frame with highest score asPutting the final prediction candidate frame list, and keeping the prediction candidate frame with the highest score as an output detection result;
step 104: taking the rest of the prediction candidate frames except the prediction candidate frame with the highest score as the arbitrary prediction candidate frameCalculating the arbitrary prediction candidate box +.>A value of a union ratio IoU with the prediction candidate frame with the highest score, and if the IoU value is determined to be greater than the overlap threshold, weakening the arbitrary prediction candidate frame by using the Gaussian penalty function>Is a score of (2);
step 105: sequencing the processed prediction candidate frames from high to low according to the score;
Step 106: repeating the steps 103 to 105 until all the prediction candidate frames are processed to obtain a preset high-score 2D image frame;
the method for constructing the improved F-Pointernet multidimensional recognition fusion network comprises the following steps:
projecting the preset high-score 2D image frame by taking a preset three-dimensional camera as an origin through a camera rotation matrix, and outputting a cone point cloud and a pipeline category vector;
based on a Pointet network, carrying out semantic segmentation on the cone-viewing point cloud by adopting the pipeline category vector, and outputting each point category score of the three-dimensional point cloud;
removing non-target point clouds from the cone-looking point clouds according to the point category scores to obtain target instance point clouds, calculating by the target instance point clouds to obtain point cloud centroids, and moving an origin of a cone-looking coordinate system to the point cloud centroids;
subtracting the point cloud centroid coordinates from the coordinates of the point cloud of the target instance, and outputting mask coordinate system point cloud data;
calculating the estimated pipeline real center of the point cloud data of the mask coordinate system by adopting a T-Net network, carrying out coordinate conversion on the estimated pipeline real center, and converting the estimated pipeline real center into a coordinate origin;
based on the coordinate origin, performing 3D non-modal frame estimation on the mask coordinate system point cloud data to obtain a pipeline 3D regression frame;
And processing the pipeline 3D regression frame by using a multi-layer perceptron MLP of the T-Net network, and outputting a pipeline 3D regression frame parameter information set by using a full connection layer.
2. The method for identifying the welded seam pipeline based on multi-dimensional identification network cascading fusion according to claim 1, wherein the steps of collecting pipeline data, labeling the pipeline data to obtain a pipeline type point cloud data set, and comprising the following steps:
acquiring an RGB image and a DEPTH image of a welded joint pipeline by using a preset three-dimensional camera, and calculating the RGB image and the DEPTH image by using an internal reference matrix of the preset three-dimensional camera to generate a visual point cloud file;
and marking the visualized point cloud file by adopting preset point cloud registration software to obtain the pipeline type point cloud data set.
3. The weld pipe identification method based on multi-dimensional identification network cascade fusion according to claim 1, wherein training the improved F-pointe multi-dimensional identification fusion network based on a pipe point cloud data set comprises:
and determining a learning rate, a learning rate momentum, a batch size, a training total round size, weight attenuation and maximum iteration times, and training the improved F-Pointnet multidimensional recognition fusion network.
4. The method for identifying the welded joint pipeline based on multi-dimensional identification network cascade fusion according to claim 1, wherein the method for obtaining the welded joint pipeline identification model by adding the regularized optimization loss function of the elastic network comprises the following steps:
obtaining an elastic network regularization function:
wherein,to improve the network loss function->For the original network loss function, +.>Is regularization coefficient, +.>Regularized weights for elastic network, +.>Training sample number, +.>Is a regularization parameter;
constructing an overall loss function by using the YOLOv5 network loss function, the Pointernet network loss function and the T-Net network loss function;
and forming the elastic network regularization optimization loss function by using the integral loss function and the elastic network regularization function.
5. The weld pipe identification method based on multi-dimensional identification network cascade fusion according to claim 4, wherein constructing an overall loss function from the YOLOv5 network loss function, the pointe network loss function, and the T-Net network loss function comprises:
wherein the method comprises the steps ofFor the whole loss function->Is a YOLOv5 network loss function, < ->Is weight(s)>For the T-Net network loss function, +.>For Pointernet network loss function, +. >For the heading angle prediction loss function,estimating a loss function for the 3D regression frame, +.>Weight value for corner loss function, +.>As an angular loss function;
correspondingly, forming the elastic network regularization optimization loss function by using the overall loss function and the elastic network regularization function, including:
optimizing a loss function for regularization of an elastic network, +.>To improve the network loss function.
6. A welded seam pipe identification system based on multi-dimensional identification network cascade fusion, comprising:
the acquisition and labeling module is used for acquiring weld joint pipeline data and labeling the weld joint pipeline data to obtain a pipeline point cloud data set;
the improved training module is used for adding an improved non-maximum suppression Soft-NMS algorithm to the YOLOv5 network, constructing the improved F-Pointnet multidimensional recognition fusion network, training the improved F-Pointnet multidimensional recognition fusion network based on a pipeline point cloud data set, and adding an elastic network regularization optimization loss function to obtain a weld pipeline recognition model;
the recognition processing module is used for inputting the weld joint pipeline data to be recognized into the weld joint pipeline recognition model to obtain a pipeline type recognition result and a pipeline size recognition result;
Wherein adding an improved non-maximum suppression Soft-NMS algorithm to the YOLOv5 network in the improved training module comprises:
step 101: calculating a predicted candidate box of the pipe point cloud dataset by the modified Soft-NMS algorithm, the Soft-NMS scoring function being:
wherein,for arbitrary prediction candidate box->Score of->For the highest scoring prediction candidate box, +.>The overlap threshold value is set to be the overlap threshold value,for Gaussian penalty function, +.>Is a super parameter selected empirically;
step 102: initializing a prediction candidate frame list, and sequencing all the prediction candidate frames according to the confidence level from high to low;
step 103: taking the prediction candidate frame with highest score asPutting the final prediction candidate frame list, and keeping the prediction candidate frame with the highest score as an output detection result;
step 104: taking the rest of the prediction candidate frames except the prediction candidate frame with the highest score as the arbitrary prediction candidate frameCalculating the arbitrary prediction candidate box +.>A value of a union ratio IoU with the prediction candidate frame with the highest score, and if the IoU value is determined to be greater than the overlap threshold, weakening the arbitrary prediction candidate frame by using the Gaussian penalty function>Is a score of (2);
step 105: sequencing the processed prediction candidate frames from high to low according to the score;
Step 106: repeating the steps 103 to 105 until all the prediction candidate frames are processed to obtain a preset high-score 2D image frame;
the improved F-Pointernet multidimensional identification fusion network constructed in the improved training module comprises the following components:
projecting the preset high-score 2D image frame by taking a preset three-dimensional camera as an origin through a camera rotation matrix, and outputting a cone point cloud and a pipeline category vector;
based on a Pointet network, carrying out semantic segmentation on the cone-viewing point cloud by adopting the pipeline category vector, and outputting each point category score of the three-dimensional point cloud;
removing non-target point clouds from the cone-looking point clouds according to the point category scores to obtain target instance point clouds, calculating by the target instance point clouds to obtain point cloud centroids, and moving an origin of a cone-looking coordinate system to the point cloud centroids;
subtracting the point cloud centroid coordinates from the coordinates of the point cloud of the target instance, and outputting mask coordinate system point cloud data;
calculating the estimated pipeline real center of the point cloud data of the mask coordinate system by adopting a T-Net network, carrying out coordinate conversion on the estimated pipeline real center, and converting the estimated pipeline real center into a coordinate origin;
Based on the coordinate origin, performing 3D non-modal frame estimation on the mask coordinate system point cloud data to obtain a pipeline 3D regression frame;
and processing the pipeline 3D regression frame by using a multi-layer perceptron MLP of the T-Net network, and outputting a pipeline 3D regression frame parameter information set by using a full connection layer.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the multi-dimensional recognition network cascade fusion-based weld pipe recognition method of any one of claims 1 to 5 when the program is executed.
8. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the multi-dimensional recognition network cascade fusion-based weld pipe recognition method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311159689.8A CN116912238B (en) | 2023-09-11 | 2023-09-11 | Weld joint pipeline identification method and system based on multidimensional identification network cascade fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311159689.8A CN116912238B (en) | 2023-09-11 | 2023-09-11 | Weld joint pipeline identification method and system based on multidimensional identification network cascade fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116912238A CN116912238A (en) | 2023-10-20 |
| CN116912238B true CN116912238B (en) | 2023-11-28 |
Family
ID=88351377
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311159689.8A Active CN116912238B (en) | 2023-09-11 | 2023-09-11 | Weld joint pipeline identification method and system based on multidimensional identification network cascade fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116912238B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117557553B (en) * | 2023-12-28 | 2024-05-24 | 中核武汉核电运行技术股份有限公司 | Nuclear power station pipeline weld joint detection method and device |
| CN118096729B (en) * | 2024-04-19 | 2024-09-03 | 南京理工大学 | Weld joint identification method based on two-dimensional-depth information fusion |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110264457A (en) * | 2019-06-20 | 2019-09-20 | 浙江大学 | Weld seam autonomous classification method based on rotary area candidate network |
| CA3042819A1 (en) * | 2018-05-09 | 2019-11-09 | Postureco, Inc. | Method and system for postural analysis and measuring anatomical dimensions from a digital image using machine learning |
| CN111145174A (en) * | 2020-01-02 | 2020-05-12 | 南京邮电大学 | 3D target detection method for point cloud screening based on image semantic features |
| AU2020101011A4 (en) * | 2019-06-26 | 2020-07-23 | Zhejiang University | Method for identifying concrete cracks based on yolov3 deep learning model |
| WO2020151109A1 (en) * | 2019-01-22 | 2020-07-30 | 中国科学院自动化研究所 | Three-dimensional target detection method and system based on point cloud weighted channel feature |
| WO2020181685A1 (en) * | 2019-03-12 | 2020-09-17 | 南京邮电大学 | Vehicle-mounted video target detection method based on deep learning |
| WO2020199834A1 (en) * | 2019-04-03 | 2020-10-08 | 腾讯科技（深圳）有限公司 | Object detection method and apparatus, and network device and storage medium |
| CN111860666A (en) * | 2020-07-27 | 2020-10-30 | 湖南工程学院 | 3D target detection method based on point cloud and image self-attention mechanism fusion |
| CN113052109A (en) * | 2021-04-01 | 2021-06-29 | 西安建筑科技大学 | 3D target detection system and 3D target detection method thereof |
| CN113255444A (en) * | 2021-04-19 | 2021-08-13 | 杭州飞步科技有限公司 | Training method of image recognition model, image recognition method and device |
| AU2021107375A4 (en) * | 2021-08-25 | 2021-12-16 | Total Drain Group Pty Ltd | Methods and systems for identifying objects in images |
| US11222217B1 (en) * | 2020-08-14 | 2022-01-11 | Tsinghua University | Detection method using fusion network based on attention mechanism, and terminal device |
| CN113989797A (en) * | 2021-10-26 | 2022-01-28 | 清华大学苏州汽车研究院（相城） | Three-dimensional dynamic target detection method and device based on voxel point cloud fusion |
| CN113989758A (en) * | 2021-10-26 | 2022-01-28 | 清华大学苏州汽车研究院（相城） | Anchor guide 3D target detection method and device for automatic driving |
| CN114565644A (en) * | 2022-03-02 | 2022-05-31 | 湖南中科助英智能科技研究院有限公司 | Three-dimensional moving object detection method, device and equipment |
| CN116402801A (en) * | 2023-04-12 | 2023-07-07 | 西南石油大学 | Pipeline weld defect identification method based on improved dense connection network |
-
2023
- 2023-09-11 CN CN202311159689.8A patent/CN116912238B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA3042819A1 (en) * | 2018-05-09 | 2019-11-09 | Postureco, Inc. | Method and system for postural analysis and measuring anatomical dimensions from a digital image using machine learning |
| WO2020151109A1 (en) * | 2019-01-22 | 2020-07-30 | 中国科学院自动化研究所 | Three-dimensional target detection method and system based on point cloud weighted channel feature |
| WO2020181685A1 (en) * | 2019-03-12 | 2020-09-17 | 南京邮电大学 | Vehicle-mounted video target detection method based on deep learning |
| WO2020199834A1 (en) * | 2019-04-03 | 2020-10-08 | 腾讯科技（深圳）有限公司 | Object detection method and apparatus, and network device and storage medium |
| CN110264457A (en) * | 2019-06-20 | 2019-09-20 | 浙江大学 | Weld seam autonomous classification method based on rotary area candidate network |
| AU2020101011A4 (en) * | 2019-06-26 | 2020-07-23 | Zhejiang University | Method for identifying concrete cracks based on yolov3 deep learning model |
| CN111145174A (en) * | 2020-01-02 | 2020-05-12 | 南京邮电大学 | 3D target detection method for point cloud screening based on image semantic features |
| CN111860666A (en) * | 2020-07-27 | 2020-10-30 | 湖南工程学院 | 3D target detection method based on point cloud and image self-attention mechanism fusion |
| US11222217B1 (en) * | 2020-08-14 | 2022-01-11 | Tsinghua University | Detection method using fusion network based on attention mechanism, and terminal device |
| CN113052109A (en) * | 2021-04-01 | 2021-06-29 | 西安建筑科技大学 | 3D target detection system and 3D target detection method thereof |
| CN113255444A (en) * | 2021-04-19 | 2021-08-13 | 杭州飞步科技有限公司 | Training method of image recognition model, image recognition method and device |
| AU2021107375A4 (en) * | 2021-08-25 | 2021-12-16 | Total Drain Group Pty Ltd | Methods and systems for identifying objects in images |
| CN113989797A (en) * | 2021-10-26 | 2022-01-28 | 清华大学苏州汽车研究院（相城） | Three-dimensional dynamic target detection method and device based on voxel point cloud fusion |
| CN113989758A (en) * | 2021-10-26 | 2022-01-28 | 清华大学苏州汽车研究院（相城） | Anchor guide 3D target detection method and device for automatic driving |
| CN114565644A (en) * | 2022-03-02 | 2022-05-31 | 湖南中科助英智能科技研究院有限公司 | Three-dimensional moving object detection method, device and equipment |
| CN116402801A (en) * | 2023-04-12 | 2023-07-07 | 西南石油大学 | Pipeline weld defect identification method based on improved dense connection network |
Non-Patent Citations (3)
| Title |
|---|
| Deep Learning Inspired Object Consolidation Approaches Using LiDAR Data for Autonomous Driving: A Review;M. S. Mekala et al;《Archives of Computational Methods in Engineering 》;全文 * |
| 基于卷积神经网络的三维目标检测研究综述;王亚东等;《模式识别与人工智能》;第34卷(第12期);全文 * |
| 基于改进YOLOv5的绝缘子缺陷检测算法;唐靓等;《华中师范大学学报(自然科学版)》;第56卷(第05期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116912238A (en) | 2023-10-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109740665B (en) | Method and system for detecting ship target with occluded image based on expert knowledge constraint | |
| CN116912238B (en) | Weld joint pipeline identification method and system based on multidimensional identification network cascade fusion | |
| CN109800689B (en) | Target tracking method based on space-time feature fusion learning | |
| CN113537208A (en) | Visual positioning method and system based on semantic ORB-SLAM technology | |
| CN111310631B (en) | Target tracking method and system for rotor operation flying robot | |
| CN112598713A (en) | Offshore submarine fish detection and tracking statistical method based on deep learning | |
| CN113312973B (en) | Gesture recognition key point feature extraction method and system | |
| Zhao et al. | Palletizing robot positioning bolt detection based on improved YOLO-V3 | |
| CN111738206A (en) | Excavator detection method for unmanned aerial vehicle inspection based on CenterNet | |
| CN114972968A (en) | Tray identification and pose estimation method based on multiple neural networks | |
| CN116229189B (en) | Image processing method, device, equipment and storage medium based on fluorescence endoscope | |
| CN117576079A (en) | Industrial product surface abnormality detection method, device and system | |
| CN114511575B (en) | Grabbing method for high-reflectivity object based on image segmentation positioning auxiliary point cloud registration | |
| CN114049318A (en) | Multi-mode fusion feature-based grabbing pose detection method | |
| CN117474979A (en) | Robot 6D grabbing gesture estimation method and system based on RGB image | |
| CN117351078A (en) | Target size and 6D gesture estimation method based on shape priori | |
| Zhu et al. | Unsupervised domain adaptive object detection for assembly quality inspection | |
| Mao et al. | Disparity filtering with 3D convolutional neural networks | |
| Zhang et al. | Dynamic Semantics SLAM Based on Improved Mask R-CNN | |
| Wu et al. | Express parcel detection based on improved faster regions with CNN features | |
| Vishnyakov et al. | Real-time semantic slam with dcnn-based feature point detection, matching and dense point cloud aggregation | |
| Lu et al. | Slicing-tracking-detection: Simultaneous multi-cylinder detection from large-scale and complex point clouds | |
| Zhang et al. | Weld Seam Detection Method with Rotational Region Proposal Network | |
| Varadarajan et al. | Attention driven grasping for clearing a heap of objects | |
| Yang et al. | Integrating Deep Learning Models and Depth Cameras to Achieve Digital Transformation: A Case Study in Shoe Company |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |