CN113421267A - Point cloud semantic and instance joint segmentation method and system based on improved PointConv - Google Patents
Point cloud semantic and instance joint segmentation method and system based on improved PointConv Download PDFInfo
- Publication number
- CN113421267A CN113421267A CN202110495434.3A CN202110495434A CN113421267A CN 113421267 A CN113421267 A CN 113421267A CN 202110495434 A CN202110495434 A CN 202110495434A CN 113421267 A CN113421267 A CN 113421267A
- Authority
- CN
- China
- Prior art keywords
- semantic
- segmentation
- instance
- feature
- pointconv
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000011218 segmentation Effects 0.000 title claims abstract description 84
- 238000000034 method Methods 0.000 title claims abstract description 22
- 238000000605 extraction Methods 0.000 claims abstract description 19
- 230000004927 fusion Effects 0.000 claims abstract description 7
- 239000011159 matrix material Substances 0.000 claims description 33
- 238000007906 compression Methods 0.000 claims description 11
- 230000006835 compression Effects 0.000 claims description 10
- 230000002776 aggregation Effects 0.000 claims description 9
- 238000004220 aggregation Methods 0.000 claims description 9
- 238000005457 optimization Methods 0.000 claims description 7
- 230000003139 buffering effect Effects 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 6
- 238000006116 polymerization reaction Methods 0.000 claims description 5
- 230000017105 transposition Effects 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 3
- 239000000203 mixture Substances 0.000 claims 1
- 238000003860 storage Methods 0.000 description 9
- 238000004590 computer program Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000009826 distribution Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001846 repelling effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a point cloud semantic and instance combined segmentation method and system based on improved PointConv, wherein point cloud obtained by a laser radar or a depth camera is used as input of an improved PointConv feature extraction module, point passing through a shared coding module is subjected to semantic segmentation decoding and instance segmentation decoding simultaneously to obtain instance feature prediction and semantic feature prediction, a double-line self-attention module performs feature fusion on the semantic feature prediction and the instance feature prediction obtained by the improved PointConv feature extraction module, and instance segmentation and semantic segmentation are performed respectively to obtain instance information containing semantic features and semantic information containing the instance features. The method and the device improve the speed of example segmentation and reduce the dependency on semantic segmentation precision.
Description
Technical Field
The invention belongs to the technical field of point cloud segmentation, and particularly relates to a point cloud semantic and instance joint segmentation method and system based on improved PointConv.
Background
The neural network has strong feature learning capability in the image feature extraction process, so that the image semantics and the example segmentation task in the computer vision field are made a significant breakthrough. After the PointNet algorithm, an end-to-end point cloud segmentation algorithm is developed rapidly, but the following defects still exist: firstly, when KNN or Radius NN search is carried out, due to the fact that the sequence of search points is different, the point cloud is disordered, most methods use mlp and maxporoling to extract features, and the extracted point features cannot acquire the interaction between the local geometric shape of the point cloud and the points; secondly, the point cloud algorithm adopts farthest point sampling in the first step, which belongs to non-uniform sampling, so that a large amount of local area points are gathered, partial area points disappear, and the feature learning ability is not strong; most networks combine semantic segmentation and instance segmentation tasks in a serial mode, and the problems of suboptimal performance, low efficiency and over-strong dependency of the semantic segmentation and the instance segmentation tasks exist.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a point cloud semantic and instance combined segmentation method and system based on improved PointConv, which can improve the speed of instance segmentation and reduce the dependency on semantic segmentation precision.
The present invention achieves the above-described object by the following technical means.
A point cloud semantic and instance joint segmentation method based on improved PointConv comprises the following steps:
inputting the obtained point cloud into an improved PointConv feature extraction module, obtaining points with a feature dimension of 512 through a shared coding module, and obtaining an example feature prediction F through semantic segmentation decoding and example segmentation decoding of the points with the feature dimension of 512 at the same timeinsAnd semantic feature prediction Fsem(ii) a The example segmentation decoding part introduces a context aggregation module and a gating transmission module to enhance the learning of features;
the double-line self-attention module predicts the semantic features obtained by the improved PointConv feature extraction modulesemAnd example feature prediction FinsAnd performing feature fusion, and performing instance segmentation and semantic segmentation respectively to obtain instance information containing semantic features and semantic information containing the instance features.
According to a further technical scheme, the point cloud input into the improved PointConv feature extraction module comprises xyz normalized absolute coordinates of points, rgb color information and relative coordinates x ' y ' z ' of the points relative to a local coordinate system.
In a further technical solution, the example partition decoding specifically includes:
the final result of the shared coding module is deconvoluted and up-sampled to N by using PointDeconvdPoint, N isdInputting the characteristics of the points into a context aggregation module, performing convolution for three times by 1x1, dividing to obtain characteristics Q, K and V, performing matrix multiplication on the characteristics Q and the converted characteristics K, and performing sigmoid compression to obtain a weight matrix W1The weight matrix W1Performing matrix multiplication with the characteristic V, and performing element-by-element addition on the obtained result and the characteristic V to obtain a final polymerization characteristic;
final polymerization characteristics FdecFeature F output by shared coding module PointConv _3encAs input to the gated propagation module, FdecAnd FencPerforming channel splicing to obtain FconThen obtaining N through convolution and sigmoid compression by 1x1dX1 weight matrix W2(ii) a Weighting matrix W2Tiling 256 times in the feature dimension, and FencElement by element multiplication to obtain Fenc', the weight matrix 1-W2Tiling 256 times in the feature dimension, and FdecElement by element multiplication to obtain Fdec', then Fenc' and Fdec' channel splicing is carried out, and the final result is output;
f is to bedecAnd FencThe two parts of features are fused to complete the first step of decoding operation to obtain 128-dimensional NcPoint; doing this twice in a similar operation, upsampling the points to N with a feature dimension of 128b(ii) a Finally, the point number is up-sampled to an input point N through one PointDeconvaKeeping the feature dimension 128 to obtain an instance feature prediction Fins。
In a further technical scheme, the semantic segmentation decoding specifically comprises: gradually up-sampling in a deconvolution mode until the number of points is sampled to an input point N with a characteristic dimension of 128aObtaining a semantic feature prediction Fsem。
In a further technical scheme, the example segmentation specifically comprises:
firstly, semantic feature prediction FsemAnd example feature prediction FinsInto STOI module, FsemMeanwhile, after two times of 1x1 convolution, the results of the two times of convolution are multiplied after transposition, and then a weight matrix is obtained through sigmoid compression, and the weight matrix is combined with FinsMultiply by FinsSplicing to obtain example characteristics F with semantic informationstoiAfter the buffering of two full-connection layers, N is finally obtaineda×NeExample of (2) is inserted into F'stoiAnd after multiple back propagation optimization, performing clustering operation once to finish instance segmentation.
In a further technical scheme, the semantic segmentation specifically comprises the following steps:
buffering of the example fractions through a full junction layer Fc1 gave NaExample feature information F 'of x 128'insAnd FsemInputting ITOS module, and converting F'insMeanwhile, after two times of 1x1 convolution, the results of the two times of convolution are multiplied after being transposed, and then a weight matrix is obtained through sigmoid compression, and the weight matrix and the FsemMultiply by FsemSplicing to obtain semantic features F with instance informationitosThen obtaining N through a full connection layera×NcAnd after multiple times of back propagation optimization, completing semantic segmentation through one argmax.
A point cloud semantic and instance joint segmentation system based on improved PointConv comprises the following steps:
improved PointConv feature extraction module for obtaining example feature prediction FinsAnd semantic feature prediction Fsem；
Two-line self-attention module, predicting F for instance featuresinsAnd semantic feature prediction FsemAnd (4) carrying out feature fusion, and respectively carrying out instance segmentation and semantic segmentation.
In the above technical solution, the input channel of the improved PointConv feature extraction module is 9, which respectively represents the xyz normalized absolute coordinate of the point, the rgb color information, and the relative coordinate x ' y ' z ' of the point with respect to the local coordinate system.
The invention has the beneficial effects that:
(1) according to the improved PointConv feature extraction module, a context aggregation module and a gating propagation module are added to an instance segmentation part, instance information is enhanced through weight learning, and the precision of joint segmentation is improved.
(2) The invention improves a PointConv feature extraction module to adopt joint segmentation parallel branches and obtain semantic feature prediction FsemAnd example feature prediction FinsWorking with this baseline improves the speed of instance segmentation, reducing the dependency on semantic segmentation accuracy.
(3) The bifilar self-attention module adopts an STOI module and an ITOS module to fuse semantic features and example features to obtain example information endowed with the semantic features and semantic information rich in the example features, and mutual promotion of the two tasks is completed in a soft constraint mode.
Drawings
FIG. 1 is a flow chart of a point cloud semantic and instance joint segmentation method based on improved PointConv according to the present invention;
FIG. 2 is a block diagram of a context aggregation module according to the present invention;
FIG. 3 is a block diagram of a gated propagation module according to the present invention;
FIG. 4 is a block diagram of the STOI module of the present invention;
FIG. 5 is a block diagram of the ITOS module of the present invention.
Detailed Description
The invention will be further described with reference to the following figures and specific examples, but the scope of the invention is not limited thereto.
As shown in FIG. 1, the point cloud semantic and instance joint segmentation system based on the improved PointConv comprises an improved PointConv feature extraction module and a double-line self-attention module. The processing objects of the point cloud semantic and instance combined segmentation system based on the improved PointConv are as follows: point clouds obtained by a laser radar or a depth camera are subjected to an improved PointConv feature extraction module to obtain an example feature prediction FinsAnd semantic feature prediction FsemExample feature prediction FinsAnd semantic feature prediction FsemThen, the feature fusion is performed by the double-line self-attention module to obtain a product containingInstance information of the semantic features and semantic information containing the instance features.
Table 1 is a specific network structure table of the point cloud semantic and instance joint segmentation system based on the improved PointConv.
Table 1 network structure table
With continued reference to fig. 1, the point cloud semantic and instance joint segmentation method based on the improved PointConv specifically includes the following steps:
the input channel of the improved PointConv feature extraction module is 9, and represents the xyz normalized absolute coordinate of the point, the rgb color information and the relative coordinate x ' y ' z ' of the point relative to the local coordinate system respectively; and introducing relative coordinates of a local coordinate system, which is mainly used for ensuring the translation invariance of the input point. N to be inputtedaThe point is convolved by a shared coding module PointConv _1 (each PointConv has BN operation) to obtain NbAnd (3) inputting points, namely inputting the points into a high-dimensional space to enable the input points to have 64-dimensional characteristics, performing convolution on the input points sequentially through PointConv _2, PointConv _3 and PointConv _4, and increasing the characteristic dimensions of the points to 512 dimensions, wherein enough characteristic information is used for a later decoding part.
The decoding part is divided into two branches: one for semantic segmentation decoding and the other for instance segmentation decoding. The example segmentation decoding part introduces a context aggregation module and a gating propagation module to enhance the learning of features, and the specific operations are as follows: the final result of the shared coding module is deconvoluted up-sampled to N using PointDeconv (with BN operation every time PointDeconv)dPoint, N isdInputting the characteristics of points (corresponding to Fin in figure 2) into a context aggregation module, performing 1x1 convolution three times, dividing to obtain characteristics Q, K and V, performing matrix multiplication on the characteristics Q and the converted characteristics K, and performing sigmoid compression to obtain a weight matrix W1The weight matrix W1Performing matrix multiplication with the characteristic V, and performing element-by-element addition on the obtained result and the characteristic V to obtain a final polymerization characteristic; the context aggregation module performs weighting on the features in a weight learning mode, so that the effective features are enhanced, and the ineffective features are weakened; the final aggregate characteristics (corresponding to F in FIG. 3) are then evaluateddec) Features of the output of the shared coding module PointConv _3 (corresponding to F in FIG. 3)enc) As input to the gated propagation module, FdecAnd FencPerforming channel splicing to obtain Fcon(512 dimensional feature), and then obtaining N through 1x1 convolution and sigmoid compressiondX1 weight matrix W2The weight matrix W2Tiling 256 times in the feature dimension, and FencElement by element multiplication to obtain Fenc', the weight matrix 1-W2Tiling 256 times in the feature dimension, and FdecElement by element multiplication to obtain Fdec', then Fenc' and Fdec' channel splicing is carried out, and the final result is output; the gating transmission module screens effective characteristics in the two parts of characteristics in a learning weight mode to reduce circulation of irrelevant information; finally FdecAnd FencThe two parts of characteristics are fused to complete the first step of decoding operation to obtain N with 256 dimensionscAnd (4) points. Go through twice in a similar operation, up-sampling the number of points to NbFeature dimension of 128; finally, the point number is up-sampled to the input point N through one time of PointDeconvaKeeping the feature dimension 128 to obtain an instance feature prediction Fins. The semantic segmentation decoding part adopts a deconvolution form to carry out gradual up-sampling until the point number is sampled to an input point Na(feature dimension 128) to obtain a semantic feature prediction Fsem。
The double-line self-attention module predicts the semantic features obtained by the improved PointConv feature extraction modulesemAnd example feature prediction FinsAnd performing feature fusion to obtain instance information containing semantic features and semantic information containing the instance features. The method specifically comprises the following steps:
example segmentation section: firstly, semantic feature prediction FsemAnd example feature prediction FinsInto the STOI module, see FIG. 4, when F is turned onsemMeanwhile, after two times of 1x1 convolution, the results of the two times of convolution are multiplied after transposition, and then a weight matrix is obtained through sigmoid compression, and the weight matrix and the example characteristic prediction FinsMultiplying and then predicting with example feature FinsSplicing to obtain example characteristics F with semantic informationstoiAfter buffering by two full-link layers (Fc1, Fc2), N is finally obtaineda×NeExample of (2) is inserted into F'stoiAnd after repeated back propagation optimization, performing mean-shift clustering operation on the part for one time to finish example segmentation.
And a semantic segmentation part: n obtained after buffering of example fractions with Fc1aExample feature information F 'of x 128'insAnd initial semantic feature prediction FsemPassed as input into the ITOS module, see FIG. 5, at which time F'insMeanwhile, after two times of 1x1 convolution, the results of the two times of convolution are multiplied after transposition, and then a weight matrix is obtained through sigmoid compression, and the weight matrix and the semantic feature prediction FsemMultiplication and semantic feature prediction FsemSplicing to obtain semantic features F with instance informationitosThen obtaining N through a full connection layer (Fc)a×NcAnd after repeated back propagation optimization, only one argmax is needed to be carried out finally to complete semantic segmentation.
When an algorithm of a point cloud semantic and example combined segmentation system based on improved PointConv is trained, an adopted loss function consists of two parts, wherein one part is the loss of a semantic segmentation part, and the other part is the loss of an example segmentation part; the two parts are optimized simultaneously to complete the training task.
The loss function is expressed as follows:
L＝Lsem+Lins
Lsemas a loss function of semantically segmented parts, LinsA loss function that is an example partition;
Lsemwith a classical cross entropy loss function, the expression is as follows:
wherein p (x) is the true probability distribution (which has been determined according to the input label of the training data set), n is the number of categories, and q (x) is the predicted probability distribution, and the smaller the difference between the two probability distributions is, the better the predicted result is, and the better the partial optimization effect is.
LinsAdopting a discrimination loss function discrete loss, wherein the expression is as follows:
Lins＝Lvar+Ldist+α·Lreg
wherein: i is the number of instances of the true value; n is a radical ofiThe number of points in example i; mu.siFor the average embedding of the example i,
is an example iAThe average of the embedding of (a) into (b),
is an example iBAverage embedding of (2); e.g. of the typejEmbedding for a certain point; deltad、δvIs a loss function threshold; a is the equilibrium coefficient, set to 0.001.
LvarThe embedding of each point instance is mainly used for clustering the embedding of each point instance to the center of each instance, so that the points belonging to the same instance can be close to each other in a feature space; and L isdistMainly used for mutually repelling points among different instances, and the distance between the points is widened; l isregTo ensure the feature embedding is bounded, the instance center is brought close to the local coordinate system origin.
Finally, when testing is carried out on the basis of the point cloud semantics of the improved PointConv and an algorithm of an example combined segmentation system, the examples generated by the example segmentation part are embedded and clustered by using a mean-shift method to obtain a final example result; and performing argmax operation on the semantic features generated by the semantic segmentation part to obtain the final semantic classification. And completing the operation of the whole algorithm of the point cloud semantic and instance combined segmentation system based on the improved PointConv.
The point cloud semantic and instance joint segmentation system based on the improved PointConv can be implemented in the form of a computer program, and the computer program can be run on a computer device, and the computer device can be a server or a terminal. The server can be an independent server or a server cluster; the terminal can be a notebook computer, a desktop computer, and other electronic equipment.
The computer device comprises a processor, a memory and a network interface which are connected through a system bus, wherein the memory can comprise a nonvolatile storage medium and an internal memory; the non-volatile storage medium may store an operating system and a computer program. The computer program includes program instructions that, when executed, cause a processor to perform any one of the methods of improving PointConv-based joint segmentation of point cloud semantics and instances. The processor is used for providing calculation and control capability and supporting the operation of the whole computer equipment. The memory provides an environment for execution of a computer program in a non-volatile storage medium, which when executed by the processor, causes the processor to perform any one of the methods for improving joint segmentation of point cloud semantics and instances based on PointConv. The network interface is used for network communication, such as sending assigned tasks and the like.
It should be understood that the Processor may be a Central Processing Unit (CPU), and the Processor may be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, etc. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The embodiment of the application further provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, the computer program comprises program instructions, and the processor executes the program instructions to realize the point cloud semantic and instance joint segmentation method based on the improved PointConv.
The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, for example, a hard disk or a memory of the computer device. The computer readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device.
The present invention is not limited to the above-described embodiments, and any obvious improvements, substitutions or modifications can be made by those skilled in the art without departing from the spirit of the present invention.
Claims (8)
1. A point cloud semantic and instance joint segmentation method based on improved PointConv is characterized by comprising the following steps:
inputting the obtained point cloud into an improved PointConv characteristic extraction module, and sharing and editingThe code module obtains 512 points with characteristic dimensionality, and the 512 points with characteristic dimensionality are subjected to semantic segmentation decoding and example segmentation decoding simultaneously to obtain example characteristic prediction FinsAnd semantic feature prediction Fsem(ii) a The example segmentation decoding part introduces a context aggregation module and a gating transmission module to enhance the learning of features;
the double-line self-attention module predicts the semantic features obtained by the improved PointConv feature extraction modulesemAnd example feature prediction FinsAnd performing feature fusion, and performing instance segmentation and semantic segmentation respectively to obtain instance information containing semantic features and semantic information containing the instance features.
2. The improved PointConv-based point cloud semantic and instance joint segmentation method according to claim 1, wherein the point cloud input into the improved PointConv feature extraction module comprises xyz normalized absolute coordinates of points, rgb color information, and relative coordinates x ' y ' z ' of points with respect to a local coordinate system.
3. The method of claim 1, wherein the example segmentation decoding specifically comprises:
the final result of the shared coding module is deconvoluted and up-sampled to N by using PointDeconvdPoint, N isdInputting the characteristics of the points into a context aggregation module, performing convolution for three times by 1x1, dividing to obtain characteristics Q, K and V, performing matrix multiplication on the characteristics Q and the converted characteristics K, and performing sigmoid compression to obtain a weight matrix W1The weight matrix W1Performing matrix multiplication with the characteristic V, and performing element-by-element addition on the obtained result and the characteristic V to obtain a final polymerization characteristic;
final polymerization characteristics FdecFeature F output by shared coding module PointConv _3encAs input to the gated propagation module, FdecAnd FencPerforming channel splicing to obtain FconThen, the mixture is convoluted by 1x1 and is compressed by sigmoidTo obtain NdX1 weight matrix W2(ii) a Weighting matrix W2Tiling 256 times in the feature dimension, and FencElement by element multiplication to obtain Fenc', the weight matrix 1-W2Tiling 256 times in the feature dimension, and FdecElement by element multiplication to obtain Fdec', then Fenc' and Fdec' channel splicing is carried out, and the final result is output;
f is to bedecAnd FencThe two parts of characteristics are fused to complete the first step of decoding operation to obtain N with 256 dimensionscPoint; doing this twice in a similar operation, upsampling the points to N with a feature dimension of 128b(ii) a Finally, the point number is up-sampled to an input point N through one PointDeconvaKeeping the feature dimension 128 to obtain an instance feature prediction Fins。
4. The point cloud semantic and instance joint segmentation method based on the improved PointConv as claimed in claim 1, wherein the semantic segmentation decoding specifically comprises: gradually up-sampling in a deconvolution mode until the number of points is sampled to an input point N with a characteristic dimension of 128aObtaining a semantic feature prediction Fsem。
5. The improved PointConv-based point cloud semantic and instance joint segmentation method according to claim 1, wherein the instance segmentation specifically comprises:
firstly, semantic feature prediction FsemAnd example feature prediction FinsInto STOI module, FsemMeanwhile, after two times of 1x1 convolution, the results of the two times of convolution are multiplied after transposition, and then a weight matrix is obtained through sigmoid compression, and the weight matrix is combined with FinsMultiply by FinsSplicing to obtain example characteristics F with semantic informationstoiAfter the buffering of two full-connection layers, N is finally obtaineda×NeExample of (2) is inserted into F'stoiAnd after multiple back propagation optimization, performing clustering operation once to finish instance segmentation.
6. The point cloud semantic and instance joint segmentation method based on the improved PointConv as claimed in claim 1, wherein the semantic segmentation is specifically as follows:
buffering of the example fractions through a full junction layer Fc1 gave NaExample feature information F 'of x 128'insAnd FsemInputting ITOS module, and converting F'insMeanwhile, after two times of 1x1 convolution, the results of the two times of convolution are multiplied after being transposed, and then a weight matrix is obtained through sigmoid compression, and the weight matrix and the FsemMultiply by FsemSplicing to obtain semantic features F with instance informationitosThen obtaining N through a full connection layera×NcAnd after multiple times of back propagation optimization, completing semantic segmentation through one argmax.
7. A segmentation system for implementing the point cloud semantic and instance joint segmentation method based on improved PointConv according to any one of claims 1 to 6, comprising:
improved PointConv feature extraction module for obtaining example feature prediction FinsAnd semantic feature prediction Fsem；
Two-line self-attention module, predicting F for instance featuresinsAnd semantic feature prediction FsemAnd (4) carrying out feature fusion, and respectively carrying out instance segmentation and semantic segmentation.
8. The segmentation system according to claim 7, wherein the input channel of the modified PointConv feature extraction module is 9, representing the xyz normalized absolute coordinates of a point, rgb color information, and the relative coordinates x ' y ' z ' of a point with respect to the local coordinate system, respectively.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110495434.3A CN113421267B (en) | 2021-05-07 | 2021-05-07 | Point cloud semantic and instance joint segmentation method and system based on improved PointConv |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110495434.3A CN113421267B (en) | 2021-05-07 | 2021-05-07 | Point cloud semantic and instance joint segmentation method and system based on improved PointConv |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113421267A true CN113421267A (en) | 2021-09-21 |
| CN113421267B CN113421267B (en) | 2024-04-12 |
Family
ID=77712141
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110495434.3A Active CN113421267B (en) | 2021-05-07 | 2021-05-07 | Point cloud semantic and instance joint segmentation method and system based on improved PointConv |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113421267B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114419078A (en) * | 2022-04-01 | 2022-04-29 | 北京矩视智能科技有限公司 | Surface defect region segmentation method and device based on convolutional neural network |
| CN116612285A (en) * | 2023-06-15 | 2023-08-18 | 重庆市测绘科学技术研究院(重庆市地图编制中心) | Building point cloud data segmentation and point cloud data semantic segmentation method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109410307A (en) * | 2018-10-16 | 2019-03-01 | 大连理工大学 | A kind of scene point cloud semantic segmentation method |
| CN110110692A (en) * | 2019-05-17 | 2019-08-09 | 南京大学 | A kind of realtime graphic semantic segmentation method based on the full convolutional neural networks of lightweight |
| CN110245709A (en) * | 2019-06-18 | 2019-09-17 | 西安电子科技大学 | Based on deep learning and from the 3D point cloud data semantic dividing method of attention |
| CN111047596A (en) * | 2019-12-12 | 2020-04-21 | 中国科学院深圳先进技术研究院 | Three-dimensional point cloud instance segmentation method and system and electronic equipment |
| CN111275026A (en) * | 2020-03-23 | 2020-06-12 | 复旦大学 | Three-dimensional point cloud combined semantic and instance segmentation method |
| CN111489358A (en) * | 2020-03-18 | 2020-08-04 | 华中科技大学 | Three-dimensional point cloud semantic segmentation method based on deep learning |
| CN111915613A (en) * | 2020-08-11 | 2020-11-10 | 华侨大学 | Image instance segmentation method, device, equipment and storage medium |
-
2021
- 2021-05-07 CN CN202110495434.3A patent/CN113421267B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109410307A (en) * | 2018-10-16 | 2019-03-01 | 大连理工大学 | A kind of scene point cloud semantic segmentation method |
| CN110110692A (en) * | 2019-05-17 | 2019-08-09 | 南京大学 | A kind of realtime graphic semantic segmentation method based on the full convolutional neural networks of lightweight |
| CN110245709A (en) * | 2019-06-18 | 2019-09-17 | 西安电子科技大学 | Based on deep learning and from the 3D point cloud data semantic dividing method of attention |
| CN111047596A (en) * | 2019-12-12 | 2020-04-21 | 中国科学院深圳先进技术研究院 | Three-dimensional point cloud instance segmentation method and system and electronic equipment |
| CN111489358A (en) * | 2020-03-18 | 2020-08-04 | 华中科技大学 | Three-dimensional point cloud semantic segmentation method based on deep learning |
| CN111275026A (en) * | 2020-03-23 | 2020-06-12 | 复旦大学 | Three-dimensional point cloud combined semantic and instance segmentation method |
| CN111915613A (en) * | 2020-08-11 | 2020-11-10 | 华侨大学 | Image instance segmentation method, device, equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114419078A (en) * | 2022-04-01 | 2022-04-29 | 北京矩视智能科技有限公司 | Surface defect region segmentation method and device based on convolutional neural network |
| CN116612285A (en) * | 2023-06-15 | 2023-08-18 | 重庆市测绘科学技术研究院(重庆市地图编制中心) | Building point cloud data segmentation and point cloud data semantic segmentation method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113421267B (en) | 2024-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112966522B (en) | Image classification method and device, electronic equipment and storage medium | |
| US11227185B2 (en) | Identifying visually similar digital images utilizing deep learning | |
| US11670071B2 (en) | Fine-grained image recognition | |
| CN112288011B (en) | Image matching method based on self-attention deep neural network | |
| CN111275107A (en) | Multi-label scene image classification method and device based on transfer learning | |
| CN112699806B (en) | Three-dimensional point cloud target detection method and device based on three-dimensional heat map | |
| US20200334287A1 (en) | Image retrieval method, image retrieval apparatus, image retrieval device and medium | |
| CN111382868A (en) | Neural network structure search method and neural network structure search device | |
| WO2020098257A1 (en) | Image classification method and device and computer readable storage medium | |
| JP7286810B2 (en) | Text intelligent cleaning method, apparatus and computer readable storage medium | |
| CN113869138A (en) | Multi-scale target detection method and device and computer readable storage medium | |
| CN110889416B (en) | Salient object detection method based on cascade improved network | |
| CN113421267A (en) | Point cloud semantic and instance joint segmentation method and system based on improved PointConv | |
| US20230394866A1 (en) | Person re-identification method and apparatus, device, and readable storage medium | |
| CN112529068B (en) | Multi-view image classification method, system, computer equipment and storage medium | |
| Lu et al. | Generalized haar filter-based object detection for car sharing services | |
| Huang et al. | Fine-art painting classification via two-channel deep residual network | |
| CN110717405A (en) | Face feature point positioning method, device, medium and electronic equipment | |
| CN114693923A (en) | Three-dimensional point cloud semantic segmentation method based on context and attention | |
| Ma et al. | Semantic segmentation using cross-stage feature reweighting and efficient self-attention | |
| CN115294343B (en) | Point cloud feature enhancement method based on cross-position and channel attention mechanism | |
| Zhou et al. | Supervised semantic segmentation based on deep learning: a survey | |
| CN114819140A (en) | Model pruning method and device and computer equipment | |
| Zhang et al. | Cross diffusion on multi-hypergraph for multi-modal 3d object recognition | |
| CN112785601B (en) | Image segmentation method, system, medium and electronic terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |