Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article
Public Access

Dynamic Graph CNN for Learning on Point Clouds

Published: 10 October 2019 Publication History

Abstract

Point clouds provide a flexible geometric representation suitable for countless applications in computer graphics; they also comprise the raw output of most 3D data acquisition devices. While hand-designed features on point clouds have long been proposed in graphics and vision, however, the recent overwhelming success of convolutional neural networks (CNNs) for image analysis suggests the value of adapting insight from CNN to the point cloud world. Point clouds inherently lack topological information, so designing a model to recover topology can enrich the representation power of point clouds. To this end, we propose a new neural network module dubbed EdgeConv suitable for CNN-based high-level tasks on point clouds, including classification and segmentation. EdgeConv acts on graphs dynamically computed in each layer of the network. It is differentiable and can be plugged into existing architectures. Compared to existing modules operating in extrinsic space or treating each point independently, EdgeConv has several appealing properties: It incorporates local neighborhood information; it can be stacked applied to learn global shape properties; and in multi-layer systems affinity in feature space captures semantic characteristics over potentially long distances in the original embedding. We show the performance of our model on standard benchmarks, including ModelNet40, ShapeNetPart, and S3DIS.

References

[1]
Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 2016. 3D semantic parsing of large-scale indoor spaces. In Proceedings of the CVPR.
[2]
Matan Atzmon, Haggai Maron, and Yaron Lipman. 2018. Point convolutional neural networks by extension operators. ACM Trans. Graph. 37, 4, Article 71 (July 2018), 12 pages.
[3]
Mathieu Aubry, Ulrich Schlickewei, and Daniel Cremers. 2011. The wave kernel signature: A quantum mechanical approach to shape analysis. In Proceedings of the ICCV Workshops.
[4]
Serge Belongie, Jitendra Malik, and Jan Puzicha. 2001. Shape context: A new descriptor for shape matching and object recognition. In Proceedings of the NIPS.
[5]
Silvia Biasotti, Andrea Cerri, A. Bronstein, and M. Bronstein. 2016. Recent trends, applications, and perspectives in 3D shape similarity assessment. Comput. Graph. Forum 35, 6 (2016), 87--119.
[6]
Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael Bronstein. 2016. Learning shape correspondence with anisotropic convolutional neural networks. In Proceedings of the NIPS.
[7]
Andrew Brock, Theodore Lim, James Millar Ritchie, and Nicholas J. Weston. 2016. Generative and discriminative voxel modeling with convolutional neural networks. In Proceedings of the NIPS.
[8]
Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric deep learning: Going beyond euclidean data. IEEE Signal Process. Mag. 34, 4 (2017), 18--42.
[9]
Michael M. Bronstein and Iasonas Kokkinos. 2010. Scale-invariant heat kernel signatures for non-rigid shape recognition. In Proceedings of the CVPR.
[10]
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2013. Spectral networks and locally connected networks on graphs. arXiv:1312.6203 (2013).
[11]
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su et al. 2015. Shapenet: An information-rich 3D model repository. arXiv:1512.03012 (2015).
[12]
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the NIPS.
[13]
Francis Engelmann, Theodora Kontogianni, Alexander Hermans, and Bastian Leibe. 2017. Exploring spatial context for 3D semantic segmentation of point clouds. In Proceedings of the CVPR.
[14]
Danielle Ezuz, Justin Solomon, Vladimir G. Kim, and Mirela Ben-Chen. 2017. GWCNN: A metric alignment layer for deep shape analysis. Comput. Graph. Forum 36, 5 (2017), 49--57.
[15]
Haoqiang Fan, Hao Su, and Leonidas J. Guibas. 2017. A point set generation network for 3D object reconstruction from a single image. In Proceedings of the CVPR.
[16]
Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. 2018. SplineCNN: Fast geometric deep learning with continuous B-spline kernels. In Proceedings of the CVPR.
[17]
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. arXiv:1704.01212 (2017).
[18]
Aleksey Golovinskiy, Vladimir G. Kim, and Thomas Funkhouser. 2009. Shape-based recognition of 3D point clouds in urban environments. In Proceedings of the ICCV.
[19]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the NIPS.
[20]
Paul Guerrero, Yanir Kleiman, Maks Ovsjanikov, and Niloy J. Mitra. 2018. PCPNet: Learning local shape properties from raw point clouds. Comput. Graph. Forum 37, 2 (2018), 75--85.
[21]
Yulan Guo, Mohammed Bennamoun, Ferdous Sohel, Min Lu, and Jianwei Wan. 2014. 3D object recognition in cluttered scenes with local surface features: A survey. Trans. PAMI 36, 11 (2014), 2270--2287.
[22]
Oshri Halimi, Or Litany, Emanuele Rodolà, Alex Bronstein, and Ron Kimmel. 2018. Self-supervised learning of dense shape correspondence. arXiv:1812.02415 (2018).
[23]
M. Henaff, J. Bruna, and Y. LeCun. 2015. Deep convolutional networks on graph-structured data. arXiv:1506.05163 (2015).
[24]
Andrew E. Johnson and Martial Hebert. 1999. Using spin images for efficient object recognition in cluttered 3D scenes. Trans. PAMI 21, 5 (1999), 433--449.
[25]
Diederik P. Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv:1312.6114 (2013).
[26]
Thomas N. Kipf and Max Welling. 2017. Semi-Supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR).
[27]
Roman Klokov and Victor Lempitsky. 2017. Escape from cells: Deep Kd-networks for the recognition of 3D point cloud models. (2017).
[28]
Ilya Kostrikov, Zhongshi Jiang, Daniele Panozzo, Denis Zorin, and Joan Bruna. 2017. Surface networks. In Proceedings of the CVPR.
[29]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of the NIPS.
[30]
Yann LeCun, Bernhard Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne Hubbard, and Lawrence D. Jackel. 1989. Backpropagation applied to handwritten ZIP code recognition. Neural Comput. 1, 4 (1989), 541--551.
[31]
Ron Levie, Federico Monti, Xavier Bresson, and Michael M. Bronstein. 2017. CayleyNets: Graph convolutional neural networks with complex rational spectral filters. arXiv:1705.07664 (2017).
[32]
Chun-Liang Li, Manzil Zaheer, Yang Zhang, Barnabas Poczos, and Ruslan Salakhutdinov. 2018b. Point cloud GAN. arXiv:1810.05795 (2018).
[33]
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. 2018a. PointCNN: Convolution On X-transformed points. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 820--830. Retrieved from http://papers.nips.cc/paper/7362-pointcnn-convolution-on-x-transformed-points.pdf.
[34]
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2016. Gated graph sequence neural networks. In Proceedings of the ICLR.
[35]
Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. 2018. Deep continuous fusion for multi-sensor 3D object detection. In Proceedings of the ECCV.
[36]
Haibin Ling and David W. Jacobs. 2007. Shape classification using the inner-distance. Trans. PAMI 29, 2 (2007), 286--299.
[37]
Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Makadia. 2017a. Deformable shape completion with graph convolutional autoencoders. arXiv:1712.00268 (2017).
[38]
Or Litany, Tal Remez, Emanuele Rodolà, Alex M. Bronstein, and Michael M. Bronstein. 2017b. Deep functional maps: Structured prediction for dense shape correspondence. In Proceedings of the ICCV.
[39]
I. Loshchilov and F. Hutter. 2017. SGDR: Stochastic gradient descent with warm restarts. In Proceedings of the ICLR.
[40]
Min Lu, Yulan Guo, Jun Zhang, Yanxin Ma, and Yinjie Lei. 2014. Recognizing objects in 3D point clouds with multi-scale local features. Sensors 14, 12 (2014), 24156--24173.
[41]
Siddharth Manay, Daniel Cremers, Byung-Woo Hong, Anthony J. Yezzi, and Stefano Soatto. 2006. Integral invariants for shape matching. Trans. PAMI 28, 10 (2006), 1602--1618.
[42]
Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope, Nadav Dym, Ersin Yumer, Vladimir G Kim, and Yaron Lipman. 2017. Convolutional neural networks on surfaces via seamless toric covers. In Proceedings of the SIGGRAPH.
[43]
Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. 2015. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the 3dRR.
[44]
Daniel Maturana and Sebastian Scherer. 2015. Voxnet: A 3D convolutional neural network for real-time object recognition. In Proceedings of the IROS.
[45]
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and Michael M. Bronstein. 2017a. Geometric deep learning on graphs and manifolds using mixture model CNNs. In Proceedings of the CVPR.
[46]
F. Monti, M. M. Bronstein, and X. Bresson. 2017b. Geometric matrix completion with recurrent multi-graph neural networks. In Proceedings of the NIPS.
[47]
Federico Monti, Karl Otness, and Michael M. Bronstein. 2018. MotifNet: A motif-based graph convolutional network for directed graphs. arXiv:1802.01572 (2018).
[48]
Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. 2012. Functional maps: A flexible representation of maps between shapes. Trans. Graph. 31, 4 (2012), 30.
[49]
Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J. Guibas. 2017a. Frustum PointNets for 3D object detection from RGB-D data. arXiv:1711.08488 (2017).
[50]
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017b. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the CVPR.
[51]
Charles R. Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J. Guibas. 2016. Volumetric and multi-view CNNs for object classification on 3D data. In Proceedings of the CVPR.
[52]
Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. 2017c. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In Proceedings of the NIPS.
[53]
Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black. 2018. Generating 3D faces using convolutional mesh autoencoders. arXiv:1807.10267 (2018).
[54]
Raif M. Rustamov. 2007. Laplace-beltrami eigenfunctions for deformation invariant shape representation. In Proceedings of the SGP.
[55]
Radu Bogdan Rusu, Nico Blodow, and Michael Beetz. 2009. Fast point feature histograms (FPFH) for 3D registration. In Proceedings of the ICRA.
[56]
Radu Bogdan Rusu, Nico Blodow, Zoltan Csaba Marton, and Michael Beetz. 2008a. Aligning point cloud views using persistent feature histograms. In Proceedings of the IROS.
[57]
Radu Bogdan Rusu, Zoltan Csaba Marton, Nico Blodow, Mihai Dolha, and Michael Beetz. 2008b. Towards 3D point cloud-based object maps for household environments. Robot. Auton. Syst. J. 56, 11 (Nov. 2008), 927--941.
[58]
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009. The graph neural network model. IEEE Tran. Neural Networks 20, 1 (2009), 61--80.
[59]
Syed Afaq Ali Shah, Mohammed Bennamoun, Farid Boussaid, and Amar A. El-Sallam. 2013. 3D-Div: A novel local surface descriptor for feature matching and pairwise range image registration. In Proceedings of the ICIP.
[60]
Yiru Shen, Chen Feng, Yaoqing Yang, and Dong Tian. 2017. Neighbors do help: Deeply exploiting local structures of point clouds. arXiv:1712.06760 (2017).
[61]
David I. Shuman, Sunil K. Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. 2013. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE Signal Process. Mag. 30, 3 (2013), 83--98.
[62]
Martin Simonovsky and Nikos Komodakis. 2017. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In Proceedings of the CVPR.
[63]
Ayan Sinha, Jing Bai, and Karthik Ramani. 2016. Deep learning 3D shape surfaces using geometry images. In Proceedings of the ECCV.
[64]
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, and Jan Kautz. 2018. SPLATNet: Sparse lattice networks for point cloud processing. In Proceedings of the CVPR. 2530--2539.
[65]
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015. Multi-view convolutional neural networks for 3D shape recognition. In Proceedings of the CVPR.
[66]
Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. 2009. A concise and provably informative multi-scale signature based on heat diffusion. Comput. Graph. Forum 28, 5 (2009), 1383--1392.
[67]
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. 2017. Octree generating networks: Efficient convolutional architectures for high-resolution 3D outputs. In Proceedings of the ICCV.
[68]
Federico Tombari, Samuele Salti, and Luigi Di Stefano. 2011. A combined texture-shape descriptor for enhanced 3D feature matching. In Proceedings of the ICIP.
[69]
Oliver Van Kaick, Hao Zhang, Ghassan Hamarneh, and Daniel Cohen-Or. 2011. A survey on shape correspondence. Comput. Graph. Forum 30, 6 (2011), 1681--1707.
[70]
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2017. Graph attention networks. arXiv:1710.10903.
[71]
Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. 2018b. Deep parametric continuous convolutional neural networks. In Proceedings of the CVPR.
[72]
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018a. Non-local neural networks. In Proceedings of the CVPR.
[73]
Lingyu Wei, Qixing Huang, Duygu Ceylan, Etienne Vouga, and Hao Li. 2016. Dense human body correspondences using convolutional networks. In Proceedings of the CVPR.
[74]
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D shapenets: A deep representation for volumetric shapes. In Proceedings of the CVPR.
[75]
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. 2018. Feature denoising for improving adversarial robustness. arXiv:1812.03411.
[76]
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. 2018. FoldingNet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the CVPR.
[77]
Li Yi, Vladimir G. Kim, Duygu Ceylan, I. Shen, Mengyan Yan, Hao Su, A. R. Cewu Lu, Qixing Huang, Alla Sheffer, Leonidas Guibas et al. 2016. A scalable active framework for region annotation in 3D shape collections. Trans. Graph. 35, 6 (2016), 210.
[78]
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. 2017. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In Proceedings of the ICRA.

Cited By

View all
  • (2025)Advanced Deep Learning–Based Hybrid Rail Extraction Algorithm Leveraging LiDAR TechnologyJournal of Infrastructure Systems10.1061/JITSE4.ISENG-245731:1Online publication date: Mar-2025
  • (2025)Class agnostic and specific consistency learning for weakly-supervised point cloud semantic segmentationPattern Recognition10.1016/j.patcog.2024.111067158(111067)Online publication date: Feb-2025
  • (2025)SemiGMMPoint: Semi-supervised point cloud segmentation based on Gaussian mixture modelsPattern Recognition10.1016/j.patcog.2024.111045158(111045)Online publication date: Feb-2025
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 38, Issue 5
October 2019
191 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3341165
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2019
Accepted: 01 June 2019
Revised: 01 May 2019
Received: 01 January 2019
Published in TOG Volume 38, Issue 5

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Point cloud
  2. classification
  3. segmentation

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

  • Air Force Office of Scientific Research
  • Toyota-CSAIL Joint Research Center
  • National Science Foundation
  • Amazon Research Award
  • MITIBM Watson AI Laboratory
  • Skoltech-MIT Next Generation Program
  • Army Research Office
  • ERC Consolidator
  • Google Faculty Research Award

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)8,238
  • Downloads (Last 6 weeks)1,189
Reflects downloads up to 19 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2025)Advanced Deep Learning–Based Hybrid Rail Extraction Algorithm Leveraging LiDAR TechnologyJournal of Infrastructure Systems10.1061/JITSE4.ISENG-245731:1Online publication date: Mar-2025
  • (2025)Class agnostic and specific consistency learning for weakly-supervised point cloud semantic segmentationPattern Recognition10.1016/j.patcog.2024.111067158(111067)Online publication date: Feb-2025
  • (2025)SemiGMMPoint: Semi-supervised point cloud segmentation based on Gaussian mixture modelsPattern Recognition10.1016/j.patcog.2024.111045158(111045)Online publication date: Feb-2025
  • (2025)Hyperbolic prototype rectification for few-shot 3D point cloud classificationPattern Recognition10.1016/j.patcog.2024.111042158(111042)Online publication date: Feb-2025
  • (2025)Revisiting 3D point cloud analysis with Markov processPattern Recognition10.1016/j.patcog.2024.110997158(110997)Online publication date: Feb-2025
  • (2025)Research and application on deep learning-based point cloud completion for marine structures with point coordinate fusion and coordinate-supervised point cloud generatorMeasurement10.1016/j.measurement.2024.116246242(116246)Online publication date: Jan-2025
  • (2025)KPTr: Key point transformer for LiDAR-based 3D object detectionMeasurement10.1016/j.measurement.2024.115820242(115820)Online publication date: Jan-2025
  • (2025)Corrupted point cloud classification based on feature compensation and cross attention mechanismMeasurement10.1016/j.measurement.2024.115721241(115721)Online publication date: Feb-2025
  • (2025)Deep learning based 3D segmentation in computer vision: A surveyInformation Fusion10.1016/j.inffus.2024.102722115(102722)Online publication date: Mar-2025
  • (2025)Advancing 3D point cloud understanding through deep transfer learning: A comprehensive surveyInformation Fusion10.1016/j.inffus.2024.102601113(102601)Online publication date: Jan-2025
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Full Access

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media