research-article

Aerial path planning for urban scene reconstruction: a continuous optimization method and benchmark

Authors:

Michael Goesele,

Wolfgang HeidrichAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 37, Issue 6

Article No.: 183, Pages 1 - 15

https://doi.org/10.1145/3272127.3275010

Published: 04 December 2018 Publication History

Abstract

Small unmanned aerial vehicles (UAVs) are ideal capturing devices for high-resolution urban 3D reconstructions using multi-view stereo. Nevertheless, practical considerations such as safety usually mean that access to the scan target is often only available for a short amount of time, especially in urban environments. It therefore becomes crucial to perform both view and path planning to minimize flight time while ensuring complete and accurate reconstructions.

In this work, we address the challenge of automatic view and path planning for UAV-based aerial imaging with the goal of urban reconstruction from multi-view stereo. To this end, we develop a novel continuous optimization approach using heuristics for multi-view stereo reconstruction quality and apply it to the problem of path planning. Even for large scan areas, our method generates paths in only a few minutes, and is therefore ideally suited for deployment in the field.

To evaluate our method, we introduce and describe a detailed benchmark dataset for UAV path planning in urban environments which can also be used to evaluate future research efforts on this topic. Using this dataset and both synthetic and real data, we demonstrate survey-grade urban reconstructions with ground resolutions of 1 cm or better on large areas (30 000 m²).

Supplementary Material

ZIP File (a183-smith.zip)

Supplemental files.

Download
14.46 MB

MP4 File (a183-smith.mp4)

Download
206.08 MB

References

[1]

Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. 2016. Large-Scale Data for Multiple-View Stereopsis. IJCV 120, 2 (01 Nov. 2016), 153--168.

Digital Library

[2]

Bashar Alsadik, Markus Gerke, and George Vosselman. 2013. Automated Camera Network Design for 3D Modeling of Cultural Heritage Objects. J. Cultural Heritage 14, 6 (2013).

[3]

Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenbaum. 2013. Simulation as an engine of physical scene understanding. Proc. National Academy of Sciences 110, 45 (2013), 18327--18332.

[4]

Christian Beder and Richard Steffen. 2006. Determining an Initial Image Pair for Fixing the Scale of a 3D Reconstruction from an Image Sequence. In Proc. Joint Pattern Recognition Symposium.

Digital Library

[5]

P. J. Besl and N. D. McKay. 1992. A Method for Registration of 3-D Shapes. IEEE Trans. PAMI 14, 2 (1992), 239--256.

Digital Library

[6]

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In Proc. CVPR.

[7]

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. In Proc. Ann. Conf. Robot Learning. 1--16.

[8]

Enrique Dunn and Jan-Michael Frahm. 2009. Next Best View Planning for Active Model Improvement. In Proc. BMVC.

[9]

Xinyi Fan, Linguang Zhang, Benedict Brown, and Szymon Rusinkiewicz. 2016. Automated View and Path Planning for Scalable Multi-Object 3D Scanning. ACM Trans. Graph. 35, 6 (Nov. 2016).

Digital Library

[10]

Christian Forster, Matia Pizzoli, and Davide Scaramuzza. 2014. Appearance-Based Active, Monocular, Dense Reconstruction for Micro Aerial Vehicles. In Robotics: Science and Systems.

[11]

Simon Fuhrmann and Michael Goesele. 2014. Floating Scale Surface Reconstruction. ACM Trans. Graph. 33, 4 (2014).

Digital Library

[12]

Simon Fuhrmann, Fabian Langguth, Nils Moehrle, Michael Waechter, and Michael Goesele. 2015. MVE - An Image-Based Reconstruction Environment. Computers & Graphics 53 (2015).

Digital Library

[13]

Yasutaka Furukawa, Brian Curless, Steven M. Seitz, and Richard Szeliski. 2010. Towards Internet-Scale Multi-View Stereo. In Proc. CVPR.

[14]

Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig. 2016. Virtual worlds as proxy for multi-object tracking analysis. In Proc. CVPR. 4340--4349.

[15]

David Gallup, Jan-Michael Frahm, Philippos Mordohai, and Marc Pollefeys. 2008. Variable Baseline/Resolution Stereo. In Proc. CVPR.

[16]

Quentin Galvane, Christophe Lino, Marc Christie, Julien Fleureau, Fabien Servant, François-Louis Tariolle, and Philippe Guillotel. 2017. Directing Cinematographic Drones. CoRR abs/1712.04216 (2017). arXiv:1712.04216 http://arxiv.org/abs/1712.04216

[17]

Christoph Gebhardt, Stefan Stevsic, and Otmar Hilliges. 2018. Optimizing for Aesthetically Pleasing Quadrotor Camera Motion. ACM Trans. Graph. 37, 4, Article 90 (Aug. 2018), 11 pages.

Digital Library

[18]

Michael Goesele, Noah Snavely, Brian Curless, Hugues Hoppe, and Steven M. Seitz. 2007. Multi-View Stereo for Community Photo Collections. In Proc. ICCV.

[19]

A. Handa, T. Whelan, J. McDonald, and A. J. Davison. 2014. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In 2014 IEEE International Conference on Robotics and Automation (ICRA). 1524--1531.

[20]

Benjamin Hepp, Matthias Niessner, and Otmar Hilliges. 2017. Plan3D: Viewpoint and Trajectory Optimization for Aerial Multi-View Stereo Reconstruction. CoRR abs/1705.09314 (2017).

[21]

Christof Hoppe, Andreas Wendel, Stefanie Zollmann, Katrin Pirker, Arnold Irschara, Horst Bischof, and Stefan Kluckner. 2012. Photogrammetric Camera Network Design for Micro Aerial Vehicles. In Proc. Computer Vision Winter Workshop, Vol. 8.

[22]

Rui Huang, Danping Zou, Richard Vaughan, and Ping Tan. 2017. Active Image-based Modeling. CoRR abs/1705.01010 (2017). arXiv:1705.01010 http://arxiv.org/abs/1705.01010

[23]

Brian Karis and Epic Games. 2013. Real Shading in Unreal Engine 4. (2013).

[24]

Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017. Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction. ACM Trans. Graph. 36, 4 (2017).

Digital Library

[25]

Fabian Langguth, Kalyan Sunkavalli, Sunil Hadap, and Michael Goesele. 2016. Shading-Aware Multi-View Stereo. In Proc. ECCV.

[26]

John J Leonard and Hugh F Durrant-Whyte. 1991. Simultaneous Map Building and Localization for an Autonomous Mobile Robot. In Proc. IROS.

[27]

Ligang Liu, Xi Xia, Han Sun, Qi Shen, Junzhan Xu, Bin Chen, Hui Huang, and Kai Xu. 2018. Object-Aware Guidance for Autonomous Scene Reconstruction. ACM Trans. Graph. 37, 4 (2018), 104:1--104:12.

Digital Library

[28]

O. Mendez, Simon Hadfield, Nicolas Pugeault, and Richard Bowden. 2016. Next-Best Stereo: Extending Next-Best View Optimisation for Collaborative Sensors. In Proc. BMVC.

[29]

P. Merrell, P. Mordohai, J. M. Frahm, and M. Pollefeys. 2007. Evaluation of Large Scale Scene Reconstruction. In Proc. ICCV. 1--8.

[30]

Pierre Moreels and Pietro Perona. 2007. Evaluation of Features Detectors and Descriptors Based on 3D Objects. IJCV 73, 3 (2007).

Digital Library

[31]

Christian Mostegel, Markus Rumpler, Friedrich Fraundorfer, and Horst Bischof. 2016. UAV-Based Autonomous Image Acquisition with Multi-View Stereo Quality Assurance by Confidence Prediction. In Proc. CVPR Workshops.

[32]

Matthias Mueller, Neil Smith, and Bernard Ghanem. 2016. A Benchmark and Simulator for UAV Tracking. In Proc. ECCV. 445--461.

[33]

Gustavo Olague and Roger Mohr. 2002. Optimal Camera Placement for Accurate Reconstruction. Pattern Recognition 35, 4 (2002).

[34]

Molly A O'Neil, Dan Tamir, and Martin Burtscher. 2011. A Parallel GPU Version of the Traveling Salesman Problem. In Proc. PDPTA.

[35]

Jeremie Papon and Markus Schoeler. 2015. Semantic Pose using Deep Networks Trained on Synthetic RGB-D. CoRR abs/1508.00835 (2015). http://arxiv.org/abs/1508.00835

Digital Library

[36]

Ken Perlin. 2002. Improving Noise. ACM Trans. Graph. 21, 3 (2002).

Digital Library

[37]

Norbert Pfeifer, Philipp Glira, and Christian Briese. 2012. Direct Georeferencing with on Board Navigation Components of Light Weight UAV Platforms. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (2012).

[38]

Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao, Zihao Xiao, Tae Soo Kim, Yizhou Wang, and Alan Yuille. 2017. UnrealCV: Virtual Worlds for Computer Vision. ACM Multimedia Open Source Software Competition (2017).

Digital Library

[39]

Stephan R. Richter, Zeeshan Hayder, and Vladlen Koltun. 2017. Playing for Benchmarks. In Proc. ICCV.

[40]

Mike Roberts, Debadeepta Dey, Anh Truong, Sudipta Sinha, Shital Shah, Ashish Kapoor, Pat Hanrahan, and Neel Joshi. 2017. Submodular Trajectory Optimization for Aerial 3D Scanning. In Proc. ICCV.

[41]

Mike Roberts and Pat Hanrahan. 2016. Generating Dynamically Feasible Trajectories for Quadrotor Cameras. ACM Trans. Graph. 35, 4 (2016).

Digital Library

[42]

German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio Lopez. 2016. The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes. In Proc. CVPR.

[43]

Markus Rumpler, Arnold Irschara, and Horst Bischof. 2011. Multi-View Stereo: Redundancy Benefits for 3D Reconstruction. In Proc. Workshop of Austrian Assoc. for Patt. Recog., Vol. 4.

[44]

Markus Rumpler, Alexander Tscharf, Christian Mostegel, Shreyansh Daftry, Christof Hoppe, Rudolf Prettenthaler, Friedrich Fraundorfer, Gerhard Mayer, and Horst Bischof. 2016. Evaluations on Multi-Scale Camera Networks for Precise and Geo-Accurate Reconstructions from Aerial and Terrestrial Images with User Guidance. Computer Vision & Image Understanding (2016).

Digital Library

[45]

Korbinian Schmid, Heiko Hirschmüller, Andreas Dömel, Iris Grixa, Michael Suppa, and Gerd Hirzinger. 2012. View Planning for Multi-View Stereo 3D Reconstruction using an Autonomous Multicopter. J. Inteligent & Robotic Sys. 65, 1--4 (2012).

Digital Library

[46]

Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR).

[47]

Thomas Schöps, Johannes L. Schönberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. 2017. A Multi-View Stereo Benchmark with High-Resolution Images and Multi-Camera Videos. In Proc. CVPR.

[48]

Steven M. Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. 2006. A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms. In Proc. CVPR, Vol. 1.

Digital Library

[49]

Ben Semerjian. 2014. A New Variational Framework for Multiview Surface Reconstruction. In Proc. ECCV.

[50]

Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. 2017. AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. In Proc. Field and Service Robotics.

[51]

Ching-Kuang Shene. 1998. Curve Global Interpolation. (1998). https://www.cs.mtu.edu/~shene/COURSES/cs3621/NOTES/INT-APP/CURVE-INT-global.html

[52]

Amarjeet Singh, Andreas Krause, and William J. Kaiser. 2009. Nonmyopic Adaptive Informative Path Planning for Multiple Robots. In IJCAI.

Digital Library

[53]

Noah Snavely, Steven M Seitz, and Richard Szeliski. 2006. Photo Tourism: Exploring Photo Collections in 3D. ACM Trans. Graph. 25, 3 (2006).

Digital Library

[54]

Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2008. Skeletal Graphs for Efficient Structure from Motion. In Proc. CVPR, Vol. 1.

[55]

Christoph Strecha, Wolfgang von Hansen, Luc van Gool, Pascal Fua, and Ulrich Thoennessen. 2008. On Benchmarking Camera Calibration and Multi-View Stereo for High Resolution Imagery. In Proc. CVPR.

[56]

W. Treible, P. Saponaro, S. Sorensen, A. Kolagunda, M. O'Neal, B. Phelan, K. Sherbondy, and C. Kambhamettu. 2017. CATS: A Color and Thermal Stereo Benchmark. In Proc. CVPR. 134--142.

[57]

J. Irving Vasquez-Gomez, L. Enrique Sucar, Rafael Murrieta-Cid, and Efrain Lopez-Damian. 2014. Volumetric Next-Best-View Planning for 3D Object Reconstruction with Positioning Error. Int. J. Adv. Robotic Sys. 11 (2014).

[58]

Michael Waechter, Mate Beljan, Simon Fuhrmann, Nils Moehrle, Johannes Kopf, and Michael Goesele. 2017. Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction. ACM Trans. Graph. 36, 1, Article 8 (Jan. 2017), 11 pages.

Digital Library

[59]

Ingo Wald. 2007. On Fast Construction of SAH-Based Bounding Volume Hierarchies. In Proc. Symp. Interactive Ray Tracing.

Digital Library

[60]

Andreas Wendel, Michael Maurer, Gottfried Graber, Thomas Pock, and Horst Bischof. 2012. Dense Reconstruction on-the-fly. In Proc. CVPR.

Digital Library

[61]

Stefan Wenhardt, Benjamin Deutsch, Elli Angelopoulou, and Heinrich Niemann. 2007. Active Visual Object Reconstruction using D-, E-, and T-Optimal Next Best Views. In Proc. CVPR.

[62]

Shihao Wu, Wei Sun, Pinxin Long, Hui Huang, Daniel Cohen-Or, Minglun Gong, Oliver Deussen, and Baoquan Chen. 2014. Quality-driven Poisson-guided Autoscanning. ACM Trans. Graph. 33, 6, Article 203 (Nov. 2014), 12 pages.

Digital Library

[63]

Ke Xie, Hao Yang, Shengqiu Huang, Dani Lischinski, Marc Christie, Kai Xu, Minglun Gong, Daniel Cohen-Or, and Hui Huang. 2018. Creating and Chaining Camera Moves for Quadrotor Videography. ACM Trans. Graph. 37, 4 (2018), 88:1--88:13.

Digital Library

Cited By

Lyu AZhao SXian CCen ZCai HFang G(2024)Accelerate Neural Subspace-Based Reduced-Order Solver of Deformable Simulation by Lipschitz OptimizationACM Transactions on Graphics10.1145/368796143:6(1-10)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687961
Trusty TFei YLevin DKaufman D(2024)Trading Spaces: Adaptive Subspace Time Integration for Contacting ElastodynamicsACM Transactions on Graphics10.1145/368794643:6(1-16)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687946
Darke AOlander IKim TBurbano AWest R(2024)More Than Killmonger Locs: A Style Guide for Black Hair (in Computer Graphics)ACM SIGGRAPH 2024 Courses10.1145/3664475.3664535(1-251)Online publication date: 27-Jul-2024
https://dl.acm.org/doi/10.1145/3664475.3664535
Show More Cited By

Index Terms

Aerial path planning for urban scene reconstruction: a continuous optimization method and benchmark
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
    2. Control methods
      1. Motion path planning

Recommendations

Learning Reconstructability for Drone Aerial Path Planning

We introduce the first learning-based reconstructability predictor to improve view and path planning for large-scale 3D urban scene acquisition using unmanned drones. In contrast to previous heuristic approaches, our method learns a model that explicitly ...
Offsite aerial path planning for efficient urban scene reconstruction

With rapid development in UAV technologies, it is now possible to reconstruct large-scale outdoor scenes using only images captured by low-cost drones. The problem, however, becomes how to plan the aerial path for a drone to capture images so that two ...
Aerial path planning for online real-time exploration and offline high-quality reconstruction of large-scale urban scenes

Existing approaches have shown that, through carefully planning flight trajectories, images captured by Unmanned Aerial Vehicles (UAVs) can be used to reconstruct high-quality 3D models for real environments. These approaches greatly simplify and cut ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 37, Issue 6

December 2018

1401 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3272127

Editor:
Takeo Igarashi
The University of Tokyo, Japan

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2018

Published in TOG Volume 37, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

68
Total Citations
View Citations
1,240
Total Downloads

Downloads (Last 12 months)134
Downloads (Last 6 weeks)31

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lyu AZhao SXian CCen ZCai HFang G(2024)Accelerate Neural Subspace-Based Reduced-Order Solver of Deformable Simulation by Lipschitz OptimizationACM Transactions on Graphics10.1145/368796143:6(1-10)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687961
Trusty TFei YLevin DKaufman D(2024)Trading Spaces: Adaptive Subspace Time Integration for Contacting ElastodynamicsACM Transactions on Graphics10.1145/368794643:6(1-16)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687946
Darke AOlander IKim TBurbano AWest R(2024)More Than Killmonger Locs: A Style Guide for Black Hair (in Computer Graphics)ACM SIGGRAPH 2024 Courses10.1145/3664475.3664535(1-251)Online publication date: 27-Jul-2024
https://dl.acm.org/doi/10.1145/3664475.3664535
Chen ALiu ZYang YYuksel C(2024)Vertex Block DescentACM Transactions on Graphics10.1145/365817943:4(1-16)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658179
Chen YHan YChen JZhang ZMcadams ATeran J(2024)Position-Based Nonlinear Gauss-Seidel for Quasistatic HyperelasticityACM Transactions on Graphics10.1145/365815443:4(1-15)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658154
Han YChen YOng CChen JHicks JTeran J(2024)A Neural Network Model for Efficient Musculoskeletal-Driven Skin DeformationACM Transactions on Graphics10.1145/365813543:4(1-12)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658135
Shen XCai RBi MLv T(2024)Preconditioned Nonlinear Conjugate Gradient Method for Real-time Interior-point HyperelasticityACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657490(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657490
Li XLi MHan XWang HYang YJiang C(2024)A Dynamic Duo of Finite Elements and Material PointsACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657449(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657449
Zhang XHu ZHu QZhao JAi MZhao PLi JZhou XChen Z(2024) A 3D urban scene reconstruction enhancement approach based on adaptive viewpoint selection of panoramic videos The Photogrammetric Record10.1111/phor.1246739:185(7-35)Online publication date: 7-Jan-2024
https://doi.org/10.1111/phor.12467
Guo JQin HZhou YChen XNan LHuang H(2024)Fast Building Instance Proxy Reconstruction for Large Urban ScenesIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.338837146:11(7267-7282)Online publication date: Nov-2024
https://doi.org/10.1109/TPAMI.2024.3388371
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents