Abstract
The recent trend of fusing complementary data from LiDARs and cameras for more accurate perception has made the extrinsic calibration between the two sensors critically important. Indeed, to align the sensors spatially for proper data fusion, the calibration process usually involves estimating the extrinsic parameters between them. Traditional LiDAR–camera calibration methods often depend on explicit targets or human intervention, which can be prohibitively expensive and cumbersome. Recognizing these weaknesses, recent methods usually adopt the autonomic targetless calibration approach, which can be conducted at a much lower cost. This paper presents a thorough review of these automatic targetless LiDAR–camera calibration methods. Specifically, based on how the potential cues in the environment are retrieved and utilized in the calibration process, we divide the methods into four categories: information theory based, feature based, ego-motion based, and learning based methods. For each category, we provide an in-depth overview with insights we have gathered, hoping to serve as a potential guidance for researchers in the related fields.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Alba M, Barazzetti L, Scaioni M, Remondino F (2012) Automatic registration of multiple laser scans using panoramic RGB and intensity images. Int Arch Photogramm Remote Sens Spat Inf Sci XXXVIII–5/W12:49–54. https://doi.org/10.5194/isprsarchives-xxxviii-5-w12-49-2011
Bai Z, Jiang G, Xu A (2020) LiDAR-camera calibration using line correspondences. Sensors 20(21):6319. https://doi.org/10.3390/s20216319
Banerjee K, Notz D, Windelen J, Gavarraju S, He M (2018) Online camera LiDAR fusion and object detection on hybrid data for autonomous driving. In: IEEE intelligent vehicles symposium (IV). IEEE. https://doi.org/10.1109/ivs.2018.8500699
Barzilai J, Borwein JM (1988) Two-point step size gradient methods. IMA J Numer Anal 8(1):141–148. https://doi.org/10.1093/imanum/8.1.141
Bay H, Tuytelaars T, Gool LV (2006) SURF: speeded up robust features. In: Computer vision – ECCV 2006. Springer, Berlin, pp 404–417. https://doi.org/10.1007/11744023_32
Belghazi MI, Baratin A, Rajeshwar S, Ozair S, Bengio Y, Courville A, Hjelm D (2018) Mutual information neural estimation. In: International conference on machine learning. PMLR, pp 531–540
Besl PJ, McKay ND (1992) Method for registration of 3-d shapes. In: Schenker PS (ed) Sensor fusion IV: control paradigms and data structures. SPIE. https://doi.org/10.1117/12.57955
Bileschi S (2009) Fully automatic calibration of LIDAR and video streams from a vehicle. In: 2009 IEEE 12th international conference on computer vision workshops, ICCV workshops. IEEE. https://doi.org/10.1109/iccvw.2009.5457439
Blaga B-C-Z, Nedevschi S (2017) Online cross-calibration of camera and LIDAR. In: 2017 13th IEEE international conference on intelligent computer communication and processing (ICCP). IEEE. https://doi.org/10.1109/iccp.2017.8117020
Böhm J, Becker S (2007) Automatic marker-free registration of terrestrial laser scans using reflectance. In: Proceedings of the 8th conference on optical 3D measurement techniques, Zurich, Switzerland, pp 9–12
CANNY J (1987) A computational approach to edge detection, pp 184–203. https://doi.org/10.1016/b978-0-08-051581-6.50024-6
Castorena J, Kamilov US, Boufounos PT (2016) Autocalibration of lidar and optical cameras via edge alignment. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE. https://doi.org/10.1109/icassp.2016.7472200
Castorena J, Puskorius GV, Pandey G (2020) Motion guided LiDAR-camera self-calibration and accelerated depth upsampling for autonomous vehicles. J Intell Robot Syst 100(3–4):1129–1138. https://doi.org/10.1007/s10846-020-01233-w
Chen X, Ma H, Wan J, Li B, Xia T (2017) Multi-view 3d object detection network for autonomous driving. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr.2017.691
Chien H-J, Klette R, Schneider N, Franke U (2016) Visual odometry driven online calibration for monocular LiDAR-camera systems. In: 2016 23rd international conference on pattern recognition (ICPR). IEEE. https://doi.org/10.1109/icpr.2016.7900068
Chou JCK, Kamel M (1991) Finding the position and orientation of a sensor on a robot manipulator using quaternions. Int J Robot Res 10(3):240–254. https://doi.org/10.1177/027836499101000305
Corsini M, Dellepiane M, Ganovelli F, Gherardi R, Fusiello A, Scopigno R (2012) Fully automatic registration of image sets on approximate geometry. Int J Comput Vis 102(1–3):91–111. https://doi.org/10.1007/s11263-012-0552-5
Cortinhal T, Tzelepis G, Aksoy EE (2020) SalsaNext: fast, uncertainty-aware semantic segmentation of LiDAR point clouds. In: Advances in visual computing. Springer, pp 207–222. https://doi.org/10.1007/978-3-030-64559-5_16
Cui Y, Chen R, Chu W, Chen L, Tian D, Li Y, Cao D (2022) Deep learning for image and point cloud fusion in autonomous driving: a review. IEEE Trans Intell Transp Syst 23(2):722–739. https://doi.org/10.1109/tits.2020.3023541
Dhall A, Chelani K, Radhakrishnan V, Krishna KM (2017) Lidar-camera calibration using 3d-3d point correspondences. arXiv preprint arXiv:1705.09785
Edelsbrunner H, Kirkpatrick D, Seidel R (1983) On the shape of a set of points in the plane. IEEE Trans Inf Theory 29(4):551–559. https://doi.org/10.1109/tit.1983.1056714
Feng D, Haase-Schutz C, Rosenbaum L, Hertlein H, Glaser C, Timm F, Wiesbeck W, Dietmayer K (2021) Deep multi-modal object detection and semantic segmentation for autonomous driving: datasets, methods, and challenges. IEEE Trans Intell Transp Syst 22(3):1341–1360. https://doi.org/10.1109/tits.2020.2972974
Förstner W, Gülch E (1987) A fast operator for detection and precise location of distinct points, corners and centres of circular features. In: Proc. ISPRS intercommission conference on fast processing of photogrammetric data, vol 6. Interlaken, pp 281–305
Geiger A, Moosmann F, Car O, Schuster B (2012) Automatic camera and range sensor calibration using a single shot. In: 2012 IEEE international conference on robotics and automation. IEEE. https://doi.org/10.1109/icra.2012.6224570
Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics: the KITTI dataset. Int J Robot Res 32(11):1231–1237. https://doi.org/10.1177/0278364913491297
González-Aguilera D, Rodríguez-Gonzálvez P, Gómez-Lahoz J (2009) An automatic procedure for co-registration of terrestrial laser scanners and digital cameras. ISPRS J Photogramm Remote Sens 64(3):308–316. https://doi.org/10.1016/j.isprsjprs.2008.10.002
Graeter J, Wilczynski A, Lauer M (2018) LIMO: Lidar-monocular visual odometry. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE. https://doi.org/10.1109/iros.2018.8594394
Guindel C, Beltran J, Martin D, Garcia F (2017) Automatic extrinsic calibration for lidar-stereo vehicle sensor setups. In: 2017 IEEE 20th international conference on intelligent transportation systems (ITSC). IEEE. https://doi.org/10.1109/itsc.2017.8317829
Guislain M, Digne J, Chaine R, Monnier G (2017) Fine scale image registration in large-scale urban LIDAR point sets. Comput Vis Image Underst 157:90–102. https://doi.org/10.1016/j.cviu.2016.12.004
Hassanein M, Moussa A, El-Sheimy N (2016) A new automatic system calibration of multi-cameras and lidar sensors. ISPRS XLI–B1:589–594. https://doi.org/10.5194/isprs-archives-xli-b1-589-2016
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr.2016.90
Hofmann S, Eggert D, Brenner C (2014) Skyline matching based camera orientation from images and mobile mapping point clouds. ISPRS Ann Photogramm Remote Sens Spat Inf Sci II–5:181–188. https://doi.org/10.5194/isprsannals-ii-5-181-2014
Horn M, Wodtko T, Buchholz M, Dietmayer K (2021) Online extrinsic calibration based on per-sensor ego-motion using dual quaternions. IEEE Robot Autom Lett 6(2):982–989. https://doi.org/10.1109/lra.2021.3056352
Hsu C-M, Wang H-T, Tsai A, Lee C-Y (2018) Online recalibration of a camera and lidar system. In: 2018 IEEE international conference on systems, man, and cybernetics (SMC). IEEE. https://doi.org/10.1109/smc.2018.00687
Huang K, Stachniss C (2017) Extrinsic multi-sensor calibration for mobile robots using the gauss-helmert model. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE. https://doi.org/10.1109/iros.2017.8205952
Hussein A, Marin-Plaza P, Martin D, de la Escalera A, Armingol JM (2016) Autonomous off-road navigation using stereo-vision and laser-rangefinder fusion for outdoor obstacles detection. In: IEEE intelligent vehicles symposium (IV). IEEE. https://doi.org/10.1109/ivs.2016.7535372
Igelbrink F, Wiemann T, Pütz S, Hertzberg J (2018) Markerless ad-hoc calibration of a hyperspectral camera and a 3d laser scanner. In: Intelligent autonomous systems, vol 15. Springer, pp 748–759. https://doi.org/10.1007/978-3-030-01370-7_58
Irie K, Sugiyama M, Tomono M (2016) Target-less camera-LiDAR extrinsic calibration using a bagged dependence estimator. In: 2016 IEEE international conference on automation science and engineering (CASE). IEEE. https://doi.org/10.1109/coase.2016.7743564
Ishikawa R, Oishi T, Ikeuchi K (2018) LiDAR and camera calibration using motions estimated by sensor fusion odometry. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE. https://doi.org/10.1109/iros.2018.8593360
Iyer G, Ram RK, Murthy JK, Krishna KM (2018) CalibNet: geometrically supervised extrinsic calibration using 3d spatial transformer networks. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE. https://doi.org/10.1109/iros.2018.8593693
Jiang J, Xue P, Chen S, Liu Z, Zhang X, Zheng N (2018) Line feature based extrinsic calibration of LiDAR and camera. In: 2018 IEEE international conference on vehicular electronics and safety (ICVES). IEEE. https://doi.org/10.1109/icves.2018.8519493
Jiang P, Osteen P, Saripalli S (2021) SemCal: semantic LiDAR-camera calibration using neural mutual information estimator. In: 2021 IEEE international conference on multisensor fusion and integration for intelligent systems (MFI). IEEE. https://doi.org/10.1109/mfi52462.2021.9591203
Jing X, Ding X, Xiong R, Deng H, Wang Y (2022) DXQ-Net: differentiable lidar-camera extrinsic calibration using quality-aware flow. arXiv preprint arXiv:2203.09385
Kelley CT (1999) Iterative methods for optimization. SIAM
Kennedy J, Eberhart R (1995) Particle swarm optimization. In: Proceedings of ICNN’95-international conference on neural networks, vol 4. IEEE, pp 1942–1948
Khurana A, Nagla KS (2021) Extrinsic calibration methods for laser range finder and camera: a systematic review. Mapan 36(3):669–690. https://doi.org/10.1007/s12647-021-00500-x
Kim A, Osep A, Leal-Taixe L (2021) EagerMOT: 3d multi-object tracking via sensor fusion. In: 2021 IEEE international conference on robotics and automation (ICRA). IEEE. https://doi.org/10.1109/icra48506.2021.9562072
Krähenbühl P, Koltun V (2011) Efficient inference in fully connected crfs with Gaussian edge potentials. Advances in neural information processing systems 24
Levenberg K (1944) A method for the solution of certain non-linear problems in least squares. Q Appl Math 2(2):164–168. https://doi.org/10.1090/qam/10666
Levinson J, Thrun S (2013) Automatic online calibration of cameras and lasers. In: Robotics: science and systems IX. Robotics: Science and Systems Foundation. https://doi.org/10.15607/rss.2013.ix.029
Li M, Chen X, Li X, Ma B, Vitányi PM (2004) The similarity metric. IEEE Trans Inf Theory 50(12):3250–3264
Li T, Fang J, Zhong Y, Wang D, Xue J (2017) Online high-accurate calibration of rgb+ 3d-lidar for autonomous driving. In: Lecture notes in computer science. Springer, pp 254–263. https://doi.org/10.1007/978-3-319-71598-8_23
Li J, Yang B, Chen C, Huang R, Dong Z, Xiao W (2018) Automatic registration of panoramic image sequence and mobile laser scanning data using semantic features. ISPRS J Photogramm Remote Sens 136:41–57. https://doi.org/10.1016/j.isprsjprs.2017.12.005
Liao Q, Liu M (2019) Extrinsic calibration of 3d range finder and camera without auxiliary object or human intervention. In: 2019 IEEE international conference on real-time computing and robotics (RCAR). IEEE. https://doi.org/10.1109/rcar47638.2019.9044146
Li-Chee-Ming J, Armenakis C, Fusion of optical and terrestrial laser scanner data. In: The (2010) Canadian geomatics conference and symposium of commission I. ISPRS Convergence in Geomatics-Shaping Canada’s Competitive Landscape, Citeseer, p 2010
Lin M, Chen Q, Yan S (2013) Network in network. arXiv preprint arXiv:1312.4400
Liu X, Deng Z, Yang Y (2018) Recent progress in semantic image segmentation. Artif Intell Rev 52(2):1089–1106. https://doi.org/10.1007/s10462-018-9641-3
Liu H, Liu Y, Gu X, Wu Y, Qu F, Huang L (2018) A deep-learning based multi-modality sensor calibration method for USV. In: 2018 IEEE fourth international conference on multimedia big data (BigMM). IEEE. https://doi.org/10.1109/bigmm.2018.8499349
Liu X, Yuan C, Zhang F (2021) Fast and accurate extrinsic calibration for multiple lidars and cameras. arXiv preprint arXiv:2109.06550
Lowe D (1999) Object recognition from local scale-invariant features. In: Proceedings of the seventh IEEE international conference on computer vision. IEEE. https://doi.org/10.1109/iccv.1999.790410
Lu X, Liu Y, Li K (2019) Fast 3d line segment detection from unorganized point cloud. arXiv preprint arXiv:1901.02532
Lv X, Wang S, Ye D (2021a) CFNet: LiDAR-camera registration using calibration flow network. Sensors 21(23):8112. https://doi.org/10.3390/s21238112
Lv X, Wang B, Dou Z, Ye D, Wang S (2021b) LCCNet: LiDAR and camera self-calibration using cost volume network. In: 2021 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW). IEEE. https://doi.org/10.1109/cvprw53098.2021.00324
Ma H, Liu K, Liu J, Qiu H, Xu D, Wang Z, Gong X, Yang S (2021a) Simple and efficient registration of 3d point cloud and image data for an indoor mobile mapping system. JOSA A 38(4):579–586. https://doi.org/10.1364/josaa.414042
Ma T, Liu Z, Yan G, Li Y (2021b) Crlf: automatic calibration and refinement based on line feature for lidar and camera in road scenes. arXiv preprint arXiv:2103.04558
Miled M, Soheilian B, Habets E, Vallet B (2016) Hybrid online mobile laser scanner calibration through image alignment by mutual information. ISPRS Ann Photogramm Remote Sens Spat Inf Sci III–1:25–31. https://doi.org/10.5194/isprsannals-iii-1-25-2016
Morel J-M, Yu G (2009) ASIFT: a new framework for fully affine invariant image comparison. SIAM J Imag Sci 2(2):438–469. https://doi.org/10.1137/080732730
Moussa W, Abdel-Wahab M, Fritsch D (2012) Automatic fusion of digital images and laser scanner data for heritage preservation. In: Progress in cultural heritage preservation. Springer, Berlin, pp 76–85. https://doi.org/10.1007/978-3-642-34234-9_8
Munoz-Banon MA, Candelas FA, Torres F (2020) Targetless camera-LiDAR calibration in unstructured environments. IEEE Access 8:143692–143705. https://doi.org/10.1109/access.2020.3014121
Mur-Artal R, Montiel JMM, Tardos JD (2015) ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans Rob 31(5):1147–1163. https://doi.org/10.1109/tro.2015.2463671
Nagy B, Benedek C (2020) On-the-fly camera and lidar calibration. Remote Sens 12(7):1137. https://doi.org/10.3390/rs12071137
Nagy B, Kovacs L, Benedek C (2019a) Online targetless end-to-end camera-LIDAR self-calibration. In: 2019 16th international conference on machine vision applications (MVA). IEEE. https://doi.org/10.23919/mva.2019.8757887
Nagy B, Kovacs L, Benedek C (2019b) SFM and semantic information based online targetless camera-LIDAR self-calibration. In: 2019 IEEE international conference on image processing (ICIP). IEEE. https://doi.org/10.1109/icip.2019.8804299
Nelder JA, Mead R (1965) A simplex method for function minimization. Comput J 7(4):308–313. https://doi.org/10.1093/comjnl/7.4.308
Nie J, Pan F, Xue D, Luo L (2021) A survey of extrinsic parameters calibration techniques for autonomous devices. In: 2021 33rd Chinese control and decision conference (CCDC). IEEE. https://doi.org/10.1109/ccdc52312.2021.9602601
Nieto JI, Monteiro ST, Viejo D (2010) 3d geological modelling using laser and hyperspectral data. In: 2010 IEEE international geoscience and remote sensing symposium. IEEE. https://doi.org/10.1109/igarss.2010.5651553
Nurunnabi A, Belton D, West G (2012) Robust segmentation in laser scanning 3d point cloud data. In: 2012 international conference on digital image computing techniques and applications (DICTA). IEEE. https://doi.org/10.1109/dicta.2012.6411672
Oishi T, Nakazawa A, Kurazume R, Ikeuchi K (2005) Fast simultaneous alignment of multiple range images using index images. In: Fifth international conference on 3-D digital imaging and modeling (3DIM’05). IEEE. https://doi.org/10.1109/3dim.2005.41
Pandey G, McBride JR, Savarese S, Eustice RM (2012) Automatic targetless extrinsic calibration of a 3d lidar and camera by maximizing mutual information. In: Twenty-sixth AAAI conference on artificial intelligence. https://doi.org/10.1609/aaai.v26i1.8379
Pandey G, McBride JR, Savarese S, Eustice RM (2014) Automatic extrinsic calibration of vision and lidar by maximizing mutual information. J Field Robot 32(5):696–722. https://doi.org/10.1002/rob.21542
Park F, Martin B (1994) Robot sensor calibration: solving AX=XB on the euclidean group. IEEE Trans Robot Autom 10(5):717–721. https://doi.org/10.1109/70.326576
Park C, Moghadam P, Kim S, Sridharan S, Fookes C (2020) Spatiotemporal camera-LiDAR calibration: a targetless and structureless approach. IEEE Robot Autom Lett 5(2):1556–1563. https://doi.org/10.1109/lra.2020.2969164
Parmehr EG, Fraser CS, Zhang C, Leach J (2014) Automatic registration of optical imagery with 3d lidar data using statistical similarity. ISPRS J Photogramm Remote Sens 88:28–40
Pascoe G, Maddern W, Newman P (2015) Direct visual localisation and calibration for road vehicles in changing city environments. In: 2015 IEEE international conference on computer vision workshop (ICCVW). IEEE. https://doi.org/10.1109/iccvw.2015.23
Peršić J, Petrović L, Marković I, Petrović I (2020) Online multi-sensor calibration based on moving object tracking. Adv Robot 35(3–4):130–140. https://doi.org/10.1080/01691864.2020.1819874
Pomerleau F, Colas F, Siegwart R, Magnenat S (2013) Comparing ICP variants on real-world data sets. Auton Robot 34(3):133–148. https://doi.org/10.1007/s10514-013-9327-2
Powell MJ (2009) The bobyqa algorithm for bound constrained optimization without derivatives. Cambridge NA Report NA2009/06, University of Cambridge, Cambridge, 26
Pusztai Z, Hajder L (2017) Accurate calibration of lidar-camera systems using ordinary boxes. In: 2017 IEEE international conference on computer vision workshops (ICCVW). IEEE. https://doi.org/10.1109/iccvw.2017.53
Quan L, Lan Z (1999) Linear n-point camera pose determination. IEEE Trans Pattern Anal Mach Intell 21(8):774–780. https://doi.org/10.1109/34.784291
Scaramuzza D, Harati A, Siegwart R (2007) Extrinsic self calibration of a camera and a 3d laser range finder from natural scenes. In: 2007 IEEE/RSJ international conference on intelligent robots and systems. IEEE. https://doi.org/10.1109/iros.2007.4399276
Schneider N, Piewak F, Stiller C, Franke U (2017) RegNet: multimodal sensor registration using deep neural networks. In: IEEE intelligent vehicles symposium (IV). IEEE. https://doi.org/10.1109/ivs.2017.7995968
Scott DW (1992) Multivariate density estimation: theory, practice and visualisation. Wiley, New York
Shannon CE (2001) A mathematical theory of communication. ACM SIGMOBILE Mobile Comput Commun Rev 5(1):3–55
Shi C, Huang K, Yu Q, Xiao J, Lu H, Xie C (2019a) Extrinsic calibration and odometry for camera-LiDAR systems. IEEE Access 7:120106–120116. https://doi.org/10.1109/access.2019.2937909
Shi S, Wang X, Li H (2019b) PointRCNN: 3d object proposal generation and detection from point cloud. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr.2019.00086
Shi J, Zhu Z, Zhang J, Liu R, Wang Z, Chen S, Liu H (2020) CalibRCNN: calibrating camera and LiDAR by recurrent convolutional neural network and geometric constraints. In: 2020 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE. https://doi.org/10.1109/iros45743.2020.9341147
Shiu Y, Ahmad S (1989) Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX=XB. IEEE Trans Robot Autom 5(1):16–29. https://doi.org/10.1109/70.88014
Sobel I, Duda R, Hart P Sobel-feldman operator
Studholme C, Hill D, Hawkes D (1999) An overlap invariant entropy measure of 3d medical image alignment. Pattern Recogn 32(1):71–86. https://doi.org/10.1016/s0031-3203(98)00091-0
Swart A, Broere J, Veltkamp R, Tan R (2011) Refined non-rigid registration of a panoramic image sequence to a LiDAR point cloud. In: Photogrammetric image analysis. Springer, Berlin, pp 73–84. https://doi.org/10.1007/978-3-642-24393-6_7
Takikawa T, Acuna D, Jampani V, Fidler S (2019) Gated-SCNN: gated shape CNNs for semantic segmentation. In: 2019 IEEE/CVF international conference on computer vision (ICCV). IEEE. https://doi.org/10.1109/iccv.2019.00533
Taylor Z, Nieto J (2012) A mutual information approach to automatic calibration of camera and lidar in natural environments. In: Australian conference on robotics and automation, pp 3–5
Taylor Z, Nieto J (2013) Automatic calibration of lidar and camera images using normalized mutual information. In: 2013 IEEE international conference on robotics and automation (ICRA). Citeseer
Taylor Z, Nieto J (2014) Parameterless automatic extrinsic calibration of vehicle mounted lidar-camera systems. In: International conference on robotics and automation: long term autonomy workshop, number October, pp 3–6
Taylor Z, Nieto J (2015) Motion-based calibration of multimodal sensor arrays. In: 2015 IEEE international conference on robotics and automation (ICRA). IEEE. https://doi.org/10.1109/icra.2015.7139872
Taylor Z, Nieto J (2016) Motion-based calibration of multimodal sensor extrinsics and timing offset estimation. IEEE Trans Rob 32(5):1215–1229. https://doi.org/10.1109/tro.2016.2596771
Taylor Z, Nieto J, Johnson D (2013) Automatic calibration of multi-modal sensor systems using a gradient orientation measure. In: 2013 IEEE/RSJ international conference on intelligent robots and systems, pp 1293–1300. IEEE
Taylor Z, Nieto J, Johnson D (2014) Multi-modal sensor calibration using a gradient orientation measure. J Field Robot 32(5):675–695. https://doi.org/10.1002/rob.21523
Teed Z, Deng J (2021) RAFT: recurrent all-pairs field transforms for optical flow (extended abstract) . In: Proceedings of the thirtieth international joint conference on artificial intelligence. International Joint Conferences on Artificial Intelligence Organization. https://doi.org/10.24963/ijcai.2021/662
Toth T, Pusztai Z, Hajder L (2020) Automatic LiDAR-camera calibration of extrinsic parameters using a spherical target. In: 2020 IEEE international conference on robotics and automation (ICRA). IEEE. https://doi.org/10.1109/icra40945.2020.9197316
Ullman S (1979) The interpretation of structure from motion. Proc R Soc Lond B 203(1153):405–426. https://doi.org/10.7551/mitpress/3877.003.0009
Unnikrishnan R, Hebert M (2005) Fast extrinsic calibration of a laser rangefinder to a camera. Robotics Institute, Pittsburgh, PA, Tech. Rep. CMU-RI-TR-05-09
Vaida A-S, Nedevschi S (2019) Automatic extrinsic calibration of LIDAR and monocular camera images. In: 2019 IEEE 15th international conference on intelligent computer communication and processing (ICCP). IEEE. https://doi.org/10.1109/iccp48234.2019.8959801
Vel’as M, Španěl M, Materna Z, Herout A (2014) Calibration of rgb camera with velodyne lidar
Vo A-V, Truong-Hong L, Laefer DF, Bertolotto M (2015) Octree-based region growing for point cloud segmentation. ISPRS J Photogramm Remote Sens 104:88–100. https://doi.org/10.1016/j.isprsjprs.2015.01.011
von Gioi RG, Jakubowicz J, Morel J-M, Randall G (2012) LSD: a line segment detector. Image Process On Line 2:35–55. https://doi.org/10.5201/ipol.2012.gjmr-lsd
Vora S, Lang AH, Helou B, Beijbom O (2020) PointPainting: sequential fusion for 3d object detection. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr42600.2020.00466
Wang R, Ferrie FP, Macfarlane J (2012) Automatic registration of mobile LiDAR and spherical panoramas. In: 2012 IEEE Computer Society conference on computer vision and pattern recognition workshops. IEEE. https://doi.org/10.1109/cvprw.2012.6238912
Wang L, Xiao Z, Zhao D, Wu T, Dai B (2018) Automatic extrinsic calibration of monocular camera and LIDAR in natural scenes. In: 2018 IEEE international conference on information and automation (ICIA). IEEE. https://doi.org/10.1109/icinfa.2018.8812555
Wang Z, Wu Y, Niu Q (2020a) Multi-sensor fusion in automated driving: a survey. IEEE Access 8:2847–2868. https://doi.org/10.1109/access.2019.2962554
Wang W, Nobuhara S, Nakamura R, Sakurada K (2020b) Soic: semantic online initialization and calibration for lidar and camera. arXiv preprint arXiv:2003.04260
Wang Y, Li J, Sun Y, Shi M (2021) A survey of extrinsic calibration of lidar and camera. In: International conference on autonomous unmanned systems. Springer, pp 933–944
Willis A, Sui Y (2009) An algebraic model for fast corner detection. In: 2009 IEEE 12th International Conference on Computer Vision. IEEE. https://doi.org/10.1109/iccv.2009.5459443
Xiao Z, Li H, Zhou D, Dai Y, Dai B (2017) Accurate extrinsic calibration between monocular camera and sparse 3d lidar points without markers. In: IEEE intelligent vehicles symposium (IV). IEEE. https://doi.org/10.1109/ivs.2017.7995755
Xu B, Jiang W, Shan J, Zhang J, Li L (2015) Investigation on the weighted RANSAC approaches for building roof plane segmentation from LiDAR point clouds. Remote Sens 8(1):5. https://doi.org/10.3390/rs8010005
Xu H, Lan G, Wu S, Hao Q (2019) Online intelligent calibration of cameras and LiDARs for autonomous driving systems. In: 2019 IEEE intelligent transportation systems conference (ITSC). IEEE. https://doi.org/10.1109/itsc.2019.8916872
Yaopeng L, Xiaojun G, Shaojing S, Bei S (2021) Review of a 3d lidar combined with single vision calibration. In: 2021 IEEE international conference on data science and computer application (ICDSCA). IEEE. https://doi.org/10.1109/icdsca53499.2021.9649726
Ye C, Pan H, Gao H (2022) Keypoint-based LiDAR-camera online calibration with robust geometric network. IEEE Trans Instrum Meas 71:1–11. https://doi.org/10.1109/tim.2021.3129882
Yoo J-S, Kim D-H, Kim G-W (2018) Improved lidar-camera calibration using marker detection based on 3d plane extraction. J Electr Eng Technol 13(6):2530–2544
Yu C, Wang J, Peng C, Gao C, Yu G, Sang N (2018) BiSeNet: bilateral segmentation network for real-time semantic segmentation. In: Computer vision – ECCV 2018. Springer, pp 334–349. https://doi.org/10.1007/978-3-030-01261-8_20
Yu H, Zhen W, Yang W, Scherer S (2020) Line-based 2-d-3-d registration and camera localization in structured environments. IEEE Trans Instrum Meas 69(11):8962–8972. https://doi.org/10.1109/tim.2020.2999137
Yuan K, Guo Z, Wang ZJ (2020) RGGNet: tolerance aware LiDAR-camera online calibration with geometric deep learning and generative model. IEEE Robot Autom Lett 5(4):6956–6963. https://doi.org/10.1109/lra.2020.3026958
Yuan C, Liu X, Hong X, Zhang F (2021) Pixel-level extrinsic self calibration of high resolution LiDAR and camera in targetless environments. IEEE Robot Autom Lett 6(4):7517–7524. https://doi.org/10.1109/lra.2021.3098923
Zhang Q, Pless R (2004) Extrinsic calibration of a camera and laser range finder (improves camera calibration) . In: 2004 IEEE/RSJ international conference on intelligent robots and systems (IROS) (IEEE Cat. No.04CH37566). IEEE. https://doi.org/10.1109/iros.2004.1389752
Zhang J, Singh S (2014) LOAM: Lidar odometry and mapping in real-time. In: Robotics: science and systems X. Robotics: Science and Systems Foundation. https://doi.org/10.15607/rss.2014.x.007
Zhang X, Zhang A, Meng X (2015) Automatic fusion of hyperspectral images and laser scans using feature points. J Sens 1–9:2015. https://doi.org/10.1155/2015/415361
Zhang W, Zhou H, Sun S, Wang Z, Shi J, Loy CC (2019) Robust multi-modality multi-object tracking. In: 2019 IEEE/CVF international conference on computer vision (ICCV). IEEE. https://doi.org/10.1109/iccv.2019.00245
Zhang X, Zhu S, Guo S, Li J, Liu H (2021) Line-based automatic extrinsic calibration of LiDAR and camera. In: 2021 IEEE international conference on robotics and automation (ICRA). IEEE. https://doi.org/10.1109/icra48506.2021.9561216
Zhao Y, Wang Y, Tsai Y (2016) 2d-image to 3d-range registration in urban environments via scene categorization and combination of similarity measurements. In: 2016 IEEE international conference on robotics and automation (ICRA). IEEE. https://doi.org/10.1109/icra.2016.7487332
Zhao G, Hu J, You S, Kuo CCJ (2021) CalibDNN: multimodal sensor calibration for perception using deep neural networks. In: Grewe LL, Blasch EP, Kadar I (eds) Signal processing, sensor/information fusion, and target recognition XXX. SPIE. https://doi.org/10.1117/12.2587994
Zhou L, Deng Z (2012) A new algorithm for computing the projection matrix between a LIDAR and a camera based on line correspondences. In: 2012 IV international congress on ultra modern telecommunications and control systems. IEEE. https://doi.org/10.1109/icumt.2012.6459706
Zhou Y, Qi H, Ma Y (2019) End-to-end wireframe parsing. In: 2019 IEEE/CVF international conference on computer vision (ICCV). IEEE. https://doi.org/10.1109/iccv.2019.00105
Zhu N, Jia Y, Ji S (2018) Registration of panoramic/fish-eye image sequence and LiDAR points using skyline features. Sensors 18(5):1651. https://doi.org/10.3390/s18051651
Zhu Y, Sapra K, Reda FA, Shih KJ, Newsam S, Tao A, Catanzaro B (2019) Improving semantic segmentation via video propagation and label relaxation. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr.2019.00906
Zhu Y, Li C, Zhang Y (2020) Online camera-LiDAR calibration with sensor semantic information. In: 2020 IEEE international conference on robotics and automation (ICRA). IEEE. https://doi.org/10.1109/icra40945.2020.9196627
Zuniga-Noel D, Ruiz-Sarmiento J-R, Gomez-Ojeda R, Gonzalez-Jimenez J (2019) Automatic multi-sensor extrinsic calibration for mobile robots. IEEE Robot Autom Lett 4(3):2862–2869. https://doi.org/10.1109/lra.2019.2922618
Zuo X, Geneva P, Lee W, Liu Y, Huang G (2019) LIC-fusion: LiDAR-Inertial-Camera Odometry. In: 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE. https://doi.org/10.1109/iros40897.2019.8967746
Acknowledgements
The work is partially supported by the 2030 National Key AI Program of China 2018AAA0100500, Guangdong Province R&D Program 2020B0909050001, Anhui Province Development and Reform Commission 2020 New Energy Vehicle Industry Innovation Development Project and 2021 New Energy and Intelligent Connected Vehicle Innovation Project, CAAI-Huawei MindSpore Open Fund, Shenzhen Yijiahe Technology R&D Co., Ltd., and Huawei Cloud Computing Technologies Co., Ltd.
Author information
Authors and Affiliations
Contributions
Had the idea for the article: XL, JJ; Performed the literature search and data analysis: XL, YX; Drafted the work: XL, YX, BW, HR; Critically revised the work: BW, YZ, JJ.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Li, X., Xiao, Y., Wang, B. et al. Automatic targetless LiDAR–camera calibration: a survey. Artif Intell Rev 56, 9949–9987 (2023). https://doi.org/10.1007/s10462-022-10317-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-022-10317-y