research-article

Vision and Inertial Sensor Cooperation Using Gravity as a Vertical Reference

Authors:

Jorge DiasAuthors Info & Claims

IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 25, Issue 12

Pages 1597 - 1608

https://doi.org/10.1109/TPAMI.2003.1251152

Published: 01 December 2003 Publication History

Abstract

Abstract--This paper explores the combination of inertial sensor data with vision. Visual and inertial sensing are two sensory modalities that can be explored to give robust solutions on image segmentation and recovery of 3D structure from images, increasing the capabilities of autonomous robots and enlarging the application potential of vision systems. In biological systems, the information provided by the vestibular system is fused at a very early processing stage with vision, playing a key role on the execution of visual movements such as gaze holding and tracking, and the visual cues aid the spatial orientation and body equilibrium. In this paper, we set a framework for using inertial sensor data in vision systems, and describe some results obtained. The unit sphere projection camera model is used, providing a simple model for inertial data integration. Using the vertical reference provided by the inertial sensors, the image horizon line can be determined. Using just one vanishing point and the vertical, we can recover the camera's focal distance and provide an external bearing for the system's navigation frame of reference. Knowing the geometry of a stereo rig and its pose from the inertial sensors, the collineation of level planes can be recovered, providing enough restrictions to segment and reconstruct vertical features and leveled planar patches.

References

[1]

H. Carpenter, Movements of the Eyes. second ed. London Pion Limited, 1988.

[2]

A. Berthoz, The Brain's Sense of Movement. Harvard Univ. Press, 2000.

[3]

K.K. Gillingham and F.H. Previc, Spatial Orientation in Flight. second ed. chapter 11, Williams and Wilkins, 1996.

[4]

R.O. Eason and R.C. Gonzalez, “Least-Squares Fusion of Multisensory Data,” Data Fusion in Robotics and Machine Intelligence, M.A. Abidi and R.C. Gonzalez, eds., chapter 9, Academic Press, 1992.

[5]

R.P.G. Collinson, Introduction to Avionics. Chapman & Hall, 1996.

[6]

G.R. Pitman, Inertial Guidance. John Wiley & Sons, 1962.

[7]

J.J. Allen R.D. Kinney J. Sarsfield M.R. Daily J.R. Ellis J.H. Smith S. Montague R.T. Howe B.E. Boser R. Horowitz A.P. Pisano M.A. Lemkin W.A. Clark and T. Juneau, “Integrated Micro-Electro-Mechanical Sensor Development for Inertial Applications,” Proc. Position Location and Navigation Symposium, Apr. 1998.

[8]

A.C. Luther, Video Camera Technology. Artech House Publishers, 1998.

[9]

T. Viéville and O.D. Faugeras, “Computation of Inertial Information on a Robot,” Proc. Fifth Int'l Symp. Robotics Research, pp. 57-65, H. Miura and S. Arimoto, eds., MIT Press, 1989.

Digital Library

[10]

T. Viéville and O.D. Faugeras, “Cooperation of the Inertial and Visual Systems,” Traditional and NonTraditional Robotic Sensors, pp. 339-350. T.C. Henderson, ed., Springer Verlag, 1990.

Digital Library

[11]

T. Viéville F. Romann B. Hotz H. Mathieu M. Buffa L. Robert P.E.D.S. Facao O. Faugeras and J.T. Audren, “Autonomous Navigation of a Mobile Robot Using Inertial and Visual Cues,” Intelligent Robots and Systems, M. Kikode, T. Sato, and K. Tatsuno, eds., 1993.

[12]

T. Viéville E. Clergue and P.E.D. Facao, “Computation of Ego-Motion and Structure from Visual and Inertial Sensor Using the Vertical Cue,” Proc. Int'l Conf. Computer Vision, pp. 591-598, 1993.

[13]

T. Viéville, A Few Steps Towards 3D Active Vision. Springer-Verlag, 1997.

Digital Library

[14]

B. Bhanu B. Roberts and J. Ming, “Inertial Navigation Sensor Integrated Motion Analysis for Obstacle Detection,” Proc. IEEE Int'l Conf. Robotics and Automation, pp. 954-959, 1990.

[15]

F. Panerai and G. Sandini, “Visual and Inertial Integration for Gaze Stabilization,” Proc. Int'l Symp. Intelligent Robotic Systems, 1997.

[16]

F. Panerai and G. Sandini, “Oculo-Motor Stabilization Reflexes: Integration of Inertial and Visual Information,” Neural Networks, vol. 11,nos. 7-8, pp. 1191-1204, 1998.

Digital Library

[17]

F. Panerai G. Metta and G. Sandini, “Visuo-Inertial Stabilization in Space-Variant Binocular Systems,” Robotics and Autonomous Systems, vol. 30, nos. 1-2, pp. 195-214, 2000.

[18]

T. Mukai and N. Ohnishi, “The Recovery of Object Shape and Camera Motion Using a Sensing System with a Video Camera and a Gyro Sensor,” Proc. Seventh Int'l Conf. Computer Vision, pp. 411-417, Sept. 1999.

[19]

T. Mukai and N. Ohnishi, “Object Shape and Camera Motion Recovery Using Sensor Fusion of a Video Camera and a Gyro Sensor,” Information Fusion, vol. 1, no. 1, pp. 45-53, 2000.

[20]

R. Kurazume and S. Hirose, “Development of Image Stabilization System for Remote Operation of Walking Robots,” Proc. IEEE Int'l Conf. Robotics and Automation, pp. 1856-1860, Apr. 2000.

[21]

S.R. Coorg, “Pose Imagery and Automated Three-Dimensional Modeling of Urban Environments,” PhD thesis, Massachusetts Inst. of Technology, Sept. 1998.

Digital Library

[22]

S. You U. Neumann and R. Azuma, “Hybrid Inertial and Vision Tracking for Augmented Reality Registration,” Proc. IEEE Virtual Realiy Conf., pp. 260-267, Mar. 1999.

Digital Library

[23]

W.A. Hoff K. Nguyen and T. Lyon, “Computer Vision-Based Registration Techniques for Augmented Reality,” Proc. Conf. Intelligent Robots and Computer Vision, pp. 538-548, Nov. 1996.

[24]

E.D. Dickmanns, “Vehicles Capable of Dynamic Vision: A New Breed of Technical Beings?” Artificial Intelligence, vol. 103, pp. 49-76, 1998.

Digital Library

[25]

J. Lobo and J. Dias, “Integration of Inertial Information with Vision towards Robot Autonomy,” Proc. IEEE Int'l Symp. Industrial Electronics, pp. 825-830, July 1997.

[26]

J. Lobo L. Marques J. Dias U. Dias and A.T. de Almeida, Sensors for Mobile Robot Navigation, pp. 50-81. Springer-Verlag, 1998.

[27]

J. Lobo and J. Dias, “Ground Plane Detection Using Visual and Inertial Data Fusion,” Proc. IEEE/RSJ Int'l Conf. Intelligent Robots and Systems, pp. 912-917, Oct. 1998.

[28]

J. Lobo C. Queiroz and J. Dias, “Vertical World Feature Detection and Mapping Using Stereo Vision and Accelerometers,” Proc. Ninth Int'l Symp. Intelligent Robotic Systems, pp. 229-238, July 2001.

[29]

J. Lobo and J. Dias, “Fusing of Image and Inertial Sensing for Camera Calibration,” Proc. Int'l Conf. Multisensor Fusion and Integration for Intelligent Systems, pp. 103-108, Aug. 2001.

[30]

J. Lobo, “Inertial Sensor Data Integration in Computer Vision Systems,” MS thesis, Univ. of Coimbra, Apr. 2002.

[31]

K. Kanatani, Geometric Computation for Machine Vision. Oxford Univ. Press, 1993.

Digital Library

[32]

M.D. Shuster, “The Kinematic Equation for the Rotation Vector,” IEEE Trans. Aerospace and Electronic Systems, vol. 29, no. 1, pp. 263-267, Jan. 1993.

[33]

P.G. Savage, Strapdown System Algorithms. chapter 3, pp. 1-30, AGARD: Advisory Group for Aerospace Research and Development, 1984.

[34]

M.J. Caruso T. Bratland C.H. Smith and R. Schneider, “A New Perspective on Magnetic Field Sensing,” technical report, Honeywell, Inc., 1998.

[35]

Summit Instruments, http://www.summitinstruments.com, 2003.

[36]

Analog Devices, Mems Integrated Micro-Electromechanical Systems, Analog Devices, iMEMS @ http://www.analog.com/, 2003.

[37]

B.K.P Horn, “Closed-Form Solution of Absolute Orientation Using Unit Quaternions,” J. Optical Soc. Am., vol. 4, no. 4, pp. 629-462, Apr. 1987.

[38]

J. Alves J. Lobo and J. Dias, “Camera-Inertial Sensor Modelling and Alignment for Visual Navigation,” Proc. 11th Int'l Conf. Advanced Robotics, pp. 1693-1698, July 2003.

[39]

J. Dias C. Paredes I. Fonseca and A.T. de Almeida, “Simulating Pursuit with Machines,” Proc. IEEE Conf. Robotics and Automation, pp. 472-477, 1995.

[40]

J. Dias C. Paredes I. Fonseca H. Araujo J. Baptista and A.T. de Almeida, “Simulating Pursuit with Machine Experiments with Robots and Artificial Vision,” IEEE Trans. Robotics and Automation, vol. 3,no. 1, pp. 1-18, Feb. 1998.

[41]

R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge Univ. Press, 2000.

Digital Library

[42]

L.-L. Wang and W.-H. Tsai, “Camera Calibration by Vanishing Lines for 3-D Computer Vision,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 4, pp. 370-376, Apr. 1991.

Digital Library

[43]

B. Caprile and V. Torre, “Using Vanishing Points for Camera Calibration,” Proc. Int'l J. Computer Vision, vol. 4, no. 2, pp. 127-140, 1990.

Digital Library

[44]

B. Brillault and B. O'Mahony, “New Method for Vanishing Point Detection,” Computer Vision, Graphics and Image Processing: Image Understanding, vol. 54, no. 2, pp. 289-300, 1991.

Digital Library

[45]

M. Li, “Camera Calibration of the Kth Head-Eye System,” Proc. European Conf. Computer Vision, pp. 543-554, 1994.

Digital Library

[46]

R.G. Willson and S.A. Shafer, “What is the Center of the Image,” J. Optical Soc. Am. A, vol. 11, no. 11, pp. 2946-2955, 1994.

[47]

B. Jahne, Digital Image Processing. Springer-Verlag, 1997.

[48]

Intel, Intel Open Source Computer Vision Library, http://www.intel.com/research/mrl/research/opencv/, 2003.

[49]

Z. Zhang, “Flexible Camera Calibration By Viewing a Plane From Unknown Orientations,” Proc. Seventh Int'l Conf. Computer Vision, pp. 666-673, Sept. 1999.

[50]

S.M. Smith and J.M. Brady, “SUSAN-A New Approach to Low Level Image Processing,” Int'l J. Computer Vision, vol. 23, no. 1, pp. 45-78, May 1997.

Digital Library

[51]

J. O'Rourke, Computational Geometry in C. Cambridge Univ. Press, 1993.

Digital Library

[52]

A.L. Ames D.R. Nadeau and J.L. Moreland, VRML 2.0 Sourcebook. second ed. John Wiley and Sons, 1997.

Digital Library

[53]

J. Lobo C. Queiroz and J. Dias, “World Feature Detection and Mapping Using Stereovision and Inertial Sensors,” Robotics and Autonomous Systems, vol. 44, no. 1, pp. 69-81, July 2003.

[54]

J. Lobo L. Almeida and J. Dias, “Segmentation of Dense Depth Maps Using Inertial Data: A Real-Time Implementation,” Proc. IEEE/RSJ Int'l Conf. Intelligent Robots and Systems, pp. 92-97, Oct. 2002.

[55]

J. Lobo and J. Dias, “Inertial Sensed Ego-Motion for 3D Vision,” Proc. InerVis Workshop, 11th Int'l Conf. Advanced Robotics, pp. 1907-1914, July 2003.

Cited By

Bi CXing G(2019)RAMTProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/33289093:2(1-21)Online publication date: 21-Jun-2019
https://dl.acm.org/doi/10.1145/3328909
Huang ZGu NHao JShen J(2018)3DLocProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/31614091:4(1-26)Online publication date: 8-Jan-2018
https://dl.acm.org/doi/10.1145/3161409
Islam BIslam MNirjon SMottola LGao JZhang P(2018)Glimpse.3DProceedings of the 17th ACM/IEEE International Conference on Information Processing in Sensor Networks10.1109/IPSN.2018.00046(176-187)Online publication date: 11-Apr-2018
https://dl.acm.org/doi/10.1109/IPSN.2018.00046
Show More Cited By

Index Terms

Vision and Inertial Sensor Cooperation Using Gravity as a Vertical Reference
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision

Recommendations

Human Pose Tracking in Monocular Sequence Using Multilevel Structured Models

Tracking human body poses in monocular video has many important applications. The problem is challenging in realistic scenes due to background clutter, variation in human appearance and self-occlusion. The complexity of pose tracking is further ...
Visual-Inertial Sensor Fusion: Localization, Mapping and Sensor-to-Sensor Self-calibration

Visual and inertial sensors, in combination, are able to provide accurate motion estimates and are well suited for use in many robot navigation tasks. However, correct data fusion, and hence overall performance, depends on careful calibration of the ...
Stereo vision using two PTZ cameras

The research of traditional stereo vision is mainly based on static cameras. As PTZ (Pan-Tilt-Zoom) cameras are able to obtain multi-view-angle and multi-resolution information, they have received more and more concern in both research and real ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Pattern Analysis and Machine Intelligence

IEEE Transactions on Pattern Analysis and Machine Intelligence Volume 25, Issue 12

December 2003

177 pages

ISSN:0162-8828

Issue’s Table of Contents

Copyright © Copyright © 2003 IEEE. All Rights Reserved.

Publisher

IEEE Computer Society

United States

Publication History

Published: 01 December 2003

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bi CXing G(2019)RAMTProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/33289093:2(1-21)Online publication date: 21-Jun-2019
https://dl.acm.org/doi/10.1145/3328909
Huang ZGu NHao JShen J(2018)3DLocProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/31614091:4(1-26)Online publication date: 8-Jan-2018
https://dl.acm.org/doi/10.1145/3161409
Islam BIslam MNirjon SMottola LGao JZhang P(2018)Glimpse.3DProceedings of the 17th ACM/IEEE International Conference on Information Processing in Sensor Networks10.1109/IPSN.2018.00046(176-187)Online publication date: 11-Apr-2018
https://dl.acm.org/doi/10.1109/IPSN.2018.00046
Saurer OVasseur PBoutteau RDemonceaux CPollefeys MFraundorfer F(2017)Homography Based Egomotion Estimation with a Common DirectionIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2016.254566339:2(327-341)Online publication date: 1-Feb-2017
https://dl.acm.org/doi/10.1109/TPAMI.2016.2545663
Wang MTayebi A(2017)Globally asymptotically stable hybrid observers design on SE (3)2017 IEEE 56th Annual Conference on Decision and Control (CDC)10.1109/CDC.2017.8264101(3033-3038)Online publication date: 12-Dec-2017
https://dl.acm.org/doi/10.1109/CDC.2017.8264101
Carlier ACalvet LNguyen DOoi WGurdjos PCharvillat VHua KRui YSteinmetz RHanjalic ANatsev AZhu W(2014)3D Interest Maps From Simultaneous Video RecordingsProceedings of the 22nd ACM international conference on Multimedia10.1145/2647868.2654947(577-586)Online publication date: 3-Nov-2014
https://dl.acm.org/doi/10.1145/2647868.2654947
Xu DWang HLi YTan M(2012)A new calibration method for an inertial and visual sensing systemInternational Journal of Automation and Computing10.1007/s11633-012-0648-y9:3(299-305)Online publication date: 1-Jun-2012
https://dl.acm.org/doi/10.1007/s11633-012-0648-y
Yang KYu WJi X(2012)Rotation estimation for mobile robot based on single-axis gyroscope and monocular cameraInternational Journal of Automation and Computing10.1007/s11633-012-0647-z9:3(292-298)Online publication date: 1-Jun-2012
https://dl.acm.org/doi/10.1007/s11633-012-0647-z
Kalantari MHashemi AJung FGuedon J(2011)A New Solution to the Relative Orientation Problem Using Only 3 Points and the Vertical DirectionJournal of Mathematical Imaging and Vision10.1007/s10851-010-0234-239:3(259-268)Online publication date: 1-Mar-2011
https://dl.acm.org/doi/10.1007/s10851-010-0234-2
Aliakbarpour HAlmeida LMenezes PDias J(2011)Multi-sensor 3D volumetric reconstruction using CUDA3D Research10.1007/3DRes.04(2011)62:4(1-14)Online publication date: 1-Dec-2011
https://dl.acm.org/doi/10.1007/3DRes.04%282011%296
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents