article

Visual Modeling with a Hand-Held Camera

Authors:

Marc Pollefeys,

Maarten Vergauwen,

Frank Verbiest,

Reinhard KochAuthors Info & Claims

International Journal of Computer Vision, Volume 59, Issue 3

Pages 207 - 232

https://doi.org/10.1023/B:VISI.0000025798.50602.3a

Published: 21 September 2004 Publication History

Abstract

In this paper a complete system to build visual models from camera images is presented. The system can deal with uncalibrated image sequences acquired with a hand-held camera. Based on tracked or matched features the relations between multiple views are computed. From this both the structure of the scene and the motion of the camera are retrieved. The ambiguity on the reconstruction is restricted from projective to metric through self-calibration. A flexible multi-view stereo matching scheme is used to obtain a dense estimation of the surface geometry. From the computed data different types of visual models are constructed. Besides the traditional geometry- and image-based approaches, a combined approach with view-dependent geometry and texture is presented. As an application fusion of real and virtual scenes is also shown.

References

[1]

Beardsley, P., Zisserman, A., and Murray, D. 1997. Sequential updating of projective and affine structure from motion. International Journal of Computer Vision, 23(3):235-259.]]

Digital Library

[2]

Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. 2001. Unstructured lumigraph rendering. In Proceedings ACM SIGGRAPH 2001, pp. 425-432.]]

Digital Library

[3]

Chai, J.-X., Tong, X., Chan, S.-C., and Shum, H.-Y. 2000. Plenoptic sampling, In Proc. Siggraph, pp. 307-318.]]

Digital Library

[4]

Chum, O. and Matas, J. 2002. Randomized ransac with td. d test. In Proceedings of the British Machine Vision Conference, P. Rosin and D. Marshall (Eds.), BMVA: London, UK, vol. 2, pp. 448-457.]]

[5]

Cornelis K., Pollefeys M. Vergauwen, M., and Van Gool, L. 2001. Augmented reality from uncalibrated video sequences. In 3D Structure from Images-SMILE 2000, M. Pollefeys, L. Van Gool, A. Zisserman, and A. Fitzgibbon (Eds.), Lecture Notes in Computer Science, Springer-Verlag, vol. 2018, pp. 150-167.]]

Digital Library

[6]

Cox, I., Hingorani, S., and Rao, S. 1996. A maximum likelihood stereo algorithm. Computer Vision and Image Understanding, 63(3).]]

Digital Library

[7]

Curless, B. and Levoy, M. 1996. A volumetric method for building complex models from range images. In Proc. SIGGRAPH '96, pp. 303-312.]]

Digital Library

[8]

Debevec, P., Borshukov, G., and Yu, Y. 1998. Efficient view-dependent image-based rendering with projective texture-mapping. In 9th Eurographics Rendering Workshop, Vienna. Austria.]]

[9]

Debevec, P., Taylor, C., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proc. SIGGRAPH'96, pp. 11-20.]]

Digital Library

[10]

Falkenhagen, L. 1997. Hierarchical block-based disparity estimation considering neighbourhood constraints. In Proceedings International Workshop on SNHC and 3D Imaging, Rhodes, Greece, pp. 115-122.]]

[11]

Faugeras, O. 1992. What can be seen in three dimensions with an uncalibrated stereo rig. In Computer Vision--ECCV'92, Lecture Notes in Computer Science, Springer-Verlag, vol. 588, pp. 563- 578.]]

[12]

Faugeras, O., Luong, Q.-T., and Maybank, S. 1992. Camera self-calibration: Theory and experiments. In Computer Vision-- ECCV'92, Lecture Notes in Computer Science, Springer-Verlag, vol. 588, pp. 321-334.]]

Digital Library

[13]

Faugeras, O., Luong, Q.-T., and Papadopoulo, T. 2001. The Geometry of Multiple Images, MIT Press.]]

Digital Library

[14]

Fischler, M. and Bolles, R. 1981. Random Sampling consensus: A paradigm for model fitting with application to image analysis and automated cartography. Commun. Assoc. Comp. Mach., 24:381- 95.]]

Digital Library

[15]

Fitzgibbon, A. and Zisserman, A. 1998. Automatic camera recovery for closed or open image sequences. In Computer Vision-- ECCV'98, Lecture Notes in Computer Science, vol. 1406, Springer-Verlag, vol. 1, pp. 311-326.]]

Digital Library

[16]

Gortler, S., Grzeszczuk, R., Szeliski, R., and Cohen, M.F. 1996. The Lumigraph. In Proc. SIGGRAPH '96, ACM Press: New York, pp. 43-54.]]

Digital Library

[17]

Harris, C. and Stephens, M. 1988. A combined corner and edge detector. In Fourth Alvey Vision Conference, pp. 147-151.]]

[18]

Hartley, R. 1994. Euclidean reconstruction from uncalibrated views. In Applications of Invariance in Computer Vision, J.L. Mundy, A. Zisserman, and D. Forsyth (Eds.), Lecture Notes in Computer Science, Springer Verlag, vol. 825, pp. 237-256.]]

[19]

Hartley, R. 1997. In defense of the eight-point algorithm. IEEE Trans. on Pattern Analysis and Machine Intelligence, 19(6):580-593.]]

Digital Library

[20]

Hartley, R., Gupta, R., and Chang, T. 1992. Stereo from uncalibrated cameras. In Proc. Conference Computer Vision and Pattern Recognition, pp. 761-764.]]

[21]

Hartley, R. 1998. Chirality. International Journal of Computer Vision , 26(1):41-61.]]

Digital Library

[22]

Hartley, R. and Sturm, P. 1997. Triangulation. Computer Vision and Image Understanding, 68(2):146-157.]]

Digital Library

[23]

Hartley, R. and Zisserman, A. 2000. Multiple View Geometry in Computer Vision. Cambridge University Press.]]

Digital Library

[24]

Heigl, B., Koch, R., Pollefeys, M., Denzler, J., and Van Gool, L. 1999. Plenoptic modeling and rendering from image sequences taken by hand-held camera. In Proc. DAGM'99, pp. 94-101.]]

Digital Library

[25]

Heyden, A. and Åström, K. 1996. Euclidean reconstruction from constant intrinsic parameters. In Proc. 13th International Conference on Pattern Recognition, IEEE Computer Soc. Press, pp. 339- 343.]]

Digital Library

[26]

Heyden, A. and Åström, K. 1997. Euclidean reconstruction from image sequences with varying and unknown focal length and principal point. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, IEEE Computer Soc. Press, pp. 438-443.]]

Digital Library

[27]

Kahl, F. 1999. Critical motions and ambiuous euclidean reconstructions in auto-calibration. In Proc. ICCV, pp. 469-475.]]

[28]

Koch, R. 1996. Automatische oberflachenmodellierung starrer drei-dimensionaler Objekte aus stereoskopischen Rundum-Ansichten, PhD thesis, University of Hannover, Germany, also published as Fortschritte-Berichte VDI, Reihe 10, No. 499, VDI Verlag, 1997.]]

[29]

Koch, R., Heigl, B., Pollefeys, M., Van Gool, L., and Niemann, H. 1999a. A geometric approach to lightfield calibration. In Proc. CAIP99, Springer-Verlag, vol. LNCS 1689, pp. 596-603.]]

Digital Library

[30]

Koch, R., Pollefeys, M., and Van Gool, L., 1998. Multi viewpoint stereo from uncalibrated video sequences. In Proc. European Conference on Computer Vision, Freiburg, Germany, pp. 55-71.]]

Digital Library

[31]

Koch, R., Pollefeys, M., Heigl, B., Van Gool, L., and Niemann, H. 1999b. Calibration of hand-held camera sequences for plenoptic modeling. In Proc. ICCV'99 (International Conference on Computer Vision), Corfu (Greece), pp. 585-591.]]

[32]

Kutulakos, K.N. and Seitz, S.M, 2000. A theory of shape by space carving. International Journal of Computer Vision, 38(3):199- 218.]]

Digital Library

[33]

Laveau, S. and Faugeras, O. 1996. Oriented projective geometry for computer vision. In Computer Vision-ECCV'96, B. Buxton and R. Cipolla (Eds.), Lecture Notes in Computer Science, Springer-Verlag. vol. 1064, pp. 147-156.]]

Digital Library

[34]

Levoy, M. and Hanrahan, P. 1996. Lightfield Rendering. In Proc. SIGGRAPH '96, ACM Press: New York, pp. 31-42.]]

Digital Library

[35]

Lorensen, W. and Cline, H. 1987. Marching cubes: A high resolution 3D surface construction algorithm. Computer Graphics (Proceedings of SIGGRAPH 87), 21(4): 163-169.]]

Digital Library

[36]

Lowe, D. 1999. Object recognition from local scale-invariant features. In Proc. International Conference on Computer Vision, pp. 1150-1157.]]

Digital Library

[37]

Matei, B. and Meer, P. 2000. A general method for errors-invariables problems in computer vision. In Proc. CVPR 2000, IEEE Computer Society Conference on Computer Vision and Pattern Recognition , IEEE Computer Society Press: Los Alamitos, CA (Hilton Head Island, South Carolina, vol. 2, pp. 18-25.]]

[38]

McMillan, L. and Bishop, G. 1995. Plenoptic modeling: An image-based rendering system. In Proc. SIGGRAPH'95, pp. 39- 46.]]

Digital Library

[39]

Ofek, E., Shilat, E., Rappopport, A., and Werman, M. 1997. Highlight and reflection independent multiresolution textures from image sequences. IEEE Computer Graphics and Applications, 17(2).]]

Digital Library

[40]

Okutomi, M. and Kanade, T. 1993. A multiple-baseline stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(4):353-363.]]

Digital Library

[41]

Pollefeys, M. 1999. Self-calibration and metric 3D reconstruction from uncalibrated image sequences, Ph.D. dissertation, ESAT-PSI, K.U. Leuven.]]

[42]

Pollefeys, M., Koch, R., and Van Gool, L. 1998. Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters. In Proc. International Conference on Computer Vision, Narosa Publishing House, pp. 90-95.]]

Digital Library

[43]

Pollefeys, M., Koch, R., and Van Gool, L. 1999. A simple and efficient rectification method for general motion. In Proc. ICCV'99 (International Conference on Computer Vision). Corfu (Greece), pp. 496-501.]]

[44]

Pollefeys, M., Koch, R., and Van Gool, L. 1999. Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters. International Journal of Computer Vision, 32(1):7-25.]]

Digital Library

[45]

Pollefeys, M. and Van Gool, L. 1999. Stratified self-calibration with the modulus constraint. IEEE transactions on Pattern Analysis and Machine Intelligence, 21(8):707-724.]]

Digital Library

[46]

Pollefeys, M., Verbiest, F., and Van Gool, L. 2002. Surviving dominant planes in uncalibrated structure and motion recovery. In Computer Vision--ECCV 2002, 7th European Conference on Computer Vision, A. Heyden, G. Sparr, M. Nielsen, P. Johansen (Eds.), Lecture Notes in Computer Science, vol. 2351, pp. 837-851.]]

[47]

Press, W., Teukolsky, S., and Vetterling, W. 1992. Numerical Recipes in C: The Art of Scientific Computing, Cambridge University Press.]]

Digital Library

[48]

Rousseeuw, P. 1987. Robust Regression and Outlier Detection. Wiley: New York.]]

Digital Library

[49]

Sawhney, H., Hsu, S, and Kumar, R. 1998. Robust video mosaicing through topology inference and local to global alignment. In Computer Vision--ECCV'98: Proc. 5th European Conference on Computer Vision, Lecture Notes in Computer Science, Springer-Verlag, vol. II, pp. 103-119.]]

Digital Library

[50]

Scharstein, D., and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Internation Journal of Computer Vision, 47(1/2/3):7-42.]]

Digital Library

[51]

Schmid, C. and Mohr, R. 1997. Local grayvalue invariants for image retrieval. IEEE Transaction on Pattern Analysis and Machine Intelligence, 19(5):530-534.]]

Digital Library

[52]

Schroeder, W., Zarge, J., and Lorensen, W. 1992. Decimation of triangle meshes. Computer Graphics (Proceedings of SIGGRAPH 92), 26(2):65-70.]]

Digital Library

[53]

Shi, J. and Tomasi, C. 1994. Good features to track. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR '94), pp. 593-600.]]

[54]

Slama, C. 1980. Manual of Photogrammetry, 4th edition. American society of photogrammetry: Church, VA, USA, Falls.]]

[55]

Soucy, M. and Laurendeau, D. 1995. A general surface approach to the integration of a set of range views. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(4):344-358.]]

Digital Library

[56]

Sturm, P. 1997a. Critical motion sequences for monocular self-calibration and uncalibrated euclidean reconstruction. In Proc. 1997 Conference on Computer Vision aud Pattern Recognition, IEEE Computer Soc. Press, pp. 1100-1105.]]

Digital Library

[57]

Sturm, P. 1997b. Vision 3D non calibrée-contributions à la reconstruction projective et étude des mouvements critiques pour l'auto-calibrage. Ph.D. Thesis, Institut National Polytechnique de Grenoble, 1997.]]

[58]

Sturm, P. 1999. Critical motion sequences for the self-calibration of cameras and stereo systems with variable focal length. In Proceedings of the tenth British Machine Vision Conference, T. Pridmore and D. Elliman (Eds.), Nottingham, England, British Machine Vision Association, pp. 63-72.]]

[59]

Tomasi, C. and Kanade, T. 1992. Shape and motion from image streams under orthography: A factorization approach, International Journal of Computer Vision, 9(2): 137-154.]]

Digital Library

[60]

Torr, P. 1995. Motion Segmentation and Outlier Detection. PhD Thesis, Dept. of Engineering Science, University of Oxford.]]

[61]

Torr, P., Fitzgibbon, A., and Zisserman, A. 1998. Maintaining multiple motion model hypotheses through many views to recover matching and structure. In Proc. ICCV, pp. 485-491.]]

Digital Library

[62]

Triggs, B. 1997. The absolute quadric. In Proc. 1997 Conference on Computer Vision and Pattern Recognition, IEEE Computer Soc. Press, pp. 609-614.]]

Digital Library

[63]

Triggs, B., McLauchlan, P., Hartley, R., and Fiztgibbon, A. 2000. Bundle adjustment--A modern synthesis. In Vision Algorithms: Theory and Practice, Triggs B., Zisserman A., Szeliski R. (Eds.), Springer-Verlag, vol. LNCS 1883, pp. 298-372.]]

Digital Library

[64]

Turk, G. and Levoy, M. 1994. Zippered polygon meshes from range images. In Proceedings of SIGGRAPH '94 pp. 311-318.]]

Digital Library

[65]

Tuytelaars, T. and Van Cool, L. 2000. Wide baseline stereo based on local, affinely invariant regions. In British Machine Vision Conference , pp. 412-422.]]

[66]

Van Meerbergen, G., Vergauwen, M., Pollefeys, M., and Van Cool, L. 2002. A hierarchical symmetric stereo algorithm using dynamic programming. International Journal on Computer Vision, 47 (1-3):275-285.]]

Digital Library

[67]

Wheeler, M., Sato, Y., and Ikeuchi, K. 1998. Consensus surfaces for modeling 3D objects from multiple range images. In Sixth International Conference on Computer Vision, pp. 917-924.]]

Digital Library

[68]

Willson, R. 1994. Modeling and Calibration of Automated Zoom Lenses. Ph.D. thesis, Department of Electrical and Computer Engineering, Carnegie Mellon University.]]

Digital Library

[69]

Zhang, Z., Deriche, R., Faugeras, O., and Luong, Q.-T. 1995. A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry. Artificial Intelligence Journal, 78:87-119.]]

Digital Library

Cited By

Wu YShi LLiu HLiao HQiu LYuan WGu XDong ZCui SHan X(2024)MVImgNet2.0: A Larger-scale Dataset of Multi-view ImagesACM Transactions on Graphics10.1145/368797343:6(1-16)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687973
Zhang KBi STan HXiangli YZhao NSunkavalli KXu Z(2024)GS-LRM: Large Reconstruction Model for 3D Gaussian SplattingComputer Vision – ECCV 202410.1007/978-3-031-72670-5_1(1-19)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72670-5_1
Snavely NSeitz SSzeliski R(2023)Photo tourism: exploring photo collections in 3DSeminal Graphics Papers: Pushing the Boundaries, Volume 210.1145/3596711.3596766(515-526)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1145/3596711.3596766
Show More Cited By

Index Terms

Visual Modeling with a Hand-Held Camera
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Image and video acquisition
        Camera calibration
        Epipolar geometry
  2. Computer graphics
    1. Image manipulation
2. Theory of computation
  1. Randomness, geometry and discrete structures
    1. Random projections and metric embeddings

Recommendations

Self-calibration of omnidirectional multi-cameras including synchronization and rolling shutter

Deal with consumer 360 cameras and spherical cameras without a privileged direction.Initialize the time offsets and intrinsic parameters using monocular structure-from-motion.Start multi-camera structure-from-motion with central and global shutter ...
Evaluating the quality of light fields computed from hand-held camera images
Special issue: In memoriam Azriel Rosenfeld

Given an image sequence recorded by a hand-held camera we examine the computation of a light field without any further input data. Using structure-from-motion algorithms and optimization techniques camera motion and a 3-D reconstruction of the scene are ...
3-D Depth Reconstruction from a Single Still Image

We consider the task of 3-d depth estimation from a single still image. We take a supervised learning approach to this problem, in which we begin by collecting a training set of monocular images (of unstructured indoor and outdoor environments which ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image International Journal of Computer Vision

International Journal of Computer Vision Volume 59, Issue 3

September-October 2004

94 pages

ISSN:0920-5691

Issue’s Table of Contents

Copyright © Copyright © 2004 Kluwer Academic Publishers.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 21 September 2004

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

155
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu YShi LLiu HLiao HQiu LYuan WGu XDong ZCui SHan X(2024)MVImgNet2.0: A Larger-scale Dataset of Multi-view ImagesACM Transactions on Graphics10.1145/368797343:6(1-16)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687973
Zhang KBi STan HXiangli YZhao NSunkavalli KXu Z(2024)GS-LRM: Large Reconstruction Model for 3D Gaussian SplattingComputer Vision – ECCV 202410.1007/978-3-031-72670-5_1(1-19)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72670-5_1
Snavely NSeitz SSzeliski R(2023)Photo tourism: exploring photo collections in 3DSeminal Graphics Papers: Pushing the Boundaries, Volume 210.1145/3596711.3596766(515-526)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1145/3596711.3596766
Jang HMeuleman AKang DKim DRichardt CKim M(2022)Egocentric scene reconstruction from an omnidirectional videoACM Transactions on Graphics10.1145/3528223.353007441:4(1-12)Online publication date: 22-Jul-2022
https://dl.acm.org/doi/10.1145/3528223.3530074
Yang YChang MTu PLyu S(2021)Seeing as it happens: Real time 3D video event visualization2015 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP.2015.7351328(2875-2879)Online publication date: 9-Mar-2021
https://dl.acm.org/doi/10.1109/ICIP.2015.7351328
Zhou YZhao JLuo C(2021)A novel method for reconstructing general 3D curves from stereo imagesThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-020-01959-637:7(2009-2021)Online publication date: 1-Jul-2021
https://dl.acm.org/doi/10.1007/s00371-020-01959-6
Lin PBao YDu WQi Y(2020)Parallel Large-Scale Structure from Motion by Distributed AveragingProceedings of the 2020 4th International Conference on Electronic Information Technology and Computer Engineering10.1145/3443467.3443817(565-572)Online publication date: 6-Nov-2020
https://dl.acm.org/doi/10.1145/3443467.3443817
Yuan HMa YSheng Y(2020)Fusing IMU Data into SfM for Image-Based 3D ReconstructionAdvances in Computer Graphics10.1007/978-3-030-61864-3_19(220-232)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1007/978-3-030-61864-3_19
Martin-Brualla RPandey RBouaziz SBrown MGoldman D(2020)GeLaTO: Generative Latent Textured ObjectsComputer Vision – ECCV 202010.1007/978-3-030-58539-6_15(242-258)Online publication date: 23-Aug-2020
https://dl.acm.org/doi/10.1007/978-3-030-58539-6_15
Teng CChuo KHsieh C(2018)Reconstructing three-dimensional models of objects using a Kinect sensorThe Visual Computer: International Journal of Computer Graphics10.5555/3288062.328806934:11(1507-1523)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.5555/3288062.3288069
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents