Abstract
This paper presents Limpid Desk that supports document search on a physical desktop by making the upper layer of a document stack transparent in a projection-based mixed reality environment. A user can visually access a lower-layer document without physically removing the upper documents. This is accomplished by superimposition of cover textures of lower-layer documents on the upper documents by projected imagery. This paper introduces a method of generating projection images that make physical documents transparent. Furthermore, a touch sensing method based on thermal image processing is proposed for the system’s input interface. Areas touched by a user on physical documents can be detected without any user-worn or handheld devices. This interface allows a user to select a stack to be made transparent by a simple touch gesture. Three document search support techniques are realized using the system. User studies are conducted, and the results show the effectiveness of the proposed techniques.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
Apple Inc., http://www.apple.com/.
Microsoft Corporation, http://www.microsoft.com/.
References
Bandyopadhyay D, Raskar R, Fuchs H (2001) Dynamic shader lamps: painting on movable objects. In: Proceedings of IEEE/ACM international symposium on augmented reality (ISAR ’01), pp 207–216
Baudisch P, Gutwin C (2004) Multiblending: displaying overlapping windows simultaneously without the drawbacks of alpha blending. In: Proceedings of of ACM conference on human factors in computing systems (CHI ’04), pp 367–374
Bimber O, Raskar R (2005) Spatial augmented reality: merging real and virtual worlds. A. K. Peters Ltd, Natick
Bimber O, Iwai D, Wetzstein G, Grundhöfer A (2008) The visual computing of projector-camera systems. Comput Graph Forum 27(8):2219–2254
Bonanni L, Lee CH, Selker T (2005) Attention-based design of augmented reality interfaces. In: Proceedings of ACM conference on human factors in computing systems (CHI ’05) Extended Abstracts, pp 1228–1231
Dietz P, Leigh D (2001) DiamondTouch: a multi-user touch technology. In: Proceedings of ACM symposium on user interface software and technology (UIST ’01), pp 219–226
Grundhöfer A, Bimber O (2008) Real-time adaptive radiometric compensation. IEEE Trans Vis Comput Graph 14(1):97–108
Ho HN, Amemiya T, Ando H (2007) Revealing invisible traces of hand-object interactions with thermal vision. In: Proceedings of the 2007 inframation conference, pp 431–438
Inami M, Kawakami N, Tachi S (2003) Optical camouflage using retro-reflective projection technology. In: Proceedings of IEEE/ACM international symposium on mixed and augmented reality (ISMAR ’03), pp 348–349
Iwai D, Sato K (2005) Heat sensation in image creation with thermal vision. In: Proceedings of ACM international conference on advances in computer entertainment technology (ACE ’05), pp 213–216
Iwai D, Sato K (2006) Limpid desk: see-through access to disorderly desktop in projection-based mixed reality. In: Proceedings of ACM symposium on virtual reality software and technology (VRST ’06), pp 112–115
Iwai D, Hanatani S, Horii C, Sato K (2006) Limpid desk: transparentizing documents on real desk in projection-based mixed reality. In: Proceedings of IEEE workshop on emerging display technologies (EDT ’06), pp 30–31
Kim J, Seitz SM, Agrawala M (2004) Video-based document tracking: unifying your physical and electronic desktops. In: Proceedings of ACM symposium on user interface software and technology (UIST ’04), pp 99–107
Koike H, Sato Y, Yoshinori K (2001) Integrating paper and digital information on enhanceddesk: a method for realtime finger tracking on an augmented desk system. ACM Trans Comput Hum Interact 8(4):307–322
Lepetit V, Fua P (2006) Keypoint recognition using randomized trees. IEEE Trans Pattern Anal Mach Intell 28(9):1465–1479
Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110
Sato K, Inokuchi S (1987) Range-imaging system utilizing nematic liquid crystal mask. In: Proceedings of IEEE international conference on computer vision (ICCV ’87), pp 657–661
Seetzen H, Heidrich W, Stuerzlinger W, Ward G, Whitehead L, Trentacoste M, Ghosh A, Vorozcovs A (2004) High dynamic range display systems. In: Proceedings of ACM international conference on computer graphics and interactive techniques (SIGGRAPH ’04), pp 760–768
Siio I, Rawan J, Mynatt E (2003) Finding objects in “strata drawer”. In: Proceedings of ACM conference on human factors in computing systems (CHI ’03) Extended Abstracts, pp 982–983
Swain MJ, Ballard DH (1991) Color indexing. Int J Comput Vis 7(1):11–32
Wilson AD (2005) PlayAnywhere: a compact interactive tabletop projection-vision system. In: Proceedings of ACM symposium on user interface software and technology (UIST ’05), pp 83–92
Yasuda K, Naemura T, Harashima H (2004) Thermo-Key: human region segmentation from video. IEEE Comput Graph Appl 24(1):26–30
Yoshida T, Horii C, Sato K (2003) A virtual color reconstruction system for real heritage with light projection. In: Proceedings of international conference on virtual systems and multimedia (VSMM ’03), pp 161–168
Zollmann S, Bimber O (2007) Imperceptible calibration for radiometric compensation. In: Proceedings of eurographics (short paper), pp 61–64
Author information
Authors and Affiliations
Corresponding author
Appendix: Radiometric compensation
Appendix: Radiometric compensation
In the real world, most objects have spatially varying reflectance properties that disturb the appearance of a projected image. We apply a radiometric compensation method (Yoshida et al. 2003), so as not to decrease the visibility of the appearance content of the projected thermal image. Note that the method assumes the spectral response of the visible camera is same as one of the human eye.
The method uses the visible camera of the system to obtain an affine correlation in color space between the projection and the reflectance captured by the camera for each camera pixel. This affine correlation can transform the desired color appearance on a physical objects surface into a projected color value. When the input RGB value for the projector is represented as (I R , I G , I B ) and the captured RGB value of the visible camera is represented as (C R , C G , C B ), the correlation between them can be represented by the following equation in the affine transformation.
K is a 3 × 4 matrix that transforms a camera’s color space to that of a projector. Therefore, K is called a color mixing matrix that takes into account the projector’s spectral characteristics, the camera’s spectral sensitivity, and the spectral reflectance of the object’s surface. K has to be calibrated for each camera pixel. Once at least four correspondences between (C R , C G , C B ) and (I R , I G , I B ) are obtained, K is calculated by a least-squares method. In the calibration process, more than four simple color patterns (e.g., red, green, blue, yellow, magenta, and cyan) are projected, and the reflectance of each projected pattern is captured. After this color calibration, images of the desired color can be displayed on surfaces under consideration of their reflectance. The color of the compensated projection image (I R , I G , I B ) is calculated by (7) for each pixel of the projection image where RGB values of the desired color are assigned to (C R , C G , C B ). As described above, the color mixing matrix K can be calibrated without any prior information about the spectral characteristics of the projector, the camera, and the object’s surface.
Rights and permissions
About this article
Cite this article
Iwai, D., Sato, K. Document search support by making physical documents transparent in projection-based mixed reality. Virtual Reality 15, 147–160 (2011). https://doi.org/10.1007/s10055-010-0159-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-010-0159-5