Augmented Reality Using High Fidelity Spherical Panorama with HDRI

Augmented Reality Using High Fidelity Spherical Panorama with HDRI Zi Siang See * University of Canterbury Mark Billinghurst Ș University of Canterbury Abstract This paper presents an experimental method and apparatus for producing spherical panoramas with high dynamic range imaging (HDRI). Our method is optimized for providing high fidelity augmented reality (AR) image-based environment recognition for mobile devices. Previous studies have shown that a pre-produced panorama image can be used to make AR tracking possible for mobile AR applications. However, there has been little research on determining the qualities of the source panorama image necessary for creating high fidelity AR experiences. Panorama image production can have various challenges that can result in inaccurate reproduction of images that do not allow correct virtual graphics to be registered in the AR scene. These challenges include using multiple angle photograph images that contain parallax error, nadir angle difficulty and limited dynamic range. For mobile AR, we developed a HDRI method that requires a single acquisition that extends the dynamic range from a digital negative. This approach that needs least acquisition time is to be used for multiple angles necessary for reconstructing accurately reproduced spherical panorama with sufficient luminance. Keywords: Augmented Reality, Spherical Panorama, High Dynamic Range Imaging (HDRI), Mobile AR. 1 Introduction Mobile Augmented Reality (AR) mixes a live real-world view with virtual interactive content on a mobile device. One of the key enablers for this is tracking technology, such as computer vision techniques for tracking off pre-defined markers or markerless images. There have been previous studies on using pre-produced panorama images for AR tracking [Arth et al 2011; DiVerdi et al. 2008; Langlotz et al. 2014]. However, most of these studies describe how the panorama images can be used for AR tracking, instead of specifying the method for high fidelity production of the source panorama images for mobile AR. In this paper we make the following contributions; We developed a HDRI method that requires a single acquisition which can achieve an exposure range usually obtainable by multiple exposures. This can then be used for reconstructing a high fidelity 360x360 degree seamless spherical panorama useful for mobile AR tracking. *e-mail: zisiang@reina.com.my, seezs@utar.edu.my Ș e-mail: mark.billinghurst@hitlabnz.org ș e-mail: adriancheok@mixedrealitylab.org Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s). Zi Siang See, Mark Billinghurst, Adrian David Cheok (2015). Augmented reality using high fidelity spherical panorama with HDRI. SIGGRAPH ASIA 2015 Mobile Graphics and Interactive Applications, November 02-06, 2015, Kobe, Japan. ACM 978-1-4503-3928-5/15/11. http://dx.doi.org/10.1145/2818427.2818445 Adrian David Cheok ș City University London 2 Related Work Using panorama images for AR tracking has the advantage of being able to assign virtual graphics at almost any viewable angle from the real-world scene relative to the prefixed position from where the spherical panorama was produced [Arth et al 2011; Langlotz et al. 2012]. This can result in a markerless AR tracking experience without needing to alter any real-world elements such as installing an AR marker on-site. For example, Langlotz et al. [2014] has shown that it is possible to generate a panorama in real-time on a mobile phone, track from it, and use it to register various forms of AR content such as text, 2D and 3D graphics, and audio and video annotations. Lieberknecht et al. [2009] has shown that it can be difficult to create source tracking images, or synthetic images that reproduce the real effects of real-world phenomena such as lighting, noise, motion blur, discretization, blooming or limited color depths. Image elements that influence real-time tracking results include texture richness, the texture repeatability of the objects to be tracked, the camera motion and speed, the changes of the object scale, and variations of the lighting conditions over time. The production of panorama image for image-based AR needs to address different uncertainties and potential errors [Arth et al 2011; Ventura and Höllerer 2013; Wagner et al. 2010; Langlotz et al. 2012]. These may include insufficient dynamic range, parallax error, inconsistent white balance, nadir angle difficulty, inconsistent lighting distribution across multiple angles of the same scene, and ghosting errors due to moving objects in same scene. Researchers have shown that handheld construction of panorama source images is superior in terms of convenience and performance [DiVerdi et al. 2008; Wagner et al. 2010; Langlotz et al. 2012]; however, in this case it may be extremely difficult to totally avoid human-handling difficulties such as camera motion drifting, exposure consistency and parallax error. Normal panorama images may not provide adequate color depth information for image-based AR tracking during high contrast lighting scenarios. This limitation can be overcome through the use of high dynamic range imaging (HDRI) using large numbers of multiple exposures [Reinhard et al. 2010]. However use of HDRI for panorama creation can increase the tendency of uncertainties and potential errors during the process. Hence, our research explores solutions for creating high fidelity spherical HDRI panoramas suitable for mobile AR. 3 Method and Apparatus Creating interactive panoramas is possible by using spherical panoramas reproduced from multiple angle images [Chen 1995; Jacobs 2004; Felinto et al 2012]. This section describes a method for creating spherical panoramas based on HDRI for high fidelity mobile AR. The ideal reproduction of panorama images for AR involves reproducing the user’s position and visual information similar to the real-world objects and environment that is to be augmented. In our approach, the panoramic image creation can be facilitated with consistently photographed multiple angle HDRI images, large resolution, accurate geometrical registration of objects, and images free from visual abnormality. (a) (b) Zenith Horizontal angles Additional 3 nadir acquisition Pixel value The first contribution that we made is to create an experimental camera mounting system for image-capture that will capture fully immersive panorama images. Figure 1(a) shows the generic configuration of multiple angle images that are stitched together for spherical panorama photographic reproduction. Normally the nadir angle acquisition is difficult because the camera mounting prevents capturing images from the bottom. Usually, hand-held acquisition is not an adequate method to overcome this as it provokes drifting, camera shake and parallax error. Figure 1(b) demonstrates a hardware configuration we have proposed that supports multiple angle image acquisition. This configuration acquires three additional nadir images at 120 degrees each, then followed by another nadir image acquired without including the base of the camera mounting as shown in figure 2. This configuration captures 360 degrees of multiple angle images (including nadir images) with high authenticity and maximum resolution, and it is free from acquisition error for seamless spherical panorama reproduction. Sequences of extendable dynamic range from single RAW Figure 3: Verify extendable dynamic range from RAW acquisition. Figure 4 describe a method for multiple angle HDRI reproduction for spherical panorama, optimized for high fidelity mobile AR. The HDRI is reproduced from a single acquired RAW image having its dynamic range extended from RAW, instead of using multiple exposures. The HDRI reproduced from a single acquisition avoids the obstacles and issues that occur in HDRI reproduced from multiple exposures. This provides an outcome with zero ghosting error, zero misalignment issue and minimum acquisition time. Proposed configuration, multiple angle images acquisition Nadir image acquired without including the base of the camera Stable mounting is required for avoiding drifting, shake and parallax error. Figure 1: (a) Generic configuration. (b) Proposed configuration for multiple angle images. RAW processing, EV parameter, extending dynamic range Extended EV (range -1ev to -4ev) 0EV (neutral /native) Extended EV (range +1ev to +4ev) Allow manual selection of extended EV range for darker /brighter scene Masking on the expanded (example -2 EV) layer on top of the neutral (0EV) layer Masking on the expanded (example +2 EV) layer on top of the neutral (0EV) layer Nadir acquisition without obstacle, having sufficient space for obtaining visual information. monotonic mask from RGB channel monotonic mask from RGB channel (inverted) Global reproduction Global reproduction Figure 2: Camera mounting system that allows nadir acquisition. Next we developed a HDRI method that uses a single acquisition that extends the dynamic range in the digital negative to be used for multiple angles for spherical panorama creation. Figure 3 shows the potential extendable dynamic range that can be obtained from single acquired RAW format digital negative. Usually a single acquired low dynamic range (LDR) image has approximately 8.5EV, which is labeled as 0EV for native EV. Pixel values usually have an RGB range of 0-255, 0 indicates total darkness and 255 shows brightest value. The example in figure 3 was calibrated for the Nikon D3X camera and used Capture One Pro v7 software for RAW processing. Multiple images that contain extended dynamic range processed from a single acquired RAW are combined together to produce a HDRI with 12.5EV~14.5EV, compared to 8.5EV from a single LDR image. Apply Image: 50% opacity on top of each layer. (can be any sequence) Allow manual opacity of HDRI on top of LDRI Blended result consistent for multiple angles, HDRI expanded from single source of RAW. Figure 4: Proposed multiple angles HDRI for high fidelity AR. The intended scenario in figure 4 takes the least amount of time for HDRI acquisition as it requires only one shot for each angle. This optimization allows for manual control of the extended dynamic range for shadow and highlights on the AR content creation stage, approximating real-world lighting phenomenon. With a 16 mm fisheye lens, figure 5 shows a complete set of multiple angle HDRI, including the additional nadir acquisition as indicated in figure 1(b). The post-processed images demonstrate a reliable source of multiple angle HDRI, ideal for panorama image stitching reconstruction with least visual abnormality. Figure 7: AR view for environment recognition. Figure 5: multiple angle images which include additional nadir. Figure 6(a) demonstrates successful reproduction examples of near-error-free spherical panorama created with our method, and figure 6(b) shows the converted cubic facade projection. The spherical panorama outcome is optimized and reproduced with an authentic nadir angle and is free from parallax error. The advantage of using HDRI in multiple angles optimized for mobile AR, is that it provides visual information of extended luminance in shadow and highlights from the real-world scene. This enhances the possibility of matching with a high contrast lighting situation during an AR experience. (a) 4 Discussion One of the important outcomes of our approach is that it produces high quality panorama imagery from the beginning, instead of needing computational compensation or correction in postproduction. High fidelity reproduction of the source content is ideal for scenarios such as showrooms, museums, heritage sites and architectural subjects where authentic reproduction of panorama content is essential. Our approach has been found to be most practical in location-based scenarios that have sufficient luminosity. This is mainly limited to the current dynamic range capability of the auxiliary camera on the mobile phone that captures image samples for matching with augmentation. During our tests, an AR browser using a mobile phone with an auxiliary camera did not perform effectively in locations with low light. In such situations the sampling images may show an insufficient dynamic range and digital noise. There are many unresolved issues to be looked into in future studies, such as managing occlusion. Heat and battery challenges are critical concerns for AR experience using a mobile device. Computer vision based AR tracking may result in fast battery consumption and heat generation on the device. Therefore, it can be reasonable to consider different approaches to have robust AR markerless tracking with lower computational requirements. (b) The following observations summarize the potential obstacles and issues, and the advantages of the solution obtained by our method. Potential obstacles and issues (before optimization):  Parallax error (especially during hand-held)  Unstable image acquisition, drifting motion (hand-held)  Nadir angle difficulty  Compromised geometrical registration in image  Low dynamic range image (LDRI)  High dynamic range requires multiple exposures  Long acquisition time if multiple exposures  High dynamic range ghosting with moving objects  Multiple exposures misalignment  Inconsistent lighting distribution for multiple angles Figure 6: (a) Spherical panorama facilitated with HDRI using proposed method and apparatus. (b) Cubic facade projection. Figure 7 shows the AR view produced using the panorama source content generated from our method and making image-based environment recognition possible for a mobile AR user experience. Augmentations can be assigned onto any cubic facade during the authoring process and scene objects can be worked with high flexibility. Benefit of using our method and apparatus:  Free from parallax error  Stable image acquisition  Nadir angle with near-error-free authenticity  Accurate geometrical registration in image  High dynamic range image (HDRI)  High dynamic range from single RAW acquisition  High dynamic range with least acquisition time  High dynamic range with no ghosting  High dynamic range with perfect alignment  Consistent high dynamic range for multiple angles Our approach covers 360x360 degrees and needs only 25~30 seconds to acquire the source images for reconstructing a seamless spherical HDRI panorama. Typically panorama creation that requires multiple exposures for HDRI may require more than 10 minutes for a similar acquisition scenario. Figures 8(a), (b) and (c) show a field test of the method. Figure 8 show the accurate AR image overlay and consistent tracking performance of a markerless mobile AR experience using the panorama capture solution described in this study. The auxiliary camera brightness of the mobile device capturing the real-world scene is changing with the panning motion of the AR user. The mobile phone’s camera image is brighter when the viewed scene is dark, and turns darker when the viewed scene is bright. For example, when the user panned the AR browser towards the sky, the camera shows darker image sampling for AR, as shown in figure 8(b) without maintaining adequate and sufficient dynamic range in the area of buildings. (a) 6 Conclusions In this paper we have described an experimental method for capturing spherical panoramas facilitated with HDRI, and providing high fidelity AR image-based environment recognition. The research outcome is ideally adaptable for working with mobile devices and wearable computers. In the future we will conduct more extensive evaluation studies to compare the tracking accuracy with the systems using our panorama images to other more traditional approaches. We will also explore other solutions suitable for HDRI panorama video and hybrid approaches that combine panorama image tracking and sensor input. References ARTH, C., KLOPSCHITZ, M., REITMAYR, G., AND SCHMALSTIEG, D. 2011. Real-Time Self-Localization From Panoramic Images on Mobile Devices, International Symposium on Mixed And Augmented Reality. CHEN, E. 1995. Quicktime VR - An Image-based Approach to Virtual Environment Navigation. ACM SIGGRAPH 1995. JACOBS, C. 2004. Interactive Panorama. Springer. LIEBERKNECHT, S., BENHIMANE, S., MEIER, P., NAVAB, N. 2009. A Dataset and Evaluation Methodology for Template-based Tracking Algorithms. International Symposium on Mixed And Augmented Reality. (b) REINHARD, E., HEIDRICH, W., DEBEVEC, P., PATTANAIK, S., WARD, G., AND MYSZKOWSKI, K. 2010. High Dynamic Range Imaging (2nd Edition), Acquisition, Display, and Image-based lighting. ELSEVIER. DIVERDI, S. WITHER, J. AND HOLLERER, T. 2008. ENVISOR: ONLINE ENVIRONMENT MAP CONSTRUCTION FOR MIXED REALITY. PROCEEDINGS OF THE IEEE VIRTUAL REALITY CONFERENCE, 19-26. VENTURA, J., AND HÖLLERER, T. 2013. Structure and motion in urban environments using upright panoramas, Virtual Reality 17, 2, 147-156. (c) WAGNER, D. MULLONI, A. LANGLOTZ, T. AND SCHMALSTIEG, D. 2010. REAL-TIME PANORAMIC MAPPING AND TRACKING ON MOBILE PHONES. IEEE VIRTUAL REALITY CONFERENCE, 211-218. LANGLOTZ, T. WAGNER, D. MULLONI, A. AND SCHMALSTIEG, D. 2012. ONLINE CREATION OF PANORAMIC AUGMENTED REALITY ANNOTATIONS ON MOBILE PHONES, IEEE PERVASIVE COMPUTING 11, 2, 56-63. Figure 8: Field test scenario of the method. 5 Implications A high fidelity spherical panorama with HDRI can provide a nearerror-free and dynamic range enhanced source of image-based tracking content for markerless AR. The tracking content is reproduced with little distortion, producing a result very similar to the original scene condition. High feature-matching AR content can operate dynamically in a real-world environment without using any visible marker, and it can work without using extra sensors such as a GPS. This allows an AR experience to be delivered on a mobile device with a low processing requirement. LANGLOTZ, T. NGUYEN, T. SCHMALSTIEG, D. AND GRASSET, R. 2014. NEXT-GENERATION AUGMENTED REALITY BROWSERS: RICH, SEAMLESS, AND ADAPTIVE. PROCEEDINGS OF THE IEEE 102, 2, 155-169. FELINTO, D., ZANG, A.R., AND VELHO, L. 2012. Production Framework for Full Panoramic Scenes with Photorealistic Augmented Reality. Latin American Conference of Informatics.