Photometric Image-Based Rendering for Virtual Lighting Image Synthesis

A concept named Photometric Image-Based Rendering (PIBR) is introduced for a seamless augmented reality. The PIBR is defined as Image-Based Rendering which covers appearance changes caused by the lighting condition changes, while Geometric Image-Based Rendering (GIBR) is defined as Image-Based Rendering which covers appearance changes caused by the view point changes. The PIBR can be applied to image synthesis to keep photometric consistency between virtual objects and real scenes in an arbitrary lighting condition. We analyze the conventional IBR algorithms, and formalize the PIBR in the whole IBR framework. A specific algorithm is also presented for realizing the PIBR. The photometric linearization makes a controllable framework for the PIBR, which consists of four processes; (1) separation of environmental illumination effects, (2) estimation of lighting directions, (3) separation of specular reflections and cast-shadows, and (4) linearization of self-shadows. After the photometric linearization of input images, we can synthesize any realistic images which include not only diffuse reflections but also self-shadows, cast-shadows and specular reflections. Experimental results show that realistic images can be successfully synthesized with keeping photometric consistency.