Photometric Stereo by Hemispherical Metric Embedding

Photometric stereo methods seek to reconstruct the 3d shape of an object from motionless images obtained with varying illumination. Most existing methods solve a restricted problem where the physical reflectance model, such as Lambertian reflectance, is known in advance. In contrast, we do not restrict ourselves to a specific reflectance model. Instead, we offer a method that works on a wide variety of reflectances. Our approach uses a simple yet uncommonly used property of the problem— the sought after normals are points on a unit hemisphere. We present a novel embedding method that maps pixels to normals on the unit hemisphere. Our experiments demonstrate that this approach outperforms existing manifold learning methods for the task of hemisphere embedding. We further show successful reconstructions of objects from a wide variety of reflectances including smooth, rough, diffuse and specular surfaces, even in the presence of significant attached shadows. Finally, we empirically prove that under these challenging settings we obtain more accurate shape reconstructions than existing methods.

Authors and Affiliations

1. Weizmann Institute of Science, Rehovot, Israel

   Ofer Bartal, Nati Ofir, Yaron Lipman & Ronen Basri

Corresponding author

Correspondence to Nati Ofir.

Appendix

Our proofs rely on a spherical harmonic representation of reflectance. Below we introduce notation, which for the most part follows the notation of [7]. Spherical harmonics are orthogonal functions from the unit sphere to the complex plain \(Y_{nm}:\mathcal{S}^2 \rightarrow \mathcal{C}\), \(n=0,1,2\ldots \) and \(-n \le m \le n\), defined as

where \(P_{nm}\) are the associated Legendre functions, defined as

Each harmonic function \(Y_{nm}\) is a polynomial of degree n of the variables (x, y, z) on the unit sphere. We will therefore interchangeably use \(Y(\theta ,\phi )\) and Y(p) with \(p=(x,y,z)^T\) and \(x=\sin \theta \cos \phi \), \(y=\sin \theta \sin \phi \), and \(z=\cos \theta \).

To work with real functions we use the notation \(Y^e_{nm}=\mathrm {Re}(Y_{nm})\) and \(Y^o_{nm}=\mathrm {Im}(Y_{nm})\) (\(1 \le m \le n\)), which denote the even and odd components of the harmonics. Note that the zonal harmonics (\(Y_{n0}\)) are real, and that the real spherical harmonics too are orthogonal. In addition, to simplify notation, we interchangeably use a single subscript notation \(Y_s\) with \(s=1,2,3,\ldots \). For this notation we order the real harmonics by their order n and within every order by m, placing each even harmonic before its corresponding odd one, i.e., \(Y_{00},Y_{10},Y^e_{11},Y^o_{11},Y_{20},\ldots \).

We will be interested in surfaces whose reflectance kernel is isotropic and band limited. An example for such kernels is the Lambertian reflectance kernel \(k(\theta )=\max (\cos \theta ,0)\), which can be approximated to a great accuracy with the first nine harmonics (\(n \le 2\)). As is shown in [7], the reflectance function for such surfaces can be expressed as a convolution of the lighting function with the reflectance kernel. As a consequence, the intensity \(I_p\) at a pixel p with surface normal \(n_p\) and albedo \(\rho _p\) can be expressed in the following form. Let \(\ell (\theta ,\phi )\) denote the environment lighting as a function of direction, and let \(\ell (\theta ,\phi ) = \sum _{s=1}^\infty \ell _s Y_s(\theta ,\phi )\) be its harmonic decomposition, then

where \(\alpha _s\) are constants due to the Funk-Hecke (convolution) theorem, \(k(\theta )=\sum _{n=0}^\infty k_{(n)} Y_{n0}(\theta ,\phi )\) is the harmonic expansion of the kernel and \(k_{s(n,m)}=k_{(n)}\).

For Lambertian surfaces we are interested in harmonics up to order 2. The kernel coefficients and the Funk-Hecke constants for these orders are

and the harmonics are

1.1 Equivalence of Distances for Isotropic Reflectance Kernels

Claim 3

Under uniformly distributed directional light sources and an isotropic, band-limited reflectance kernel, the \(\ell _2\) distances of normalized intensity vectors \(\hat{v}_p\) and \(\hat{v}_q\) of points p and q with nearby normals \(n_p\) and \(n_q\) are proportional to the spherical distance \(d(n_p,n_q)\) between the corresponding surface normals, i.e.,

where the constant c depends on the reflectance kernel.

Proof

We prove the claim by evaluating the expression:

The (un-normalized) vector \(v_{p}\) contains the intensities observed at pixel \(p\) over a set of images \(k=1..K\), each lit by a point light source from direction \(l_k \in \mathcal{S}^2\) whose magnitudes are equal. Using (17), the intensity \(I_{kp}\) of p can be written as,

where \(\rho _p\) and \(n_p\) respectively are the albedo and normal at p, \(\ell _{ks}\) are the harmonic coefficients of the light in the k’th image, and \(c_s=\alpha _s k_s\) are constants whose values depend on the reflectance kernel. As we will be interested in normalized intensity vectors \(\hat{v}_p\), we can assume w.l.o.g. that \(\rho _p=1\) and likewise that the lights are of unit magnitudes.

The inner product of two intensity vectors \(v_{p}\) and \(v_{q}\) is given by,

where,

Under the assumption that the light in the K images is distributed uniformly \(L_{st}=\mathbbm {1}_{s=t}\). This is because for a point source light at direction \(l_{k}\), which is expressed as a Dirac delta function \(\delta _{l_{k}}\), \(\ell _{ks}=<\delta _{l_{k}},Y_s>=Y_s(l_{k})\), so that,

and for uniform light of unit intensity, we get

due to the orthonormality of the spherical harmonics. The inner product \(v_{p}^Tv_{q}\) therefore simplifies to

where \(N_s=c^2_s Y_s(n_p) Y_s(n_q)\).

We notice next that \(N_s\) can be expressed as a univariate polynomial in \(z=\cos \theta \), where \(\theta \) is the angle between \(n_p\) and \(n_q\). As the inner product \(v_{p}^Tv_{q}\) and the spherical harmonics are invariant to a global rotation of the normals, we can orient our coordinate system so that \(n_p=[0,0,1]^T\) and \(n_q=[\sin \theta , 0, \cos \theta ]^T\) (and \(\phi =0\)). In this coordinate frame \(Y_s(n_p)\) is constant and \(Y_s(n_q)\) is a (scaled) associated Legendre function, which is polynomial of degree n in \(z=\cos \theta \). We can therefore write

with coefficients \(a_{sr}\) that depend on the reflectance kernel and the harmonic order, s. Since \(n_p\) and \(n_q\) are nearby, we use the Taylor approximation \(\cos \theta =1-\frac{\theta ^2}{2}+O(\theta ^4)\), which yields,

where \(a_s=\sum _{r=0}^n a_{sr}\) and \(b_s=\sum _{r=1}^n r a_{sr}\). We now have,

We further assume that the reflectance kernel is band limited, so that harmonic terms of orders \(n>N\) for a finite N can be omitted. Therefore,

where \(S=(N+1)^2\). We further denote \(a=\sum _{s=1}^S a_s\) and \(b=\sum _{s=1}^S b_s\).

\(v_{p}^Tv_{p}\) can be computed simply by plugging \(\theta =0\) in the previous expression, obtaining \(v_{p}^Tv_{p} = a\), and \(v_q^Tv_q=v_p^Tv_p\) due to rotation invariance.

We can now evaluate the inner product between the normalized intensity vectors,

where \(c=\sqrt{b/a}\). Note that a is positive since \(a=\Vert v_p\Vert ^2\) and b is positive for sufficiently small \(\theta \), since \(v_p^T v_q<a\). We conclude that, up to \(O(N^2\theta ^4)\) terms,

\(\square \)

1.2 Intensity Distances to Geodesic Distances for the Lambertian Reflectance Kernel

Claim 4

The reflectance-dependent constant c can be computed analytically for Lambertian reflectance, and is approximately 0.93.

Proof

As proved in the previous claim,

We now wish to calculate the constant c for the Lambertian kernel. To this end we assume \(n_p=(0,0,1)^T\), \(n_q=(\sin \theta ,0,\cos \theta )^T\) and compute \(N_{s}=\alpha ^2_{(n)} k^2_{(n)} Y_s(n_p) Y_s(n_q)\) as a function of \(\theta \). Using (18)–(19) we get

and the rest of the terms vanish. Therefore,

Replacing \(\cos \theta \) by its Taylor approximation, \(\cos \theta \simeq 1-\frac{\theta ^2}{2}\), we get that

and \(v_p^Tv_p=v_q^Tv_q\) are obtained by plugging \(\theta =0\). Denote, \(a=127\pi /192\) and \(b=109\pi /192\), we can now calculate the inner product of the normalized observation vectors \(\hat{v}_p^T \hat{v}_q\),

Let \(c=\sqrt{\frac{b}{a}} \approx 0.93\). We obtain,

Reprints and permissions