View-Independent Adjoint Light Tracing for Lighting Design Optimization

In the following, we classify related work in four categories, as illustrated in the inset image. Path tracing, one of the most fundamental methods for physically based rendering, applies Monte Carlo integration to solve the rendering equation [Kajiya 1986; Veach 1998]. Similar methods have been extensively used to render physically realistic images (category a). Pre-computed light-transport methods, however, such as radiosity [Goral et al. 1984; Greenberg et al. 1986], focus on solving global illumination for all surfaces in a 3D scene (category b) rather than for a view-dependent image. More recently, these radiosity methods have been extended to incorporate glossy surfaces [Hašan et al. 2009; Krivanek et al. 2005; Omidvar et al. 2015; Sillion et al. 1991; Sloan et al. 2002] and also enable fast incremental updates allowing for interactive frame rates [Luksch et al. 2019]. Similarly, photon mapping [Jensen 1995] is a two-pass process, where illumination data are first cast into the 3D scene by tracing light paths, while the second pass gathers these data to compute the final image. We take inspiration from these latter approaches, constructing a spatio-directional radiance data structure to enable view-independent differentiable rendering (category d). The key ideas of our adjoint light-tracing approach would in principle also be applicable to other pre-computed light-transport methods and radiance-field data structures, such as the recent work by Yu et al. [2022], as well as photon tracing, virtual point light, or light probe approaches.

Previous work in (physically based) inverse rendering has, so far, focused on camera-based methods (category c), where the goal is to optimize scene parameters (e.g., materials, textures, or geometry) [Gkioulekas et al. 2016, 2013; Khungurn et al. 2015; Li et al. 2018; Liu et al. 2018; Nimier-David et al. 2020, 2019; Zeltner et al. 2021; Zhang et al. 2019]. These methods define an optimization objective on the difference between a rendered and a target image. A strong focus has been on optimizing textures, while only few methods have considered optimizing light sources, which affect the result much more globally throughout the scene. Nimier-David et al. [2021] describe a texture-space sampling strategy to improve efficiency for multi-camera setups, while our approach removes cameras altogether.

Pioneering works in differentiable path tracing [Loubet et al. 2019; Nimier-David et al. 2019] have relied on code-level AD, recording a large computation graph, which is then traversed in reverse to compute the objective function gradient. However, AD fails for standard light tracing (Algorithm 1; also known as particle tracing), because each ray carries a constant amount of radiative flux (power) along the ray, independently of its direction or distance travelled. Consequently, parameters that affect these quantities (but not the per-ray flux) cannot be correctly differentiated with AD. The PSDR approach [Zhang et al. 2020] sidesteps this issue by evaluating radiative transport in a path-space formulation before differentiation. One of our key insights is that we can take inspiration from their “material-form differential path integral” and construct a differentiation of light tracing itself for lighting parameters. Instead of using automatic differentiation, we formulate an analytical adjoint state per ray, which allows for an efficient GPU-based implementation.

The basic idea of adjoint methods is to reverse the flow of information through a simulation to find objective function gradients with respect to optimization parameters [Bradley 2019; Geilinger et al. 2020; Stam 2020]. For path tracing, Mitsuba 3 [Jakob et al. 2022] provides a similar adjoint method, called “radiative backpropagation” [Nimier-David et al. 2020], which avoids the prohibitive memory cost of AD. They extend this approach to the reparameterized “path-replay backpropagation” integrator [Vicini et al. 2021].

In our view-independent case, we find that differentiable light tracing often outperforms path tracing in terms of optimization convergence behaviour (Figure 2). Instead of transferring one radiance sample per path from a light source to a sensor, we update our radiance data at each ray-surface intersection and accordingly collect objective function derivatives from each of these locations in our adjoint tracing step, thereby using samples more efficiently. To our knowledge, we present the first view-independent, analytically differentiable light tracing, inverse rendering method. For comparisons to related work, we have implemented a reference differentiable path tracer working on our view-independent data structure. We also compare to existing image-based methods in a baseline test scenario, by restricting our objective function to consider only surfaces that are visible by their camera. In this way we separate the performance of the differentiation method from the benefits of operating on a larger global target.

One problem that has received a lot of attention recently is how to differentiate pixel values in the presence of discontinuous integrands due to silhouette edges or hard shadows. Various strategies to resolve these issues have been proposed, such as reparametrizing the discontinuous integrands [Loubet et al. 2019], repositioning the samples around discontinuous edges [Bangaru et al. 2020; Zeltner et al. 2021], or separating the affected integrals into interior and boundary terms and applying separate Monte Carlo estimators to each part [Yan et al. 2022; Zhang et al. 2020]. The main focus of these methods is to compute how a pixel value changes as a sharp edge or hard shadow moves within this pixel. Our light-tracing gradient formulation keeps ray-surface intersection points constant while moving light sources. Therefore, rays cannot move across feature edges when their source is perturbed, although lights could still move behind an occluding object. The resulting moving shadows are currently not explicitly handled in our formulation; we leave them for future work. Nevertheless, we achieve good optimization convergence using our gradients. While previous work often deals with situations where optimization parameters only affect a very small number of pixels, in our case, lighting parameters affect many receiving 3D surfaces simultaneously, thereby diminishing the relevance of shadows moving through a few surface elements, see also Section 6.4. Our comparisons show that Mitsuba 3’s reparametrization method, and PSDR’s edge sampling, resolve discontinuities but introduce noise, leading to reduced optimization convergence rates compared to their standard version. Using our spatio-directional data structure could in principle allow for extension of their code bases to the view-independent case, potentially including advanced discontinuity handling. For the specific case of lighting design optimization, however, our adjoint formulation is more efficient than these methods (even before considering the overhead of discontinuity handling), as shown in Figures 11–13.

The problem of designing lighting configurations has also been previously explored in computer graphics and tackled from different perspectives. Many approaches use variations of the sketching metaphor, such as sketching the shape of shadows to indirectly move light sources [Poulin et al. 1997] or painting highlights and dark spots to control light parameters in the scene [Pellacini et al. 2007; Shesh and Chen 2007]. Our view-independent differentiable rendering framework brings the sketching metaphor to 3D, providing a more intuitive way to paint the desired illumination directly into the scene. Therefore, the user-painted objective is guaranteed to be free of contradictions that might otherwise occur when editing images from different views corresponding to the same 3D location.

Procedural or hierarchical optimization approaches to luminaire placement have also been explored [Gkaravelis 2016; Jin and Lee 2019; Lin et al. 2013; Schwarz and Wonka 2014]. As we focus on continuous optimization, these methods could be used to estimate a starting configuration before further optimizing the parameters of the generated light sources.

More recent approaches include the user in the optimization process [Sorger et al. 2016; Walch et al. 2019] directly, instead of automatically operating on a predefined target. These methods interactively display the current illumination and provide information about where and how this configuration could be improved. The user is then free to choose which measures to take to bring the scene closer to the desired state. In the future, our approach could be integrated into such a user-centred editing framework to improve the design interaction between the user and the optimization system.

To represent the spatio-directional radiance field L with a finite number of variables (or degrees of freedom), we first construct an appropriate interpolation scheme. We then use this scheme to find approximate solutions to Equation (1).

For the spatial component, we choose a piecewise linear interpolation, as often used in finite-element (and specifically radiosity) methods [Greenberg et al. 1986; Larson and Bengzon 2013],Instead of single nodal values, however, we consider \(L_k\) to be a directional function associated with the kth mesh vertex and its nodal basis function \(\varphi _k\).

For the directional component, we discretize each per-vertex function \(L_k(\omega _o)\) using a hemi-spherical harmonic (HSH) basis [Gautron et al. 2004; Green 2003; Krivanek et al. 2005; Sillion et al. 1991], specifically, the \(2\pi\)-normalized hemi-spherical harmonics \({H_m^l}\) [Wieczorek and Meschede 2018]. Each directional function is consequently represented aswhere n is the maximal order used in this approximation, which implies that for each vertex k, we require \((n+1)^2\) directional coefficients. Consequently, our spatio-directional approximation (using three colour channels, RGB) has \(3 m (n+1)^2\) degrees of freedom, where m is the number of mesh vertices.

For completeness, we summarize the construction of \({H_m^l}\) in the supplementary material. Furthermore, for the special case \(n=0\), which is sufficient for diffuse surfaces, we effectively ignore the directional component and simply store vertex colours. Note that even if we choose \(n=0\), our light tracing method may still include glossy materials when simulating indirect lighting.

Our primary goal is to solve inverse problems based on this discretization, whereas we produce high-quality output images via standard rendering methods. Therefore, we favour the simplicity of this discretization over alternatives such as texturelike bilinear interpolation or meshless bases [Lehtinen et al. 2008]. In contrast to previous work, we store exitant rather than incident radiance. We also define our optimization objective on the exitant radiance, which can be intuitively painted by the user (Figure 4). Discretizing the target radiance field \(L^*(\boldsymbol {\mathbf {x}},\omega _o)\) using the projection derived in the supplement, yields target coefficients \({L_{klm}^*}\) and subsequently a discrete analog of the objective function, Equation (2),where \(A_k\) is the area associated with vertex k, i.e, one third of the sum of adjacent triangle areas. Here, we use a shorthand notation for the nested summation over vertices and HSH coefficients according to Equations (4) and (5). We choose to split the weights into a spatial and a directional component, \(\alpha _k\) and \(\alpha _l\), respectively. Thus, the weights \(\alpha _l\) adjust the influence of each HSH band: Over-weighting lower bands for example would emphasize low-frequency components of the reflected radiance. Similarly, \(\alpha _k\) allows us to focus the attention of the optimizer on specific parts of the scene geometry. We use a painting interface to allow the user to specify \(L_{klm}^*\) as well as \(\alpha _k\). The HSH coefficients can be efficiently painted using a “directional” brush combined with an angular blur kernel, which is easy to compute, because a closed-form solution to Laplacian smoothing exists in HSH space.

For differentiable rendering, note that the derivative of Equation (6), \({\partial O / \partial L_{klm}}\) is straightforward to evaluate, which we use in Section 4.3.

One key feature of our discretization is the orthogonality of all basis functions. Therefore, we can project each radiance sample computed during light tracing into our data structure quickly and independently. In this section, we discuss how we solve the rendering equation using light tracing and update the coefficients \(\lbrace L_{klm}\rbrace\) of our data structure accordingly. We summarize this procedure in Algorithm 1 and provide a detailed derivation in the supplementary document.

Theoretically, we could regard our data structure as a generalized sensor and construct a path-space formulation of its measurement integral. Here, we instead consider light transport in the simpler particle tracing form, directly discretizing each light source’s radiative power into a set of light rays. In Section 4.4, we then use ideas from the path-space theory to formulate the required derivatives for our light tracing approach. For every light source, we sample N exitant rays, each representing a radiant flux \(\Phi _r\) leaving the light source, distributed according to the light’s emission profile, such that the sum of all these exitant samples equals the total radiative power of the light. Let \({{\mathbf {x}}_0}\) be the origin of a ray. We then construct a light path up to a maximal number of indirect bounces, denoted as a sequence of ray-surface intersection points \({{\mathbf {x}}_1},\ldots ,{{\mathbf {x}}_i},\ldots , {{\mathbf {x}}_\text{max}}\).

The incident flux \(\Phi _i\) arriving at each hit point \({{\mathbf {x}}_i}\) is \({\Phi _1} = {\Phi _r}\) (direct illumination) and(indirect illumination), where \(f_i\) denotes the BRDF evaluated at \({{\mathbf {x}}_i}\), \({{\mathbf {n}}_i}\) is the surface normal at \({{\mathbf {x}}_i}\) and \({p_{i+1}}\) is the probability density of the exitant sample in direction \({\omega _{i +1}} = ({{\mathbf {x}}_{i+1}} - {{\mathbf {x}}_{i}}) / \left\Vert {{{\mathbf {x}}_{i+1}} - {{\mathbf {x}}_{i}}} \right\Vert\).

At every intersection point \(\boldsymbol {\mathbf {x}}_i\), we sample the exitant radiance distribution due to the incident flux \({\Phi _i}\) along \({N^{\prime }}\) randomly selected outward directions \({\omega _o}\). We then update the coefficients \(\lbrace L_{klm}\rbrace\) by adding the contribution of each sample \({\omega _o}\) to each vertex k, adjacent to \(\boldsymbol {\mathbf {x}}_i\), as follows:where k refers to the vertices of the triangle containing \({{\mathbf {x}}_i}\), with vertex-associated area \(A_k\), while l and m refer to each HSH basis function up to the selected maximal order n. Please refer to our supplement for a detailed derivation of this update rule.

So far we have discussed how we store the exitant radiance field (Section 4.1) and how each light path affects the degrees of freedom of this field (Section 4.2). We now turn our attention to improving the lighting configuration of a scene via gradient-based optimization and an adjoint formulation for computing the required gradients.

In contrast to previous work on image-based differentiable rendering, we do not handle derivatives along discontinuous edges in our method. This means we primarily ignore direct shadows and instead focus on an efficient derivative formulation for the continuous case. Our results show that these discontinuity problems have limited influence in common lighting design tasks; even in specifically designed test cases (Figures 20 and 21), the optimization algorithm sometimes recovers from receiving these biased gradients.

In the context of our light tracer, parameters that affect the lighting configuration may cause changes in brightness or colour (\(\Phi _r\)), as well as the ray origin, \(\boldsymbol {\mathbf {x}}_0\). In case the origin shifts (infinitesimally), the question is how this change propagates through the scene. Does the first hit, \(\boldsymbol {\mathbf {x}}_1\), also shift, and if so, what about the rest of the light path, \(\boldsymbol {\mathbf {x}}_i\)? Automatic differentiation of Algorithm 1 would answer “yes” and compute derivatives of all hit points, \({ d \boldsymbol {\mathbf {x}}_i / d \boldsymbol {\mathbf {x}}_0 }\), as well as derivatives of the interpolation basis functions. Our test cases (Figures 6 and 7) show that this naïve approach does not lead to useful results. Zhang et al. [2020] (PSDR) work around this issue by moving the evaluation to path space before applying AD. Following a similar idea, we compute derivatives by treating the light path \(\lbrace \boldsymbol {\mathbf {x}}_i\rbrace\) as a parameter-independent constant sample in path space when computing derivatives. Consequently, if a light source moves, then only the origin, \(\boldsymbol {\mathbf {x}}_0\), and hence the first incident direction, \(\omega _1 = {(\boldsymbol {\mathbf {x}}_1-\boldsymbol {\mathbf {x}}_0) / \Vert \boldsymbol {\mathbf {x}}_1-\boldsymbol {\mathbf {x}}_0 \Vert }\), are parameter dependent. All other intersection points, \(\boldsymbol {\mathbf {x}}_i\), remain fixed to the scene geometry. We first derive the general structure of our adjoint tracing method in this section and then compute the required parametric derivatives in Section 4.4.

Our goal is to compute the objective function gradient with respect to the parameters \(\mathbf {p}\), which affect the solution \(L_{klm}(\mathbf {p})\) as follows:The partial derivative \({\partial O / \partial {L_{klm}}}\) can be interpreted as the desired illumination change in the 3D scene per degree of freedom of the discretized radiance field.

We first consider the direct differentiation for \(d {L_{klm}} / d \boldsymbol {\mathbf {p}}\) and then re-arrange the resulting terms to formulate an adjoint state per light path. Every light path resulting from a primary light ray r affects some coefficients of the solution \({L_{klm}}\) according to Equation (8). Consequently, we can split the sensitivity term \(d {L_{klm}} / d \boldsymbol {\mathbf {p}}\) into contributions per path as follows:Note that for direct illumination, the direct incident flux \(\Phi _1\) does not depend on the BRDF \(f_1\), and therefore the second term vanishes when \(i=1\), as \(\partial {\Phi _1}/\partial {f_1} = 0\). Similarly, only \(f_1\) depends on the direct incident angle \({\omega _1}\), but later BRDF evaluations do not; therefore, the third term vanishes for indirect illumination, because \(\partial f_i/\partial {\omega _1} = 0\) for \(i \gt 1\). While we have omitted function arguments for brevity, it is important to point out that the direct BRDF is sampled along local outward directions \(\omega _o\), see Equation (8), whereas the indirect path evaluates the BRDF toward the exitant direction of the first bounce, \(\omega _2\), Equation (7). Both of these terms depend on the first incident direction \(\omega _1\), which changes when the origin of the light path moves. This dependence is stated explicitly in Equation (10); in the following we use the notation \({{{df_1({{\omega _2}})}}/{{d{\mathbf {p}}}}}\) and \({{{df_1({{\omega _o}})}}/{{d{\mathbf {p}}}}}\) to distinguish these terms.

Storing the full sensitivity matrix \(d {L_{klm}} / d \boldsymbol {\mathbf {p}} = \sum \nolimits _r (d {L_{klm}} / d\boldsymbol {\mathbf {p}})_r\) by summation over contributions from all paths, Equation (10), would consume a lot of memory as the number of parameters grows. Our formulation avoids this memory cost and instead uses adjoint states per path, which fit into the local memory of each GPU thread.

Instead of tracing derivatives of radiant flux forward along a light path, we collect partial objective derivatives backwards along that path and compute a weighted sum of partial derivatives \({\partial O}/{\partial {\Phi _r}}\). Nimier-David et al. [2020] also refer to a similar quantity as adjoint radiance in their work. Their concept of adjoint radiance projects partial derivatives of the objective function from an image into the scene along camera paths, whereas we collect objective derivatives along light paths. Combining Equations (9) and (10), including summation over light paths noted above, and then re-arranging terms yieldsHere, \({\partial O}/{\partial {\Phi _r}}\) and \({\partial O}/{\partial f_1({{\omega _2}})}\) are adjoint states, collecting objective function derivatives along the light path. We update these terms at every ray-surface intersection point \(\boldsymbol {\mathbf {x}}_i\), analogously to the primal simulation, Equation (8),andFigure 5 illustrates this idea, which we also outline in Algorithm 2.

Furthermore, the derivative wrt. the BRDF under direct illumination, \({{{\partial O}}/{{\partial f_1({{\omega _o}})}}}\), only occurs at \(\boldsymbol {\mathbf {x}}_1\) and can be calculated directly from Equations (8) and (6). Similarly, the derivative of the BRDF itself wrt. parameters \(\boldsymbol {\mathbf {p}}\) follows directly from the BRDF’s definition. Additionally, we find the derivative terms required to evaluate Equations (12) and (13) as follows: \({\partial O / \partial L_{klm}}\) is the derivative of Equation (6), while \({\partial L_{klm} / \partial \Phi _i}\) follows from differentiating Equation (8). Finally, \({\partial \Phi _i / \partial \Phi _r}\) and \({\partial \Phi _i / \partial f_1(\omega _2)}\) are tracked along the light path, effectively expanding the recursive product induced by Equation (7) and collecting factors.

Note that in our formulation, primal and adjoint tracing are entirely separate rendering passes. In contrast, automatic differentiation correlates primal rendering and derivative calculation, and previous methods work around this issue by running the primal pass twice (“decorrelation”). Here, whether we evaluate the adjoint step in a correlated or decorrelated way is simply a question of whether we choose the same or a different random seed compared to the primal step. In validation tests and finite difference comparisons, we use the correlated evaluation (i.e., keeping the same random seed in all rendering passes), while for all other results we use decorrelated evaluation.

The final step to evaluating the objective function gradient according to Equation (11) is computing \(d\Phi _r/d\boldsymbol {\mathbf {p}}\) for all parameters \(\boldsymbol {\mathbf {p}}\) of the lighting configuration. Note that this is the only term that depends on the type of each light source (point, spot, area, etc.) and on the meaning of each parameter (position, intensity, rotation, etc.). As \(\Phi _r\) denotes the direct illumination, these derivatives can generally be calculated analytically, and we briefly summarize them here while deferring details to the supplementary document.

We first verify that our differentiable light-tracing system is capable of solving various inverse problems, where a well-defined solution exists. We perform ground-truth recovery tests, where the illumination target is copied from the result corresponding to a specific lighting configuration in various scenes: a large office (Figure 1), a glossy coin, using hemi-spherical harmonics up to order \(n=5\) (Figure 9), as well as the Stanford bunny (Figure 11). For these examples, we visualize the differences to the ground-truth target before and after optimization or compare convergence of the optimization to related methods (see details in Section 6.2).

Apart from ground-truth tests, we verify that our method correctly finds solutions for well-defined objectives on specific surfaces, Figures 7 and 10, and compare our gradient calculation to finite difference approximations in Figures 6, 8, and 14. In our supplementary material we also provide a visual comparison of different orders of HSH interpolation, which shows that our radiance data structure converges under refinement.

Finally, we test the capabilities of our system in dealing with indirect illumination. In Figure 10, we place a spot light into a small “labyrinth,” such that light can only reach the “exit” via multiple bounces. The side walls of the labyrinth have a dark target value to push the light away from the walls, with a very low weight (\(\alpha _k = 0.004\)), whereas the exit has a very bright target with unit weight to attract the light source. We trace light paths with 10 indirect bounces in this example. Optimizing for position and rotation simultaneously using ADAM with step size \(\alpha = 0.2\) successfully navigates the light through the labyrinth in ca. 10 s and 200 gradient evaluations. In principle, L-BFGS also successfully solves this problem, but the momentum estimated by ADAM results in a smoother, visually appealing motion of the light source.

Current work on inverse rendering always considers images to define the optimization objective. We establish a baseline for comparing our (camera-free) view-independent approach to existing methods by restricting our objective function to data available to image-based methods (camera-visible surfaces) on a test scene, where one camera sufficiently captures most surfaces (except of course the back of the bunny, Figure 12). In this test, we use ideally diffuse materials for all objects in the scene, and no directional data (\(n=0\)) for our light tracer and reference path tracer, to allow for a fair comparison. As shown in Figure 11, our adjoint light tracing matches state-of-the-art methods in this setting in terms of optimization convergence, while our adjoint formulation, combined with an efficient GPU-based implementation outperforms other methods in terms of runtime. We use an equal number of samples for these comparisons. Finally, note that for the unrestricted case (‘full’) with \(n=0\), Figure 11 also contains an “object-space” baseline comparison between our light tracer and reference path tracer, which shows that light tracing (0.38 s) outperforms path tracing (0.9 s) for an equal number of samples (delivering equal optimization convergence).

Mitsuba’s light tracing method (‘ptracer’), using automatic differentiation, fails to converge, similar to our naïve differentiation approach in Figure 7. The differentiable path-integral formulation of PSDR¹ [Zhang et al. 2020], however, produces good results in this simple setting for both light and path tracing (with slightly more noise and reduced convergence rate of their light tracer).

While related image-based methods (Mitsuba, PSDR) handle discontinous edges in various ways, they also introduce additional noise into the gradient (compared to Mitsuba’s standard AD path tracer), causing reduced optimization convergence (Figure 11). As discontinuity handling also comes with a runtime cost, we choose speed over (theoretical) accuracy for the scope of this article; we analyze the effects of the resulting gradient bias in Section 6.4. Overall, our approach produces less noisy gradient data compared to existing differentiable light tracing methods.

For further comparison, we also implement a reference differentiable path tracer following the ideas of Stam [2020] on top of our view-independent data structure: We uniformly sample each triangle and trace paths with next event estimation to find the incident radiance. Finally, we sample exitant directions locally, apply the BRDF and update our data structure according to Equation (8). This path tracer uses the same sampling strategies to construct indirect illumination paths as our light tracer for a fair comparison of our light tracing method against “object-space” path tracing. The gradients produced by this path tracer closely match Mitsuba’s AD path tracer (Figure 12). For the baseline test scene we observe similar convergence behaviour between restricted objective function data and full view-independent data (i.e., camera-visible vs. all surfaces in the scene), as intended, while our light-tracing approach is substantially faster than path tracing. We show an extension of this baseline test in the supplement, where adding an off-camera box around the scene highlights the advantage of our view-independent approach. Furthermore, Figure 15 shows an equal-time comparison on the more complex optimization problem in Figure 14, where our light-tracing method yields improved convergence rates, due to reduced (gradient) noise (similar behaviour is also visible in Figure 2, as well as the indirect illumination “labyrinth” example in our video).

Finally, we compare our approach to Mitsuba 3 on a large office scene (Figure 13), where Mitsuba uses four cameras to cover most of the scene. We again restrict our method to surfaces visible by any camera for comparison (Figure 13). Mitsuba’s reparmetrization shows reduced optimization performance (likely due to increased noise in the gradient estimates). Using our method without restriction to visible surface improves convergence, even at lower sample count. This behaviour is due to the global coupling between parameters in the lighting design problem, which is in this sense a more challenging optimization problem than, for instance, optimizing texture colours given a target image.

In summary, in terms of optimization convergence, our adjoint light tracing method at least matches, and on complex scenes outperforms, state-of-the art inverse rendering systems on the lighting design problem. Our formulation enables a fast GPU implementation, which results in far better runtime performance.

In this section, we analyze specific situations where our method may fail to converge properly due to not handling occlusion discontinuities (moving shadows). First, we show a modified lighting design example in Figure 20, where the optimizer should navigate the light (in the horizontal plane) in between the occluding panels on the ceiling of a small office. In a second test case, we construct a scene exhibiting a high degree of occlusion, Figure 21, where the light must move through a staggered array of cubes. In both cases, we clearly observe that our gradients are lacking information about how the moving occlusion boundaries (shadows) affect the objective function, causing most optimization attempts to fail. Especially gradient descent or L-BFGS exhibit convergence problems in these tests, as they rely on being provided with a valid descent direction. Adding momentum, as in ADAM, however, can sometimes recover and “skip over” areas exposed to problematic gradients. In fortunate cases, where momentum compensates for gradient bias, ADAM sometimes finds acceptable solutions, even with inaccurate gradient data.

One additional problem caused by noisy or wrong gradients is the increased chance of a light entering a closed-off scene object, effectively trapping it and stalling optimization. Note, however, that lights passing through a wall cause a truly non-differentiable discontinuity that could still occur even with known discontinuity handling methods and should be addressed by continuous collision detection instead. We leave this line of investigation for future work.