Keywords

1 Introduction

It is effective to measure the spatial reflectance distributions on the surfaces of real materials, such as paper, fabric and metal materials, for the purpose of reproducing realistic computer graphics. The reflectance at a position on a surface can be expressed using the BRDF (Bidirectional Reflectance Distribution Function). The BRDF is a function that defines how light is reflected at an opaque surface, and it is a four-dimensional reflectance function of the incoming direction and the outgoing direction , where \(\theta \) and \(\phi \) are the elevation and azimuthal angles, respectively.

Many researchers developed reflectance measurement systems by setting up existing apparatus in various ways. Some systems use large apparatus, such as a robot-based gonioreflectometer [3, 13], and three stepper motors-based gonioreflectometer [9]. They used many motors and controllers, which need to be oriented carefully. Their systems can measure the reflectance of a single point, but the spatially-varying reflectance cannot be measured. This system was extended by additionally using a pan-tilt-roll motor unit [14] to measure the spatial reflectance. An imaging gonioreflectometer was developed using a silicon photodiodes and an incandescent lamp [15]. Some systems were accordingly extended using a half-silvered hemispherical mirror [19], and a curved mirror [5]. Another apparatus like gonioreflectometer was developed using a photometer and a CCD video camera [4]. Their systems can measure the spatially-varying reflectance, but the different combinations of viewing and illumination directions are required. Therefore, they require high cost apparatus and take long capturing time.

On the other hand, some researchers developed their own acquisition apparatus by combining a gantry with robot arms, rotating arms, lenses, cameras, and light arrays in a variety of ways [6,7,8, 10, 16,17,18, 20]. Their systems can also measure the spatial reflectance but require many light sources, controllers for moving the devices, and wide space to set up the apparatus. Other systems using portable apparatus consisting of a single light source, mobile phone camera and built-in flashlight were developed [1, 2, 12]. Their acquisition apparatus can measure the spatially-varying reflectance but has limitations in the movement of cameras, the number of images to be captured, and the type of materials to be measured.

In this study, we propose a system that can measure the spatially-varying reflectance of real materials using two handheld cameras as a light and view cameras. The light camera is used to capture the incoming light directions, and the view camera is used to capture the outgoing directions, and the brightness of each pixel on the surface of the material sample. The light camera is attached to by an LED light and is placed on a stand, and the view camera is moved by hand. The attached LED light is used to illuminate the target material. A chess board with markers is used to acquire the camera poses of two cameras, and additionally the positions of all pixels on the surface. A turning table is used to acquire the different views of the material. The Ward BRDF model is used to approximate the reflectance. The spherical polar coordinate system is used to calculate the distributions of normal directions across the surface. All model parameters as well as the normal directions are estimated by non-linear optimization. Our system can be used to easily measure the spatially-varying reflectance of real materials.

2 Related Work

Measuring the reflectance on the surface highly depends on the configuration of apparatus, and many measurement systems were built by researchers. We categorized those systems into three groups.

Measurement Systems Using Gonioreflectometer. Baribeau et al. [3] configured a robot-based gonioreflectometer in which a five-axis robot was used to tilt the material sample, and a rotation stage was used to hold the light source. An array spectroradiometer was used to detect the reflected light from the sample and also detect the direct light from the light source. Their system can measure the reflected spectral radiance but requires to synchronically orient the sample and light source. Lyngby et al. [13] set up a gonioreflectometer using a six-axis industrial robotic arm, an arc-shaped light source used as a static light array, and an RGB camera. The sample was placed on a stand under the static light array. The robotic arm was used to position and orient a camera to capture images of the material sample. Their system can reduce the number of controllers to one, but their systems can only measure the reflectance at one point.

Murray-Coleman et al. [15] developed an imaging gonioreflectometer by using silicon photodiodes as two photodetectors, and an incandescent lamp. All components and the sample were positioned by stepper motors. Their system can measure the spatial reflectance, but the configuration is complex and takes long measurement time. Ward [19] extended it by using a half-silvered hemispherical mirror instead of using silicon photodiodes, a quartz-halogen lamp, and a CCD camera with a fish-eye lens. The sample was moved by hand and illuminated by a quartz-halogen lamp. The light source was controlled by a controller to move along the hemisphere. His system can cover the entire hemisphere of reflected light directions simultaneously, but the configuration is still complex and expensive.

Dana [4] did not use the hemispherical mirror arrangement, and developed an apparatus like gonioreflectometer, in which a robot, a photometer, a CCD video camera, a halogen bulb with a Fresnel lens, and a personal computer were used. The camera was mounted on a tripod, and the texture sample and halogen bulb were oriented by the robot arm. A few years later, Dana et al. [5] used a curved mirror instead of using the hemisphere and proposed another type of gonioreflectometer including a beam splitter, a concave parabolic mirror, a CCD camera, a collimated beam of light, and translation stages. The stages were used for automatic material scanning by placing the illumination aperture on an X–Z stage and the mirror on an X–Y–Z stage. Their systems can obtain multiple views of the same surface points simultaneously, but many devices are required. Mcallister et al. [14] designed a spatial gonioreflectometer based on the system [9] using a pan-tilt-roll motor unit, which held the surface sample with fiducial markers, a calibrated light mounted on a motorized rail, and a CCD camera. All components were mounted on an optical bench and controlled by four motors separately. Their system can acquire the spatial reflectance distribution, but many devices and controllers are required.

Measurement Systems Using Light Array. Filip et al. [8] proposed an acquisition setup by using a mechanical gantry, a consumer digital camera, and two LED lights. One of the two arms held two LEDs and another arm held the camera. These two arms were synchronically controlled by the gantry to capture images of the material sample placed below the setup. Their system can capture sparse texture data but limits the size of the material sample. Tunwattanapong et al. [17] used a hemispherical gantry, in which a motion control motor was used to rotate a 1 m diameter semi-circular arc attached with 105 white LEDs series. A sample was placed on a platform that was rotated to eight positions by a motor. An array of five machine vision cameras was configured in the form of a mathematical plus sign, and it was placed in front of the sample. Their system can acquire the geometry and spatially-varying reflectance, but many light sources and cameras were used.

Gardner et al. [10] used a translation gantry, a white neon tube as a linear light source, and a digital camera. The gantry is used to move the light source horizontally across the surface of a planar material. The camera was fixed on a tripod at approximately a 55-degree incidence angle to the material. Their system can estimate the spatial reflectance by a single pass of a linear light source, but it limits the materials, such as printed paper and fabrics. Wang et al. [18] adapted the system in [10] by replacing the white neon tube with a linear array of 40 LEDs and using a color checker pattern for camera calibration. A stepping motor was used to control the light array to move horizontally over the material sample. A static camera was placed about 1.0 m away and 0.6 m above the center of the sample surface. Their system can capture images of the material under different illuminations from one view, but motor and many light sources are still required.

Measurement Systems Using Portable Apparatus. Aittala et al. [1] developed a system using an LCD display (laptop screen) and a single-lens reflex camera. The material sample was illuminated by the LCD screen by placing it in a static position below the screen, and the camera took the reflected light from the sample. Their system is portable and can measure the spatial reflectance. Aittala et al. [2] also used only a mobile phone, that was hold in parallel with a material sample and captured a flash-no-flash image pair under a point light source (built-in flash-light). Li et al. [12] also used only one mobile camera set-up that took a pair of images under the ambient light, and a point light source (built-in flash-light). Their systems are light-weight and can measure the spatially-varying reflectance but has limitations in the movement of cameras, the number of images to be captured, and the type of materials to be measured.

Fig. 1.
figure 1

Measurement setup.

Full size image

3 Measuring the Spatially-Varying Reflectance

3.1 Measurement Apparatus

The measurement setup is shown in Fig. 1. The system consists of a PC, two handheld cameras, a turning table, a small LED light, and a ChArUco board [11]. The two handheld cameras are used as a light camera and a view camera, respectively. The light camera is mounted with a small LED light, and is placed on a stand. The LED light is used to illuminate the target material placed on the turning table. The view camera is moved by hand from 0 to \(90\) \(^\circ \) of polar angles, and 0 to \(360\) \(^\circ \) of azimuthal angles of the outgoing light directions. The ChArUco board, a chessboard with markers, is placed under the material on the turning table. While conducting the experiment, the room was set dark except the small LED light to prevent entering light from the environment. Using this apparatus, dynamically changing incoming and outgoing light angles and different views of each spatial point can be acquired.

The color images were captured by the two handheld cameras at the frame rate of 15 frames per second. The cameras were 0.5 m away from the material, and 50 frames were captured for each material. The resolution of color images was \(1920\,\times \,1024\) pixels. The specifications of the PC were Intel Core \(i7-7700\) CPU, 16 GB memory and NVIDIA GeForce GTX 1070 6 GB PC for GPU.

Fig. 2.
figure 2

Measurement method: The system measures the incoming and outgoing light directions \(\omega _i(t)\) and \(\omega _o(t)\), and the intensity I(x, t) on a point x at time t. Then, the model parameters as well as the normal direction are estimated by non-linear optimization.

Full size image

3.2 Measurement Method

Our system measures the incoming and outgoing light directions, \(\omega _i(t)\) and \(\omega _o(t)\) using the chessboard with markers, and the intensity I(x, t) on a point x at time t. Then, the system estimates the reflectance at several points on the material according to the BRDF model as shown in Fig. 2. The model parameters, the diffuse albedo \(\rho _d\), the specular albedo \(\rho _s\), and the roughness \(\alpha \), as well as the normal direction N are simultaneously estimated using the Levenberg–Marquardt optimization method, which finds the parameters that minimizes the square difference between the observed intensity I(x, t) and the estimated luminance on a pixel x.

The system represents the reflectance using the Ward BRDF model [19], which determines how much light is reflected from the incoming direction i to the outgoing direction o at a surface point x as follows:

$$\begin{aligned} f_r(\theta _i, \phi _i, \theta _o, \phi _o) = \frac{\rho _d}{\pi } + \frac{\rho _s}{4 \pi \alpha ^2 \sqrt{\cos \theta _i \cos \theta _o}} e^{-\frac{\tan ^2 \theta _h}{\alpha ^2}} \end{aligned}$$
(1)

where \(\theta _i\) and \(\theta _o\) are the incoming and outgoing light angles, \(\theta _h\) is the halfway angle between the normal and the halfway vector, respectively.

The ChArUco marker board is used to acquire the calibration parameters, which is used to estimate the poses of the two cameras. Moreover, it is used to know the pixel positions of the material sample.

As the view camera is moved by hand while capturing the color images, some of markers can be missed out. Therefore, more than four detected markers in each frame are used to interpolate the marker corners, otherwise the frames that do not have enough number of markers are skipped. In this way, the calibration parameters for each camera in the world coordinate system defined by the markers are acquired. Then, the camera pose \(\mathbf{x} _{c}\) for each 3D position \(\mathbf{p} \) is estimated as follows:

$$\begin{aligned} \mathbf{x} _{c} = R^{-1}{} \mathbf{p} - R^{-1}{} \mathbf{t} \end{aligned}$$
(2)

where R is the rotation matrix, and t is the translation vector of each camera.

Fig. 3.
figure 3

Variables used in the Ward BRDF model.

Full size image

Estimation of the BRDF Parameters. The variables used in the BRDF model are shown in Fig. 3, where i and o represent the incoming and outgoing light directions on a surface point, H is the halfway vector between i and o, N is the normal direction, \(\theta _i\) is the angle between the normal and the incident light direction, \(\theta _o\) is the angle between the normal and the outgoing light direction, and \(\theta _h\) is the angle between N and H. The reflectance parameters are estimated using the Levenberg–Marquardt optimization. It is a non-linear optimization method and accordingly it is required to give the initial values.

As the object is rotating and the view camera is moving, the intensity on the vertex is changing from frame to frame. To obtain the diffuse and specular albedos, the correspondence between the object vertex and the color pixel is calculated according to the projection of the vertex to the image plane. The initial values for the diffuse and specular albedos are initialized by the average of the color values at the reprojected points over all frames. The roughness parameter is initialized from 0 to 1, and the best-fitted one is chosen.

Normal Estimation. The distribution of normal directions for all points across the material surface is estimated using the spherical polar coordinate system as shown in Fig. 4, where \({\varvec{{N}}}(\theta _n,\phi _n)\) is the estimated normal direction at a pixel position, which is described by two angles, the polar angle \(\theta _n\) and the azimuthal angle \(\phi _n\), respectively. The normal direction for each pixel position is estimated by giving the initial values to the two angles. The initial values range from 0 to \(180^\circ \) for \(\theta _n\), and from 0 to \(360^\circ \) for \(\phi _n\), respectively, and the best-fitted pair is chosen.

Fig. 4.
figure 4

Estimation of normal direction using the spherical polar coordinate system: \({\varvec{{N}}}(\theta _n,\phi _n)\) is the estimated normal direction, which is described by two angles, the polar angle \(\theta _n\) and the azimuthal angle \(\phi _n\), respectively.

Full size image
Fig. 5.
figure 5

Fitting result for the estimated reflectance over observed brightness of RGB components with regard to frame numbers at a spatial point of metallic-colored paper.

Full size image
Fig. 6.
figure 6

The rendering results for metallic-colored paper and paper with golden texture: (a) Original materials (b) Estimated normal maps (c–d) Rendered materials in different views. (Color figure online)

Full size image

4 Experimental Result

In our experiment, real materials, such as metallic-colored paper and paper with golden texture were used as samples. The colored frames were captured about 50 frames for each sample, and the reflectance is approximated using the Ward BRDF model. The reflectance of totally 62901 points were estimated for each material. It took totally about 12.5 min of capturing time, and about 9 min of computation time for each material. Therefore, our system took less than 22 min of total measuring time for each material.

Fitting Result. The fitting result for the estimated reflectance, which is expressed using the parameters of the Ward BRDF model, over observed brightness of RGB components with regard to frame numbers at a spatial point, is shown in Fig. 5. As the model is well fitted with observed brightness of each component, we can confirm that the estimated BRDF parameters are optimized correctly.

Rendering Result. Figure 6 shows the rendering results for metallic-colored paper and paper with golden texture. Figure 6 (a) shows the appearance of the original materials, Fig. 6 (b) shows the estimated normal maps, and Fig. 6 (c)(d) shows the rendered materials in different views. We can see the correct colors of rendered materials that correspond to those of the original materials. Additionally, highlight changes on the surfaces were observed when we moved the light source or the rendered materials.

5 Conclusion

We proposed a portable system that can easily measure the spatially-varying reflectance of real materials. This system requires only a small space compared to the apparatus using gonioreflectometer and gantry, and can move the cameras freely by hand to capture images. We conducted the experiment using only planner materials; however, this configuration could be applied to the measurement of non-planner materials as well. In the future, we will extend our method to measure more complex reflectance such as the anisotropic reflectance.