Digitizing the Parthenon: Estimating Surface Reflectance under Measured Natural Illumination (Digital Imaging) Part 2

Natural Illumination Capture

Each time a photograph of the site was taken, we used a device to record the corresponding incident illumination within the environment. The lighting capture device was a digital camera aimed at three spheres: one mirrored, one shiny black, and one diffuse gray. We placed the device in a nearby accessible location far enough from the principal structure to obtain an unshadowed view of the sky, and close enough to ensure that the captured lighting would be sufficiently similar to that incident upon the structure. Measuring the incident illumination directly and quickly enabled us to make use of photographs taken under a wide range of weather including sunny, cloudy, and partially cloudy conditions, and also in changing conditions.

Apparatus Design

The lighting capture device is designed to measure the color and intensity of each direction in the upper hemisphere. A challenge in capturing such data for a natural illumination environment is that the sun’s intensity can exceed that of the sky by over five orders of magnitude, which is significantly beyond the range of most digital image sensors. This dynamic range surpassing 17 stops also exceeds that which can conveniently be captured using high dynamic range capture techniques. Our solution was to take a limited dynamic range photograph and use the mirrored sphere to image the sky and clouds, the shiny black sphere to indicate the position of the sun (if visible), and the diffuse grey sphere to indirectly measure the intensity of the sun. We placed all three spheres on a board so that they could be photographed simultaneously (Figure 6.5). We painted the majority of the board gray to allow a correct exposure of the device to be derived from the camera’s auto-exposure function, but surrounded the diffuse sphere by black paint to minimize the indirect light it received. We also included a sundial near the top of the board to validate the lighting directions estimated from the black sphere. Finally, we placed four fiducial markers on the board to estimate the camera’s relative position to the device.


(a) The incident illumination measurement device at its chosen location on the site; (b) An incident illumination dataset.

FIGURE 6.5

(a) The incident illumination measurement device at its chosen location on the site; (b) An incident illumination dataset.

We used a Canon D30 camera with a resolution of 2,174 x 1,446 pixels to capture images of the device. Since the site photography took place up to 300m from the incident illumination measurement station, we used a radio transmitter to trigger the device at the appropriate times. Though the technique we describe can work with a single image of the device, we set the camera’s internal auto-exposure bracketing function to take three exposures for each shutter release at -2, +0, and +2 stops. This allowed somewhat higher dynamic range to better image brighter clouds near the sun, and to guard against any problems with the camera’s automatic light metering.

Sphere Reflectance Calibration

To achieve accurate results, we calibrated the reflectance properties of the spheres. The diffuse sphere was painted with flat gray primer paint, which we measured as having a reflectivity of (0.30,0.31, 0.32) in the red, green, and blue color channels. We further verified it to be nearly spectrally flat using a spectroradiometer. We also exposed the paint to several days of sunlight to verify its color stability. In the above calculations, we divide all pixel values by the sphere’s reflectance, producing values that would result from a perfectly reflective white sphere.

We also measured the reflectivity of the mirrored sphere, which was made of polished steel. We measured this reflectance by using a robotic arm to rotate a rectangular light source in a circle around the sphere and taking a long-exposure photograph of the resulting reflection (Figure 6.6(a)). We found that the sphere was 52% reflective at normal incidence, becoming more reflective toward grazing angles due to Fresnel reflection (Figure 6.6(b)). From the measured reflectance data we used a nonlinear optimization to fit a Fresnel curve to the data, arriving at a complex index of refraction of (2.40 + 2.98i, 2.40 + 3.02i, 2.40 + 3.02i) for the red, green, and blue channels of the sphere.

Light from a clear sky can be significantly polarized, particularly in directions perpendicular to the direction of the sun. In our work we assume that the surfaces in our scene are not highly specular, which makes it reasonable for us to disregard the polarization of the incident illumination in our reflectometry process. However, since Fresnel reflection is affected by the polarization of the incoming light, the clear sky may reflect either more or less brightly toward the grazing angles of the mirrored sphere than it should if it were photographed directly. To quantify this potential error, we photographed several clear skies reflected in the mirrored sphere and at the same time took hemispherical panoramas with a 24mm lens. Comparing the two, we found an RMS error of 5% in sky intensity between the sky photographed directly and the sky photographed as reflected in the mirrored sphere (Figure 6.7).

(a) Mirrored sphere photographed under an even ring of light, showing an increase in brightness at extreme grazing angles (the dark gap in the center is due to light source occluding the camera). (b) Fitted Fresnel reflectance curves.

FIGURE 6.6

(a) Mirrored sphere photographed under an even ring of light, showing an increase in brightness at extreme grazing angles (the dark gap in the center is due to light source occluding the camera). (b) Fitted Fresnel reflectance curves.

In most situations, however, unpolarized light from the sun, clouds, and neighboring surfaces dominates the incident illumination on surfaces, which minimizes the effect of this error. In Section 6.6, we suggest techniques for eliminating this error through improved optics.

(a) Sky photographed as reflected in a mirrored sphere; (b) Stitched sky panorama from 16 to 24mm photographs, showing slightly different reflected illumination due to sky polarization.

FIGURE 6.7

(a) Sky photographed as reflected in a mirrored sphere; (b) Stitched sky panorama from 16 to 24mm photographs, showing slightly different reflected illumination due to sky polarization.

Image Processing and Deriving Sun Intensity

To process these images, we assemble each set of three bracketed images into a single higher dynamic range image, and derive the relative camera position from the fiducial markers. The fiducial markers are indicated manually in the first image of each day and then tracked automatically through the rest of the day, compensating for small motions due to wind. Then, the reflections in both the mirrored and shiny black spheres are transformed to 512 x 512 images of the upper hemisphere. This is done by forward-tracing rays from the camera to the spheres (whose positions are known) and reflecting the rays into the sky, noting for each sky point the corresponding location on the sphere. The image of the diffuse sphere is also mapped to the sky’s upper hemisphere, but based on the sphere’s normals rather the reflection vectors. In the process, we also adjust for the reflectance properties of the spheres as described in Section 6.3.3, creating the images that would have been produced by spheres with unit albedo. Examples of these unwarped images are shown in Figure 6.8.

Sphere images unwarped to the upper hemisphere for the (a) Mirrored sphere; (b) Shiny black sphere; (c) Diffuse sphere D. Saturated pixels are shown in black.

FIGURE 6.8

Sphere images unwarped to the upper hemisphere for the (a) Mirrored sphere; (b) Shiny black sphere; (c) Diffuse sphere D. Saturated pixels are shown in black.

If the sun is below the horizon or occluded by clouds, no pixels in the mirrored sphere image will be saturated and it can be used directly as the image of the incident illumination. We can validate the accuracy of this incident illumination map by rendering a synthetic diffuse image D’ with this lighting and checking that it is consistent with the appearance of the actual diffuse sphere image D. As described in [35], this lighting operation can be performed using a diffuse convolution filter on the incident lighting environment. For our data, the root mean square illumination error for our diffuse sphere images agreed to within 2% percent for a variety of environments.

When the sun is visible, it usually saturates a small region of pixels in the mirrored sphere image. Since the sun’s bright intensity is not properly recorded in this region, performing a diffuse convolution of the mirrored sphere image will produce a darker image than actual appearance of the diffuse sphere (Compare D’ to D in Figure 6.9). In this case, we reconstruct the illumination from the sun as follows. We first measure the direction of the sun as the center of the brightest spot reflected in the shiny black sphere (with its darker reflection, the black sphere exhibits the most sharply defined image of the sun). We then render an image of a diffuse sphere D* lit from this direction of illumination, using a unit-radiance infinite light source 0.53 degrees in diameter to match the subtended angle of the real sun. Such a rendering can be seen in the center of Figure 6.9.

We can then write that the appearance of the real diffuse sphere D should equal the sphere lit by the light captured in the mirrored sphere D’ plus an unknown factor α times the sphere illuminated by the unit sun D*, i.e.,

tmp8745121_thumb[2]

Since there are many pixels in the sphere images, this system is overdetermined, and we compute the red, green, and blue components of α using least squares as αD* « D – D’. Since D* was rendered using a unit radiance sun, α indicates the radiance of the sun disk for each channel. For efficiency, we keep the solar illumination modeled as the directional disk light source, rather than updating the mirrored sphere image M to include this illumination. As a result, when we create renderings with the measured illumination, the solar component is more efficiently simulated as a direct light source.

Solving for sun intensity a based on the appearance of the diffuse sphere D and the convolved mirrored sphere D'.

FIGURE 6.9

Solving for sun intensity a based on the appearance of the diffuse sphere D and the convolved mirrored sphere D’.

(a) Real photograph of the lighting capture device; (b) Synthetic rendering of a 3D model of the lighting capture device to validate the lighting measurements.

FIGURE 6.10

(a) Real photograph of the lighting capture device; (b) Synthetic rendering of a 3D model of the lighting capture device to validate the lighting measurements.

We note that this process does not reconstruct correct values for the remaining saturated pixels near the sun; the missing illumination from these regions is effectively added to the sun’s intensity. Also, if the sun is partially obscured by a cloud, the center of the saturated region might not correspond precisely to the center of the sun. However, for our data the saturated region has been sufficiently small that this error has not been significant. Figure 6.10 shows a lighting capture dataset and a comparison rendering of a model of the capture apparatus, showing consistent captured illumination.

3D Scanning

To obtain 3D geometry for the scene, we used a time-of-flight panoramic range scanner manufactured by Quantapoint, Inc., which uses a 950nm infrared laser measurement component [36]. In high-resolution mode, the scanner acquires scans of 18,000 by 3,000 3D points in 8 minutes, with a maximum scanning range of 40m and a field of view of 360 degrees horizontal by 74.5 degrees vertical. Some scans from within the structure were scanned in low-resolution, acquiring one-quarter the number of points. The data returned is an array of (x,y,z) points as well as a 16-bit monochrome image of the infrared intensity returned to the sensor for each measurement. Depending on the strength of the return, the depth accuracy varied between 0.5cm and 3cm.

Range measurements, shaded according to depth (top), and infrared intensity return (bottom) for one of 53 panoramic laser scans used to create the model. A fiducial marker appears at right.

FIGURE 6.11

Range measurements, shaded according to depth (top), and infrared intensity return (bottom) for one of 53 panoramic laser scans used to create the model. A fiducial marker appears at right.

Scan Processing

Our scan processing followed the traditional process of alignment, merging, and decimation. Scans from outside the structure were initially aligned during the scanning process through the use of checkerboard fiducial markers placed within the scene. After the site survey, the scans were further aligned using an iterative closest point (ICP) algorithm [9,10] implemented in the CNR-Pisa 3D scanning toolkit [37].To speed the alignment process, three or more subsections of each scan corresponding to particular scene areas were cropped out and used to determine the alignment for the entire scan.

For merging, the principal structure of the site was partitioned into an 8 x 17 x 5 lattice of voxels 4.3 meters on a side. For convenience, the grid was chosen to align with the principal architectural features of the site. The scan data within each voxel was merged by a volumetric merging algorithm [12] also from the CNR-Pisa toolkit using a volumetric resolution of 1.2cm. Finally, the geometry of a 200m x 200m area of surrounding terrain was merged as a single mesh with a resolution of 40cm.

Several of the merged voxels contained holes due to occlusions or poor laser return from dark surfaces. Since such geometric inconsistencies would affect the reflectometry process, they were filled using semi-automatic tools with Geometry Systems, Inc. GSI Studio software (Figure 6.12).

Our reflectometry technique determines surface reflectance properties which are stored in texture maps. We used a texture atlas generator [38] based on techniques in [39] to generate a 512 x 512 texture map for each voxel. Then, a low-resolution version of each voxel was created using the Qslim software [40] based on techniques in [41]. This algorithm was chosen since it preserves edge polygons, allowing low-resolution and high-resolution voxels to connect without seams, and since it preserves the texture mapping space, allowing the same texture map to be used for either the high- or low-resolution geometry.

The complete high-resolution model of the main structure used 89 million polygons in 442 non-empty voxels (Figure 6.13). The lowest-resolution model contained 1.8 million polygons, and the surrounding environment used 366K polygons.

Photograph Acquisition and Alignment

Images were taken of the scene from a variety of viewpoints and lighting conditions using the Canon 1Ds camera. We used a semi-automatic process to align the photographs to the 3D scan data. We began by marking approximately 15 point correspondences between each photo and the infrared intensity return image of one or more 3D scans, forming a set of 2D to 3D correspondences. From this we estimated the camera pose using Intel’s OpenCV library, achieving a mean alignment error of between 1 and 3 pixels at 4,080 x 2, 716 pixel resolution. For photographs with higher alignment error, we use an automatic technique to refine the alignment based on comparing the structure’s silhouette in the photograph to the model’s silhouette seen through the recovered camera as in [42], using a combination of gradient-descent and simulated annealing.

(a) Geometry for a voxel colored according to texture atlas regions; (b) The corresponding texture atlas.

FIGURE 6.12

(a) Geometry for a voxel colored according to texture atlas regions; (b) The corresponding texture atlas.

Next post:

Previous post: