Mapping Complex Photographic Detail on 3D Models
Another issue concerning the high-quality mapping of color information over a 3D model is how to manage the very high resolution and redundant color sampling. Just using plain standard or consumer digital cameras one can easily obtain a very high resolution color sampling. Currently, the resolution of consumer digital cameras allows one to obtain a sampling that is an order of magnitude higher than the one of triangulation-based scanning devices; therefore, we might have at least 10 color/reflectance samples falling in the vicinity of each geometric sampled point. A medium size object (e.g., a statue) can be easily sampled with 50-100 images leading to hundred (or even thousand) of million pixels. The need to manage such texturing resolutions in real time opens several issues concerning the selection of the best texture mapping representation, how to extend multiresolution representation to hold both geometry and texture data, and the impact on the rendering speed. Beside the issues concerning the size of the raw data, the problem of mapping complex photographic detail on 3D models raises at least two main problems:
• Efficient storing of color. The possible higher density of the color data with respect to geometric data can make unsuitable direct approaches that store the color information over the primitives of the mesh (e.g., color-per-vertex). Texture mapping techniques would be more appropriate,but scanned meshes are usually highly complex and topologically unclean and these conditions make difficult the task of building a good texture parameterization.
• Coherently mixing the photographic information. Creating a high-quality texture content is a critical step. Merging all the photos by a resampling process is a valid option, but it should not decrease the quality of the color information (e.g., resampling from multiple images often introduces blurring and ghost effects).
Texture Parameterization
For the specific task of building a mesh parameterization (taking in input the 3D mesh and the mapped photos) one approach could be to determine, given a set of images, an optimal partition of the mesh such that each surface patch could be mapped with (a portion of) one single input image. This approach has been used in a number of early papers (see for example [58, 59]). The main disadvantage is that the number of pieces in which the mesh is partitioned grows up rapidly with the number of involved photos [48].
Another possibility is to adopt classical mesh parameterization techniques that are able to build a good-quality parameterization of the mesh over a texture space. Then, we can resample a texture from the input set of images, at the proper resolution required by the specific application. These approaches (see [60] for a survey on this subject) have often the drawback that, to avoid great distortion in the parameterization, the result is composed by many small texture pieces whose discontinuities can give rise to many visualization issues. A different approach has been recently proposed [61] that tries to overcome these limitations by computing the resulting parameterization in a continuous abstract texture domain that has the same topology of the input mesh and presents no discontinuity.
Resampling Color Data
Given a 3D model and the cameras associated to each input photo, we need a robust algorithm for color resampling from several input photographs, which can produce in output a coherently mixed blending of all the input images either as a texture map or as resampled per-vertex colors. A multivariate blending approach has been recently proposed [62]. This blending framework allows one: to use many different heuristics to characterize the image content (pixel by pixel) in terms of intrinsic quality of each sample; to produce, for each vertex on the surface or pixel on the output texture space, an optimal weighted average of all the available pixel samples which maps on the requested geometric location/parcel.
Advanced Color Data Acquisition: Sampling Surface Reflection Properties
Obviously, we envision future systems able to encode not only the reflected radiation (apparent color), but also able to sample the reflection properties of the surface. The goal is to move from apparent color acquisition towards BRDF sampling or at least the acquisition of approximations of the surface BRDF. To make those solutions practical we need to improve current methods in terms of ease of operation, reduced dependency from highly controlled lighting environment, and reduced processing complexity.
An approximate approach has been recently proposed, with the aim of adopting a very easy to deploy lighting setup that should allow one to remove lighting artifacts from the acquired photos. Flash light is a very easy way to illuminate an object or an environment, but it is rarely considered in most of the computer graphics and computer vision literature concerning color acquisition. This is due to the large amount of artifacts introduced by this specific light source, and to the difficulty in modeling its behavior in space. A simple method of using flash light in the context of color acquisition and mapping on digital 3D models has been recently proposed [63], based on a technique that allows one to characterize the emission in the 3D space of the flash light (see Figure 3.11). The calibration process allows one to acquire accurate information on how the light produced by a given flash is distributed in the 3D space sampled by the camera and how the color values sampled in that region are modified by the flash light. Moreover, given a calibrated flash and the extrinsic parameters for each single photo (location and direction of view of the camera), we are able to accurately estimate the flash position with respect to the framed object. We are thus able to apply automatic methods to detect and remove artifacts (specular highlights and shadows, see Figures 3.12 and 3.13) from a set of images which are registered to a 3D model. These methods are integrated in the context of a color mapping framework and allow one to obtain improved colored 3D models. These results fit very well the context of cultural heritage, where the acquisition of color has often to be performed on site, with very few constraints on the lighting setup. This approach can be considered as a first result to fill the gap between basic reflected color versus complex BRDF acquisition methods.
FIGURE 3.11
Some examples of the flash characterization obtained on three different cameras. The images show plots of an orizontal plane (top) and a vertical plane (bottom) intersecting the working camera space, for a Nikon reflex camera(Left), a Canon reflex camera (Center) and a compact Casio camera (Right).
FIGURE 3.12
An example of highlights detection. Upper row: an input flash image and the same image after detection of highlights (blue is the highlight border, cyan is the internal highlight region). Lower row: two detail views, where highlights candidates selected by taking into account just geometric criteria (surface normals) are rendered in green and the ones more robustly detected are rendered in cyan.
FIGURE 3.13
An example of shadows detection: left, the original image; right, the shadow detection map.
Other promising approaches support sophisticated sampling of the reflection properties by adopting image-based rendering approaches: instead of producing accurate 3D encoding with poor-quality surface reflection data, we could aim at acquiring and using the classical 2D media, but enhanced with sophisticated reflection encoding. We have a spectrum of possibilities, from the one-view and multiple-lighting approach of Polynomial Texture Maps (PTM) [64], to the more demanding multi-view approach granted by Light Field rendering [65]. In all these cases (see [66] for an overview), we get rid off the 3D model and adopt approaches which sample the artifact with a multitude of images. To produce any view requested at visualization time we process/interpolate the knowledge granted by the sampled image set. In most cases, this implies a huge quantity of image data to be acquired (thus long acquisition times are usually needed), stored, and accessed in real time, for example making web-based visualization not easy.
MeshLab: An Open Source Tool for Processing 3D Scanned Data
As we have described in previous sections, processing 3D sampled data is not an easy task. We need several algorithms and the pipeline is neither fixed nor easy to be managed, particularly in the case the operator is not an ICT expert. Even if a number of commercial solutions target the market of processing 3D sampled data, we found that the community is also searching for complete software solutions, possibly much cheaper than current SW tools. The introduction of new 3D scanners sold at the cost of a good digital camera makes the request of cheap SW solutions even more pressing. Moreover, most of the commercial packages in this field target the reverse engineering application domain, which has different needs and priorities with respect to the requirements of the CH domain,so there is a demand for a set of tools that are more tailored to the processing of sampled 3D historical artifacts.
FIGURE 3.14
Snapshots of MeshLab in action: the mesh under processing is interactively displayed and the user can work on it by means of a large set of unattended parametric filters, or by means of interactive tools, like the one shown in the image on the left, where the user is smoothing out some features of the object with some simple mouse strokes; on the right we show a range maps alignment session.
This objective was at the base of the ISTI-CNR effort in designing and implementing MeshLab as an open source and extendable mesh processing tool [19,41]. MeshLab was designed as a general 3D mesh processing system tool with the following primary objectives in mind:
• Mesh processing oriented. The system should try to stay focused on mesh processing tasks instead of aspiring to contribute to mesh editing and mesh design, where a number of fierce competitors already crowd the software arena (notably Blender, 3D Max, Maya, and many others).
• Ease of use. The tool should be designed so that users without high 3D modeling skills could use it (at least for the most basic functionalities).
• Depth of use. The tool should be designed so that advanced users can tweak and extend it by adding functionality and or by modifying all the involved parameters.
• Efficiency. 3D scanning meshes easily reach several million primitives, so the tool should be able to manage efficiently very large meshes.
As a result, MeshLab presents itself as an intuitive mesh viewer application, where digital 3D models, stored in a variety of formats, can be loaded and interactively inspected in an easy way, by simply dragging and clicking on the mesh itself. MeshLab supports an ever growing variety of 3D formats (all the most common formats are supported) to accommodate the broadest set of users. Once having loaded a mesh, the user can work on it by means of a large set of direct parametric filters that perform unattended automatic tasks like surface reconstruction, smoothing, re-meshing or simplification, or by mean of interactive tools (for example, range maps registration).
The system has proved a success over any initial prediction. MeshLab is usually downloaded more than 6,000 — 8,000 times every month; MeshLab’s user community is thousands of users from all over the world. Users come from hundred of universities and renowned commercial companies that have found MeshLab useful in many different contexts, widening the original CH domain we were focusing on.
One of the interesting characteristic of MeshLab is the presence of all the basic resources for processing 3D scanning data. MeshLab already provides tools for: cleaning sampled range maps, performing range maps alignment, merging/reconstruction, simplification, transferral of color information from range maps to meshes, measuring differences between objects, and providing texture parameterization algorithms. MeshLab is an evolving system, being one of the technologies under further development in the framework of the EC IP “3D-COFORM” project (2009-2012) [21]. Some of the color management algorithms described in the previous section are among the new functionalities that we are including in MeshLab.
