Efficient Visualization and Management of Sampled 3D Data
Some issues arise from the very dense sampling resolution granted by modern scanning devices. Being able to sample in the order of ten points per squared millimeter or more (in the case of triangulation-based systems) is of paramount value in many applications which need a very accurate and dense digital description. On the other hand, this information is not easy to process, render, and transfer. Therefore, excessive data density becomes a problem for many applications. In effect, the availability of 3D scanned data was a driving force for intense research on more efficient geometric data management and rendering solutions. Some issues arising from the impressive increase in data complexity (and richness) provided by the evolution of 3D scanning technology are as follows: how to manage/visualize those data on commodity computers; how to improve the ease of use of the visualization tools (as potential users are often not expert with interactive graphics); finally, due to the accuracy of the 3D medium, we can think to use it as the main representation media, able both to represent an artifact but also to integrate other multimedia information.
Simplification and Multiresolution Management of Huge Models
Data complexity can be managed by adopting a data simplification approach, reducing the data resolution at the expenses of a (controlled) loss of geometric accuracy. Many solutions have been proposed for the accurate simplification of 3D triangulated surfaces, usually based on the iterative elimination of selected vertices or faces, driven by an error-minimization cost function. This approach allows the construction of any level of resolution we need, usually with a rather expensive computation (from a few seconds to a few hours, depending on the solution used and the complexity of the initial surface) which has to be executed only once. Simplification is very handy to produce models which fit the specific application requirements (e.g., a simple small model for a web presentation which should be downloadable in a given short time, or a model to be used for an accurate rapid reproduction by a specific 3D printer).
Another approach is the so-called multiresolution encoding (i.e., storing not just the final simplified model but all the intermediate results reached during the iterative simplification). All these intermediate results have to be encoded in an efficient data structure (the specific multiresolution representation scheme) that should allow the interactive application to extract different resolutions on the fly, supporting data feeding rates to the GPU compatible with real-time applications (visualization is an example of those). A view-dependent variable resolution representation can be produced for each frame from these schemes, according to the current view specification (e.g., higher resolution for the portions in foreground, progressively lower resolution for data in the background) and the requested visualization accuracy.
Recent research on multiresolution schemes has produced a number of solutions based on higher granularity than previous methods (i.e., instead of focusing on single triangles, patches of triangles become the elementary entity). These solutions allow one to manage huge 3D scanned models at interactive frame rate on consumer PC’s [67,68]. Virtual Inspector [56] is an example of a visualization system which adopts this approach to support inspection of large complex 3D models in real time (see Figure 3.9).
FIGURE 3.15
An example of two different visualization modes: on the left we render just the sampled points, on the right we present a rendering where shading of the surface element plays an important role for an improved insight over the details of the represented architecture (from a scanned model of the Cathedral of Pisa, Italy).
After around fifteen years of intense research on simplification and multiresolution technologies, we can nowadays consider those technologies sufficiently mature. But there is still some lack of information in the 3D scanning community (at the application level) on the potential of those methods. One of the more common negative concerns raised by practitioners against 3D scanning is the size of the models obtained, which according to this common believe makes them unusable in real applications. This is not true, since the availability of simplification and multiresolution technologies can successfully cope with the data complexity of 3D scanned data. In this context, multiresolution encoding is of paramount value, supporting transparent real-time selection of the level of accuracy and producing at interactive rates a data density that best fits the current application requirements.
Mesh-Based versus Point-Based Encoding and Rendering
We have presented briefly the triangle-based simplification and multiresolution methodologies (i.e., data optimization/encoding strategies based on geometric processing applied to a triangle mesh). Point-based representations have also been used a lot in the 3D scanning domain, especially to present data acquired with TOF scanning devices. It is important to distinguish between naive point-based and the more sophisticated point-based representation and rendering approaches available. With the term naive point-based we mean the use of simple 3D points to render sampled data. This is the usual approach of many scanning devices, which usually render the samples as colored points, using the color channel to map the perceived reflectivity of the sampled surface or the estimated accuracy of the sample.
Naive point rendering is very easy to implement, but it is also a major cause of the very poor acceptance of 3D scanning in several applicative domains. Just splatting points on the screen gives very poor images, where it is not easy at all to understand the relations between different elements, perception of depth is lacking, it is very hard to spot errors or inaccuracies of the data (see Figure 3.15). This does not mean that point-based rendering is a wrong solution, but that choosing a naive implementation can be a very bad decision.
First, we should remember that even if we endorse a point-based approach, all the processing phases presented in Section 3.2 are valid and have to be performed in an accurate and correct way.
The several point clouds (or range maps) acquired have to be aligned precisely one to the other and globally (note that the alignment results can be checked and assessed only when we have the possibility to render the points as a surface, adding shading). A merging phase should be applied even if we want to endorse a point-based representation: it is not sufficient to simply join the several point clouds in a bigger set, because sampled data are redundant and contain noise; we have already seen that a merging phase (performing an intelligent interpolation among sampled data) can improve the data and reduce unnecessary redundancy. Moreover, we need to adopt multiresolution to make large dataset tractable (as an example, the Pisa Dome shown in Figure 3.15 is around 200 mega samples after merging). Now that we have established that we have to process the dataset even when a point-based approach is endorsed, let us focus on visualization constraints. Even when we use points, we should be able to render color (this is very easy, since we can assign a single color to each point) but also to render the surface slope associated to each point. This means that the representation should store a normal vector for each sample, and this vector has to be computed by integrating a small region around the given sample. Many different advanced point-based representations and rendering methodologies have been presented in the last few years. Presenting an overview of the extensive research results on point-based graphics is well beyond the focus of this topic; this specific domain is the topic of a series of successful symposia, the EG Point-Based Graphics series; interested readers can consult those symposia proceedings or a recent tutorial on point-based techniques [69]. Point-based representation as well can be managed in an efficient manner by endorsing simplification or multiresolution methods. The latter are usually based on hierarchical structures that allow one to encode the dataset at different levels of detail and to extract view-dependent representation very easily in real time [70-72].
Usability of Virtual Heritage Worlds
Ease of use of visual CH tools oriented to ordinary people (that are still not very competent with 3D graphics and computer games, especially if not part of the young generation) is an important factor for their success. One of the most complicated actions to perform nowadays with 3D tools is to drive navigation in the virtual space, especially when we have a clear objective (e.g., I want to reach a specific location in the scene and see my focus of interest from a specific view). Therefore, free navigation should be requested only in those cases where this action really adds something to the learning experience. The risk is to have the visitor losing orientation (e.g., discovering himself lost in void space, maybe just because he turned his back to the scene and is erroneously looking at the sky), losing faith that he can drive the inspection or navigation session and quitting the system.
Other important characteristics of a visualization system are its flexibility, completeness, and configurability. To fulfill this objective developers could be induced to design complicated systems characterized by a very complete set of functionalities and involute user interfaces (for an example, consider the commercial 3D modeling systems). Conversely, while designing our Virtual Inspector tool [56] as a system oriented to non-expert users (e.g., museum visitors), our approach was to define a restricted set of functionalities and to provide the tool with an easy installation interface for the selection of the subset of these functionalities that the designer of the specific installation wants to endorse (e.g., to build up a new museum kiosk).
Another important factor of success nowadays is web availability of those resources, to allow a much wider community of users to experiment, navigate and enjoy the reconstructed scenes. Implementing a Virtual Heritage reconstruction to make it deployable on the web introduces a number of design constraints, but the technology is almost ready to support that type of distribution channel.
Not Just 3D Data: Adding Other Knowledge
Visualization tools usually focus on just the visual presentation of the shape characteristics of the artifact. This is not enough if we want to provide a comprehensive presentation of the global knowledge available on the artifact. On the other hand, the 3D shape can become some sort of visual 3D map which allows us to integrate, links and present all the available information in an intuitive and visually pleasing way. The goal is therefore to transform standard 3D browsers into tools able to connect the shape (or region of the latter) with all the available multimedia (MM) data that is linked with the artifact. This opens the wider topic on how to support and implement data annotation on 3D models.
Hot spots can be a very handy resource to associate multimedia data to any point or region of a 3D model. This allows one to design interactive presentations where the 3D model becomes a natural visual index to historical/artistic information, for example presented using standard HTML format and browsers. But hot spots are only a first step, which implements a point-wise association metaphor. We should devise more flexible and powerful instruments to associate information to 3D meshes (e.g., supporting links between sub-regions of the surface to any possible MM document). Therefore, tools should provide easy-to-use features for the segmentation of the shape model in components. Segmentation can be driven by pure shape-oriented criteria (usually, working on surface curvature is one of the most diffuse approaches) but it should also include the user in the loop, by giving him the lead in driving the segmentation process according to the metadata that characterize the specific artifact and the message he wants to convey with the specific segmentation.
Data presentation of a multitude of information tokens can become an issue (many different types of point-wise links, associated to different types of information; several different segmentations available for the same object, each one focusing on a different interpretation or context). Intellingent approaches to data hiding and visualization will have to be endorsed also in this type of applications, as it has already been the case of other scientific visualization or data analytics contexts. Moreover, visualization instruments should be extended to provide tools supporting the easy integration and update of the data associated to the 3D digital model, supporting a democratic and cooperative approach such as the one at the base of the Wikipedia effort.
Presenting 3D Data on the Web
In the modern world, we cannot avoid considering the issues related to the web-based access and distribution of 3D data. The peculiar aspect of scanned data with respect to other 3D models is the average size of the models. To make these data usable on the web we should either deploy efficient geometric compression technology [73], or adopt remote rendering approaches [74]. A web-based application which could boost considerably the usage of 3D scanning is GoogleEarth (or similar systems), since those applications could allow very nice opportunities for announcing on the web the availability of 3D models, showing to the users their availability while navigating the geographical space. Another very promising platform is the WebGL specifications for the inclusion of 3D graphics among the data directly managed by common web browsers [75]. The new generation of web browsers (now in beta version) will therefore support natively digital 3D content, without the need to install specific plug-ins. Some examples of how 3D content could be represented and processed using WebGL are presented on [76]
3D Digitization: How to Improve Current Procedures and Make It More Practical and Successful
While 3D scanning hardware and software technologies have considerably evolved in the last years and their use in the cultural heritage field has gained consent and acceptance, these technologies are far from being easy to use and error-free. Many problems can arise during a scanning campaign; most of these issues are related to various deficiencies in the technologies and the improper background/skills of the operators. Some of these problems have already been briefly mentioned through the topic; we discuss them in detail in this section.
Technology – Limitations Perceived by Practitioners
First of all, 3D scanning technologies are not able to capture many kinds of materials that may occur in the CH field: transparent or semi-transparent surfaces (like glass, jewels, and some stones); mirroring, polished and very shiny objects (like metals); fluff and fuzzy substances like feathers, furs, or some tissues. All those materials are quite difficult to sample with the current off-the-shelf optical technologies. For some of the previous cases, experimental research projects have shown the feasibility of technological solutions that overcome these limitations (in some cases, by adopting enhanced 2D image media, rather than a pure 3D encoding); in several cases, their practical applicability to real projects has still to be assessed.
The working space of the devices is another issue. The CH domain is a very good example of an application where the objects of interest can span the entire interval: from tiny objects (few millimeters) to an entire building or even an archaeological site or a city. With such a wide domain, an important feature of a scanning device would be the flexibility of the working space supported. Conversely, the devices often offer only a very restricted working space (or require a time-consuming calibration to select a slightly different working range). This forces one to select different devices for the different acquisition contexts, increasing the technical skills required to master technology and the overall cost. In this sense, current acquisition methodologies based on pure images (multi stereo matching and similar approaches) present a big advantage with respect to the classical scanning devices, due to the much wider working space of digital cameras (changing focal lenses is much easier than switching to a different scanning device).
Moreover, as already noted before, most of the current scanning technologies focus only on shape, ignoring the issues related to the acquisition of the optical appearance of the scanned surface. Even when the adopted technologies are able to capture the “color” (for example by mapping photos over the surface), in almost all cases this leads only to the acquisition of the apparent reflected color, without considering the reflectance properties of the surface (e.g., its shininess). The acquisition of a correct representation of the reflectance properties of a surface together with its 3D shape is still the domain of research experiments and it is not an off-the-shelf technology. Another sensible issue of 3D scanning is the cost of the hardware equipment: usually it is very high and it can become prohibitive for many low-budgeted cultural heritage institutions. On the other hand, the field is growing and very cheap solutions are appearing on the market. On the positive side we have to note that the research in this field is very active, so we can easily hope that in the future the capabilities of 3D scanning hardware will improve, gradually covering the shortcomings listed above.
Misuse of Technology
The people involved in a scanning campaign can be roughly divided in two sets: technical staff, who actually perform the scanning task with a not-so-strong CH background and sensibility; CH operators/experts, who know very well the specific field and the digitization objectives, but often lack a deep knowledge of the technological details. Obviously there are some notable exceptions, typically in most of the successful 3D scanning projects run so far. But, especially if the field increases in terms of number of devices sold and application testbeds run, the situation will turn more and more towards the case of users with a limited technological background. Designers of software tools should keep this in mind while designing the tools.
Selecting the Wrong Device
As we have sketched in the previous sections, there are many possible technologies available, each one with its own pros and cons. We should underline that scanning system producers are usually very poor in illustrating the conditions which make a given device not fit for the planned task. External constraints (like for example the availability of a given device or of consolidated esperience with a specific technology) might affect the choice of the preferred hardware, leading to the selection of a non-optimal device.
Wrong Data Post-Processing
The wish to provide clean and nice results to the purchaser of a scanning service often causes the undocumented (and often not required) editing of the acquired data. 3D scanning technologies are often unable to completely recover the whole surface, leaving holes and unsampled regions in the final model. Smoothing can reduce the sampling noise in the final model, but it can also cancel important high-frequency details (e.g., traces of deterioration of the surface, traces of the tools used to produce the artwork). Any editing action over the originally acquired data should be documented in the most evident way in order to make it possible to distinguish between ground truth sampled data and the parts that have been added or heavily edited by the subsequent geometric processing phases. Excessive data smoothing or inaccurate simplification are clear examples of actions which could destroy data accuracy and trustability.
Better Management of 3D Data
On the other hand, lack or incomplete technical knowledge of 3D scanning can cause various inconveniences in the overall process of creating trusted digital reproductions of works of art.
Evaluating and Assessing Data Quality
Being able to evaluate in an objective way the final quality of a 3D scanned model is a basic resource for a wide adoption of this technology. In contrast to a more traditional media such as photography, where established protocols and procedures exist and the quality can be determined by an analysis of the (digital) photo and its provenance data, quality assessment of the final digital 3D model is much more complex. Quality depends on a large number of factors: scanning HW characteristics, modality of use of the scanning device, pipeline and parameters of the SW postprocessing which transforms the raw data in the final 3D model. Therefore the whole pipeline, including HW and SW used to produce the digital object, has to be considered while assessing the quality of the digital model. Provenance data are therefore more complex than the ones characterizing the 2D media (photographs). While common practices for the use of general 3D visualization in the research and communication of cultural heritage have been already discussed in the London Charter [77], there is still an absence of clear protocols for the management of 3D scanning technologies and data. Some attempts should be made to define the concept of provenance data for 3D models, in a sufficiently general and standardized way. This is one of the major focuses of the EC research initiative IST IP “3D-COFORM” [21]. 3D-COFORM technlogies are steered towards a comprehensive management of provenance data, by providing integrated management at the level of both the modeling tools and the repository for 3D assets and metadata [78].
The availability of free and standard tools supporting the quality assessment (ideally, by considering both geometry and reflectance data) would be of paramount value, especially for the usually not-technical people who order a scanning campaign. This is an area that desperately needs research and practical results made available to the community. Even a well informed consumer at this point does not have the tools and standards available to make a purchasing decision and a good assessment of the results that are produced by a (usually expensive) project commissioned to an external service company.
Preserving the Data for the Future
An interesting point of discussion is which could be the most fruitful use of the acquired data; a given artifact can be scanned now for a specific application, but once we have the digital model we can envision many different applications of the same 3D model in the future. Therefore, the results of scanning efforts should be made available for future uses, even by different entities. This can be also a way to recover from the cost of the digitization: CH institutions should develop policies to play a role in the market of digital 3D content, finding on the market the financial resources to perform 3D digitization.
Another consideration, that is often neglected, is the long-term preservation of these data. While for traditional media, like written material, photography, and films, the issues about the preservation, classification, and future access to the acquired data is a rather well studied subject, there are no established practices for the preservation of digital 3D content. Without going into details, let us just mention here the issues related to the format of the delivered 3D data: often the results of a 3D scanning campaign are delivered only as files encoded in a closed format, accessible only through proprietary software whose longevity is not guaranteed at all. We are seriously risking that the data produced in several important initiatives are buried in a hidden format without the possibility of being utilized in the near future.
Conclusions
The topic presented the software technologies available for processing and visualization of sampled 3D data. Due to the vast amount of material published in the current literature on this subject, we were forced to present here just a selection of the approaches proposed. Therefore, the bibliography cites only a very small subset of the available literature (otherwise the bibliography could have easily become excessively long). One of the focuses of the presentation has been to give more emphasis to the open problems or to subjects which are currently under study, rather than presenting in detail the algorithms mentioned. Therefore, the current topic should be read by consulting also the papers cited in literature, according to the interests of the reader. Some of those papers are very good resources for gathering a state of the art of the specific problem and for getting a more complete list of citations to the other works in the selected sub-domain. Finally, as usually happens with review papers on a subject where the authors have dedicated time and efforts, we have probably given an emphasis on our own results and papers; we apologize to the reader for that but, as we explicitly mentioned in the text, some results are presented as a representative of several other research efforts which produced comparable outputs.
