Open AccessArticle

Enhancing Image-Based Multiscale Heritage Recording with Near-Infrared Data

Efstathios Adamopoulos

and

Fulvio Rinaudo

^2,*

Università degli Studi di Torino - DI, Corso Svizzera, 185 – 10149 Torino, Italy

Politecnico di Torino - DAD, Viale P.A. Mattioli, 39 – 10125 Torino, Italy

Author to whom correspondence should be addressed.

ISPRS Int. J. Geo-Inf. 2020, 9(4), 269; https://doi.org/10.3390/ijgi9040269

Submission received: 16 March 2020 / Revised: 1 April 2020 / Accepted: 16 April 2020 / Published: 20 April 2020

Download

Browse Figures

Figure 1
Case studies: (a) part of the archaeological site of ancient Kymissala in Rhodes, (b) interior courtyard façades at the Venaria Center of Conservation and Restoration, (c) coromandel folding screen from Castello Cavour in Santena, (d) wooden furniture painted with flowers from Palazzo Chiablese in Turin. "> Figure 2
Pre-signaling and distribution of fixed-location points for geo-referencing and metric checks at the archaeological site of ancient Kymissala. "> Figure 3
Near-infrared (NIR) image acquisition: (a) dataset 1, (b) dataset 2, and (c) dataset 4. "> Figure 4
Number of overlapping images on the area of interest for scenarios 3 (left) and 4 (right). "> Figure 5
Textured models for dataset 1, (a) scenario 1, (b) scenario 2, (c) scenario 3, (d) scenario 4. "> Figure 6
NIR models for dataset 2: Metashape Professional (left) and Zephyr Aerial (right). "> Figure 7
Dataset 3 meshes produced with Metashape Professional, (a) untextured VIS mesh, (b) textured VIS mesh, (c) untextured NIR mesh, (d) textured NIR mesh. "> Figure 8
Variation between VIS and NIR Metashape Professional models for dataset 3 (Hausdorff distances shown on a scale up to 8 mm, with 0.5 mm scalar field intervals). "> Figure 9
Dataset 3 meshes produced with Metashape Professional, (a) untextured VIS mesh, (b) untextured NIR mesh, (c) vertices textured NIR mesh, (d) distances between NIR and VIS meshes. "> Figure 10
Details from the high-resolution orthomosaics produced for dataset 1 from Metashape Professional: scenario 1—VIS (left), scenario 2—NIR (right). "> Figure 11
Data terrain models, constructed after vegetation filtering, produced for dataset 1 from Metashape Professional: scenario 1—VIS (left), scenario 2—NIR (right). "> Figure 12
Comparison of details from the VIS (left) and NIR (right) dataset 2 textured models to evaluate identification of bio-deterioration on the façade’s surface. "> Figure 13
Detail from the NIR dataset 3 textured model, where retouching and defects on the painted surface can be observed. "> Figure 14
Detail from dataset 4 VIS (left) and NIR (right) textures to showcase how restored and repainted areas can be observed with the beyond-VIS acquisition. ">

Versions Notes

Abstract

Passive sensors, operating in the visible (VIS) spectrum, have widely been used towards the trans-disciplinary documentation, understanding, and protection of tangible cultural heritage (CH). Although, many heritage science fields benefit significantly from additional information that can be acquired in the near-infrared (NIR) spectrum. NIR imagery, captured for heritage applications, has been mostly investigated with two-dimensional (2D) approaches or by 2D-to-three-dimensional (3D) integrations following complicated techniques, including expensive imaging sensors and setups. The availability of high-resolution digital modified cameras and software implementations of Structure-from-Motion (SfM) and Multiple-View-Stereo (MVS) algorithms, has made the production of models with spectral textures more feasible than ever. In this research, a short review of image-based 3D modeling with NIR data is attempted. The authors aim to investigate the use of near-infrared imagery from relatively low-cost modified sensors for heritage digitization, alongside the usefulness of spectral textures produced, oriented towards heritage science. Therefore, thorough experimentation and assessment with different software are conducted and presented, utilizing NIR imagery and SfM/MVS methods. Dense 3D point clouds and textured meshes have been produced and evaluated for their metric validity and radiometric quality, comparing to results produced from VIS imagery. The datasets employed come from heritage assets of different dimensions, from an archaeological site to a medium-sized artwork, to evaluate implementation on different levels of accuracy and specifications of texture resolution.

Keywords:

cultural heritage survey; close-range photogrammetry; multi-view stereo; spectral imaging; near-infrared; modified camera

1. Introduction

Close-Range Photogrammetry (CRP) and Technical Photography (TP) constitute two digital-recording techniques that have been widely used in the framework of the integrated documentation and study of tangible CH. The capacity of CRP to digitize three-dimensional (3D) geometrical features, providing accurate representations of the visible surfaces, along with its versatility, makes feasible the interdisciplinary analyses of CH. This technique can provide valuable textural information for the examination of the historical surfaces’ characteristics [1]. It can also produce data necessary to document archaeological sites of different proportions and geometries [2,3,4,5], support excavation activities [6,7], plan the conservation interventions [8,9], and in general, create reference models and systems to assist the three-dimensional integration of various multi-sensor diagnostical data [10,11]. Furthermore, the latest algorithmic developments in the domain of metric exploitation of digital images have enabled increased automation, processing velocities, accuracy, and precision [12,13,14]. Thus, facilitating the implementation of software for the straightforward production of dense point clouds, models, and other digital reconstruction metric derivatives. TP includes a wide range of techniques applicable to historic art examination [15]. Specifically, near-infrared (NIR) imaging has been implemented to enhance archaeological observation [16], to determine the state of conservation of buildings [17], to inspect mural paintings [18], to assist the identification of pigments [19], to investigate underdrawings of panel paintings [20], underprintings [21], and palimpsests [22], to examine rock art [23], and to study feature characteristics of painted artifacts [24]. Applications of integrated heritage CRP and TP can be found in recent bibliography, showcasing a promising combination that should be further evaluated.

Past approaches regarding the integration of metric heritage modeling and information in the near-infrared spectrum have primarily concentrated on separate data acquisitions. Some methods explored for the production of models enhanced with NIR texture through two-dimensional (2D)-to-3D registration are (1) mathematical transformation of the spectral images using corresponding points [25], (2) mutual information methods, as with utilization of depth maps [26] or silhouette maps, reflection maps, and other illumination-based renderings [27], and (3) registration based on known sensor position, as can be performed with optical tracking of the cameras implemented for spectral acquisition [28]. However, the implementation of these approaches is often expensive due to the multiple sensors used and time-consuming due to the frequent need to develop application-specific algorithms.

The recent introduction of consumer-grade digital cameras modified for full-spectrum or single wavelength acquisition to heritage science has provided a less expensive, higher-resolution alternative [29,30]. This solution has spectral imaging capabilities while retaining user-friendly features and interfaces to a wide range of photographic accessories and image processing software. In combination with the automated or semi-automated photogrammetric software implementing Structure-from-Motion (SfM) and Multiple-View-Stereo (MVS) algorithms, which are becoming extremely popular for heritage applications, it can make feasible metric heritage modeling in near-infrared [31,32,33].

The current study focuses on briefly reviewing the modeling of multiscale tangible heritage from NIR imagery with state-of-the-art algorithm SfM/MVS implementations. The presented research has a dual aim, metric and radiometric. The metric aspect of the research refers to the assessment of geometric results that can be achieved with the hybrid CRP and TP approach. The metric evaluations are performed by comparing to the classic CRP approach (using visible spectrum digital images) and to scanning results whenever available. The radiometric aspect concerns the evaluation of NIR textured 3D results, on their capacity to be further exploited towards archaeological or diagnostic observations.

The second section of the paper presents the different cultural heritage case studies, equipment, and methods of the various conducted experimentations. We give special attention to the characteristics of sensors involved and the capturing and processing parameters. It should be highlighted that in order to increase the comparability of the metric results, we attempted to maintain most parameters of the spectral imaging and photogrammetric reconstruction as constants. The third section focuses on the results of image-based 3D modeling and accuracy. The fourth section is devoted to exploring the use of acquired results towards the possible enhancing of archaeological and diagnostic observations. The final section discusses some concluding remarks and future perspectives.

2. Materials and Methods

2.1. Case Studies

The first case study (dataset 1) is the ruins of Vassilika settlement, part of the archaeological site of ancient Kymissala, located about 70 km south-west of Rhodes city in Greece. Kymissala is one of the most important archaeological sites in rural Rhodes, as indicated by the extensive visible ruins scattered in various places, dating from the Mycenaean period to Late Antiquity (see Figure 1a).

The second case study (dataset 2, see Figure 1b) refers to a part of the inner courtyard brick walls of the Center for Conservation and Restoration “La Venaria Reale” (owned by the Consorzio delle Residenze Reali Sabaude). This foundation was established in 2005 as part of the broader restoration works on the Reggia di Venaria Reale palace outside of Turin in Italy. They are located in the former 18th century Scuderie and Maneggio, designed by Benedetto Alfieri (approximate dimensions, 5 × 2.5 m).

The third case study (dataset 3, see Figure 1c) is a detail from a Chinese four-panel Coromandel folding screen from Castello Cavour in Santena (Turin), owned by the Città di Torino—Fondazione Cavour and dated to the 18th century (approximate full dimensions, 2.6 × 2.2 m).

The fourth case study (dataset 4, see Figure 1d) is a wooden furniture part painted with flowers from Palazzo Chiablese (Sala dell’Alcova) in Turin, owned by the Ministero per i Beni e le Attività Culturali dating between the late 19th and early 20th centuries (approximate dimensions, 2.4 × 0.8 m).

2.2. Datasets

Dataset 1 from the archaeological site of ancient Kymissala was acquired during the Erasmus Intensive Program HERICT 2013, an international Summer School for the documentation support of the archaeological excavation in Vassilika settlement in Rhodes [34]. This settlement lies within the wider archaeological site and is the ruins of an organized urban network covering an area of approximately 200 × 250 m² with some 10 m of height differences. For this study, we used only the data captured with a Swinglet fixed wing Unmanned Aircraft System (UAS) by Sensefly. Two 12 Mega-Pixel (MP) camera sensors were used (Table 1); a Canon Ixus 220HS compact camera (sensor size 6.14 × 4.55 mm², pixel size 1.55 μm) for VIS images and a Canon PowerShot ELPH300HS compact camera (sensor size 6.14 × 4.55 mm², pixel size 1.55 μm) for NIR acquisition. The latter was modified by removing the infrared cut filter and placing an internal NIR-only filter [35]. The mission planning with both sensors had been automated for the application using e-motion software, for four flights of approximately 90 m height. Ground control points had been measured using the Global Navigation Satellite System (GNSS) and the Real-Time Kinematic method (RTK) with an accuracy of 2–3 cm. They had been signalized with a 20 × 20 cm² black and white checkerboard pattern and distributed in order to cover the entire area in the most effective way possible (Figure 2). We decided to process the data from each of the four flight scenarios separately.

The datasets for the rest of the case studies (Figure 1b–d) were acquired with a 17.9 MP Canon Rebel SL1 digital single-lens reflex camera (DSLR) camera (sensor size 22.30 × 14.90 mm², pixel size 4.38 μm) converted by ‘Life Pixel Infrared’ for full-spectrum acquisition by removing the IR cut filter. For the VIS and NIR acquisitions, two different external filters were utilized. For the interior case studies, we used flash and a tripod.

A standard capturing workflow was followed to acquire rigid imagery datasets (Figure 3) with large overlaps for image-based 3D reconstruction. It was additionally attempted to maintain capturing conditions (focal length, aperture, exposure, camera positions) and ground sample distances (GSDs) constant between VIS and NIR spectra, for every case study. Referencing for the brick wall case study was realized with a set of 18 pre-signalized control and check points, measured with a total station theodolite (TST) GeoMax Zoom30 3”, producing results with an accuracy of 4–5 mm at the x-axis, 2–3 mm at the y-axis, and 5–6 mm at the z-axis. For the panels and furniture case studies, scaling was performed with an invar scale bar of 1.000165 m (±13 nm). The characteristics of all the datasets are summarized in Table 1.

2.3. Processing Software and Hardware

Photogrammetric processing was conducted through two SfM/MVS-based commercial software. Agisoft Metashape Pro 1.5.1, which uses a scale-invariant feature transform (SIFT)-like algorithm to detect and describe features, a greedy algorithm to find approximate camera locations, and a Global bundle-adjustment matching algorithm to refine them. It employs a form of MVS disparity calculation for dense reconstruction and Screened Poisson surface reconstruction for meshing. 3DFlow Zephyr Aerial 4.519 implements a modified Difference-of-Gaussian (DoG) detector and a combination of Approximate Nearest Neighbor Searching, M-estimator SAmple Consensus, and Geometric Robust Information Criterion for matching, then, performs hierarchical SfM and Incremental bundle-adjustment. The dense MVS reconstruction is achieved with fast visibility integration and tight disparity bounding. The triangulated irregular network construction by an edge-preserving algorithmic approach was selected to differentiate from Agisoft Metashape Professional. All processing was performed with a SANTECH laptop, with a 6-core Intel i7-8750H CPU at 2.2 GHz (Max 4.1 GHz), 32 GB RAM, and NVIDIA GeForce RTX 2070 GPU.

To effectively evaluate the performance of implemented software and the effects of 3D image-based modeling on different spectra, similar parameters, when applicable, were selected for the 3D reconstruction workflows, as summarized in Table 2. The parameters were selected after experimentation to optimize the final 3D-textured results. They were selected in order to maximize preserved surface details, while not producing results of unnecessarily high density, meaning duplicate points—considering each GSD, thus slowing down the processing steps.

The geometric comparisons between vertices of final models were made by measuring the Hausdorff distances in Cloud Compare software. No local model was used for calculating these distances.

We should underline that for the processing of datasets 1 and 4, only specific areas of the dense clouds were selected before the 3D mesh reconstruction step to better showcase the image-based modeling results on areas of higher interest for archaeological/archaeometric observation. Specifically, for the archaeological site of Kymissala, an area of approximately 230 × 180 m² and for the wooden furniture part painted with flowers, an area of 60 × 60 cm² was selected. Consequently, computational steps and results of meshing and texturing refer to those areas only.

3. Results

For dataset 1, full reconstructions were produced for scenarios 1 and 2 (see supplementary file PDF-S1). For the other flight scenarios, due to some existing irregularities of flight conditions and due to smaller overlaps, specific areas were not depicted in two pictures, at least, and therefore, not reconstructed (Figure 4). Scenarios of similar flight altitudes produced similar photogrammetric results with Metashape Professional, in terms of cloud densities, preservation of surface detail on meshes, quality of textures, and required processing times (Table 3). Reconstruction with NIR imagery produced half the root mean square (RMS) Errors on control and check points for the lower flight scenarios but the same levels of RMS Errors for the higher altitude scenarios (Table 4). Figure 5 shows the texturing results achieved.

With Zephyr Aerial, the scene of the archaeological site was not fully reconstructed in any of the acquisition scenarios, producing sparser point clouds (Table 5) with many discontinuities. As a result, in this specific case, it was decided not to continue with the mesh reconstruction phase because fragmentary results would be produced in the area of interest.

Image-based 3D reconstruction for dataset 2 produced very dense modeling results of high-fidelity surface detail (see supplementary file PDF-S2). Overall, Metashape Pro produced denser results with less processing time required (Table 6), but with close examination, it was found that all four processing combinations provided similar 3D detail preservation, notwithstanding that the Zephyr Aerial NIR model had a small number of holes on its upper part.

Metashape Pro also resulted in smaller reprojection errors and measured points’ RMS Errors for both VIS and NIR imagery. Additionally, the two different software resulted in the same level of errors between VIS and NIR imagery processing. Metashape Pro produced control and check RMS Errors of about 1–1.5 mm and Zephyr Aerial of approximately 2.5–3 mm.

For dataset 2, geometric comparisons between the VIS and NIR model from Metashape Pro showed differences of 0.9 mm mean and 0.4 mm RMS and for Zephyr Aerial, of 1.2 mm mean and 0.2 mm RMS. Differences between the two NIR models were below 2 GSDs. Additionally, differences measured between VIS and NIR imagery had 2.6 mm mean, and 1.1 mm RMS distances for Metashape Pro, the same magnitude of variation that was measured between the VIS model and a mesh produced by a Leica BLK 360 scanner point cloud. Also, similar 2.5 mm mean and 1.0 mm RMS distances were present for Zephyr Aerial, comparing to the same laser scanning 3D point cloud after performing down-sampling. Figure 6 showcases the NIR texturing results achieved with both software.

From dataset 3, Zephyr Aerial was not able to reconstruct the scene, neither for the VIS nor for the NIR scenario. Textured meshes produced with Metashape Professional from the two different spectra were of similar density and quality (Table 7). Although, the VIS mesh contained a small amount of noise compared to the NIR (Figure 7), which can be mainly identified through the calculation of geometric differences between them since the same level of detail was preserved on both (Figure 8). The Hausdorff distances calculated between the two models were 0.4 mm mean and 0.3 mm RMS, for an inspected area of approximately 45 × 75 cm².

Image-based reconstruction for dataset 4 produced very high-density results with Metashape Pro, compared to Zephyr Aerial, and overall, performed better with the non-VIS imagery, since Zephyr Aerial produced sparser, noisier, and less complete results in longer processing times (Table 8). For better visualization purposes, part of the mesh and texture results are shown in Figure 9. The geometric differences calculated between VIS and NIR 3D models were 0.5 mm mean and 0.7 mm RMS for Metashape Professional, and 1.0 mm mean and 1.0 RMS for Zephyr Aerial, while Hausdorff distances between the two VIS reconstructions with different algorithmic approaches were 0.9 mm mean and 0.8 mm RMS, and between NIR reconstructions, 1.0 mean and 1.1 RMS for a 0.1 mm sampling distance of original images. Again, VIS mesh contained a small amount of noise compared to the NIR, identifiable through the calculation of geometric differences between them, since the same level of detail was preserved on both. The distances between all 3D models and the mesh produced from a Stonex F6 Short Range structured light scanner were in the range of 1.0 ± 1.0 mm.

4. Discussion

Near-infrared modeling of the archaeological site of Kymissala resulted in slight enhancement of the archaeological observation, without giving any significant insight compared to the visible 3D documentation (Figure 10). Although, using the dense NIR reconstruction results, we were able to construct a fine approximate of the digital terrain model. It should be mentioned that the digital terrain models were constructed by removing canopy only by color filtering. For the VIS and NIR dense point clouds, we used color values at the same coordinates, corresponding to higher, lower, and shadowed vegetation, to classify and then erase vegetation (maintaining constant tolerance values). As showcased in Figure 11, the terrain model produced by NIR imagery is almost noiseless, facilitating the identification of the archaeological remains. Therefore, we could claim that for this case study, NIR modeling made the separation of the canopy easier to create a more accurate terrain model (see supplementary files TIF-S3 and TIF-S4).

For the case study of the brick walls, near-infrared modeling made it possible to perform a rough identification of the areas of bio-deterioration on the surfaces, since these decayed areas have a different response at the NIR spectrum than healthy materials. On the lower areas of the NIR model (Figure 12), decay is easily identifiable and can be discriminated from areas of high-moisture content, which also appear dark on the VIS model. The results were verified by in-situ inspections.

For the case study of the detail from the Chinese four-panel Coromandel Screen, NIR modeling assisted the identification of retouched and defected areas (Figure 13), which appear darker than the uncolored lacquerware background surface. Additionally, NIR modeling helped the production of a noiseless surface 3D model, as described above.

Near infrared modeling of the wooden furniture part painted with flowers helped to better identify defects and restored areas (Figure 14). On the NIR model, we were able to observe undersurface characteristics such as previous restorations, which were performed by filling with new materials and by repainting, cracks, and small deteriorated areas. Those characteristics could not otherwise be detected by visual inspection and VIS modeling only.

5. Conclusions

This paper suggested how the use of near infrared imagery from modified consumer DSLR cameras can be used to enhance the geometry and texture of 3D heritage models at different scales, using image-based modeling software. Considering GSDs and the precision of each methodology, modeling with NIR datasets produced very accurate results, compared to ones produced with VIS datasets. Furthermore, for the very-large-scale case studies, direct modeling from dense NIR imagery resulted in high-resolution noiseless models, compensating for the glaring problems in visible imagery caused by lighting conditions and highly reflective materials. For all CRP reconstructions, both in the visible and in the near-infrared spectrum, Agisoft Metashape performed better than 3DFlow Zephyr. Reconstructions produced by 3DFlow Zephyr were overall sparser, noisier, and had discontinuities. Although, the algorithmic implementations used were intentionally vastly different and some of them were not suitable for every case study. Additionally, we should mention that Agisoft Metashape showed excellent noise-filtering capabilities. Also, in cases of very-large-scale applications, of millimetric or sub-millimetric requirements, 3DFlow Zephyr can perform better, producing a meshing result closer to the raw reconstruction results, with the edge-preserving algorithm, which does not interpolate any data. On the contrary, 3DFlow Zephyr is not recommended for areas of large dimensions as it usually malfunctions. Furthermore, the use of NIR imagery did not seem to have a significant impact on the processing durations or reconstruction errors on any of the employed commercial software.

The reconstructed high-resolution near-infrared textures helped enhance archaeological observation and evaluation of the state of preservation, depending on the heritage case study. For the case study of the archaeological site, NIR modeling facilitated the classification of the canopy to create an approximate of the digital terrain model. For the rest of the case studies, it provided valuable conservation-oriented information, regarding the surface and subsurface characteristics of the historical materials. It should be mentioned that unlike the other spectral imaging techniques, this approach cannot replace the laboratory characterizations when detailed information is required about the characteristics of materials and decay products on historical architecture or artifacts. Although, it can be used as the first diagnostical step to identify areas of higher interest and plan sampling.

To conclude, the 3D modeling approach showcased here proved to be a simple and flexible alternative to previously implemented methodologies for the NIR enhancement of heritage models. This approach can potentially benefit the rapid diagnostics and conservation of multiscale tangible heritage, ranging from small artifacts to historical architecture. Thus, our future research will focus on the implementation of near-infrared and thermal 3D modeling for identification, mapping, and quantification of materials and their deterioration on heritage assets.

Supplementary Materials

The following are available at https://www.mdpi.com/2220-9964/9/4/269/s1. PDF-S1: 3D pdf file of the near-infrared textured model from the archaeological site of Ancient Kymissala, produced with Agisoft Metashape Professional, decimated to 1,000,000 triangles and textured with a single-file 4096 × 4096 pixel² texture. PDF-S2: 3D pdf file of near-infrared textured model from an inner courtyard brick wall of the Center for Conservation and Restoration “La Venaria Reale”, produced with Agisoft Metashape Professional, decimated to 1,000,000 triangles and textured with a single-file 4096 × 4096 pixel² texture. TIF-S3: TIF file of the digital terrain model from the archaeological site of Ancient Kymissala, produced with Agisoft Metashape Professional from visible UAV imagery, with a GSD of 5.5 cm. TIF-S4: TIF file of the digital terrain model from the archaeological site of Ancient Kymissala, produced with Agisoft Metashape Professional from near-infrared UAV imagery, with a GSD of 5.5 cm.

Author Contributions

Conceptualization, Efstathios Adamopoulos; Methodology, Efstathios Adamopoulos; Validation, Efstathios Adamopoulos; Formal analysis, Efstathios Adamopoulos; Investigation, Efstathios Adamopoulos; Resources, Efstathios Adamopoulos and Fulvio Rinaudo; Writing—Original draft, Efstathios Adamopoulos; Visualization, Efstathios Adamopoulos; Supervision, Fulvio Rinaudo; Project administration, Fulvio Rinaudo; Funding acquisition, Fulvio Rinaudo; Writing—review & editing, Fulvio Rinaudo. All authors have read and agreed to the published version of the manuscript.

Funding

This project has received funding from the European Union’s Framework Programme for Research and Innovation Horizon 2020 (2014–2020) under the Marie-Skłodowska Curie Grant Agreement No. 754511 and from the Compagnia di San Paolo.

Acknowledgments

The datasets of the archaeological site of Kymissala, Rhodes were provided by the coordinator and the partners of the Erasmus IP HERICT (http://herict.survey.ntua.gr/2013). The authors would like to acknowledge Consorzio delle Residenze Reali Sabaude for the courteous concession of permission to publish the results about the inner courtyard walls of Fondazione Centro Conservazione e Restauro dei Beni Culturali La Venaria Reale. The authors would like to acknowledge the Italian Ministry of Cultural Heritage and Activities for the courteous concession of permission to publish the results about the wooden furniture part. The authors would like to acknowledge the City of Turin—Cavour Foundation for the courteous concession of permission to publish the results about the 4-panel Coromandel Screen. The authors would also like to acknowledge the contribution of Marianna Ferrero and Dr. Stefania De Blasi, responsible for Labs and Documentation, and Programming and Communications respectively, at the Dept. of Development and Programming at Fondazione Centro Conservazione e Restauro dei Beni Culturali La Venaria Reale in making possible the publishing of results produced at the foundation mentioned above. Finally, the authors would like to acknowledge the assistance of Alessandro Bovero in the acquisition of 3D laser scans for the case study of the inner courtyard walls at Fondazione Centro Conservazione e Restauro dei Beni Culturali La Venaria Reale.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

Dostal, C.; Yamafune, K. Photogrammetric texture mapping: A method for increasing the fidelity of 3D models of cultural heritage materials. J. Archeol. Sci. Rep. 2018, 18, 430–436. [Google Scholar] [CrossRef]
Soile, S.; Tsilimantou, E.; Keramidas, V.; Karoglou, M.; Bourexis, F.; Adamopoulos, E.; Delegou, T.E.; Lampropoulos, K.; Ioannidis, C.; Moropoulou, A.; et al. Multidisciplinary Documentation Using Non-destructive Testing Techniques for the Diagnostic Study of an Ancient Temple. In Nondestructive Evaluation and Monitoring Technologies, Documentation, Diagnosis and Preservation of Cultural Heritage; Osman, A., Moropoulou, A., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 17–36. [Google Scholar]
Aiello, D.; Buscemi, F.; D’Agostino, G.; Galizia, M.; Militello, P.; Santagati, C. Low cost techniques for the digital survey of a Minoan architecture in the archaeological site of Phaistos (Crete). Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2019, XLII-2/W17, 31–38. [Google Scholar] [CrossRef] [Green Version]
Manajitprasert, S.; Tripathi, N.K.; Arunplod, S. Three-Dimensional (3D) Modeling of Cultural Heritage Site Using UAV Imagery: A Case Study of the Pagodas in Wat Maha That, Thailand. Appl. Sci. 2019, 9, 3640. [Google Scholar] [CrossRef] [Green Version]
Chiabrando, F.; D’Andria, F.; Sammartano, G.; Spanò, A. UAV photogrammetry for archaeological site survey. 3D models at the Hierapolis in Phrygia (Turkey). Virtual Archaeol. Rev. 2018, 9, 28. [Google Scholar] [CrossRef] [Green Version]
Rossi, C.; Achille, C.; Fassi, F.; Lori, F.; Rechichi, F.; Fiorillo, F. Digital Workflow to Support Archaeological Excavation: From the 3D Survey to the Websharing of Data. In Innovative Models for Sustainable Development in Emerging African Countries; Aste, N., Della Torre, S., Talamo, C., Adhikari, R.S., Rossi, C., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 149–157. [Google Scholar]
Spanò, A.; Chiabrando, F.; Sammartano, G.; Teppati Losè, L. Integrated imaging approaches supporting the excavation activities. Multi-scale geospatial documentation in Hierapoli (TK). Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2018, XLII–2, 1075–1082. [Google Scholar]
Mirabella Roberti, G.; Nannei, V.M.; Azzola, P.; Cardaci, A. Preserving the Venetian Fortress of Bergamo: Quick Photogrammetric Survey for Conservation Planning. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2019, XLII-2/W11, 873–879. [Google Scholar] [CrossRef] [Green Version]
Adamopoulos, E.; Tsilimantou, E.; Keramidas, V.; Apostolopoulou, M.; Karoglou, M.; Tapinaki, S.; Ioannidis, C.; Georgopoulos, A.; Moropoulou, A. Multi-sensor documentation of metric and qualitative information of historic stone structures. ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. 2017, IV-2/W2, 1–8. [Google Scholar] [CrossRef] [Green Version]
Apollonio, F.I.; Basilissi, V.; Callieri, M.; Dellepiane, M.; Gaiani, M.; Ponchio, F.; Rizzo, F.; Rubino, A.R.; Scopigno, R.; Sobra, G. A 3D-centered information system for the documentation of a complex restoration intervention. J. Cult. Herit. 2018, 29, 89–99. [Google Scholar] [CrossRef]
Tsilimantou, E.; Delegou, E.T.; Nikitakos, I.A.; Ioannidis, C.; Moropoulou, A. GIS and BIM as Integrated Digital Environments for Modeling and Monitoring of Historic Buildings. Appl. Sci. 2020, 10, 1078. [Google Scholar] [CrossRef] [Green Version]
Nikolov, I.; Madsen, C. Benchmarking Close-range Structure from Motion 3D Reconstruction Software Under Varying Capturing Conditions. In Digital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection; Lecture Notes in Computer Science; Ioannides, M., Fink, E., Moropoulou, A., Hagedorn-Saupe, M., Fresa, A., Liestøl, G., Rajcic, V., Grussenmeyer, P., Eds.; Springer International Publishing: Cham, Switzerland, 2016; Volume 10058, pp. 15–26. [Google Scholar]
Adamopoulos, E.; Rinaudo, F. An Updated Comparison on Contemporary Approaches for Digitization of Heritage Objects. In Proceedings of the 5th IMEKO TC-4 International Conference on Metrology for Archaeology and Cultural Heritage, Florence, Italy, 4–6 December 2019; Catelani, M., Daponte, P., Eds.; IMEKO: Florence, Italy, 2019; pp. 1–6. [Google Scholar]
Stathopoulou, E.K.; Welponer, M.; Remondino, F. Open-source image-based 3D reconstruction pipelines: Revie, comparison and evaluation. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2019, XLII-2/W17, 331–338. [Google Scholar] [CrossRef] [Green Version]
Cosentino, A.; Gil, M.; Ribeiro, M.; Di Mauro, R. Technical photography for mural paintings: The newly discovered frescoes in Aci Sant’Antonio (Sicily, Italy). Cons. Patrim. 2014, 20, 23–33. [Google Scholar] [CrossRef]
Verhoeven, G.J.; Smet, P.F.; Poelman, D.; Vermeulen, F. Spectral Characterization of a Digital Still Camera’s NIR Modification to Enhance Archaeological Observation. IEEE Trans. Geosci. Remote Sens. 2009, 47, 3456–3468. [Google Scholar] [CrossRef]
Lerma, J.L.; Cabrelles, M.; Akasheh, T.S.; Haddad, N.A. Documentation of Weathered Architectural Heritage with Visible, near Infrared, Thermal and Laser Scanning Data. Int. J. Herit. Digit. Era 2012, 1, 251–275. [Google Scholar] [CrossRef]
Cosentino, A.; Stout, S.; Scandurra, C. Innovative imaging techniques for examination and documentation of mural paintings and historical graffiti in the catacombs of San Giovanni, Syracuse. Int. J. Conserv. Sci. 2015, 6, 23–34. [Google Scholar]
Delaney, J.K.; Zeibel, J.G.; Thoury, M.; Littleton, R.; Palmer, M.; Morales, K.M.; de la Rie, E.R.; Hoenigswald, A. Visible and Infrared Imaging Spectroscopy of Picasso’s Harlequin Musician: Mapping and Identification of Artist Materials In Situ. Appl. Spectrosc. 2010, 64, 584–594. [Google Scholar] [CrossRef]
Bendada, A.; Sfarra, S.; Ibarra-Castanedo, C.; Akhloufi, M.; Caumes, J.P.; Pradere, C.; Batsale, J.C.; Maldague, X. Subsurface imaging for panel paintings inspection: A comparative study of the ultraviolet, the visible, the infrared and the terahertz spectra. Optoelectron. Rev. 2015, 23, 88–99. [Google Scholar] [CrossRef]
Miguel, C.; Bottura, S.; Ferreira, T.; Conde, A.F.; Barrocas-Dias, C.; Candeias, A. Unveiling the underprintings of a late-fifteenth-early-sixteenth century illuminated French incunabulum by infrared reflectography. J. Cult. Herit. 2019, 40, 34–42. [Google Scholar] [CrossRef] [Green Version]
Easton, R.L., Jr.; Noël, W. The multispectral imaging of the Archimedes Palimpsest. Gazette du Livre Médiéval 2004, 45, 39–49. [Google Scholar] [CrossRef]
Zainuddin, K.; Majid, Z.; Ariff, M.F.M.; Idris, K.M.; Abbas, M.A.; Darwin, N. 3D modeling for Rock Art documentation using lightweight multispectral camera. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2019, XLII-2/W9, 787–793. [Google Scholar] [CrossRef] [Green Version]
Verhoeven, G. Imaging the invisible using modified digital still cameras for straightforward and low-cost archaeological near-infrared photography. J. Archaeol. Sci. 2008, 35, 3087–3100. [Google Scholar] [CrossRef]
Kedzierski, M.; Walczykowski, P.; Wojtkowska, M.; Fryskowska, A. Integration of point clouds and images acquired from a low-cost NIR camera sensor for cultural heritage purposes. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2017, XLII-2/W5, 407–414. [Google Scholar] [CrossRef] [Green Version]
Pelagotti, A.; Mastio, A.D.; Uccheddu, F.; Remondino, F. Automated Multispectral Texture Mapping of 3D Models. In Proceedings of the 17th European Signal Processing Conference (EUSIPCO-2009), Glasgow, Scotland, 24–28 August 2009; IEEE: Piscataway, NJ, USA, 2009; pp. 1215–1219. [Google Scholar]
Corsini, M.; Dellepiane, M.; Ponchio, F.; Scopigno, R. Image-to-Geometry Registration: A Mutual Information Method exploiting Illumination-related Geometric Properties. Comput. Graph. Forum 2009, 28, 1755–1764. [Google Scholar] [CrossRef]
Simon Chane, C.; Mansouri, A.; Marzani, F.S.; Boochs, F. Integration of 3D and multispectral data for cultural heritage applications: Survey and perspectives. Image Vis. Comput. 2013, 31, 91–102. [Google Scholar] [CrossRef] [Green Version]
Blazek, J.; Soukup, J.; Zitova, B.; Flusser, J.; Tichy, T.; Hradilova, J. Low-cost mobile system for multispectral cultural heritage data acquisition. In Proceedings of the Digital Heritage International Congress (DigitalHeritage), Marseille, France, 28 October–1 November 2013; IEEE: Piscataway, NJ, USA, 2013; pp. 73–79. [Google Scholar]
Webb, E.K.; Robson, S.; MacDonald, L.; Garside, D.; Evans, R. Spectral and 3D cultural heritage documentation using a modified camera. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2018, XLII–2, 1183–1190. [Google Scholar] [CrossRef] [Green Version]
Mathys, A.; Jadinon, R.; Hallot, P. Exploiting 3D multispectral texture for a better feature identification for cultural heritage. ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. 2019, IV-2/W6, 91–97. [Google Scholar] [CrossRef] [Green Version]
Adamopoulos, E.; Rinaudo, F.; Bovero, A. First assessments on heritage science-oriented image-based modeling using low-cost modified and mobile cameras. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2019, XLII-2/W17, 23–30. [Google Scholar] [CrossRef] [Green Version]
Pamart, A.; Guillon, O.; Faraci, S.; Gattet, E.; Genevois, M.; Vallet, J.M.; De Luca, L. Multispectral Photogrammetric data acquisition and processing for wall paintings studies. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2017, XLII-2/W3, 559–566. [Google Scholar] [CrossRef] [Green Version]
Pardo, C.A.; Farjas, M.; Georgopoulos, A.; Mielczarek, M.; Parenti, R.; Schramm, T.; Skarlatos, D.; Stefanakis, E.; Tapinaki, S.; Tucci, G.; et al. Experiences gained from the ERASMUS intensive programme HERICT 2013. In Proceedings of the 6th International Conference of Education, Research and Innovation, Seville, Spain, 18–20 November 2013; IATED: Valencia, Spain, 2013; pp. 4424–4431. [Google Scholar]
Nijland, W.; de Jong, R.; de Jong, S.M.; Wulder, M.A.; Bater, C.W.; Coops, N.C. Monitoring plant condition and phenology using infrared sensitive consumer grade digital cameras. Agric. For. Meteorol. 2014, 184, 98–106. [Google Scholar] [CrossRef] [Green Version]

Figure 1. Case studies: (a) part of the archaeological site of ancient Kymissala in Rhodes, (b) interior courtyard façades at the Venaria Center of Conservation and Restoration, (c) coromandel folding screen from Castello Cavour in Santena, (d) wooden furniture painted with flowers from Palazzo Chiablese in Turin.

Figure 2. Pre-signaling and distribution of fixed-location points for geo-referencing and metric checks at the archaeological site of ancient Kymissala.

Figure 3. Near-infrared (NIR) image acquisition: (a) dataset 1, (b) dataset 2, and (c) dataset 4.

Figure 4. Number of overlapping images on the area of interest for scenarios 3 (left) and 4 (right).

Figure 5. Textured models for dataset 1, (a) scenario 1, (b) scenario 2, (c) scenario 3, (d) scenario 4.

Figure 6. NIR models for dataset 2: Metashape Professional (left) and Zephyr Aerial (right).

Figure 7. Dataset 3 meshes produced with Metashape Professional, (a) untextured VIS mesh, (b) textured VIS mesh, (c) untextured NIR mesh, (d) textured NIR mesh.

Figure 8. Variation between VIS and NIR Metashape Professional models for dataset 3 (Hausdorff distances shown on a scale up to 8 mm, with 0.5 mm scalar field intervals).

Figure 9. Dataset 3 meshes produced with Metashape Professional, (a) untextured VIS mesh, (b) untextured NIR mesh, (c) vertices textured NIR mesh, (d) distances between NIR and VIS meshes.

Figure 10. Details from the high-resolution orthomosaics produced for dataset 1 from Metashape Professional: scenario 1—VIS (left), scenario 2—NIR (right).

Figure 11. Data terrain models, constructed after vegetation filtering, produced for dataset 1 from Metashape Professional: scenario 1—VIS (left), scenario 2—NIR (right).

Figure 12. Comparison of details from the VIS (left) and NIR (right) dataset 2 textured models to evaluate identification of bio-deterioration on the façade’s surface.

Figure 13. Detail from the NIR dataset 3 textured model, where retouching and defects on the painted surface can be observed.

Figure 14. Detail from dataset 4 VIS (left) and NIR (right) textures to showcase how restored and repainted areas can be observed with the beyond-VIS acquisition.

Table 1. Description of the datasets.

Dataset	Scenario	Camera Model	Mega- Pixels	Focal Length (mm)	Pixel Size (μm)	Distance (m)	Ground Sample Distance (mm)	Spectrum	Image Count
1	1	IXUS 220HS	12.0	4.0	1.55	86.4	27.8	VIS	82
	2	ELPH 300 HS	12.0	4.0	1.55	86.9	27.7	NIR	82
	3	IXUS 220HS	12.0	4.0	1.55	95.3	31.6	VIS	77
	4	ELPH 300 HS	12.0	4.0	1.55	109.0	36.0	NIR	77
2	1	REBEL-SL1	17.9	18	4.38	1.69	0.36	VIS	110
2	2	REBEL-SL1	17.9	18	4.38	1.67	0.36	NIR	110
3	1	REBEL-SL1	17.9	18	4.38	0.67	0.58	VIS	10
3	2	REBEL-SL1	17.9	18	4.38	0.57	0.58	NIR	10
4	1	REBEL-SL1	17.9	55	4.38	0.86	0.07	VIS	100
4	2	REBEL-SL1	17.9	55	4.38	0.86	0.07	NIR	100

Table 2. Photogrammetric processing parameters.

Dataset	1	2	3	4
Sparse reconstruction
Key point density	highest	high	highest	highest
Matching type	accurate	fast	accurate	accurate
Pair preselection	reference	unordered	unordered	unordered
Key point limits	100 K	50 K	100 K	100 K
Dense reconstruction
Masking	no	no	yes	yes
Point density	high	medium	very high	high
Depth filtering	moderate	moderate	moderate	moderate
Mesh generation
Max faces number	25 M	15 M	10 M	20 M
Quality	high	very high	high	very high
Interpolation	enabled	disabled	enabled	disabled
Texture generation
Mapping mode	ortho	generic	ortho	generic
Blending mode	mosaic	mosaic	average	average
Texture size	3840	16,384	4096	8192
Hole filling	yes	no	yes	no

Table 3. Photogrammetric results for dataset 1 from Agisoft Metashape Professional.

Scenario	1	2	3	4
Sparse Cloud
Aligned images	82	82	76	73
Matching time ¹	0:02:11	0:02:09	0:03:58	0:04:29
Alignment time	0:00:21	0:00:17	0:00:13	0:00:28
Tie point count	87,183	90,028	51,542	63,833
Projections	191,675	197,004	109,756	139,906
Adjustment error (pixels)	0.66	0.55	0.58	0.72
Dense Cloud
Densification time	0:10:24	0:10:06	0:04:42	0:05:53
Point count	75,572	75,785	47,705	50,942
Triangle Mesh
Meshing time	0:13:23	0:14:33	0:06:18	0:06:04
Face count	23,856,573	23,805,413	24,145,080	24,347,432
Vertices count	11,930,446	11,904,337	12,076,150	12,177,636
Texture
Texturing time	0:03:10	0:03:06	0:02:26	0:02:35
Overall Results
Total time	0:29:29	0:30:11	0:17:37	0:19:29

¹ Processing durations in hh:mm:ss.

Table 4. RMS Errors for dataset 1 from Agisoft Metashape Professional.

Scenario	1	2	3	4
Control Points
Count	13	13	10	10
X error (cm)	2.5	1.4	6.4	5.7
Y error (cm)	3.4	1.6	4.1	4.9
Z error (cm)	5.5	3.2	3.7	4.9
Total (cm)	7.0	3.8	8.4	9.0
Check Points
Count	8	8	7	7
X error (cm)	3.1	1.9	5.8	4.6
Y error (cm)	4.1	1.5	4.1	6.6
Z error (cm)	5.6	2.7	8.0	6.1
Total (cm)	7.6	3.6	10.7	10.1

Table 5. Photogrammetric results for dataset 1 from Zephyr Aerial.

Scenario	1	2	3	4
Sparse Cloud
Aligned images	68	69	76	73
Tie point count	21,067	17,435	31,740	25,690
Projections	56,168	50,025	96,064	78,329
Adjustment error (pixels)	0.58	0.56	0.70	0.67
Dense Cloud
Point count	5,762,125	6,846,175	9,315,791	9,959,981

Table 6. Photogrammetric results and accuracy for dataset 2.

Software	Agisoft Metashape		3DFlow Zephyr
Scenario	1	2	1	2
Sparse Cloud
Aligned images	110/110	110/110	110/110	110/110
Matching time ¹	0:01:34	00:01:33	0:31:11	0:36:47
Alignment time	0:03:06	00:02:37	0:03:58	0:02:52
Tie point count ²	443	455	57	64
Projections	1928	1850	481	423
Adjustment error (pixels)	0.51	0.42	0.88	0.91
Dense Cloud
Densification time	0:28:57	00:21:02	0:38:17	0:36:11
Point count	6286	7252	2842	2321
Triangle Mesh
Meshing time	00:02:09	00:01:49	00:00:35	00:00:27
Texture
Texturing time	00:17:26	00:07:46	0:11:48	0:16:05
Overall Results
Total time	00:53:12	00:34:47	1:25:49	1:32:22
Control RMS Error (mm)	1.3	1.6	2.8	2.3
Check RMS Error (mm)	1.1	1.3	3.1	2.7

¹ Processing durations in hh:mm:ss, ² Point counts in 1000.

Table 7. Photogrammetric results from dataset 3 from Metashape Professional.

Scenario	1	2
Sparse Cloud
Aligned images	10	10
Tie point count	37,119	43,105
Projections	165,662	192,583
Adj. error (pixels)	0.21	0.34
Dense Cloud
Point count	1,692,727	1,757,642
Overall Results
Total time (mm:ss)	02:14	01:57

Table 8. Photogrammetric results for dataset 4.

Software	Agisoft Metashape		3DFlow Zephyr
Scenario	1	2	1	2
Sparse Cloud
Aligned images	100	99	100	80
Matching time (hh:mm:ss)	00:02:54	00:02:42	00:22:36	00:21:56
Alignment time (hh:mm:ss)	00:01:20	00:00:35	00:01:30	00:01:16
Tie point count	212,285	110,606	123,262	81,541
Projections	488,205	254,553	380,900	325,400
Adjustment error (pixels)	0.36	0.49	0.45	0.50
Dense Cloud
Densification time (hh:mm:ss)	00:09:32	00:11:45	00:31:26	0:24:12
Point count	120,664,933	198,431,339	11,185,124	8,787,781
Triangle Mesh
Meshing time (hh:mm:ss)	00:11:53	00:07:42	00:28:18	00:21:53
Texture
Texturing time (hh:mm:ss)	00:05:35	00:03:36	00:06:04	00:05:48
Total time (hh:mm:ss)	00:31:14	00:26:20	01:29:54	01:15:05

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Adamopoulos, E.; Rinaudo, F. Enhancing Image-Based Multiscale Heritage Recording with Near-Infrared Data. ISPRS Int. J. Geo-Inf. 2020, 9, 269. https://doi.org/10.3390/ijgi9040269

AMA Style

Adamopoulos E, Rinaudo F. Enhancing Image-Based Multiscale Heritage Recording with Near-Infrared Data. ISPRS International Journal of Geo-Information. 2020; 9(4):269. https://doi.org/10.3390/ijgi9040269

Chicago/Turabian Style

Adamopoulos, Efstathios, and Fulvio Rinaudo. 2020. "Enhancing Image-Based Multiscale Heritage Recording with Near-Infrared Data" ISPRS International Journal of Geo-Information 9, no. 4: 269. https://doi.org/10.3390/ijgi9040269

APA Style

Adamopoulos, E., & Rinaudo, F. (2020). Enhancing Image-Based Multiscale Heritage Recording with Near-Infrared Data. ISPRS International Journal of Geo-Information, 9(4), 269. https://doi.org/10.3390/ijgi9040269

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Menu