ImageOP: The Image Dataset with Religious Buildings in the World Heritage Town of Ouro Preto for Deep Learning Classification
<p>Methodology for the development of the ImageOP dataset.</p> "> Figure 2
<p>Historic Town of Ouro Preto. (<b>a</b>) Chapel of the Governors Palace and Museum of Inconfidence. (<b>b</b>) Church of Saint Francis of Assisi and Pico do Itacolomi. (<b>c</b>) Church of Saint Efigenia and historic houses. (<b>d</b>) Mountains and historic buildings.</p> "> Figure 3
<p>Regions of the historic town of Ouro Preto visited for the development of the ImageOP dataset. Source: modified from Google Maps.</p> "> Figure 4
<p>Religious monuments of Ouro Preto (Part I): (<b>a</b>) Chapel of Lord of Bonfim; (<b>b</b>) Chapel of the Dry Bridge Pass; (<b>c</b>) Chapel of the Governors Palace; (<b>d</b>) Chapel of Saint Anthony; (<b>e</b>) Chapel of the Saint Kings; (<b>f</b>) Chapel of Saint Joseph; (<b>g</b>) Chapel of Our Lady of Piety; (<b>h</b>) Chapel of Our Lady of Conception; (<b>i</b>) Church of Our Lady of Mercy; (<b>j</b>) Chapel of Our Lady of Good Dispatch; (<b>k</b>) Chapel of Saint Luzia; (<b>l</b>) Church of Our Lady of Piety.</p> "> Figure 5
<p>Religious monuments of Ouro Preto (Part II): (<b>a</b>) Church of Our Lady of Nazareth; (<b>b</b>) Church of Our Lady of Sorrows; (<b>c</b>) Basilica of Our Lady of Pilar; (<b>d</b>) Church of Saint Francis of Assisi; (<b>e</b>) Church of Our Lady of Mercy and Pardons; (<b>f</b>) Church of Saint Francis of Paula; (<b>g</b>) Church of Our Lady of Mount Carmel; (<b>h</b>) Sanctuary of Our Lady of Conception; (<b>i</b>) Church of Saint Anthony of Leite; (<b>j</b>) Church of Saint Anthony of Casa Branca; (<b>k</b>) Church of Good Jesus of Matosinhos and Saint Michael and Souls; (<b>l</b>) Church of Saint Efigenia; (<b>m</b>) Church of Our Lady of Mercy and Compassion; (<b>n</b>) Church of Saint Gonçalo; (<b>o</b>) Church of Our Lady of the Rosary; (<b>p</b>) Church of Saint Bartholomew; (<b>q</b>) Church of Our Lady of Mercy (Cachoeira do Campo); (<b>r</b>) Church of Our Lady of Sorrows of Mount Calvary; (<b>s</b>) Church of Saint Joseph; (<b>t</b>) Church of Our Lady of Mercy (São Bartolomeu).</p> "> Figure 6
<p>Image collection process in the historic town of Ouro Preto. (<b>a</b>) Data collection of small church. (<b>b</b>) Data collection of church.</p> "> Figure 7
<p>Kodak<sup>®</sup> PIXPRO AZ255 digital camera. (<b>a</b>) Front view of the camera. (<b>b</b>) Camera display.</p> "> Figure 8
<p>(<b>a</b>) Building components in a church: (1) fronton; (2) door; (3) window; (4) tower. (<b>b</b>) Building components in a small church (chapel): (1) fronton; (2) door.</p> "> Figure 9
<p>Examples of images from the fronton class. (<b>a</b>–<b>l</b>) Photographs of church pediments in the Historic Town of Ouro Preto.</p> "> Figure 10
<p>Examples of images from the church class. (<b>a</b>–<b>l</b>) Photographs of churches in the Historic Town of Ouro Preto.</p> "> Figure 11
<p>Examples of images from the window class. (<b>a</b>–<b>l</b>) Photographs of church windows in the Historic Town of Ouro Preto.</p> "> Figure 12
<p>Examples of images from the door class. (<b>a</b>–<b>l</b>) Photographs of church doors in the Historic Town of Ouro Preto.</p> "> Figure 13
<p>Examples of images from the tower class. (<b>a</b>–<b>l</b>) Photographs of church towers in the Historic Town of Ouro Preto.</p> "> Figure 14
<p>Method of dataset benchmarking for deep learning classification.</p> "> Figure 15
<p>Graph of the train and validation history for the MobileNet architecture: (<b>a</b>) accuracy; (<b>b</b>) loss.</p> "> Figure 16
<p>Confusion matrix for the MobileNet architecture.</p> "> Figure 17
<p>Graph of the train and validation history for the EfficientNet architecture: (<b>a</b>) accuracy; (<b>b</b>) loss.</p> "> Figure 17 Cont.
<p>Graph of the train and validation history for the EfficientNet architecture: (<b>a</b>) accuracy; (<b>b</b>) loss.</p> "> Figure 18
<p>Confusion matrix for the EfficientNet architecture.</p> "> Figure 19
<p>Proposed procedure for applying computer vision using smartphones to recognize elements of religious buildings.</p> "> Figure 20
<p>Process of computer vision using mobile device in the historic town of São João del-Rei. (<b>a</b>) Church door detection. (<b>b</b>) Church Detection.</p> "> Figure 21
<p>Example of deep learning classification using computer vision with a mobile device and Edge Impulse software. Detected class: window (<span class="html-italic">janela</span> in Portuguese).</p> "> Figure 22
<p>Examples of real-time classification from screenshots of the Edge Impulse graphical interface accessed on the mobile device.</p> ">
Abstract
:1. Introduction
- A new image dataset featuring religious buildings in the World Heritage Town of Ouro Preto;
- Deep learning for the recognition of five classes pertaining to historical temples of Brazilian religious architecture: fronton, door, window, tower, and church;
- Simulated experiments and real-world applications using computer vision and mobile devices (smartphones) for the detection of components in historical religious buildings.
2. Related Work
- Dataset: the literature lacks new datasets of religious buildings, especially with images of Brazilian Baroque and Rococo architecture;
- Classes: in general, previous work does not address the set of common building elements on the facades of Brazilian churches from the 18th century: fronton, window, door, and tower;
- Experiments: the literature also needs new deep learning approaches for practical experiments in real environments using smartphones.
3. ImageOP: Our Contribution
3.1. Scientific Motivation
3.2. Historic Town of Ouro Preto
3.3. Religious Buildings
3.4. Data Collection of Building Components
3.5. Dataset Result
- 1.
- Fronton (Pediment): the upper part of religious buildings, usually accompanied by a cross;
- 2.
- Church: class containing photographs of religious monuments (normal and small size) featuring multiple building components in the same image;
- 3.
- Window: images of windows (front or side) of religious buildings;
- 4.
- Door: images of doors (front or side) of religious buildings;
- 5.
- Tower: photographs of towers of religious buildings, typically featuring a bell.
4. Dataset Benchmarking for Deep Learning Classification
4.1. Simulation Experiments
4.2. Simulation Results
4.3. Computer Vision Using Mobile Device
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
1 | https://www.ouropreto.mg.gov.br/turismo/atrativo/113 (accessed on 24 October 2024). |
2 | https://www.ouropreto.mg.gov.br/turismo/atrativo/139 (accessed on 24 October 2024). |
3 | |
4 |
References
- Piccialli, F.; Casolla, G.; Cuomo, S.; Giampaolo, F.; Prezioso, E.; Di Cola, V.S. Unsupervised learning on multimedia data: A Cultural Heritage case study. Multimed. Tools Appl. 2020, 79, 34429–34442. [Google Scholar] [CrossRef]
- Spennemann, D.H. Will Artificial Intelligence Affect How Cultural Heritage Will Be Managed in the Future? Responses Generated by Four genAI Models. Heritage 2024, 7, 1453–1471. [Google Scholar] [CrossRef]
- Mishra, M.; Lourenço, P.B. Artificial intelligence-assisted visual inspection for cultural heritage: State-of-the-art review. J. Cult. Herit. 2024, 66, 536–550. [Google Scholar] [CrossRef]
- Silva, C.; Oliveira, L. Artificial Intelligence at the Interface between Cultural Heritage and Photography: A Systematic Literature Review. Heritage 2024, 7, 3799–3820. [Google Scholar] [CrossRef]
- Karimi, N.; Mishra, M.; Lourenço, P.B. Deep learning-based automated tile defect detection system for Portuguese cultural heritage buildings. J. Cult. Herit. 2024, 68, 86–98. [Google Scholar] [CrossRef]
- Li, Y.; Zhao, M.; Mao, J.; Chen, Y.; Zheng, L.; Yan, L. Detection and recognition of Chinese porcelain inlay images of traditional Lingnan architectural decoration based on YOLOv4 technology. Herit. Sci. 2024, 12, 137. [Google Scholar] [CrossRef]
- Maria, M.D.; Fiumi, L.; Mazzei, M.; V, B.O. A system for monitoring the environment of historic places using convolutional neural network methodologies. Heritage 2021, 4, 1429–1446. [Google Scholar] [CrossRef]
- Liu, Z.; Brigham, R.; Long, E.R.; Wilson, L.; Frost, A.; Orr, S.A.; Grau-Bové, J. Semantic segmentation and photogrammetry of crowdsourced images to monitor historic facades. Herit. Sci. 2022, 10, 1–17. [Google Scholar] [CrossRef]
- Yazdi, H.; Sad Berenji, S.; Ludwig, F.; Moazen, S. Deep learning in historical architecture remote sensing: Automated historical courtyard house recognition in Yazd, Iran. Heritage 2022, 5, 3066–3080. [Google Scholar] [CrossRef]
- Samhouri, M.; Al-Arabiat, L.; Al-Atrash, F. Prediction and measurement of damage to architectural heritages facades using convolutional neural networks. Neural Comput. Appl. 2022, 34, 18125–18141. [Google Scholar] [CrossRef]
- Kumar, P.; Ofli, F.; Imran, M.; Castillo, C. Detection of disaster-affected cultural heritage sites from social media images using deep learning techniques. J. Comput. Cult. Herit. (JOCCH) 2020, 13, 1–31. [Google Scholar] [CrossRef]
- Zou, Z.; Zhao, X.; Zhao, P.; Qi, F.; Wang, N. CNN-based statistics and location estimation of missing components in routine inspection of historic buildings. J. Cult. Herit. 2019, 38, 221–230. [Google Scholar] [CrossRef]
- Llamas, J.; Lerones, P.M.; Medina, R.; Zalama, E.; Gómez-García-Bermejo, J. Classification of architectural heritage images using deep learning techniques. Appl. Sci. 2017, 7, 992. [Google Scholar] [CrossRef]
- Janković, R. Machine learning models for cultural heritage image classification: Comparison based on attribute selection. Information 2020, 11, 12. [Google Scholar] [CrossRef]
- Siountri, K.; Anagnostopoulos, C.N. The classification of cultural heritage buildings in athens using deep learning techniques. Heritage 2023, 6, 3673–3705. [Google Scholar] [CrossRef]
- Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; Chen, L.C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 4510–4520. [Google Scholar]
- Tan, M. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv 2019, arXiv:1905.11946. [Google Scholar]
- Abayomi-Alli, O.O.; Damaševičius, R.; Misra, S.; Abayomi-Alli, A. FruitQ: A new dataset of multiple fruit images for freshness evaluation. Multimed. Tools Appl. 2024, 83, 11433–11460. [Google Scholar] [CrossRef]
- Ali, A.; Sarkar, R.; Das, D.K. IRUVD: A new still-image based dataset for automatic vehicle detection. Multimed. Tools Appl. 2024, 83, 6755–6781. [Google Scholar] [CrossRef]
- Lamas, A.; Tabik, S.; Cruz, P.; Montes, R.; Martínez-Sevilla, Á.; Cruz, T.; Herrera, F. MonuMAI: Dataset, deep learning pipeline and citizen science based app for monumental heritage taxonomy and classification. Neurocomputing 2021, 420, 266–280. [Google Scholar] [CrossRef]
- de Alvarenga Pereira Costa, S.; Barci Castriota, L.; Salgado, M. The World Heritage site of Ouro Preto. Facilities 2011, 29, 339–351. [Google Scholar] [CrossRef]
- UNESCO. United Nations Educational, Scientific and Cultural Organization—World Heritage List: Historic Town of Ouro Preto; United Nations Educational, Scientific and Cultural Organization. 2024. Available online: https://whc.unesco.org/en/list/124 (accessed on 19 September 2024).
- Barbosa, V.d.S.B.; de Lima, H.M.; Fonseca, B.M. Assessing risks of abandoned urban mines in the UNESCO World Heritage City of Ouro Preto, Brazil. Appl. Geogr. 2022, 139, 102648. [Google Scholar] [CrossRef]
- Valerio, M.A. Architects of their own humanity: Race, devotion, and artistic agency in Afro-Brazilian confraternal churches in eighteenth-century Salvador and Ouro Preto. Colon. Lat. Am. Rev. 2021, 30, 238–271. [Google Scholar] [CrossRef]
- Bhakti, T.; Rossi, F.; de Oliveira Mafia, P.; de Almeida, E.F.; Fujaco, M.A.G.; de Azevedo, C.S. Preservation of historical heritage increases bird biodiversity in urban centers. Environ. Dev. Sustain. 2021, 23, 8758–8773. [Google Scholar] [CrossRef]
- Boniek, D.; de Abreu, C.S.; Dos Santos, A.F.B.; de Resende Stoianoff, M.A. Evaluation of microbiological air parameters and the fungal community involved in the potential risks of biodeterioration in a cultural heritage of humanity, Ouro Preto, Brazil. Folia Microbiol. 2021, 66, 797–807. [Google Scholar] [CrossRef]
- Santos, B.H.d.; Fonseca Filho, R.E.; Castro, P.d.T.A. Potential Urban Geotourist profile of Ouro Preto (MG, Brazil). Anu. Inst. Geociencias 2021, 44, 1–14. [Google Scholar]
- Froner, Y.A.; de Barros Gonçalves, W.; Souza, L.A.C.; Costa, A.G.; Rosado, A.; Mizrahy Cuperschmid, A.R.; de Oliveira Walter, G.; Neves Miranda, A.C.; Michelin, G.; Montalvão, A.C.; et al. Data Collection for Cultural Heritage Risk Management: The Damage Map through Heritage Building Information Modeling (HBIM) Project Applied to the Façade of St Francis of Assisi, Ouro Preto, Brazil. Stud. Conserv. 2024, 69, 98–107. [Google Scholar] [CrossRef]
- Cuperschmid, A.R.M.; Neves de Oliveira, G.; Froner, Y.A. Exploring the Use of LiDAR in Smartphones: Documenting the Frontispiece of Saint Francis of Assisi Church in Ouro Preto, Brazil. Int. J. Archit. Herit. 2024, 1–18. [Google Scholar] [CrossRef]
- Ottoni, A.L.C.; Souza, A.M.; Novo, M.S. Automated hyperparameter tuning for crack image classification with deep learning. Soft Comput. 2023, 27, 18383–18402. [Google Scholar] [CrossRef]
- Almujally, N.A.; Alzahrani, A.; Hakeem, A.M.; Attiah, A.; Umer, M.; Alsubai, S.; Polsinelli, M.; Ashraf, I. Selective feature-based ovarian cancer prediction using MobileNet and explainable AI to manage women healthcare. Multimed. Tools Appl. 2024, 1–22. [Google Scholar] [CrossRef]
- Liu, L.; Wang, X.; Bao, Q.; Li, X. Behavior detection and evaluation based on multi-frame MobileNet. Multimed. Tools Appl. 2024, 83, 15733–15750. [Google Scholar] [CrossRef]
- Marques, G.; Ferreras, A.; de la Torre-Diez, I. An ensemble-based approach for automated medical diagnosis of malaria using EfficientNet. Multimed. Tools Appl. 2022, 81, 28061–28078. [Google Scholar] [CrossRef]
- Nair, A.; Vadher, H.; Patel, P.; Vyas, T.; Bhatt, C.; Bruno, A. Lung sound disease detection using attention over pre-trained efficientnet architecture. Multimed. Tools Appl. 2024, 1–35. [Google Scholar] [CrossRef]
Pr. | I | II | III | IV | ||
---|---|---|---|---|---|---|
[13] | [14] | [15] | [20] | |||
Dataset | New image dataset | ✓ | ✓ | – | ✓ | ✓ |
Religious buildings | ✓ | ✓ | ✓ | – | ✓ | |
Brazilian cultural heritage | ✓ | – | – | – | – | |
Classes | Fronton | ✓ | – | – | – | – |
Church | ✓ | – | – | – | – | |
Window | ✓ | – | – | ✓ | – | |
Door | ✓ | – | – | ✓ | – | |
Tower | ✓ | ✓ | – | – | – | |
Others | – | ✓ | ✓ | ✓ | ✓ | |
Experiments | Simulations with deep learning | ✓ | ✓ | ✓ | ✓ | ✓ |
Computer vision using smartphone | ✓ | – | – | – | ✓ |
Regions | Distance (km) | Religious Buldings |
---|---|---|
Center of Ouro Preto | – | 15 |
Cachoeira do Campo | 18 | 5 |
St. Antônio do Leite | 25 | 5 |
Glaura | 22 | 3 |
Amarantina | 25 | 2 |
São Bartolomeu | 15 | 2 |
Total | – | 32 |
# | Religious Building | Region |
---|---|---|
1 | Church of Our Lady of Nazareth | Cachoeira do Campo |
2 | Church of Our Lady of Sorrows | Cachoeira do Campo |
3 | Basilica of Our Lady of Pilar | Center of Ouro Preto |
4 | Church of Saint Francis of Assisi | Center of Ouro Preto |
5 | Church of Our Lady of Mercy and Pardons | Center of Ouro Preto |
6 | Church of Saint Francis of Paula | Center of Ouro Preto |
7 | Church of Our Lady of Mount Carmel | Center of Ouro Preto |
8 | Sanctuary of Our Lady of Conception | Center of Ouro Preto |
9 | Church of Saint Anthony of Leite | St. Antônio do Leite |
10 | Church of Saint Anthony of Casa Branca | Glaura |
11 | Church of G. Jesus of Matosinhos, St. Michael and Souls | Center of Ouro Preto |
12 | Church of Saint Efigenia | Center of Ouro Preto |
13 | Church of Our Lady of Mercy and Compassion | Center of Ouro Preto |
14 | Church of Saint Gonçalo | Amarantina |
15 | Church of Our Lady of the Rosary | Center of Ouro Preto |
16 | Church of Saint Bartholomew | São Bartolomeu |
17 | Church of Our Lady of Mercy | Cachoeira do Campo |
18 | Church of Our Lady of Sorrows of Mount Calvary | Center of Ouro Preto |
19 | Church of Saint Joseph | Center of Ouro Preto |
20 | Church of Our Lady of Mercy | São Bartolomeu |
21 | Chapel of Lord of Bonfim | Center of Ouro Preto |
22 | Chapel of the Dry Bridge Pass | Center of Ouro Preto |
23 | Chapel of the Governors Palace | Center of Ouro Preto |
24 | Chapel of Saint Anthony | Cachoeira do Campo |
25 | Chapel of the Saint Kings | St. Antônio do Leite |
26 | Chapel of Saint Joseph | St. Antônio do Leite |
27 | Chapel of Our Lady of Piety | Amarantina |
28 | Chapel of Our Lady of Conception | Glaura |
29 | Chapel of Our Lady of Mercy | Glaura |
30 | Chapel of Our Lady of Good Dispatch | Cachoeira do Campo |
31 | Chapel of Saint Luzia | St. Antônio do Leite |
32 | Chapel of Our Lady of Piety | St. Antônio do Leite |
Class | Number of Images |
---|---|
Fronton | 290 |
Church | 353 |
Window | 341 |
Door | 292 |
Tower | 337 |
Total | 1613 |
Class | Train (≈) | Validation (≈) | Test (≈) | Total () |
---|---|---|---|---|
Fronton | 176 | 59 | 55 | 290 |
Church | 223 | 61 | 69 | 353 |
Window | 232 | 44 | 65 | 341 |
Door | 191 | 44 | 57 | 292 |
Tower | 219 | 53 | 65 | 337 |
Total | 1041 | 261 | 311 | 1613 |
Architecture | Accuracy | Precision | Recall | F-Score |
---|---|---|---|---|
MobileNet | 92.6 | 94.0 | 94.0 | 93.0 |
EfficientNet | 94.5 | 96.0 | 96.0 | 96.0 |
Architecture | Fronton | Church | Window | Door | Tower |
---|---|---|---|---|---|
MobileNet | 92.7 | 97.1 | 81.5 | 98.2 | 93.8 |
EfficientNet | 96.4 | 97.1 | 89.2 | 94.7 | 95.4 |
Class | Sample | Fronton | Church | Window | Door | Tower |
---|---|---|---|---|---|---|
Fronton | 100.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
Fronton | 100.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
Church | 0.0 | 95.0 | 2.5 | 0.0 | 2.5 | |
Church | 0.0 | 87.0 | 0.0 | 4.0 | 9.0 | |
Window | 0.0 | 0.0 | 100.0 | 0.0 | 0.0 | |
Window | 0.0 | 0.0 | 81.0 | 3.0 | 16.0 | |
Door | 0.0 | 0.0 | 4.0 | 96.0 | 0.0 | |
Door | 0.0 | 0.0 | 0.0 | 100.0 | 0.0 | |
Tower | 2.0 | 0.0 | 2.0 | 0.0 | 96.0 | |
Tower | 1.0 | 2.0 | 0.0 | 0.0 | 97.0 |
Monument | Name | Historic Town | Acc |
---|---|---|---|
Church of Saint Francis of Assisi | São João del-Rei | 100.0 | |
Church of Our Lady of Mount Carmel | São João del-Rei | 88.9 | |
Cathedral Basilica of Our Lady of Pilar | São João del-Rei | 93.8 | |
Church of Our Lady of the Rosary | São João del-Rei | 92.3 | |
Chapel of Divine Holy Spirit | São João del-Rei | 75.0 | |
Chapel of Saint Anthony | São João del-Rei | 80.0 | |
Church of Our Lady of Mercy | São João del-Rei | 85.7 | |
Church of Saint Anthony | Lagoa Dourada | 92.9 | |
Church of Saint Bras | São Brás do Suaçuí | 83.3 | |
Avg. Accuracy | 88.0 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Ottoni, A.L.C.; Ottoni, L.T.C. ImageOP: The Image Dataset with Religious Buildings in the World Heritage Town of Ouro Preto for Deep Learning Classification. Heritage 2024, 7, 6499-6525. https://doi.org/10.3390/heritage7110302
Ottoni ALC, Ottoni LTC. ImageOP: The Image Dataset with Religious Buildings in the World Heritage Town of Ouro Preto for Deep Learning Classification. Heritage. 2024; 7(11):6499-6525. https://doi.org/10.3390/heritage7110302
Chicago/Turabian StyleOttoni, André Luiz Carvalho, and Lara Toledo Cordeiro Ottoni. 2024. "ImageOP: The Image Dataset with Religious Buildings in the World Heritage Town of Ouro Preto for Deep Learning Classification" Heritage 7, no. 11: 6499-6525. https://doi.org/10.3390/heritage7110302
APA StyleOttoni, A. L. C., & Ottoni, L. T. C. (2024). ImageOP: The Image Dataset with Religious Buildings in the World Heritage Town of Ouro Preto for Deep Learning Classification. Heritage, 7(11), 6499-6525. https://doi.org/10.3390/heritage7110302