Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Toward an Automated Pipeline for a Browser-Based, City-Scale Mobile 4D VR Application Based on Historical Images

  • Conference paper
  • First Online:
Research and Education in Urban History in the Age of Digital Libraries (UHDL 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1501))

  • 472 Accesses

Abstract

The process for automatically creating 3D city models from contemporary photographs and visualizing them on mobile devices is well established. 4D city models that can display a temporal dimension are far more complex to generate automatically. In this article, we focus on major challenges in the process of developing an automated pipeline, starting from content-based image retrieval applied to historical images, via automatic historical image orientation, up to visualization of the 4D data in Virtual Reality (VR). The result is an interactive browser-based device-rendered 4D visualization and information system for mobile devices. This pipeline has been in development since 2015. In this article, we present initial results and early-stage findings in the process of overcoming three major challenges on the way to 4D city models: (1) to identify photographs with corresponding views, (2) to reconstruct the position and orientation of photographs and (3) to design a user-centered, browser-based 4D mobile application.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Münster, S., Maiwald, F., Lehmann, C., Lazariv, T., Hofmann, M., Niebling, F.: Introducing an automated pipeline for a browser-based, city-scale mobile 4D VR application based on historical images. Paper Presented at the ACM Multimedia - SUMAC Workshop, Seattle (2020)

    Google Scholar 

  2. Ceconello, M., Spagnoli, A., Spallazzo, D., Tolino, U.: Playing design- mobile serious games to valorize design culture in the urban space. Paper Presented at the Digital Heritage 2015, Granada, Spain (2015)

    Google Scholar 

  3. Breitenstein, M., Münster, S., Niebling, F.: Gamifizierte augmented reality-anwendungen im tourismuskontext: ein literaturreview zu Gestaltungsansätzen, Chancen und Risiken. In: Köhler, T. (ed.) Communities in New Media. Researching the Digital Transformation in Science, Business, Education & Public Administration, pp 197–209. TUDPress, Dresden (2019)

    Google Scholar 

  4. ViMM WG 2.2: Meaningful Content connected to the Real World (Report) (2017)

    Google Scholar 

  5. Mathis, R.: Freiburg Zeitreise - die Stadtjubiläum 2020 app (version 1.1.0) (2020). https://play.google.com/store/apps/details?id=com.extendedvision.futurehistory.freiburg2020&hl=de. Accessed 21 Dec 2020

  6. Youssef, M.: Museum of London: streetmuseum app (version 2.03) (2016). https://apkpure.com/de/museum-of-london-streetmuseum/com.streetmuseum. Accessed 21 Dec 2020

  7. Schücking, B.A.: Zeitfenster app - friedliche revolution leipzig (version 1.1.1) (2018). https://zeitfenster.uni-leipzig.de/. Accessed 21 Dec 2020

  8. Burkert, P., Straubinger, S., Schaufler, B.: Zeitfenster app (2018). https://www.zeitfenster-app.de/. Accessed 21 Dec 2020

  9. Berdin, J., Helder, M., Fridhi, A.: Applications of the urban timetravel project (2019). https://www.urbantimetravel.com/project. Accessed 21 Dec 2020

  10. Buhrow, T.: WDR 360° VR app (2017). https://play.google.com/store/apps/details?id=de.WDR.VR&hl=de. Accessed 10 Oct 2020

  11. Buhrow, T.: NRW2go app (version 1.0.3) (2019). https://play.google.com/store/apps/details?id=de.WDR.NRW2go&hl=de. Accessed 10 Oct 2020

  12. Bellut, T.: History 360° (2019). https://history360.zdf.de/. Accessed 10 Oct 2020

  13. Schulte-Kellinghaus, J.: The Berlin wall – a multimedia history (2013). https://www.the-berlin-wall.com/. Accessed 10 Oct 2020

  14. Mager, T., Hein, C.: Digital excavation of mediatized urban heritage: automated recognition of buildings in image sources. Urban Plann. 5(2), 24–34 (2020)

    Article  Google Scholar 

  15. Maiwald, F., Schneider, D., Henze, F., Münster, S., Niebling, F.: Feature matching of historical images based on geometry of quadrilaterals. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLII-2, 643–650 (2018). https://doi.org/10.5194/isprs-archives-XLII-2-643-2018

  16. Pomaska, G.: Zur Dokumentation und 3D-modellierung von denkmalen mit digitalen fotografischen verfahren. In: Heine, K., Rheidt, K., Henze, F., Riedel, A. (eds.) Von Handaufmaß bis High Tech III - 3D in der historischen Bauforschung, pp. 26–32. Verlag Philipp von Zabern, Mainz (2011)

    Google Scholar 

  17. Maiwald, F., Vietze, T., Schneider, D., Henze, F., Münster, S., Niebling, F.: Photogrammetric analysis of historical image repositories for virtual reconstruction in the field of digital humanities. Int. Arch. Photogr. Remote Sens. Spat. Inf. Sci. 42, 447 (2017). https://doi.org/10.5194/isprs-archives-XLII-2-W3-447-2017

    Article  Google Scholar 

  18. Wood, J., Isenberg, P., Isenberg, T., Dykes, J., Boukhelifa, N., Slingsby, A.: Sketchy rendering for information visualization. IEEE Trans. Vis. Comput. Graphics 18(12), 2749–2758 (2012). https://doi.org/10.1109/TVCG.2012.262

    Article  Google Scholar 

  19. Glaser, M., Lengyel, D., Toulouse, C., Schwan, S.: Designing computer-based learning contents: influence of digital zoom on attention. Educ. Tech. Res. Dev. 65(5), 1135–1151 (2016). https://doi.org/10.1007/s11423-016-9495-9

    Article  Google Scholar 

  20. Münster, S.: Cultural heritage at a glance. Four case studies about the perception of digital architectural 3D models. In: Alonso, F. (ed.) 2018 3rd Digital Heritage International Congress (DigitalHERITAGE) held jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018). IEEE, San Francisco (2018)

    Google Scholar 

  21. Burmester, M., et al.: Lost in space? 3D-interaction-patterns für einfache und positive nutzung von 3D interfaces. In: Hess, S., Fischer, H. (eds.) Mensch und Computer 2018 – Usability Professionals (Electronic Book). Gesellschaft für Informatik e.V. und German UPA e.V., Bonn (2018)

    Google Scholar 

  22. Oliveira, S.A., Lenardo, I.D., Kaplan, F.: Machine vision algorithms on cadaster plans. In: Conference of the International Alliance of Digital Humanities Organizations (DH 2017), Montreal, Canada, 8–11 August 2017 (2017)

    Google Scholar 

  23. Razavian, A.S.: CNN features off-the-shelf: an astounding baseline for recognition. Paper Presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2014)

    Google Scholar 

  24. Wan, J., et al.: Deep learning for content-based image retrieval: a comprehensive study. Proceedings of the 22nd ACM International Conference on Multimedia, pp. 157–166 (2014)

    Google Scholar 

  25. Razavian, A.S., et al.: Visual instance retrieval with deep convolutional networks. ITE Trans. Media Technol. Appl. 4(3), 251–258 (2016)

    Article  Google Scholar 

  26. Simonyan, K., Zisserman, A.: Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  27. Chollet, F.: Deep Learning with Python. Manning, Shelter Island (2018)

    Google Scholar 

  28. Ting, K.M.: Confusion matrix. In: Sammut, C., Webb, G.I. (eds.) Encyclopedia of Machine Learning and Data Mining. Springer, Boston (2016). https://doi.org/10.1007/978-1-4899-7502-7_50-1

  29. Ting, K.M.: Precision and recall. In: Sammut, C., Webb, G.I. (eds.) Encyclopedia of Machine Learning and Data Mining. Springer, Boston (2016). https://doi.org/10.1007/978-1-4899-7502-7_659-1

    Chapter  Google Scholar 

  30. Davis, J., Goadrich, M.: The relationship between precision-recall and ROC curves. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 233–240 (2006)

    Google Scholar 

  31. Schindler, G., Dellaert, F.: 4D cities: analyzing, visualizing, and interacting with historical urban photo collections. J. Multimed. 7(2), 124–131 (2012). https://doi.org/10.4304/jmm.7.2.124-131

    Article  Google Scholar 

  32. Zawieska, D., Markiewicz, J.: Development of photogrammetric documentation of the borough at biskupin based on archival photographs - first results. In: Ioannides, M., et al. (eds.) EuroMed 2016. LNCS, vol. 10059, pp. 3–9. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48974-2_1

    Chapter  Google Scholar 

  33. Bevilacqua, M.G., Caroti, G., Piemonte, A., Ulivieri, D.: Reconstruction of lost architectural volumes by integration of photogrammetry from archive imagery with 3-D models of the status quo. Int. Arch. Photogr. Remote Sens. Spat. Inf. Sci. XLII-2/W9, 119–125 (2019). https://doi.org/10.5194/isprs-archives-XLII-2-W9-119-2019

  34. Maiwald, F., Vietze, T., Schneider, D., Henze, F., Münster, S., Niebling, F.: Photogrammetric analysis of historical image repositories for virtual reconstruction in the field of digital humanities. ISPRS Int. Arch. Photogr. Remote Sens. Spat. Inf. Sci. WG V/5 447–452 (2017). 3D-Arch 2017 – 3D Virtual Reconstruction and Visualization of Complex Architectures (XL-5/W5)

    Google Scholar 

  35. Bitelli, G., Dellapasqua, M., Girelli, V.A., Sbaraglia, S., Tinia, M.A.: Historical photogrammetry and terrestrial laser scanning for the 3D virtual reconstruction of destroyed structures: a case study in Italy. ISPRS – Inte. Arch. Photogr. Remote Sens. Spat. Inf. Sci. XLII-5/W1, 113–119 (2017). https://doi.org/10.5194/isprs-archives-XLII-5-W1-113-2017

  36. n.b.: Agisoft Metashape (2020). www.agisoft.com. Accessed 10 Oct 2020

  37. AliceVision: Meshroom: a 3D reconstruction software (2018)

    Google Scholar 

  38. Schönberger, J.L., Frahm, J.-M.: Structure-from-motion revisited. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  39. Condorelli, F., Rinaudo, F.: Cultural heritage reconstruction from historical photographs and videos. Int. Arch. Photogr. Remote Sens. Spat. Inf. Sci. XLII-2, 259–265 (2018)

    Google Scholar 

  40. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. ACM Trans. Graph. (TOG) 25, 835–846 (2006)

    Article  Google Scholar 

  41. Maiwald, F.: Generation of a benchmark dataset using historical photographs for an automated evaluation of different feature matching methods. Int. Arch. Photogr. Remote Sens. Spat. Inf. Sci. XLII-2/W13, 87–94 (2019). https://doi.org/10.5194/isprs-archives-XLII-2-W13-87-2019

  42. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  43. Alcantarilla, P.F., Solutions, T.: Fast explicit diffusion for accelerated features in nonlinear scale spaces. IEEE Trans. Pattern Anal. Mach. Intell. 34(7), 1281–1298 (2011)

    Google Scholar 

  44. Bay, H., Tuytelaars, T., Van Gool, L.: Surf: Speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_32

    Chapter  Google Scholar 

  45. Moulon, P., Monasse, P., Marlet, R.: Adaptive structure from motion with a contrario model estimation. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012. LNCS, vol. 7727, pp. 257–270. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37447-0_20

    Chapter  Google Scholar 

  46. Wu, C.: Towards linear-time incremental structure from motion. In: 2013 International conference on 3D Vision-3DV, pp 127–134. IEEE (2013)

    Google Scholar 

  47. Maiwald, F., Bruschke, J., Lehmann, C., Niebling, F.: A 4D information system for the exploration of multitemporal images and maps using photogrammetry, web technologies and VR/AR. Virtual Archaeol. Rev. 10(21), 1–13 (2019)

    Article  Google Scholar 

  48. Li, J., Hu, Q., Ai, M.: RIFT: multi-modal image matching based on radiation-invariant feature transform. arXiv preprint arXiv:180409493 (2018)

  49. Wu, Y., Ma, W., Gong, M., Su, L., Jiao, L.: A novel point-matching algorithm based on fast sample consensus for image registration. IEEE Geosci. Remote Sens. Lett. 12(1), 43–47 (2015). https://doi.org/10.1109/LGRS.2014.2325970

    Article  Google Scholar 

  50. Li, J., Hu, Q., Ai, M.: Robust feature matching for geospatial images via an affine-invariant coordinate system. Photogram. Rec. 32(159), 317–331 (2017)

    Article  Google Scholar 

  51. Mishkin, D., Matas, J., Perdoch, M.: MODS: fast and robust method for two-view matching. Comput. Vis. Image Underst. 141, 81–93 (2015). https://doi.org/10.1016/j.cviu.2015.08.005

    Article  Google Scholar 

  52. Mishkin, D., Matas, J., Perdoch, M., Lenc, K.: WxBS: wide baseline stereo generalizations. arXiv preprint arXiv:150406603 (2015)

  53. Chum, O., Matas, J.: Matching with PROSAC-progressive sample consensus. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, pp 220–226. IEEE (2005). https://doi.org/10.1109/CVPR.2005.221

  54. Dusmanu, M., et al.: D2-Net: a trainable CNN for joint description and detection of local features. In: 2019 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8092–8101 (2019)

    Google Scholar 

  55. Noh, H., Araujo, A., Sim, J., Weyand, T., Han, B.: Large-scale image retrieval with attentive deep local features. In: Proceedings of the IEEE International Conference on Computer Vision, pp 3456–3465 (2017)

    Google Scholar 

  56. X3DOM (2018). https://x3dom.org/

  57. Three.js (2019). https://threejs.org/

  58. Smithsonian 3D Labs (2019). https://3d.si.edu/labs

  59. 3D HOP (2019). http://vcg.isti.cnr.it/3dhop/

  60. Hexalab (2019). https://www.hexalab.net/

  61. INCEPTION (2018). https://www.inception-project.eu/en. https://www.inception-project.eu/en

  62. Champion, E., Rahaman, H.: Survey of 3D digital heritage repositories and platforms. Virtual Archaeol. Rev. 11(23) (2020). https://doi.org/10.4995/var.2020.13226

  63. Fernie, K., et al.: 3D content in Europeana task force. The Hague (2020)

    Google Scholar 

  64. Prechtel, N., Münster, S., Kröber, C., Schubert, C., Schietzold, S.: Presenting cultural heritage landscapes – from gis via 3D models to interactive presentation frameworks. ISPRS Ann. Photogr. Remote Sens. Spat. Inf. Sci. II-5/W1, 2013 (2013)

    Google Scholar 

  65. NIMA – National Imagery and Mapping Agency: Department of Defense World Geodetic System. Technical report, TR 8350.2 (1984)

    Google Scholar 

  66. Borda, A., Bowen, J.P.: Smart cities and digital culture: models of innovation. In: Giannini, T., Bowen, J.P. (eds.) Museums and Digital Culture. SSCC, pp. 523–549. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-97457-6_27

    Chapter  Google Scholar 

  67. Nielsen, J., Budiu, R.: Mobile Usability. Academic Press, Salt Lake City (2013)

    Google Scholar 

  68. Harrison, R., Flood, D., Duce, D.: Usability of mobile applications: literature review and rationale for a new usability model. J. Interact. Sci. 1(1), 1 (2013). https://doi.org/10.1186/2194-0827-1-1

    Article  Google Scholar 

  69. Hassenzahl, M., Burmester, M., Koller, F.: AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In: Ziegler, J., Szwillus (eds.) Mensch & Computer 2003. Interaktion in Bewegung, pp 187–196. B.G. Teubner., Leipzig (2003)

    Google Scholar 

  70. Schrepp, M., Hinderks, A., Thomaschewski, J.: Design and evaluation of a short version of the user experience questionnaire (UEQ-S). IJIMAI 4(6), 103–108 (2017)

    Article  Google Scholar 

  71. Laugwitz, B., Held, T., Schrepp, M.: Construction and evaluation of a user experience questionnaire. In: Holzinger, A. (ed.) USAB 2008. LNCS, vol. 5298, pp. 63–76. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-89350-9_6

    Chapter  Google Scholar 

  72. Klamert, K., Münster, S.: Child’s play - A literature based survey on gamified tools and methods for fostering public participation in urban planning. In: Parycek, P., et al. (eds.) Electronic Participation. LNCS, pp. 24-33. Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-319-64322-9_3

  73. Niebling, F., Maiwald, F., Barthel, K., Latoschik, M.E.: 4D augmented city models, photogrammetric creation and dissemination. In: Münster, S., Friedrichs, K., Niebling, F., Seidel-Grzesinska, A. (eds.) UHDL/DECH -2017. CCIS, vol. 817, pp. 196–212. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-76992-9_12

    Chapter  Google Scholar 

  74. Jylhäa, H., Hamari, J.: An icon that everyone wants to click: how perceived aesthetic qualities predict app icon successfulness. Int. J. Hum Comput Stud. 130, 73–85 (2019)

    Article  Google Scholar 

  75. Böhmer, M., Krüger, A.: A study on icon arrangement by smartphone users. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp 2137–2146. Association for Computing Machinery (2013). https://doi.org/10.1145/2470654.2481294

  76. Lavid Ben Lulu, D., Kuflik, T.: Wise mobile icons organization: apps taxonomy classification using functionality mining to ease apps finding. Mob. Inf. Syst. 2016, 3083450 (2016). https://doi.org/10.1155/2016/3083450

    Article  Google Scholar 

  77. Colley, A., Häkkilä, J.: Exploring finger specific touch screen interaction for mobile phone user interfaces. Paper Presented at the Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: the Future of Design, Sydney, New South Wales, Australia (2014)

    Google Scholar 

  78. Gao, Q., Sun, Q.: Examining the usability of touch screen gestures for older and younger adults. Hum. Factors 57(5), 835–863 (2015). https://doi.org/10.1177/0018720815581293

    Article  Google Scholar 

  79. Noh, H., et al.: Large-scale image retrieval with attentive deep local features. In: IEEE International Conference on Computer Vision, pp. 3456–3465 (2017)

    Google Scholar 

  80. Xie, L., et al.: Image classification and retrieval are one. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, pp. 3–10 (2015)

    Google Scholar 

Download references

Acknowledgments

The research for this paper was carried out in the projects TMPC (Sächsische Aufbaubank, 100377090), TMPCJ (Thüringische Aufbaubank, 220FGI0045), and Denkmalschutz4D (Deutsche Bundesstiftung Umwelt, 35654) as well as the junior research group UrbanHistory4D (German Federal Ministry of Education and Research, 01UG1630). Furthermore, this work was supported by the German Federal Ministry of Education and Research (BMBF, 01/S18026A-F) by funding the competence center for Big Data and AI “ScaDS.AI Dresden/Leipzig.” The authors gratefully acknowledge the Gemeinsame Wissenschaftkonferenz’s support for this project by providing computing time through the Center for Information Services and HPC (ZIH) at TU Dresden on HRSK-II.

Author information

Authors and Affiliations

  1. Digital Humanities, Friedrich-Schiller-Universität Jena, 07743, Jena, Germany

    Sander Münster, Ferdinand Maiwald & Susanne Karsten

  2. Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI), ZIH TU Dresden, 01062, Dresden, Germany

    Christoph Lehmann & Taras Lazariv

Authors
  1. Sander Münster
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christoph Lehmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Taras Lazariv
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ferdinand Maiwald
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Susanne Karsten
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sander Münster .

Editor information

Editors and Affiliations

  1. Universität Würzburg, Würzburg, Germany

    Florian Niebling

  2. Friedrich-Schiller-Universität Jena, Jena, Germany

    Sander Münster

  3. TU Dresden, Dresden, Germany

    Heike Messemer

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Münster, S., Lehmann, C., Lazariv, T., Maiwald, F., Karsten, S. (2021). Toward an Automated Pipeline for a Browser-Based, City-Scale Mobile 4D VR Application Based on Historical Images. In: Niebling, F., Münster, S., Messemer, H. (eds) Research and Education in Urban History in the Age of Digital Libraries. UHDL 2019. Communications in Computer and Information Science, vol 1501. Springer, Cham. https://doi.org/10.1007/978-3-030-93186-5_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-93186-5_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-93185-8

  • Online ISBN: 978-3-030-93186-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation