research-article

An Automated Pipeline for a Browser-based, City-scale Mobile 4D VR Application based on Historical Images

Authors:

Sander Münster,

Ferdinand Maiwald,

Christoph Lehmann,

Mathias Hofmann,

Florian NieblingAuthors Info & Claims

SUMAC'20: Proceedings of the 2nd Workshop on Structuring and Understanding of Multimedia heritAge Contents

Pages 33 - 40

https://doi.org/10.1145/3423323.3425748

Published: 12 October 2020 Publication History

Abstract

The process for automatically creating 3D city models from contemporary photographs and visualizing them on mobile devices is now well established, but historical 4D city models are more challenging. The fourth dimension here is time. This article describes an automated VR pipeline based on historical photographs and resulting in an interactive browser-based device-rendered 4D visualization and information system for mobile devices. Since the pipeline shown is currently still under development, initial results for stages of the process will be shown and assessed for accuracy and usability.

References

[1]

Schindler, G. and Dellaert, F. 4D cities: analyzing, visualizing, and interacting with historical urban photo collections. Journal of Multimedia, 7, 2 (2012), 124--131.

[2]

Bitelli, G., Dellapasqua, M., Girelli, V. A., Sbaraglia, S. and Tinia, M. A. Historical Photogrammetry and Terrestrial Laser Scanning for the 3d Virtual Reconstruction of Destroyed Structures: a Case Study in Italy. The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 42 (2017), 113.

[3]

n.b. ArchiMediaL. Enriching and linking historical architectural and urban image collections. 2019.

[4]

Radovic, M., Adarkwa, O. and Wang, Q. Object Recognition in Aerial Images Using Convolutional Neural Networks. Journal of Imaging, 3 (2017).

[5]

Aiger, D., Allen, B. and Golovinskiy, A. Large-Scale 3D Scene Classification With Multi-View Volumetric CNN (2017).

[6]

Snavely, N., Seitz, S. M. and Szeliski, R. Photo tourism: exploring photo collections in 3d. ACM transactions on graphics (TOG), 25 (2006), 835--846.

[7]

Snavely, N., Seitz, S. M. and Szeliski, R. Modeling the World from Internet Photo Collection. International Journal of Computer Vision, 80, 2 (2007), 189--210.

[8]

Maiwald, F., Schneider, D., Henze, F., Niebling, F. and Münster, S. Feature matching of historical images based on geometry of quadrilaterals. ISPRS - International Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (2018).

[9]

Pomaska, G. Zur Dokumentation und 3D?Modellierung von Denkmalen mit digitalen fotografischen Verfahren. Verlag Philipp von Zabern, 2011.

[10]

Smithsonian 3D Labs https://3d.si.edu/labs. 2019.

[11]

X3DOM https://x3dom.org/. 2018.

[12]

Three.js https://threejs.org/. 2019.

[13]

3D HOP http://vcg.isti.cnr.it/3dhop/, 2019.

[14]

Hexalab https://www.hexalab.net/. 2019.

[15]

INCEPTION https://www.inception-project.eu/en. 2018.

[16]

N. Prechtel, S. M., C. Kröber, C. Schubert, S. Schietzold Presenting cultural heritage landscapes -- from gis via 3d models to interactive presentation frameworks. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-5/W1, 2013 (2013).

[17]

NIMA -- National Imagery and Mapping Agency: Department of Defense World Geodetic System Technical Report, TR 8350.2, 1984.

[18]

Borda, A. and Bowen, J. P. Smart Cities and Digital Culture: Models of Innovation. Springer International Publishing, 2019.

[19]

Friedrichs, K., Münster, S., Kröber, C. and Bruschke, J. Creating Suitable Tools for Art and Architectural Research with Historic Media Repositories. Springer International Publishing, 2017.

[20]

Sivic, J., Russell, B. C., Efros, A. A., Zisserman, A. and Freeman, W. T. Discovering objects and their location in images. IEEE, 2005.

Digital Library

[21]

Wan, J., Wang, D., Hoi, S. C. H., Wu, P., Zhu, J., Zhang, Y. and Li, J. Deep Learning for Content-Based Image Retrieval (2014), 157--166.

[22]

Goodfellow, I., Bengio, Y. and Courville, A. Deep learning. MIT Press, Cambridge, 2016.

Digital Library

[23]

Pan, S. J. and Yang, Q. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22, 10 (2009), 1345--1359.

[24]

Simonyan, K. and Zisserman, A. Very deep convolutional networks for largescale image recognition. arXiv preprint, 1409.1556 (2014).

[25]

Selvaraju et al., R. R. Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE international conference on computer vision (2017), 618--626.

[26]

Azizpour, H., Sullivan, J. and Carlsson, S. Cnn features off-the-shelf: An astounding baseline for recognition. CVPRW (2014).

[27]

Wan et al., J. Deep learning for content-based image retrieval: A comprehensive study. Proceedings of the 22nd ACM international conference on Multimedia (2014), 157--166.

Digital Library

[28]

Razavian et al., A. S. Visual instance retrieval with deep convolutional networks. ITE Transactions on Media Technology and Applications, 4, 3 (2016), 251--258.

[29]

Zhou, W., Li, H. and Tian, Q. Recent advance in content-based image retrieval: A literature survey. arXiv preprint, 1706.06064 (2017).

[30]

Noh et al., H. Large-scale image retrieval with attentive deep local features. IEEE international conference on computer vision (2017), 3456--3465.

[31]

Xie et al., L. Image classification and retrieval are one. Proceedings of the 5th ACM on International Conference on Multimedia Retrieval (2015), 3--10.

Digital Library

[32]

Callieri, M., Cignoni, P., Corsini, M. and Scopigno, R. Masked photo blending: Mapping dense photographic data set on high-resolution sampled 3D models. Computers & Graphics, 32, 4 (2008), 464--473.

Digital Library

[33]

Pani Paudel, D., Habed, A., Demonceaux, C. and Vasseur, P. Robust and optimal sum-of-squares-based point-to-plane registration of image sets and structured scenes. 2015.

[34]

Lowe, D. G. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60, 2 (2004), 91--110.

[35]

Bay, H., Tuytelaars, T. and Van Gool, L. Surf: Speeded up robust features. Springer, 2006.

[36]

Ali, H. K. and Whitehead, A. Feature Matching for Aligning Historical and Modern Images. International Journal of Computer Applications, 21, 3 (2014), 188--201.

[37]

Wolfe, R. Modern to historical image feature matching. Published online: http://robbiewolfe. ca/programming/honoursproject/report. pdf (2013).

[38]

Arandjelovic, R. and Zisserman, A. Three things everyone should know to improve object retrieval. IEEE, 2012.

[39]

Morel, J.-M. and Yu, G. ASIFT: A new framework for fully affine invariant image comparison. SIAM journal on imaging sciences, 2, 2 (2009), 438--469.

[40]

Moulon, P., Monasse, P. and Marlet, R. Adaptive structure from motion with a contrario model estimation. Springer, 2012.

[41]

Schönberger, J. L. and Frahm, J.-M. Structure-from-Motion Revisited. 2016.

[42]

Wu, C. Towards linear-time incremental structure from motion. IEEE, 2013.

Digital Library

[43]

Wang, L., Neumann, U. and You, S. Wide-baseline image matching using line signatures. 2009.

[44]

Micusik, B., Wildenauer, H. and Kosecka, J. Detection and matching of rectilinear structures. 2008.

[45]

Maiwald, F., Schneider, D. and Henze, F. Bestimmung der Korrespondenz zwischen historischen Gebäudeaufnahmen basierend auf der Zuordnung geometrischer Merkmale. Publikationen der Deutschen Gesellschaft für Photogrammetrie, Fernerkundung und Geoinformation eV, 27, 38 (2018), 103--116.

[46]

Li, J., Hu, Q. and Ai, M. RIFT: Multi-modal Image Matching Based on Radiation-invariant Feature Transform. arXiv preprint arXiv:1804.09493 (2018).

[47]

Mishkin, D., Matas, J. and Perdoch, M. MODS: Fast and robust method for two-view matching. Computer Vision and Image Understanding, 141 (2015), 81--93.

Digital Library

[48]

Mishkin, D., Matas, J., Perdoch, M. and Lenc, K. Wxbs: Wide baseline stereo generalizations. arXiv preprint arXiv:1504.06603 (2015).

[49]

Maiwald, F., Bruschke, J., Lehmann, C. and Niebling, F. A 4D information system for the exploration of multitemporal images and maps using photogrammetry, web technologies and VR/AR. Virtual Archaeology Review, 10, 21 (2019), 1--13.

[50]

Maiwald, F., Vietze, T., Schneider, D., Henze, F., Münster, S. and Niebling, F. Photogrammetric analysis of historical image repositories for virtual reconstruction in the field of digital humanities. The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 42 (2017), 447.

[51]

Dusmanu, M., Rocco, I., Pajdla, T., Pollefeys, M., Sivic, J., Torii, A. and Sattler, T. D2-Net: A Trainable CNN for Joint Description and Detection of Local Features. 2019.

[52]

Noh, H., Araujo, A., Sim, J., Weyand, T. and Han, B. Large-scale image retrieval with attentive deep local features. 2017.

[53]

Herold, H. and Hecht, R. 3D Reconstruction of Urban History Based on Old Maps. Springer International Publishing, 2018.

[54]

Oliveira, S. A., Lenardo, I. d. and Kaplan, F. Machine Vision Algorithms on Cadaster Plans. 2017.

[55]

Münster, S. Cultural Heritage at a Glance. Four case studies about the perception of digital architectural 3D models. IEEE, 2018.

[56]

Special Interest Group 3D CityGML Specification. 2007.

[57]

Stadt Dresden Virtuelles 3D-Stadtmodell (2018).

[58]

Harrison, R., Flood, D. and Duce, D. Usability of mobile applications: literature review and rationale for a new usability model. Journal of Interaction Science, 1, 1 (May 07 2013), 1.

[59]

Zhang, D. and Adipat, B. Challenges, methodologies, and issues in the usability testing of mobile applications. International Journal of Human-Computer Interaction, 18, 3 (2005), 293--308.

[60]

Nielsen, J. and Budiu, R. Mobile Usability. Academic Press, Salt Lake City, 2013.

Cited By

Münster SMaiwald FBruschke JKröber CSun YDworak DKomorowicz DMunir IBeck CMünster D(2024)A Digital 4D Information System on the World Scale: Research Challenges, Approaches, and Preliminary ResultsApplied Sciences10.3390/app1405199214:5(1992)Online publication date: 28-Feb-2024
https://doi.org/10.3390/app14051992
Colin CGautier CVinasco-Alvarez DDelanoy JGesquière GSamuel JServigne SBoix ÉDupont TLivebardon MMachado VMarnat L(2023)UD-SV : Plateforme d’exploration de données urbaines à n-dimensions — Espace, Temps, ThématiquesMappemonde10.4000/mappemonde.8265Online publication date: 7-Mar-2023
https://doi.org/10.4000/mappemonde.8265
Münster S(2023)Advancements in 3D Heritage Data Aggregation and Enrichment in Europe: Implications for Designing the Jena Experimental Repository for the DFG 3D ViewerApplied Sciences10.3390/app1317978113:17(9781)Online publication date: 29-Aug-2023
https://doi.org/10.3390/app13179781
Show More Cited By

Index Terms

An Automated Pipeline for a Browser-based, City-scale Mobile 4D VR Application based on Historical Images

Recommendations

Software and Content Design of a Browser-based Mobile 4D VR Application to Explore Historical City Architecture
SUMAC'21: Proceedings of the 3rd Workshop on Structuring and Understanding of Multimedia heritAge Contents

The Kulturerbe4D project aims at making the diversity and change processes of architectural monuments in the urban context virtually visible and experienceable, especially for children and young people, but also for residents and tourists. A virtual ...
4D Geo Modelling from Different Sources at Large Scale
SUMAC '24: Proceedings of the 6th workshop on the analySis, Understanding and proMotion of heritAge Contents

Imagine you're exploring the historic center of a city with its impressive town houses, churches and monuments. What if you could just use your mobile device to find out about the historic buildings around you, making history come alive before your eyes? ...
Data Visualization on Mobile Devices
CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems

As mobile visualization is increasingly used and new mobile device form factors and hardware capabilities continuously emerge, it is timely to reflect on what has been discovered to date and to look into the future. This workshop will bring together ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SUMAC'20: Proceedings of the 2nd Workshop on Structuring and Understanding of Multimedia heritAge Contents

October 2020

70 pages

ISBN:9781450381550

DOI:10.1145/3423323

General Chairs:
Valerie Gouet-Brunet
Univ. Gustave Eiel, IGN-ENSG/LASTIG, France
,
Margarita Khokhlova
Univ. Gustave Eiel, IGN-ENSG/LASTIG, France, Centrale Lyon/LIRIS, France
,
Ronak Kosti
Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany
,
Liming Chen
Centrale Lyon/LIRIS, France
,
Xu-Cheng Yin
University of Science and Technology Beijing, China

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Thringische Aufbaubank
German Federal Ministry of Education and Research

Conference

MM '20

Sponsor:

SIGMM

MM '20: The 28th ACM International Conference on Multimedia

October 12, 2020

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 5 of 6 submissions, 83%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
157
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Münster SMaiwald FBruschke JKröber CSun YDworak DKomorowicz DMunir IBeck CMünster D(2024)A Digital 4D Information System on the World Scale: Research Challenges, Approaches, and Preliminary ResultsApplied Sciences10.3390/app1405199214:5(1992)Online publication date: 28-Feb-2024
https://doi.org/10.3390/app14051992
Colin CGautier CVinasco-Alvarez DDelanoy JGesquière GSamuel JServigne SBoix ÉDupont TLivebardon MMachado VMarnat L(2023)UD-SV : Plateforme d’exploration de données urbaines à n-dimensions — Espace, Temps, ThématiquesMappemonde10.4000/mappemonde.8265Online publication date: 7-Mar-2023
https://doi.org/10.4000/mappemonde.8265
Münster S(2023)Advancements in 3D Heritage Data Aggregation and Enrichment in Europe: Implications for Designing the Jena Experimental Repository for the DFG 3D ViewerApplied Sciences10.3390/app1317978113:17(9781)Online publication date: 29-Aug-2023
https://doi.org/10.3390/app13179781
Korkut ESurer E(2023)Visualization in virtual reality: a systematic reviewVirtual Reality10.1007/s10055-023-00753-827:2(1447-1480)Online publication date: 17-Jan-2023
https://doi.org/10.1007/s10055-023-00753-8
Bruschke JKröber CUtescher RNiebling F(2023)Towards Querying Multimodal Annotations Using GraphsResearch and Education in Urban History in the Age of Digital Libraries10.1007/978-3-031-38871-2_5(65-87)Online publication date: 29-Jul-2023
https://doi.org/10.1007/978-3-031-38871-2_5
Schaffland AHeidemann G(2022)Heritage and Repeat Photography: Techniques, Management, Applications, and PublicationsHeritage10.3390/heritage50402205:4(4267-4305)Online publication date: 18-Dec-2022
https://doi.org/10.3390/heritage5040220
Muenster S(2022)Digital 3D Technologies for Humanities Research and Education: An OverviewApplied Sciences10.3390/app1205242612:5(2426)Online publication date: 25-Feb-2022
https://doi.org/10.3390/app12052426
Münster SLehmann CLazariv TMaiwald FKarsten S(2022)Toward an Automated Pipeline for a Browser-Based, City-Scale Mobile 4D VR Application Based on Historical ImagesResearch and Education in Urban History in the Age of Digital Libraries10.1007/978-3-030-93186-5_5(106-128)Online publication date: 11-Jan-2022
https://doi.org/10.1007/978-3-030-93186-5_5
Maiwald FLehmann CLazariv T(2021)Fully Automated Pose Estimation of Historical Images in the Context of 4D Geographic Information Systems Utilizing Machine Learning MethodsISPRS International Journal of Geo-Information10.3390/ijgi1011074810:11(748)Online publication date: 4-Nov-2021
https://doi.org/10.3390/ijgi10110748
Fanini BFerdani DDemetrescu E(2021)Temporal Lensing: An Interactive and Scalable Technique for Web3D/WebXR Applications in Cultural HeritageHeritage10.3390/heritage40200404:2(710-724)Online publication date: 30-Apr-2021
https://doi.org/10.3390/heritage4020040
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents