research-article

Mixed-Reality World Exploration Using Image-Based Rendering

Authors:

Masayuki Kanbara,

Naokazu YokoyaAuthors Info & Claims

Journal on Computing and Cultural Heritage (JOCCH), Volume 8, Issue 2

Article No.: 9, Pages 1 - 26

https://doi.org/10.1145/2700428

Published: 05 March 2015 Publication History

Abstract

This article describes a Mixed-Reality (MR) application that superimposes lost buildings of a historical site onto real scenes virtualized using spherical aerial images. The proposed application is set at a UNESCO World Heritage site in Japan, and is based on a novel framework that supports the photorealistic superimposition of virtual objects onto virtualized real scenes. The proposed framework utilizes Image-Based Rendering (IBR), which enables users to freely change their viewpoint in a real-world virtualization constructed using precaptured images. This framework combines the offline rendering of virtual objects and IBR to take advantage of the higher quality of offline rendering without the additional computational cost of online processing; that is, it incurs only the cost of online lightweight IBR, which is simplified through the pregeneration of structured viewpoints (e.g., at grid points).

References

[1]

Sameer Agarwal, Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2010. Bundle adjustment in the large. In Proceedings of the 11th European Conference on Computer Vision (ECCV’10). Springer-Verlag, 29--42.

Digital Library

[2]

Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz, and Richard Szeliski. 2009. Building Rome in a day. In Proceedings of the 12th IEEE International Conference on Computer Vision (ICCV’09). IEEE, 72--79.

[3]

Ronald Azuma. 1997. A survey of augmented reality. Presence: Teleoperators and Virtual Environments 6, 4 (Aug. 1997), 355--385.

Digital Library

[4]

Ronald Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner, Simon Julier, and Blair MacIntyre. 2001. Recent advances in augmented reality. IEEE Comput. Graph. Appl. 21, 6 (Aug. 2001), 34--47.

Digital Library

[5]

Atsuhiko Banno, Tomohito Masuda, Takeshi Oishi, and Katsushi Ikeuchi. 2008. Flying laser range sensor for large-scale site-modeling and its applications in Bayon digital archival project. Int. J. Comput. Vis. 78, 2 (July 2008), 207--222.

Digital Library

[6]

Andrea Bottino and Aldo Laurentini. 2006. What’s NEXT&quest; An interactive next best view approach. Pattern Recognit. 39, 1 (Jan. 2006), 126--132.

Digital Library

[7]

Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2001. Unstructured lumigraph rendering. In Proceedings of ACM SIGGRAPH’01. ACM, 425--432.

Digital Library

[8]

Declan Butler. 2006. Virtual globes: The web-wide world. Nature 439, 7078 (Feb. 2006), 776--778.

[9]

Paul Debevec. 1998. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of ACM SIGGRAPH’98. ACM, 189--198.

Digital Library

[10]

Paul Debevec, Yizhou Yu, and George Borshukov. 1998. Efficient view-dependent image-based rendering with projective texture-mapping. In Proceedings of the 9th Eurographics Workshop on Rendering (EGWR’98). Eurographics, 105--116.

[11]

Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik. 1996. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In Proceedings of ACM SIGGRAPH’96. ACM, 11--20.

Digital Library

[12]

Marco Di Benedetto, Fabio Ganovelli, M. Balsa Rodriguez, A. Jaspe Villanueva, Roberto Scopigno, and Enrico Gobbetti. 2014. Exploremaps: Efficient construction and ubiquitous exploration of panoramic view graphs of complex 3d environments. Comput. Graphics Forum 33, 2 (May 2014), 459--468.

Digital Library

[13]

Alessandro E. Foni, George Papagiannakis, and Nadia Magnenat-Thalmann. 2010. A taxonomy of visualization strategies for cultural heritage applications. ACM J. Comput. Cultural Heritage 3, 1 (June 2010), 1:1--1:21.

Digital Library

[14]

Michael Garland and Paul S. Heckbert. 1997. Surface simplification using quadric error metrics. In Proceedings of ACM SIGGRAPH’97. ACM, 209--216.

Digital Library

[15]

Payam Ghadirian and Ian D. Bishop. 2008. Integration of augmented reality and GIS: A new approach to realistic landscape visualisation. Landscape Urban Plan. 86 (June 2008), 226--232.

[16]

Simon Gibson, Jon Cook, Toby Howard, and Roger Hubbold. 2003. Rapid shadow generation in real-world lighting environments. In Proceedings of the 14th Eurographics Symposium on Rendering (EGSR’03). Eurographics, 219--229.

Digital Library

[17]

Simon Gibson and Alan Murta. 2000. Interactive rendering with real-world illumination. In Proceedings of the 11th Eurographics Workshop on Rendering (EGWR’00). Eurographics, 365--376.

Digital Library

[18]

Thomas Gierlinger, Daniel Danch, and André Stork. 2010. Rendering techniques for mixed reality. J. Real-Time Image Process. 5, 2 (June 2010), 109--120.

[19]

Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The Lumigraph. In Proceedings of ACM SIGGRAPH’96. ACM, 43--54.

Digital Library

[20]

Thorsten Grosch. 2005. PanoAR: Interactive augmentation of omnidirectional images with consistent lighting. In Proceedings of the Computer Vision/Computer Graphics Collaboration Techniques and Applications (Mirage’05). 25--34.

[21]

Michael Haller, Stephan Drab, and Werner Hartmann. 2003. A real-time shadow approach for an augmented reality application using shadow volumes. In Proceedings of the 10th ACM Symposium on Virtual Reality Software and Technology (VRST’03). ACM, 56--65.

Digital Library

[22]

Maiya Hori, Masayuki Kanbara, and Naokazu Yokoya. 2010. Arbitrary stereoscopic view generation using multiple omnidirectional image sequences. In Proceedings of the 20th IAPR International Conference on Pattern Recognition (ICPR’10). IEEE, 286--289.

Digital Library

[23]

Norio Igawa, Yasuko Koga, Tomoko Matsuzawa, and Hiroshi Nakamura. 2004. Models of sky radiance distribution and sky luminance distribution. Sol. Energy 77 (2004), 137--157.

[24]

Michal Jancosek and Tomás Pajdla. 2011. Multi-view reconstruction preserving weakly-supported surfaces. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). IEEE, 3121--3128.

Digital Library

[25]

Henrik Wann Jensen. 1996. Global illumination using photon maps. In Proceedings of the 7th Eurographics Workshop on Rendering (EGWR’96). Eurographics, 21--30.

Digital Library

[26]

Peter Kán and Hannes Kaufmann. 2012. High-quality reflections, refractions, and caustics in augmented reality and their contribution to visual coherence. In Proceedings of the 11th IEEE International Symposium on Mixed and Augmented Reality (ISMAR’12). IEEE, 99--108.

Digital Library

[27]

Masayuki Kanbara and Naokazu Yokoya. 2002. Geometric and photometric registration for real-time augmented reality. In Proceedings of the 1st International Symposium on Mixed and Augmented Reality (ISMAR’02). IEEE, 279--280.

Digital Library

[28]

Sing Bing Kang, Richard Szeliski, and P. Anandan. 2000. The geometry-image representation tradeoff for rendering. In Proceedings of the 2000 IEEE International Conference on Image Processing (ICIP’00), Vol. 2. IEEE, 13--16.

[29]

Martin Knecht, Christoph Traxler, Oliver Mattausch, Werner Purgathofer, and Michael Wimmer. 2010. Differential instant radiosity for mixed reality. In Proceedings of the 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR’10). IEEE, 99--107.

[30]

Hideyuki Kume, Takafumi Taketomi, Tomokazu Sato, and Naokazu Yokoya. 2010. Extrinsic camera parameter estimation using video images and GPS considering GPS positioning accuracy. In Proceedings of the 20th IAPR International Conference on Pattern Recognition (ICPR’10). IEEE, 3923--3926.

Digital Library

[31]

Pierre-Yves Laffont, Adrien Bousseau, and George Drettakis. 2013. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Trans. Vis. Comput. Graphics 19, 2 (April 2013), 210--224.

Digital Library

[32]

Ping-Man Lam, Chi-Sing Leung, and Tien-Tsin Wong. 2004. A compression method for a massive image data set in image-based rendering. Signal Process.: Image Commun. 19, 8 (Sept. 2004), 741--754.

[33]

Philipp Lensing and Wolfgang Broll. 2012. Instant indirect illumination for dynamic mixed reality scenes. In Proceedings of the 11th IEEE International Symposium on Mixed and Augmented Reality (ISMAR’12). IEEE, 109--118.

Digital Library

[34]

Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of ACM SIGGRAPH’96. ACM, 31--42.

Digital Library

[35]

Maxime Lhuillier. 2011. Fusion of GPS and structure-from-motion using constrained bundle adjustments. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). IEEE, 3025--3032.

Digital Library

[36]

Marcus Magnor, Prashant Ramanathan, and Bernd Girod. 2003. Multi-view coding for image-based rendering using 3-D scene geometry. IEEE Trans. Circuits Syst. Video Technol. 13, 11 (Oct. 2003), 1092--1106.

Digital Library

[37]

Takeshi Naemura, Takahide Takano, Masahide Kaneko, and Hiroshi Harashima. 1997. Ray-based creation of photo-realistic virtual world. In Proceedings of the 3rd International Conference on Virtual Systems and Multimedia (VSMM’97). IEEE, 59--68.

Digital Library

[38]

Fumio Okura, Masayuki Kanbara, and Naokazu Yokoya. 2010. Augmented telepresence using autopilot airship and omni-directional camera. In Proceedings of the 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR’10). IEEE, 259--260.

[39]

Fumio Okura, Masayuki Kanbara, and Naokazu Yokoya. 2011. Fly-through Heijo Palace Site: Augmented telepresence using aerial omnidirectional videos. In Proceedings of ACM SIGGRAPH’11 Posters. ACM, 78.

Digital Library

[40]

Fumio Okura, Masayuki Kanbara, and Naokazu Yokoya. 2014. Aerial full spherical imaging and display. Virtual Reality 18, 4 (Nov. 2014), 255--269.

Digital Library

[41]

Matt Pharr and Greg Humphreys. 2010. Physically Based Rendering: From Theory to Implementation. Morgan Kaufmann, Burlington, MA.

Digital Library

[42]

Richard Pito. 1999. A solution to the next best view problem for automated surface acquisition. IEEE Trans. Pattern Anal. Machine Intell. 21, 10 (Oct. 1999), 1016--1030.

Digital Library

[43]

Marc Pollefeys, David Nistér, Jan-Michael Frahm, Amir Akbarzadeh, Philippos Mordohai, Brian Clipp, Chris Engels, David Gallup, S.-J. Kim, and Paul Merrell. 2008. Detailed real-time urban 3D reconstruction from video. Int. J. Comput. Vis. 78, 2 (July 2008), 143--167.

Digital Library

[44]

Damien Porquet, Jean-Michel Dischler, and Djamchid Ghazanfarpour. 2005. Real-time high-quality view-dependent texture mapping using per-pixel visibility. In Proceedings of the 3rd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia (GRAPHITE’05). ACM, 213--220.

Digital Library

[45]

Steven M. Seitz and Charles R. Dyer. 1996. View morphing. In Proceedings of ACM SIGGRAPH’96. ACM, 21--30.

Digital Library

[46]

Heung-Yeung Shum, Sing Bing Kang, and Shing-Chow Chan. 2003. Survey of image-based representations and compression techniques. IEEE Trans. Circ. Syst. Video Technol. 13, 11 (Oct. 2003), 1020--1037.

Digital Library

[47]

Greg J. Ward. 1994. The RADIANCE lighting simulation and rendering system. In Proceedings of ACM SIGGRAPH’94. ACM, 459--472.

Digital Library

[48]

Robert M. Wolk. 2008. Utilizing Google Earth and Google Sketchup to visualize wind farms. In Proceedings of the 2008 IEEE International Symposium on Technology and Society (ISTAS’08). IEEE, 1--8.

[49]

Changchang Wu. 2013. Towards linear-time incremental structure from motion. In Proceedings of the 2013 International Conference on 3D Vision (3DV’13). IEEE, 127--134.

Digital Library

[50]

Changchang Wu, Sameer Agarwal, Brian Curless, and Steven M. Seitz. 2011. Multicore bundle adjustment. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). IEEE, 3057--3064.

Digital Library

[51]

Hanwei Xu, Rami Badawi, Xiaohu Fan, Jiayong Ren, and Zhiqiang Zhang. 2009. Research for 3D visualization of digital city based on SketchUp and ArcGIS. In Proceedings of SPIE International Symposium on Spatial Analysis, Spatial-Temporal Data Modeling, and Data Mining, Vol. 7492. SPIE, 74920Z:1--74920Z:12.

[52]

Michael Zoellner, Jens Keil, Timm Drevensek, and Harald Wuest. 2009. Cultural heritage layers: Integrating historic media in augmented reality. In Proceedings of the 15th International Conference on Virtual Systems and Multimedia (VSMM’09). IEEE, 193--196.

Digital Library

Cited By

Phoong SPhoong SKhek S(2024)Virtual Reality in the Tourism Sector: A Bibliometric Systematic Literature Review of the Development and Future TrendsSage Open10.1177/2158244024129093314:4Online publication date: 17-Oct-2024
https://doi.org/10.1177/21582440241290933
Yu TLai SZhang WCui JTao J(2024)RelicCARD: Enhancing cultural relics exploration through semantics-based augmented reality tangible interaction designVisual Informatics10.1016/j.visinf.2024.06.0038:3(32-41)Online publication date: Sep-2024
https://doi.org/10.1016/j.visinf.2024.06.003
Zhang JXiong KLiu ZHe L(2022)Research progress and knowledge system of world heritage tourism: a bibliometric analysisHeritage Science10.1186/s40494-022-00654-010:1Online publication date: 23-Mar-2022
https://doi.org/10.1186/s40494-022-00654-0
Show More Cited By

Index Terms

Mixed-Reality World Exploration Using Image-Based Rendering
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality

Recommendations

Image-based rendering of diffuse, specular and glossy surfaces from a single image
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques

In this paper, we present a new method to recover an approximation of the bidirectional reflectance distribution function (BRDF) of the surfaces present in a real scene. This is done from a single photograph and a 3D geometric model of the scene. The ...
Photorealistic rendering of knitwear using the lumislice
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques

We present a method for efficient synthesis of photorealistic free-form knitwear. Our approach is motivated by the observation that a single cross-section of yarn can serve as the basic primitive for modeling entire articles of knitwear. This primitive, ...
Realistic Rendering and Animation of Knitwear

We present a framework for knitwear modeling and rendering that accounts for characteristics that are particular to knitted fabrics. We first describe a model for animation that considers knitwear features and their effects on knitwear shape and ...

Comments

Information & Contributors

Information

Published In

cover image Journal on Computing and Cultural Heritage

Journal on Computing and Cultural Heritage Volume 8, Issue 2

March 2015

114 pages

ISSN:1556-4673

EISSN:1556-4711

DOI:10.1145/2743013

Editor:
Roberto Scopigno
CNR-ISTI, Italy

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 March 2015

Accepted: 01 September 2014

Revised: 01 July 2014

Received: 01 April 2014

Published in JOCCH Volume 8, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Grant-in-Aid for JSPS Fellows No. 25-7448
“Ambient Intelligence” project funded by Ministry of Education, Culture, Sports, Science and Technology (MEXT)
Japan Society for the Promotion of Science (JSPS) Grant-in-Aid for Scientific Research (A), No. 23240024

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
587
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)2

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Phoong SPhoong SKhek S(2024)Virtual Reality in the Tourism Sector: A Bibliometric Systematic Literature Review of the Development and Future TrendsSage Open10.1177/2158244024129093314:4Online publication date: 17-Oct-2024
https://doi.org/10.1177/21582440241290933
Yu TLai SZhang WCui JTao J(2024)RelicCARD: Enhancing cultural relics exploration through semantics-based augmented reality tangible interaction designVisual Informatics10.1016/j.visinf.2024.06.0038:3(32-41)Online publication date: Sep-2024
https://doi.org/10.1016/j.visinf.2024.06.003
Zhang JXiong KLiu ZHe L(2022)Research progress and knowledge system of world heritage tourism: a bibliometric analysisHeritage Science10.1186/s40494-022-00654-010:1Online publication date: 23-Mar-2022
https://doi.org/10.1186/s40494-022-00654-0
Trček D(2022)Cultural heritage preservation by using blockchain technologiesHeritage Science10.1186/s40494-021-00643-910:1Online publication date: 10-Jan-2022
https://doi.org/10.1186/s40494-021-00643-9
Lin JShe YLin LChen SChen JLiu HXu XShen YLi J(2022)Out of theater: Interactive Mixed-reality Performance for Intangible Culture Heritage Glove PuppetryProceedings of the Tenth International Symposium of Chinese CHI10.1145/3565698.3565782(181-189)Online publication date: 22-Oct-2022
https://dl.acm.org/doi/10.1145/3565698.3565782
Evangelidis KPapadopoulos TSylaiou S(2021)Mixed Reality: A Reconsideration Based on Mixed Objects and Geospatial ModalitiesApplied Sciences10.3390/app1105241711:5(2417)Online publication date: 9-Mar-2021
https://doi.org/10.3390/app11052417
Kiyokawa TKatayama HTatsuta YTakamatsu JOgasawara T(2021)Robotic Waste Sorter With Agile Manipulation and Quickly Trainable DetectorIEEE Access10.1109/ACCESS.2021.31107959(124616-124631)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3110795
Caggianese GGallo L(2021)Field of View Limitation-Driven Design of a Mixed Reality Game for Heritage SitesHuman-Computer Interaction – INTERACT 202110.1007/978-3-030-85607-6_51(426-430)Online publication date: 30-Aug-2021
https://dl.acm.org/doi/10.1007/978-3-030-85607-6_51
Maasthi MGururaj HJanhavi VHarshitha KSwathi B(2020)An Interactive Approach Deployed for Rhinoplasty using Mixed Reality2020 International Conference on COMmunication Systems & NETworkS (COMSNETS)10.1109/COMSNETS48256.2020.9027491(680-682)Online publication date: Jan-2020
https://doi.org/10.1109/COMSNETS48256.2020.9027491
Zhang JLiu SFan JHuang JPei M(2020)NK-CDS: A Creative Design System for Museum Art DerivativesIEEE Access10.1109/ACCESS.2020.2972561(1-1)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2972561
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents