survey

Real-time Illumination and Visual Coherence for Photorealistic Augmented/Mixed Reality

Authors:

A’Aeshah Alhakamy,

Mihran TuceryanAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 53, Issue 3

Article No.: 49, Pages 1 - 34

https://doi.org/10.1145/3386496

Published: 28 May 2020 Publication History

Abstract

A realistically inserted virtual object in the real-time physical environment is a desirable feature in augmented reality (AR) applications and mixed reality (MR) in general. This problem is considered a vital research area in computer graphics, a field that is experiencing ongoing discovery. The algorithms and methods used to obtain dynamic and real-time illumination measurement, estimating, and rendering of augmented reality scenes are utilized in many applications to achieve a realistic perception by humans. We cannot deny the powerful impact of the continuous development of computer vision and machine learning techniques accompanied by the original computer graphics and image processing methods to provide a significant range of novel AR/MR techniques. These techniques include methods for light source acquisition through image-based lighting or sampling, registering and estimating the lighting conditions, and composition of global illumination. In this review, we discussed the pipeline stages with the details elaborated about the methods and techniques that contributed to the development of providing a photo-realistic rendering, visual coherence, and interactive real-time illumination results in AR/MR.

References

[1]

Nor Azlina Ab Aziz, Kamarulzaman Ab Aziz, Avijit Paul, Anuar Mohd Yusof, and Noor Shuhailie Mohamed Noor. 2012. Providing augmented reality based education for students with attention deficit hyperactive disorder via cloud computing: Its advantages. In Proceedings of the 2012 14th International Conference on Advanced Communication Technology (ICACT’12). 577--581.

[2]

Kusuma Agusanto, Li Li, Zhu Chuangui, and Ng Wan Sing. 2003. Photorealistic rendering for augmented reality using environment illumination. In Proceedings of the 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. 208--216.

[3]

A’aeshah Alhakamy and Mihran Tuceryan. 2019. AR360: Dynamic illumination for augmented reality with real-time interaction. In Proceedings of the 2019 IEEE 2nd International Conference on Information and Computer Technologies (ICICT’19). 170--174.

[4]

A’aeshah Alhakamy and Mihran Tuceryan. 2019. CubeMap360: Interactive global illumination for augmented reality in dynamic environment. In Proceedings of the 2019 IEEE Region 3 Technical, Professional, and Student Conference (SoutheastCon’19). IEEE.

[5]

A’aeshah Alhakamy and Mihran Tuceryan. 2019. An empirical evaluation of the performance of real-time illumination approaches: Realistic scenes in augmented reality. In Proceedings of the 6th Augmented Reality, Virtual Reality, and Computer Graphics (SALENTO AVR’19), Lucio Tommaso De Paolis and Patrick Bourdot (Eds.). Springer International, Cham, 179--195.

[6]

A’aeshah Alhakamy and Mihran Tuceryan. 2019. Polarization-based illumination detection for coherent augmented reality scene rendering in dynamic environments. In Advances in Computer Graphics through the 36th Computer Graphics International (CGI’19) in Cooperation with ACM SIGGRAPH and Eurographics, Marina Gavrilova, Jian Chang, Nadia Magnenat Thalmann, Eckhard Hitzer, and Hiroshi Ishikawa (Eds.). Springer International, Cham, 3--14.

[7]

Francesco Banterle, Marco Callieri, Matteo Dellepiane, Massimiliano Corsini, Fabio Pellacini, and Roberto Scopigno. 2013. EnvyDepth: An interface for recovering local natural illumination from environment maps. Comput. Graph. Forum 32, 7, 411--420.

[8]

Yoram Baram. 2004. Closed-loop augmented reality apparatus. US Patent 6,734,834.

[9]

João Barreira, Maximino Bessa, Luís Barbosa, and Luís Magalhães. 2018. A context-aware method for authentically simulating outdoors shadows for mobile augmented reality. IEEE Trans. Vis. Comput. Graph. 24, 3 (Mar. 2018), 1223--1231.

[10]

Jonathan T. Barron and Jitendra Malik. 2015. Shape, illumination, and reflectance from shading. IEEE Trans. Pattern Anal. Mach. Intell. 37, 8 (Aug. 2015), 1670--1687.

Digital Library

[11]

Bastiaan J. Boom, Sergio Orts-Escolano, Xin X. Ning, Steven McDonagh, Peter Sandilands, and Robert B. Fisher. 2017. Interactive light source position estimation for augmented reality with an RGB-D camera. Comput. Anim. Virt. Worlds 28, 1 (2017), e1686. e1686 cav.1686.

[12]

Derek Bradley, Gerhard Roth, and Prosenjit Bose. 2009. Augmented reality on cloth with realistic illumination. Mach. Vis. Appl. 20, 2 (2009), 85--92.

Digital Library

[13]

David E. Breen, Ross T. Whitaker, Eric Rose, and Mihran Tuceryan. 1996. Interactive occlusion and automatic object placement for augmented reality. Comput. Graph. Forum 15, 3, 11--22.

[14]

Xavier Casas, Gerardo Herrera, Inmaculada Coma, and Marcos Fernández. 2012. A kinect-based augmented reality system for individuals with autism spectrum disorders. In Proceedings of the International Conference on Computer Graphics Theory and Applications and International Conference on Information Visualization Theory and Applications - Volume 1: GRAPP, (VISIGRAPP’12). INSTICC, SciTePress, 440--446.

[15]

Andrew I. Comport, Éric Marchand, and François Chaumette. 2003. A real-time tracker for markerless augmented reality. In Proceedings of the 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. 36--45.

[16]

Jon Cook, Simon Gibson, Toby Howard, and Roger Hubbold. 2003. Real-time photo-realistic augmented reality for interior design. In Proceedings of the ACM SIGGRAPH 2003 Sketches 8 Applications (SIGGRAPH’03). Association for Computing Machinery, New York, NY, 1.

Digital Library

[17]

Massimiliano Corsini, Marco Callieri, and Paolo Cignoni. 2008. Stereo light probe. Comput. Graph. Forum 27, 2, 291--300.

[18]

Paul Debevec. 2008. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of the ACM SIGGRAPH 2008 Classes (SIGGRAPH’08). Association for Computing Machinery, New York, NY.

Digital Library

[19]

Paul Debevec, Paul Graham, Jay Busch, and Mark Bolas. 2012. A single-shot light probe. In Proceedings of the ACM SIGGRAPH 2012 Talks (SIGGRAPH’12). Association for Computing Machinery, New York, NY.

Digital Library

[20]

Paul Debevec, Chris Tchou, Andrew Gardner, Tim Hawkins, Charis Poullis, Jessi Stumpfel, Andrew Jones, Nathaniel Yun, Per Einarsson, Therese Lundgren, and Marcos Fajardo. 2004. Estimating Surface Reflectance Properties of a Complex Scene Under Captured Natural Illumination. USC ICT Technical Report ICT-TR-06.2004.

[21]

Cass Everitt and Mark J. Kilgard. 2003. Practical and Robust Stenciled Shadow Volumes for Hardware-Accelerated Rendering. CoRRcs.GR/0301002(2003). http://arxiv.org/abs/cs/0301002.

[22]

Alain Fournier, Atjeng S. Gunawan, and Chris Romanzin. 1993. Common illumination between real and computer generated scenes. In Graphics Interface. Canadian Information Processing Society, 254--254.

[23]

Jan-Michael Frahm, Kevin Koeser, Daniel Grest, and Reinhard Koch. 2005. Markerless augmented reality with light source estimation for direct illumination. In Proceedings of the Conference on Visual Media Production (CVMP’05). IET, 211--220.

[24]

Banterle Francesco, Artusi Alessandro, Debattista Kurt, and Chalmers Alan. 2017. Advanced High Dynamic Range Imaging (2nd ed.). A K Peters/CRC Press. 328 pages.

[25]

Tobias Alexander Franke. 2013. Delta light propagation volumes for mixed reality. In Proceedings of the 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR’13). 125--132.

[26]

Tobias Alexander Franke. 2014. Delta voxel cone tracing. In Proceedings of the 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR’14). 39--44.

[27]

Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde. 2017. Learning to predict indoor illumination from a single image. ACM Trans. Graph. 36, 6, Article 176 (Nov. 2017), 14 pages.

Digital Library

[28]

Simon Gibson, Jon Cook, Toby Howard, and Roger Hubbold. 2003. Rapid shadow generation in real-world lighting environments. In Proceedings of the 14th Eurographics Workshop on Rendering (EGRW’03). Eurographics Association, Goslar, DEU, 219--229.

Digital Library

[29]

Simon Gibson and Alan Murta. 2000. Interactive rendering with real-world illumination. In Rendering Techniques 2000, Bernard Péroche and Holly Rushmeier (Eds.). Springer, Vienna, 365--376.

[30]

Robin Green. 2003. Spherical harmonic lighting: The gritty details. In Proceedings of the Archives of the Game Developers Conference, Vol. 56. 4.

[31]

Thorsten Grosch. 2005. Differential photon mapping-consistent augmentation of photographs with correction of all light paths. In Proceedings of the Annual Conference of the European Association for Computer Graphics (Eurographics’05). 53--56.

[32]

Thorsten Grosch. 2005. PanoAR: Interactive augmentation of omnidirectional images with consistent lighting. Proceedings of the Annual Conference on Computer Vision/Computer Graphics Collaboration Techniques and Applications (Mirage’05). 25--34.

[33]

Thorsten Grosch, Tobias Eble, and Stefan Mueller. 2007. Consistent interactive augmentation of live camera images with correct near-field illumination. In Proceedings of the 2007 ACM Symposium on Virtual Reality Software and Technology (VRST’07). Association for Computing Machinery, New York, NY, 125--132.

Digital Library

[34]

Lukas Gruber, Thomas Richter-Trummer, and Dieter Schmalstieg. 2012. Real-time photometric registration from arbitrary geometry. In Proceedings of the 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR’12). 119--128.

Digital Library

[35]

Lukas Gruber, Jonathan Ventura, and Dieter Schmalstieg. 2015. Image-space illumination for augmented reality in dynamic environments. In Proceedings of the 2015 Conference on IEEE Virtual Reality (VR’15). 127--134.

[36]

Michael Haller, Stephan Drab, and Werner Hartmann. 2003. A real-time shadow approach for an augmented reality application using shadow volumes. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST’03). Association for Computing Machinery, New York, NY, 56--65.

Digital Library

[37]

Jassim Happa, Thomas Bashford-Rogers, Kurt Debattista, and Alan Chalmers. 2011. Rendering interior cultural heritage scenes using image-based shooting. In Proceedings of the Annual Conference of the European Association for Computer Graphics (Eurographics’11). 49--56.

[38]

Vlastimil Havran, Miloslaw Smyk, Grzegorz Krawczyk, Karol Myszkowski, and Hans-Peter Seidel. 2005. Importance sampling for video environment maps. In Proceedings of the ACM SIGGRAPH 2005 Sketches (SIGGRAPH’05). Association for Computing Machinery, New York, NY, 109--es.

Digital Library

[39]

Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. 2011. KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST’11). Association for Computing Machinery, New York, NY, 559--568.

Digital Library

[40]

Jan Jachnik, Richard A. Newcombe, and Andrew J. Davison. 2012. Real-time surface light-field capture for augmentation of planar specular surfaces. In Proceedings of the 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR’12). 91--97.

Digital Library

[41]

Katrien Jacobs and Céline Loscos. 2006. Classification of illumination methods for mixed reality. Comput. Graph. Forum 25, 1, 29--51.

[42]

Henrik Wann Jensen and Niels Jørgen Christensen. 1995. Photon maps in bidirectional Monte Carlo ray tracing of complex objects. Comput. Graph. 19, 2 (1995), 215--224.

[43]

Salma Jiddi, Philippe Robert, and Eric Marchand. 2016. Reflectance and illumination estimation for realistic augmentations of real scenes. In Proceedings of the 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct’16). IEEE, 244--249.

[44]

M. Carmen Juan, Mariano Alcaniz, Carlos Monserrat, Cristina Botella, Rosa María Baños, and Belen Guerrero. 2005. Using augmented reality to treat phobias. IEEE Comput. Graph. Appl. 25, 6 (Nov. 2005), 31--37.

Digital Library

[45]

James T. Kajiya. 1986. The rendering equation. In Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’86). Association for Computing Machinery, New York, NY, 143--150.

Digital Library

[46]

Divya Kamboj and Wankui Liu. 2014. Improved variance cut algorithm for light source estimation in augmented reality. Int. J. Comput. Appl. 85, 19 (2014).

[47]

Peter Kán and Hannes Kaufmann. 2012. High-quality reflections, refractions, and caustics in augmented reality and their contribution to visual coherence. In Proceedings of the 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR’12). 99--108.

Digital Library

[48]

Peter Kán and Hannes Kaufmann. 2013. Differential irradiance caching for fast high-quality light transport between virtual and real worlds. In Proceedings of the 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR’13). 133--141.

[49]

Masayuki Kanbara and Naokazu Yokoya. 2004. Real-time estimation of light source environment for photorealistic augmented reality. In Proceedings of the 17th International Conference on Pattern Recognition (ICPR’04), Vol. 2. IEEE Cambridge, UK, 911--914.

[50]

Kevin Karsch, Varsha Hedau, David Forsyth, and Derek Hoiem. 2011. Rendering synthetic objects into legacy photographs. In Proceedings of the 2011 SIGGRAPH Asia Conference (SA’11). Association for Computing Machinery, New York, NY.

Digital Library

[51]

Kevin Karsch, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, Hailin Jin, Rafael Fonte, Michael Sittig, and David Forsyth. 2014. Automatic scene inference for 3D object compositing. ACM Trans. Graph. 33, 3 (June 2014).

Digital Library

[52]

Jan Kautz, John Snyder, and Peter-Pike J. Sloan. 2002. Fast arbitrary BRDF shading for low-frequency lighting using spherical harmonics. Render. Techn. 2, 291--296 (2002), 1.

[53]

Martin Knecht, Christoph Traxler, Oliver Mattausch, Oliver Mattausch, Purgathofer Werner, and Wimmer Michael. 2010. Differential instant radiosity for mixed reality. In Proceedings of the 2010 IEEE International Symposium on Mixed and Augmented Reality. 99--107.

[54]

Sebastian B. Knorr and Daniel Kurz. 2014. Real-time illumination estimation from faces for coherent rendering. In Proceedings of the 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR’14). 113--122.

[55]

Hoshang Kolivand and Mohd Shahrizal Sunar. 2013. Survey of shadow volume algorithms in computer graphics. IETE Techn. Rev. 30, 1 (2013), 38--46.

[56]

Hoshang Kolivand and Mohd Shahrizal Sunar. 2014. Realistic real-time outdoor rendering in augmented reality. PLoS ONE 9, 9 (2014), e108334.

[57]

Hoshang Kolivand, Mohd Shahrizal Sunar, Samira Y. Kakh, Riyadh Al-Rousan, and Ismahafezi Ismail. 2018. Photorealistic rendering: A survey on evaluation. Multimedia Tools Appl. 77, 19 (2018), 25983--26008.

Digital Library

[58]

Hoshang Kolivand, Mohd Shahrizal Sunar, Tanzila Saba, and Hatem Ali. 2019. ReLiShaft: Realistic real-time light shaft generation taking sky illumination into account. Multimedia Tools Appl. 78, 5 (2019), 6073--6092.

Digital Library

[59]

Matthias Korn, Maik Stange, Andreas von Arb, Lisa Blum, Michael Kreil, Kathrin-Jennifer Kunze, Jens Anhenn, Timo Wallrath, and Thorsten Grosch. 2008. Interactive augmentation of live images using a HDR stereo camera. J. Virt. Real. Broadcast. 4, 12 (2008).

[60]

Joel Kronander, Francesco Banterle, Andrew Gardner, Ehsan Miandji, and Jonas Unger. 2015. Photorealistic rendering of mixed reality scenes. Comput. Graph. Forum 34, 2, 643--665. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.12591

Digital Library

[61]

Jean-François Lalonde, Alexei A. Efros, and Srinivasa G. Narasimhan. 2009. Estimating natural illumination from a single outdoor image. In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision. 183--190.

[62]

Philipp Lensing and Wolfgang Broll. 2012. Instant indirect illumination for dynamic mixed reality scenes. In Proceedings of the 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR’12). 109--118.

Digital Library

[63]

Philipp Lensing and Wolfgang Broll. 2013. LightSkin: Real-time global illumination for virtual and mixed reality. In Proceedings of the 5th Joint Virtual Reality Conference. Eurographics Association, 17--24.

[64]

Yang Liu, Theo Gevers, and Xueqing Li. 2015. Estimation of sunlight direction using 3D object models. IEEE Trans. Image Process. 24, 3 (Mar. 2015), 932--942.

[65]

Yanli Liu and Xavier Granier. 2012. Online tracking of outdoor lighting variations for augmented reality with moving cameras. IEEE Trans. Vis. Comput. Graph. 18, 4 (Apr. 2012), 573--580.

[66]

Céline Loscos, George Drettakis, and Luc Robert. 2000. Interactive virtual relighting of real scenes. IEEE Trans. Vis. Comput. Graph. 6, 4 (Oct. 2000), 289--305.

Digital Library

[67]

Céline Loscos, Marie-Claude Frasson, George Drettakis, Bruce Walter, Xavier Granier, and Pierre Poulin. 1999. Interactive virtual relighting and remodeling of real scenes. In Proceedings of the Conference on Rendering Techniques 1999, Dani Lischinski and Greg Ward Larson (Eds.). Springer, Vienna, 329--340.

[68]

Claus B. Madsen and Michael Nielsen. 2008. Towards probe-less augmented reality. Position Paper, Computer Vision and Media Technology Lab, Aalborg University, Aalborg, Denmark (2008).

[69]

David Mandl, Kwang Moo Yi, Peter Mohr, Peter M. Roth, Pascal Fua, Vincent Lepetit, Dieter Schmalstieg, and Denis Kalkofen. 2017. Learning lightprobes for mixed reality illumination. In Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR’17). 82--89.

[70]

Stephen Robert Marschner and Donald P. Greenberg. 1998. Inverse Rendering for Computer Graphics. Citeseer.

[71]

Stephen R. Marschner, Stephen H. Westin, Eric P. F. Lafortune, Kenneth E. Torrance, and Donald P. Greenberg. 1999. Image-based BRDF measurement including human skin. In Proceedings of the Conference on Rendering Techniques 1999, Dani Lischinski and Greg Ward Larson (Eds.). Springer, Vienna, 131--144.

[72]

Vincent Masselus, Pieter Peers, Philip Dutré, and Yves D. Willems. 2003. Relighting with 4D incident light fields. ACM Trans. Graph. 22, 3 (2003), 613--620.

Digital Library

[73]

Soham Uday Mehta, Kihwan Kim, Dawid Pajak, Kari Pulli, Jan Kautz, and Ravi Ramamoorthi. 2015. Filtering environment illumination for interactive physically-based rendering in mixed reality. In Proceedings of the Eurographics Symposium on Rendering, Vol. 1.

[74]

Maxime Meilland, Christian Barat, and Andrew Comport. 2013. 3D high dynamic range dense visual SLAM and its application to real-time object re-lighting. In Proceedings of the 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR’13). 143--152.

[75]

Rafael Monroy, Matis Hudon, and Aljosa Smolic. 2018. Dynamic environment mapping for augmented reality applications on mobile devices. In Proceedings of the Conference on Vision, Modeling 8 Visualization (VMV’18). 21--28.

Digital Library

[76]

Alexandre Morgand, Mohamed Tamaazousti, and Adrien Bartoli. 2018. A geometric model for specularity prediction on planar surfaces with multiple light sources. IEEE Trans. Vis. Comput. Graph. 24, 5 (May 2018), 1691--1704.

[77]

Brian Mount, Stephen Latta, Daniel McCulloch, Kevin Geisner, Jason Scott, Jonathan Steed, Arthur Tomlin, and Mark Mihelich. 2013. Augmented reality with realistic occlusion. US Patent App. 13/309,372.

[78]

Ko Nishino and Shree K. Nayar. 2004. Eyes for relighting. ACM Trans. Graph. 23, 3 (Aug. 2004), 704--711.

Digital Library

[79]

Derek Nowrouzezahrai, Stefan Geiger, Kenny Mitchell, Robert Sumner, Wojciech Jarosz, and Markus Gross. 2011. Light factorization for mixed-frequency shadows in augmented reality. In Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality. 173--179.

Digital Library

[80]

Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, et al. 2010. OptiX: A general purpose ray tracing engine. ACM Trans. Graph. 29, 4.

Digital Library

[81]

Gustavo Patow and Xavier Pueyo. 2003. A survey of inverse rendering problems. Comput. Graph. Forum 22, 4, 663--687.

[82]

Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically Based Rendering: From Theory to Implementation. Morgan Kaufmann.

[83]

Pierre Poulin, Mathieu Ouimet, and Marie-Claude Frasson. 1998. Interactively modeling with photogrammetry. In Proceedings of the Annual Confernence on Rendering Techniques 1998, George Drettakis and Nelson Max (Eds.). Springer, Vienna, 93--104.

[84]

Ravi Ramamoorthi and Pat Hanrahan. 2001. An efficient representation for irradiance environment maps. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’01). Association for Computing Machinery, New York, NY, 497--500.

Digital Library

[85]

Erik Reinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward, and Karol Myszkowski. 2010. High Dynamic Range Imaging: Acquisition, Display, and Image-based Lighting. Morgan Kaufmann.

[86]

Thomas Richter-Trummer, Denis Kalkofen, Jinwoo Park, and Dieter Schmalstieg. 2016. Instant mixed reality lighting from casual scanning. In Proceedings of the 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR’16). 27--36.

[87]

Tobias Ritschel, Elmar Eisemann, Inwoo Ha, James D. K. Kim, and Hans-Peter Seidel. 2011. Making imperfect shadow maps view-adaptive: High-quality global illumination in large dynamic scenes. Comput. Graph. Forum 30, 8 (2011), 2258--2269.

[88]

Kai Rohmer, Wolfgang Büschel, Raimund Dachselt, and Grosch Thorsten. 2015. Interactive near-field illumination for photorealistic augmented reality with varying materials on mobile devices. IEEE Trans. Vis. Comput. Graph. 21, 12 (Dec. 2015), 1349--1362.

Digital Library

[89]

Kai Rohmer and Thorsten Grosch. 2015. Tiled frustum culling for differential rendering on mobile devices. In Proceedings of the 2015 IEEE International Symposium on Mixed and Augmented Reality. 37--42.

Digital Library

[90]

Kai Rohmer, Johannes Jendersie, and Thorsten Grosch. 2017. Natural environment illumination: Coherent interactive augmented reality for mobile and non-mobile devices. IEEE Trans. Vis. Comput. Graph. 23, 11 (Nov. 2017), 2474--2484.

Digital Library

[91]

Imari Sato, Yoichi Sato, and Katsushi Ikeuchi. 1999. Acquiring a radiance distribution to superimpose virtual objects onto a real scene. IEEE Trans. Vis. Comput. Graph. 5, 1 (Jan. 1999), 1--12.

Digital Library

[92]

I. Sato, Y. Sato, and K. Ikeuchi. 2003. Illumination from shadows. IEEE Trans. Pattern Anal. Mach. Intell. 25, 3 (Mar. 2003), 290--300.

Digital Library

[93]

Yoichi Sato, Mark D. Wheeler, and Katsushi Ikeuch. 1997. Object shape and reflectance modeling from observation. In Proceedings of the 24thAnnual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’97). ACM Press/Addison-Wesley Publishing Co., USA, 379--387.

Digital Library

[94]

Dieter Schmalstieg and Tobias Höllerer. 2016. Augmented Reality: Principles and Practice. Addison-Wesley Professional.

Digital Library

[95]

Tobias Schwandt and Wolfgang Broll. 2016. A single camera image based approach for glossy reflections in mixed reality applications. In Proceedings of the 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR’16). 37--43.

[96]

Weiqi Shi, Zeyu Wang, Metin Sezgin, Julie Dorsey, and Holly Rushmeier. 2017. Material design in augmented reality with In-Situ visual feedback. In Proceedings of the Eurographics Symposium on Rendering: Experimental Ideas 8 Implementations (EGSR’17). Eurographics Association, Goslar, DEU, 93--103.

Digital Library

[97]

Wonyoung Son, Bodam Nam, Taehyub Kim, and Hyunki Hong. 2012. Using environment illumination information in mobile augmented reality. In Proceedings of the 2012 IEEE International Conference on Consumer Electronics (ICCE’12). 588--589.

[98]

Kevin Soulier, Matías Nicolás Selzer, and Martín Leonardo Larrea. 2016. Real-time estimation of illumination direction for augmented reality with low-cost sensors. In Proceedings of the XXII Congreso Argentino de Ciencias de la Computación (CACIC’16).

[99]

Jürgen Stauder. 1999. Augmented reality with automatic illumination control incorporating ellipsoidal models. IEEE Trans. Multimedia 1, 2 (Jun. 1999), 136--143.

Digital Library

[100]

Natsuki Sugano, Hirokazu Kato, and Keihachiro Tachibana. 2003. The effects of shadow representation of virtual objects in augmented reality. In Proceedings of the 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality. 76--83.

Digital Library

[101]

Peter Supan, Ines Stuppacher, and Michael Haller. 2006. Image based shadowing in real-time augmented reality. Int. J. Virt. Real. 5, 3 (Jan. 2006), 1--7.

[102]

Norimichi Tsumura, Minh Nguyet Dang, Takao Makino, and Yoichi Miyake. 2003. Estimating the directions to light sources using images of eye for reconstructing 3D human face. In Proceedings of the Color and Imaging Conference 2003, 1, 77--81.

[103]

Jonas Unger, Stefan Gustavson, and Anders Ynnerman. 2007. Spatially varying image based lighting by light probe sequences. Vis. Comput. 23, 7 (2007), 453--465.

Digital Library

[104]

Jonas Unger, Joel Kronander, Per Larsson, Stefan Gustavson, Joakim Löw, and Anders Ynnerman. 2013. Spatially varying image based lighting using HDR-video. Comput. Graph. 37, 7 (2013), 923--934.

Digital Library

[105]

Jonas Unger, Joel Kronander, Per Larsson, Stefan Gustavson, and Anders Ynnerman. 2013. Temporally and spatially varying image based lighting using HDR-video. In Proceedings of the 21st European Signal Processing Conference (EUSIPCO’13). 1--5.

[106]

Henrique Weber, Donald Prévost, and Jean-François Lalonde. 2018. Learning to estimate indoor lighting from 3D objects. In Proceedings of the 2018 International Conference on 3D Vision (3DV’18). 199--207.

[107]

Harald Wuest, Florent Vial, and Didier Stricker. 2005. Adaptive line tracking with multiple hypotheses for augmented reality. In Proceedings of the 4th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR’05). 62--69.

Digital Library

[108]

Guanyu Xing, Yanli Liu, Xueying Qin, and Qunsheng Peng. 2011. On-line illumination estimation of outdoor scenes based on area selection for augmented reality. In Proceedings of the 2011 12th International Conference on Computer-Aided Design and Computer Graphics. 439--442.

Digital Library

Cited By

Wang HWang WZhou HXu HWu SZhu LCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Language-Driven Interactive Shadow DetectionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681192(5527-5536)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681192
Alhakamy A(2024)Extended Reality (XR) Toward Building Immersive Solutions: The Key to Unlocking Industry 4.0ACM Computing Surveys10.1145/365259556:9(1-38)Online publication date: 25-Apr-2024
https://dl.acm.org/doi/10.1145/3652595
Pointecker FFriedl-Knirsch JJetter HAnthes C(2024)From Real to Virtual: Exploring Replica-Enhanced Environment Transitions along the Reality-Virtuality ContinuumProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642844(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642844
Show More Cited By

Index Terms

Real-time Illumination and Visual Coherence for Photorealistic Augmented/Mixed Reality
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

Towards Indistinguishable Augmented Reality: A Survey on Optical See-through Head-mounted Displays
Invited Tutorial

Adding virtual information that is indistinguishable from reality has been a long-awaited goal in Augmented Reality (AR). While already demonstrated in the 1960s, only recently have Optical See-Through Head-Mounted Displays (OST-HMDs) seen a reemergence, ...
Real Time Ray Tracing for Augmented Reality
SVR '12: Proceedings of the 2012 14th Symposium on Virtual and Augmented Reality

This paper introduces a novel graphics rendering pipeline applied to augmented reality, based on a real time ray tracing paradigm. Ray tracing techniques process pixels independently from each other, allowing an easy integration with image-based ...
An Empirical Evaluation of the Performance of Real-Time Illumination Approaches: Realistic Scenes in Augmented Reality
Augmented Reality, Virtual Reality, and Computer Graphics
Abstract
Augmented, Virtual, and Mixed Reality (AR/VR/MR) systems have been developed in general, with many of these applications having accomplished significant results, rendering a virtual object in the appropriate illumination model of the real ... $^{}$

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 53, Issue 3

May 2021

787 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/3403423

Editor:
Albert Zomaya
University of Sydney, Australia

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 May 2020

Online AM: 07 May 2020

Accepted: 01 March 2020

Revised: 01 July 2019

Received: 01 September 2018

Published in CSUR Volume 53, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Survey
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
934
Total Downloads

Downloads (Last 12 months)135
Downloads (Last 6 weeks)19

Reflects downloads up to 06 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang HWang WZhou HXu HWu SZhu LCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Language-Driven Interactive Shadow DetectionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681192(5527-5536)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681192
Alhakamy A(2024)Extended Reality (XR) Toward Building Immersive Solutions: The Key to Unlocking Industry 4.0ACM Computing Surveys10.1145/365259556:9(1-38)Online publication date: 25-Apr-2024
https://dl.acm.org/doi/10.1145/3652595
Pointecker FFriedl-Knirsch JJetter HAnthes C(2024)From Real to Virtual: Exploring Replica-Enhanced Environment Transitions along the Reality-Virtuality ContinuumProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642844(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642844
Feld NBimberg PWeyers BZielasko D(2024)Simple and Efficient? Evaluation of Transitions for Task-Driven Cross-Reality ExperiencesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.335694930:12(7601-7618)Online publication date: Dec-2024
https://doi.org/10.1109/TVCG.2024.3356949
Alhakamy A(2024)"XR Is A Game": Exploring People’s Reasons for Using, Limiting, and Abandoning Extended Reality2024 IEEE International Conference on Real-time Computing and Robotics (RCAR)10.1109/RCAR61438.2024.10670927(271-278)Online publication date: 24-Jun-2024
https://doi.org/10.1109/RCAR61438.2024.10670927
Feld NPointecker FAnthes CZielasko D(2024)Perceptual Issues in Mixed Reality: A Developer-oriented Perspective on Video See-Through Head-Mounted Displays2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct64951.2024.00044(170-175)Online publication date: 21-Oct-2024
https://doi.org/10.1109/ISMAR-Adjunct64951.2024.00044
Wang YSong SZhao LXia HYuan ZZhang Y(2024)CGLight: An effective indoor illumination estimation method based on improved convmixer and GauGANComputers & Graphics10.1016/j.cag.2024.104122125(104122)Online publication date: Dec-2024
https://doi.org/10.1016/j.cag.2024.104122
Kritikos YGiariskanis FProtopapadaki EPapanastasiou APapadopoulou EMania K(2023)Audio Augmented Reality OutdoorsProceedings of the 2023 ACM International Conference on Interactive Media Experiences10.1145/3573381.3597028(199-204)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3573381.3597028
Sabie DSheta HFerdous HKopalakrishnan VAhmed S(2023)Be Our Guest: Intercultural Heritage Exchange through Augmented Reality (AR)Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581005(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581005
Xu YJiang YWang SLi KGeng G(2023)Delta Path Tracing for Real-Time Global Illumination in Mixed Reality2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00020(44-52)Online publication date: Mar-2023
https://doi.org/10.1109/VR55154.2023.00020
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents