article

Foveated Real-Time Ray Tracing for Head-Mounted Displays

Authors:

André Hinkenjann,

Arsène Pérard-Gayot,

Philipp Slusallek,

Yongmin LiAuthors Info & Claims

Computer Graphics Forum, Volume 35, Issue 7

Pages 289 - 298

Published: 01 October 2016 Publication History

Abstract

Head-mounted displays with dense pixel arrays used for virtual reality applications require high frame rates and low latency rendering. This forms a challenging use case for any rendering approach. In addition to its ability of generating realistic images, ray tracing offers a number of distinct advantages, but has been held back mainly by its performance. In this paper, we present an approach that significantly improves image generation performance of ray tracing. This is done by combining foveated rendering based on eye tracking with reprojection rendering using previous frames in order to drastically reduce the number of new image samples per frame. To reproject samples a coarse geometry is reconstructed from a G-Buffer. Possible errors introduced by this reprojection as well as parts that are critical to the perception are scheduled for resampling. Additionally, a coarse color buffer is used to provide an initial image, refined smoothly by more samples were needed. Evaluations and user tests show that our method achieves real-time frame rates, while visual differences compared to fully rendered images are hardly perceivable. As a result, we can ray trace non-trivial static scenes for the Oculus DK2 HMD at 1182 × 1464 per eye within the the VSync limits without perceived visual differences.

References

[1]

<label>{ALK12}¿¿</label> Aila T., Laine S., Karras T.: Understanding the Efficiency of Ray Traversal on GPUs - Kepler and Fermi Addendum. Technical Report NVR-2012-02, NVIDIA, 2012. HPG2012 poster. 3, 4

[2]

<label>{Dc07}¿¿</label> Duchowski A.T., Clöltekin A.: Foveated Gaze-contingent Displays for Peripheral LOD Management, 3D Visualization, and Stereo Imaging. ACM Trans. Multimedia Comput. Commun. Appl. Volume 3, Issue 4 Dec. 2007, pp.6:1-6:18. 2

Digital Library

[3]

<label>{DWWL05}¿¿</label> Dayal A., Woolley C., Watson B., Luebke D.: Adaptive Frameless Rendering. In SIGGRAPH 2005 Courses 2005, ACM. 3

Digital Library

[4]

<label>{FH03}¿¿</label> Field A., Hole G.: How to Design and Report Experiments. Sage publications Limited, 2003. 7, 8

[5]

<label>{FH14}¿¿</label> Fujita M., Harada T.: Foveated Real-Time Ray Tracing for Virtual Reality Headset, May 2014. Poster Siggraph Asia. 2, 6

[6]

<label>{FS93}¿¿</label> Funkhouser T.A., Séquin C.H.: Adaptive display algorithm for interactive frame rates during visualization of complex virtual environments. In Computer graphics and interactive techniques 1993, ACM, pp. pp.247-254. 2

Digital Library

[7]

<label>{GFD*12}¿¿</label> Guenter B., Finch M., Drucker S., Tan D., Snyder J.: Foveated 3D Graphics. ACM Transactions on Graphics Volume 31, Issue 6 2012, pp.164. 2, 3, 6

Digital Library

[8]

<label>{GP98}¿¿</label> Geisler W.S., Perry J.S.: A real-time foveated multiresolution system for low-bandwidth video communication</otherTitle>. In <otherTitle>in Proc. SPIE 1998, pp. pp.294-305. 2

[9]

<label>{Hun15}¿¿</label> Hunt W.: Virtual Reality: The Next Great Graphics Revolution</otherTitle>. <otherTitle>Keynote Talk HPG 2015, Oculus Research, Aug 2015. 1, 2

[10]

<label>{JESG12}¿¿</label> Jimenez J., Echevarria J.I., Sousa T., Gutierrez D.: SMAA: Enhanced subpixel morphological antialiasing. In Computer Graphics Forum 2012, vol. Volume 31, pp. pp.355-364. 5

Digital Library

[11]

<label>{JW02}¿¿</label> Jeschke S., Wimmer M.: Textured Depth Meshes for Realtime Rendering of Arbitrary Scenes</otherTitle>. In <otherTitle>13th Eurographics Workshop on Rendering 2002, pp. pp.181-190. 3

Digital Library

[12]

<label>{LK87}¿¿</label> Legge G.E., Kersten D.: Contrast discrimination in peripheral vision. J. Opt. Soc. Am. A4, Volume 8 Aug 1987, pp.1594-1598. 2, 7

[13]

<label>{LMR*12}¿¿</label> Lou C.I., Migotina D., Rodrigues J.P., Semedo J., Wan F., Mak P.U., Mak P.I., Vai M.I., Melicio F., Pereira J.G., Rosa A.: Object Recognition Test in Peripheral Vision: A Study on the Influence of Object Color, Pattern and Shape</otherTitle>. In <otherTitle>International Conference on Brain Informatics 2012, pp. pp.18-26. 2, 7

Digital Library

[14]

<label>{LW90}¿¿</label> Levoy M., Whitaker R.: Gaze-directed Volume Rendering. In Proceedings of the 1990 Symposium on Interactive 3D Graphics 1990, I3D '90, ACM, pp. pp.217-223. 2

Digital Library

[15]

<label>{MD07}¿¿</label> Murphy H., Duchowski A.T.: Hybrid Image-/Model-based Gaze-contingent Rendering. In 4th Symposium on Applied Perception in Graphics and Visualization 2007, ACM, pp. pp.107-114. 2

Digital Library

[16]

<label>{Miu86}¿¿</label> Miura T.: Coping with situational demands: A study of eye movements and peripheral vision performance</otherTitle>. <otherTitle>Vision in Vehicles 1986, pp.206-216. 2, 8, 9

[17]

<label>{MMB97}¿¿</label> Mark W.R., McMillan L., Bishop G.: Post-rendering 3D Warping. In Proceedings of the 1997 Symposium on Interactive 3D Graphics 1997, ACM, pp. pp.7-ff. 3, 5

Digital Library

[18]

<label>{MN84}¿¿</label> McKee S., Nakayama K.: The detection of motion in the peripheral visual field. Vision research Volume 24, Issue 1 jan 1984, pp.25-32. 2

[19]

<label>{NSL*07}¿¿</label> Nehab D., Sander P.V., Lawrence J., Tatarchuk N., Isidoro J.R.: Accelerating Real-time Shading with Reverse Reprojection Caching. In 22Nd ACM Symposium on Graphics Hardware 2007, Eurographics Association, pp. pp.25-35. 3

Digital Library

[20]

<label>{ODH02}¿¿</label> O'Sullivan C., Dingliana J., Howlett S.: Gazecontingent algorithms for interactive graphics</otherTitle>. <otherTitle>Mind's Eyes: Cognitive and Applied Aspects of Eye Movement Research 2002. 2

[21]

<label>{PBNG15}¿¿</label> Pohl D., Bolkart T., Nickels S., Grau O.: Using astigmatism in wide angle HMDs to improve rendering. In Virtual Reality 2015, IEEE, pp. pp.263-264. 2

[22]

<label>{Ree15}¿¿</label> Reed N.: Gameworks VR. Tech. rep., NVIDIA, 2015. 6

[23]

<label>{RLMS03}¿¿</label> Reingold E.M., Loschky L.C., McConkie G.W., Stampe D.M.: Gaze-Contingent Multiresolutional Displays: An Integrative Review. Human Factors Volume 45 2003, pp.307-328. 2

[24]

<label>{SFD09}¿¿</label> Stich M., Friedrich H., Dietrich A.: Spatial Splits in Bounding Volume Hierarchies. In High Performance Graphics 2009 2009, ACM, pp. pp.7-13. 3

Digital Library

[25]

<label>{SGEM16}¿¿</label> Stengel M., Grogorick S., Eisemann M., Magnor M.: Adaptive image-space sampling for gaze-contingent real-time rendering. Eurographics Symposium on Rendering Volume 35, Issue 4 2016. 2, 6

[26]

<label>{Sim00}¿¿</label> Simmons M.: A Dynamic Mesh Display Representation for the Holodeck Ray Cache System. Tech. Rep. UCB/CSD-00-1090, EECS Department, University of California, Berkeley, Jan 2000. 3

Digital Library

[27]

<label>{SRJ11}¿¿</label> Strasburger H., Rentschler I., Jüttner M.: Peripheral vision and pattern recognition: A review. Journal of vision Volume 11, Issue 5 jan 2011, pp.1-82. 3

[28]

<label>{SS00}¿¿</label> Simmons M., Séquin C.H.: Tapestry: A Dynamic Meshbased Display Representation for Interactive Rendering</otherTitle>. In <otherTitle>Eurographics Workshop on Rendering Techniques 2000 2000, pp. pp.329-340. 3, 4

Digital Library

[29]

<label>{TPWG02}¿¿</label> Tole P., Pellacini F., Walter B., Greenberg D.P.: Interactive Global Illumination in Dynamic Scenes. ACM Trans. Graph. Volume 21, Issue 3 July 2002, pp.537-546. 3, 4

Digital Library

[30]

<label>{Wan95}¿¿</label> Wandell B.A.: Foundations of Vision. Stanford University, 1995. 2

[31]

<label>{WDP99}¿¿</label> Walter B., Drettakis G., Parker S.: Interactive Rendering using the Render Cache. In Proceedings of the Eurographics Workshop on Rendering Jun 1999, vol. Volume 10, pp. pp.235-246. 3

Digital Library

[32]

<label>{WS99}¿¿</label> Ward G., Simmons M.: The Holodeck Ray Cache: An Interactive Rendering System for Global Illumination in Nondiffuse Environments. ACM Trans. Graph Volume 18, Issue 4 Oct. 1999, pp.361-368. 3

Digital Library

[33]

<label>{WWH04}¿¿</label> Watson B., Walker N., Hodges L.F.: Supra-threshold Control of Peripheral LOD. ACM Trans. Graph Volume 23, Issue 3 Aug. 2004, pp.750-759. 2

Digital Library

[34]

<label>{YNS*09}¿¿</label> Yang L., Nehab D.F., Sander P.V., Sitthi-amorn P., Lawrence J., Hoppe H.: Amortized supersampling. ACM Trans. Graph. Volume 28, Issue 5 2009, pp.135:1-135:12. 3, 5

Digital Library

[35]

<label>{YPG01}¿¿</label> Yee H., Pattanaik S., Greenberg D.P.: Spatiotemporal Sensitivity and Visual Attention for Efficient Rendering of Dynamic Environments. ACM Trans. Graph Volume 20, Issue 1 Jan. 2001, pp.39-65. 2

Digital Library

Cited By

Fan CHung THsu C(2022)Modeling the User Experience of Watching 360° Videos with Head-Mounted DisplaysACM Transactions on Multimedia Computing, Communications, and Applications10.1145/346382518:1(1-23)Online publication date: 27-Jan-2022
https://dl.acm.org/doi/10.1145/3463825
Kaplanyan ASochenov ALeimkühler TOkunev MGoodall TRufo G(2019)DeepFoveaACM Transactions on Graphics10.1145/3355089.335655738:6(1-13)Online publication date: 8-Nov-2019
https://dl.acm.org/doi/10.1145/3355089.3356557
Hadnett-Hunter JNicolaou GO'Neill EProulx M(2019)The Effect of Task on Visual Attention in Interactive Virtual EnvironmentsACM Transactions on Applied Perception10.1145/335276316:3(1-17)Online publication date: 6-Sep-2019
https://dl.acm.org/doi/10.1145/3352763
Show More Cited By

Index Terms

Foveated Real-Time Ray Tracing for Head-Mounted Displays
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
    2. Rendering
      1. Ray tracing

Recommendations

Foveated Depth-of-Field Filtering in Head-Mounted Displays

In recent years, a variety of methods have been introduced to exploit the decrease in visual acuity of peripheral vision, known as foveated rendering. As more and more computationally involved shading is requested and display resolutions increase, ...
Virtual Spherical Gaussian Lights for Real-time Glossy Indirect Illumination

Virtual point lights VPLs are well established for real-time global illumination. However, this method suffers from spiky artifacts and flickering caused by singularities of VPLs, highly glossy materials, high-frequency textures, and discontinuous ...
Consistent Scene Editing by Progressive Difference Images

Even though much research was dedicated to the acceleration of consistent, progressive light transport simulations, the computation of fully converged images is still very time-consuming. This is problematic, as for the practical use in production ...

Comments

Information & Contributors

Information

Published In

cover image Computer Graphics Forum

Computer Graphics Forum Volume 35, Issue 7

October 2016

513 pages

ISSN:0167-7055

EISSN:1467-8659

Issue’s Table of Contents

Publisher

The Eurographs Association & John Wiley & Sons, Ltd.

Chichester, United Kingdom

Publication History

Published: 01 October 2016

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Fan CHung THsu C(2022)Modeling the User Experience of Watching 360° Videos with Head-Mounted DisplaysACM Transactions on Multimedia Computing, Communications, and Applications10.1145/346382518:1(1-23)Online publication date: 27-Jan-2022
https://dl.acm.org/doi/10.1145/3463825
Kaplanyan ASochenov ALeimkühler TOkunev MGoodall TRufo G(2019)DeepFoveaACM Transactions on Graphics10.1145/3355089.335655738:6(1-13)Online publication date: 8-Nov-2019
https://dl.acm.org/doi/10.1145/3355089.3356557
Hadnett-Hunter JNicolaou GO'Neill EProulx M(2019)The Effect of Task on Visual Attention in Interactive Virtual EnvironmentsACM Transactions on Applied Perception10.1145/335276316:3(1-17)Online publication date: 6-Sep-2019
https://dl.acm.org/doi/10.1145/3352763
Friston SRitschel TSteed A(2019)Perceptual rasterization for head-mounted display image synthesisACM Transactions on Graphics10.1145/3306346.332303338:4(1-14)Online publication date: 12-Jul-2019
https://dl.acm.org/doi/10.1145/3306346.3323033
Weier MRoth THinkenjann ASlusallek P(2018)Foveated Depth-of-Field Filtering in Head-Mounted DisplaysACM Transactions on Applied Perception10.1145/323830115:4(1-14)Online publication date: 19-Sep-2018
https://dl.acm.org/doi/10.1145/3238301
Koskela MImmonen KViitanen TJääskeläinen PMultanen JTakala JGutierrez DHuang H(2017)Foveated instant preview for progressive renderingSIGGRAPH Asia 2017 Technical Briefs10.1145/3145749.3149423(1-4)Online publication date: 27-Nov-2017
https://dl.acm.org/doi/10.1145/3145749.3149423

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents