Article

Operating system support for augmented reality applications

Authors:

Loris D'Antoni,

Tadayoshi Kohno,

Benjamin Livshits,

Alexander Moshchuk,

Franziska Roesner,

Helen J. WangAuthors Info & Claims

HotOS'13: Proceedings of the 14th USENIX conference on Hot Topics in Operating Systems

Page 21

Published: 13 May 2013 Publication History

Abstract

Augmented reality (AR) takes natural user input (NUI), such as gestures, voice, and eye gaze, and produces digital visual overlays on top of reality seen by a user. Today, multiple shipping AR applications exist, most notably titles for the Microsoft Kinect and smartphone applications such as Layar, Wikitude, and Junaio. Despite this activity, little attention has been paid to operating system support for AR applications. Instead, each AR application today does its own sensing and rendering, with the help of user-level libraries like OpenCV or the Microsoft Kinect SDK.

In this paper, we explore how operating systems should evolve to support AR applications. Because AR applications work with fundamentally new inputs and outputs, an OS that supports AR applications needs to re-think the input and display abstractions exposed to applications. Unlike mouse and keyboard, which form explicit, separate channels for user input, NUI requires continuous sensing of the real-world environment, which often has sensitive data mixed with user input. Hence, the OS input abstractions must ensure that user privacy is not violated, and the OS must provide a fine-grained permission system for access to recognized objects like a user's face and skeleton. In addition, because visual outputs of AR applications mix real-world and virtual objects, the synthetic window abstraction in traditional GUIs is no longer viable, and OSes must rethink the display abstractions and their management. We discuss research directions for solving these and other issues and building an OS that let multiple applications share one (augmented) reality.

References

[1]

Michael Abrash. Latency - the sine qua non of ar and vr, 2012. http://blogs.valvesoftware.com/abrash/ latency-the-sine-qua-non-of-ar-and-vr/.

[2]

Mark Aiken, Manuel Fähndrich, Chris Hawblitzel, Galen Hunt, and James Larus. Deconstructing process isolation. In Proceedings of the 2006 workshop on Memory system performance and correctness, MSPC '06, pages 1-10, New York, NY, USA, 2006. ACM.

[3]

Ronald T. Azuma. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6(4):355-385, August 1997.

[4]

Ronald T. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre. Recent advances in augmented reality. Computer Graphics and Applications, 21(6):34-47, 2001.

[5]

D. Chu, A. Kansal, J. Liu, and F. Zhao. Mobile apps: It's time to move up to condOS. May 2011. http://research.microsoft.com/ apps/pubs/default.aspx?id=147238.

[6]

Loris D'Antoni, Margus Veanes, Benjamin Livshits, and David Molnar. FAST: A transducer-based language for tree manipulation, 2012. MSR Technical Report 2012-123 http://research.microsoft.com/apps/ pubs/default.aspx?id=179252.

[7]

Micah Dowty and Jeremy Sugerman. GPU virtualization on VMware's hosted I/O architecture. In First Workshop on I/O Virtualization, 2008. http://static.usenix.org/event/wiov08/ tech/full_papers/dowty/dowty_html/.

[8]

Cynthia Dwork. The differential privacy frontier. In 6th Theory of Cryptography Conference (TCC), 2009.

[9]

Dawson R. Engler and M. Frans Kaashoek. Exterminate all operating system abstractions. In Workshop on Hot Topics in Operating Systems - HotOS, 1995.

[10]

Adrienne Porter Felt, Serge Egelman, Matthew Finifter, Devdatta Akhawe, and David Wagner. How to ask for permission. In USENIX Workshop on Hot Topics in Security, 2012.

[11]

Adrienne Porter Felt, Elizabeth Ha, Serge Egelman, Ariel Haney, Erika Chin, and David Wagner. Android permissions: User attention, comprehension, and behavior. In Symposium on Usable Privacy and Security (SOUPS), 2012.

[12]

Robert Hansen. Clickjacking, 2008. http://ha.ckers.org/blog/20080915/ clickjacking/.

[13]

Emma Hutchings in psfk. Augmented reality lets shoppers see how new furniture would look at home, 2012. http://www.psfk.com/2012/05/ augmented-reality-furniture-app.html.

[14]

Eliza Gauger in WIRED Magazine. Sonicstyle grey goo cripples second life, 2006. http://www.wired.com/gamelife/2006/11/ sonicstyle_grey/.

[15]

M. Frans Kaashoek, Dawson R. Engler, Gregory R. Ganger, Hector M. Briceno, Russell Hung, Davied Mazieres, Thomas Pinckney, Robert Grimm, John Jannotti, and Kenneth Mackenzie. Application performance and flexibility on exokernel systems. In SOSP, 1997. http://stanford.edu/~engler/sosp-97.ps.

[16]

Layar. Layar catalogue, 2013. http://www.layar.com/layers.

[17]

LSL Labs Wiki. Grey goo fence, 2012. http://lslwiki.net/lslwiki/wakka.php? wakka=GreyGooFence.

[18]

Kevin Poulsen WIRED Magazine. Hackers assault epilepsy patients via computer, 2008. http://www.wired.com/politics/ security/news/2008/03/epilepsy.

[19]

Edmund B. Nightingale, Orion Hodson, Ross McIlroy, Chris Hawblitzel, and Galen Hunt. Helios: Heterogeneous multiprocessing with satellite kernels. In SOSP, 2009. http://research.microsoft.com/apps/ pubs/default.aspx?id=81154.

[20]

M.Z. Poh, D.J. MacDuff, and R.W. Picard. Advancements in non-contact, multiparameter physiological measurements using a webcam. IEEE Trans Biomed Engineering, 58(1):7-11, 2011.

[21]

Project Glass. https://plus.google.com/ +projectglass/posts.

[22]

Qualcomm. Augmented Reality SDK, 2011. http://www.qualcomm.com/products_ services/augmented_reality.html.

[23]

Franziska Roesner, Tadayoshi Kohno, Alexander Moshchuk, Bryan Parno, Helen J. Wang, and Crispin Cowan. User-driven access control: Rethinking permission granting in modern operating systems. In IEEE Symposium on Security and Privacy, 2011.

[24]

J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. Real-time human pose recognition in parts from a single depth image. In Computer Vision and Pattern Recognition, June 2011.

[25]

Paul Viola and Michael Jones. Robust Realtime Object Detection. In International Journal of Computer Vision, 2001.

Cited By

Ruth KKohno TRoesner FHeninger NTraynor P(2019)Secure multi-user content sharing for augmented reality applicationsProceedings of the 28th USENIX Conference on Security Symposium10.5555/3361338.3361349(141-158)Online publication date: 14-Aug-2019
https://dl.acm.org/doi/10.5555/3361338.3361349
Lebeck KKohno TRoesner FWolman AZhong L(2019)Enabling Multiple Applications to Simultaneously Augment RealityProceedings of the 20th International Workshop on Mobile Computing Systems and Applications10.1145/3301293.3302362(81-86)Online publication date: 22-Feb-2019
https://dl.acm.org/doi/10.1145/3301293.3302362
Wójtowicz AWojciechowski RRumiński DWalczak K(2018)Securing ubiquitous AR servicesMultimedia Tools and Applications10.1007/s11042-018-5892-x77:20(26881-26899)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1007/s11042-018-5892-x
Show More Cited By

Operating system support for augmented reality applications
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms

Recommendations

Augmented reality technologies, systems and applications

This paper surveys the current state-of-the-art of technology, systems and applications in Augmented Reality. It describes work performed by many different research groups, the purpose behind each new Augmented Reality system, and the difficulties and ...
Haptics in Augmented Reality
ICMCS '99: Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2

An augmented reality system merges synthetic sensory information into a user's perception of a three-dimensional environment. An important performance goal for an augmented reality system is that the user perceives a single seamless environment. In most ...
Augmented reality as perceptual reality
VSMM'06: Proceedings of the 12th international conference on Interactive Technologies and Sociotechnical Systems

As shown in Paul Milgram et al’s Reality-Virtuality Continuum (1994), Augmented Reality occupies a very unique status in the spectrum of Mixed Reality. Unlike Virtual Reality, which is completely made up of the virtual and has been the most important ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

HotOS'13: Proceedings of the 14th USENIX conference on Hot Topics in Operating Systems

May 2013

27 pages

General Chair:
Petros Maniatis
Intel Labs

Sponsors

VMware
Google Inc.
Microsoft Research: Microsoft Research

Publisher

USENIX Association

United States

Publication History

Published: 13 May 2013

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ruth KKohno TRoesner FHeninger NTraynor P(2019)Secure multi-user content sharing for augmented reality applicationsProceedings of the 28th USENIX Conference on Security Symposium10.5555/3361338.3361349(141-158)Online publication date: 14-Aug-2019
https://dl.acm.org/doi/10.5555/3361338.3361349
Lebeck KKohno TRoesner FWolman AZhong L(2019)Enabling Multiple Applications to Simultaneously Augment RealityProceedings of the 20th International Workshop on Mobile Computing Systems and Applications10.1145/3301293.3302362(81-86)Online publication date: 22-Feb-2019
https://dl.acm.org/doi/10.1145/3301293.3302362
Wójtowicz AWojciechowski RRumiński DWalczak K(2018)Securing ubiquitous AR servicesMultimedia Tools and Applications10.1007/s11042-018-5892-x77:20(26881-26899)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1007/s11042-018-5892-x
Chhetri SRashid NFaezi SFaruque MParameswaran S(2017)Security trends and advances in manufacturing systems in the era of industry 4.0Proceedings of the 36th International Conference on Computer-Aided Design10.5555/3199700.3199844(1039-1046)Online publication date: 13-Nov-2017
https://dl.acm.org/doi/10.5555/3199700.3199844
Chhetri SRashid NFaezi SFaruque M(2017)Security trends and advances in manufacturing systems in the era of industry 4.02017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)10.1109/ICCAD.2017.8203896(1039-1046)Online publication date: 13-Nov-2017
https://dl.acm.org/doi/10.1109/ICCAD.2017.8203896
Aditya PSen RDruschel PJoon Oh SBenenson RFritz MSchiele BBhattacharjee BWu TBalan RMisra AAgarwal SMascolo C(2016)I-PicProceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services10.1145/2906388.2906412(235-248)Online publication date: 20-Jun-2016
https://dl.acm.org/doi/10.1145/2906388.2906412
Lebeck KKohno TRoesner FChu DDutta P(2016)How to Safely Augment RealityProceedings of the 17th International Workshop on Mobile Computing Systems and Applications10.1145/2873587.2873595(45-50)Online publication date: 23-Feb-2016
https://dl.acm.org/doi/10.1145/2873587.2873595
McPherson RJana SShmatikov VGangemi ALeonardi SPanconesi A(2015)No Escape From RealityProceedings of the 24th International Conference on World Wide Web10.1145/2736277.2741657(743-753)Online publication date: 18-May-2015
https://dl.acm.org/doi/10.1145/2736277.2741657
Egelman SKannavara RChow RBegole BKim JInkpen KWoo W(2015)Is This Thing On?Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems10.1145/2702123.2702251(1669-1678)Online publication date: 18-Apr-2015
https://dl.acm.org/doi/10.1145/2702123.2702251
Roesner FKohno TMolnar D(2014)Security and privacy for augmented reality systemsCommunications of the ACM10.1145/2580723.258073057:4(88-96)Online publication date: 1-Apr-2014
https://dl.acm.org/doi/10.1145/2580723.2580730

View Options

View options

Media

Figures

Other

Tables

View Table of Contents