research-article

OverLay: Practical Mobile Augmented Reality

Authors:

Justin Manweiler,

Romit Roy ChoudhuryAuthors Info & Claims

MobiSys '15: Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services

Pages 331 - 344

https://doi.org/10.1145/2742647.2742666

Published: 18 May 2015 Publication History

Abstract

The idea of augmented reality - the ability to look at a physical object through a camera and view annotations about the object - is certainly not new. Yet, this apparently feasible vision has not yet materialized into a precise, fast, and comprehensively usable system. This paper asks: What does it take to enable augmented reality (AR) on smartphones today? To build a ready-to-use mobile AR system, we adopt a top-down approach cutting across smartphone sensing, computer vision, cloud offloading, and linear optimization. Our core contribution is in a novel location-free geometric representation of the environment - from smartphone sensors - and using this geometry to prune down the visual search space. Metrics of success include both accuracy and latency of object identification, coupled with the ease of use and scalability in uncontrolled environments. Our converged system, OverLay, is currently deployed in the engineering building and open for use to regular public; ongoing work is focussed on campus-wide deployment to serve as a "historical tour guide" of UIUC. Performance results and user responses thus far have been promising, to say the least.

References

[1]

Amazon fire phone. https://developer.amazon.com/public/solutions/devices/fire-phone.

[2]

Google goggles. https://play.google.com/store/apps/details?id=com.google.android.apps.unveil&hl=en.

[3]

Illinois distributed museum. http://distributedmuseum.blogspot.com/.

[4]

Instagram hyperlapse. http://hyperlapse.instagram.com/.

[5]

Mobile augmented reality, video demonstration. http://synrg.csl.illinois.edu/projects/MobileAR/.

[6]

Project tango. https://www.google.com/atap/projecttango/.

[7]

Qualcomm vuforia. https://www.qualcomm.com/products/vuforia.

[8]

Videoguide, antoni gaudi modernist museum in barcelona. http://www.casabatllo.es/en/visit/videoguide.

[9]

Wikitude. http://www.wikitude.com/.

[10]

S. Agarwal, Y. Furukawa, N. Snavely, I. Simon, B. Curless, S. M. Seitz, and R. Szeliski. Building rome in a day. Communications of the ACM, 54(10):105--112, 2011.

Digital Library

[11]

C. Arth and D. Schmalstieg. Challenges of large-scale augmented reality on smartphones. Graz University of Technology, Graz, pages 1--4, 2011.

[12]

A. Ashok, M. Gruteser, N. Mandayam, J. Silva, M. Varga, and K. Dana. Challenge: mobile optical networks through visual mimo. In MobiCom, pages 105--112. ACM, 2010.

Digital Library

[13]

R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre. Recent advances in augmented reality. Computer Graphics and Applications, IEEE, 21(6):34--47, 2001.

Digital Library

[14]

H. Bay, T. Tuytelaars, and L. Van Gool. Surf: Speeded up robust features. In ECCV, pages 404--417. Springer, 2006.

Digital Library

[15]

D. M. Chen, S. S. Tsai, R. Vedantham, R. Grzeszczuk, and B. Girod. Streaming mobile augmented reality on mobile phones. In ISMAR, pages 181--182. IEEE, 2009.

Digital Library

[16]

E. Cuervo, A. Balasubramanian, D.-k. Cho, A. Wolman, S. Saroiu, R. Chandra, and P. Bahl. Maui: making smartphones last longer with code offload. In MobiSys, pages 49--62. ACM, 2010.

Digital Library

[17]

S. Feiner, B. MacIntyre, T. Höllerer, and A. Webster. A touring machine: Prototyping 3d mobile augmented reality systems for exploring the urban environment. Personal Technologies, 1(4):208--217, 1997.

[18]

P. Föckler, T. Zeidler, B. Brombach, E. Bruns, and O. Bimber. Phoneguide: museum guidance supported by on-device object recognition on mobile phones. In Proceedings of the 4th international conference on Mobile and ubiquitous multimedia, pages 3--10. ACM, 2005.

Digital Library

[19]

S. Gammeter, A. Gassmann, L. Bossard, T. Quack, and L. Van Gool. Server-side object recognition and client-side object tracking for mobile augmented reality. In CVPR Workshops, pages 1--8. IEEE, 2010.

[20]

B. Girod and C. et al. Mobile visual search. Signal Processing Magazine, IEEE, 28(4):61--76, 2011.

[21]

A. Henrysson and M. Ollila. Umar: Ubiquitous mobile augmented reality. In Proceedings of the 3rd international conference on Mobile and ubiquitous multimedia, pages 41--45. ACM, 2004.

Digital Library

[22]

M. Hwangbo, J.-S. Kim, and T. Kanade. Inertial-aided klt feature tracking for a moving camera. In Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on, pages 1909--1916. IEEE, 2009.

Digital Library

[23]

S. Izadi et al. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In UIST. ACM, 2011.

Digital Library

[24]

P. Jain, J. Manweiler, A. Acharya, and K. Beaty. Focus: clustering crowdsourced videos by line-of-sight. In SenSys, page 8. ACM, 2013.

Digital Library

[25]

G. Klein and D. Murray. Parallel tracking and mapping on a camera phone. In ISMAR, pages 83--86. IEEE, 2009.

Digital Library

[26]

J. Kopf, M. F. Cohen, and R. Szeliski. First-person hyper-lapse videos. TOG, 33(4):78, 2014.

Digital Library

[27]

R. LiKamWa, B. Priyantha, M. Philipose, L. Zhong, and P. Bahl. Energy characterization and optimization of image sensing toward continuous mobile vision. In MobiSys, pages 69--82. ACM, 2013.

Digital Library

[28]

J. G. Manweiler, P. Jain, and R. Roy Choudhury. Satellites in our pockets: an object positioning system using smartphones. In MobiSys, pages 211--224. ACM, 2012.

Digital Library

[29]

P. Mistry and P. Maes. Sixthsense: a wearable gestural interface. In ACM SIGGRAPH ASIA 2009 Sketches, page 11. ACM, 2009.

Digital Library

[30]

M. Muja and D. G. Lowe. Fast approximate nearest neighbors with automatic algorithm configuration. In VISAPP (1), pages 331--340, 2009.

[31]

H. E. Pence. Smartphones, smart objects, and augmented reality. The Reference Librarian, 52(1-2):136--145, 2010.

[32]

W. Piekarski and B. Thomas. Arquake: the outdoor augmented reality gaming system. Communications of the ACM, 45(1):36--38, 2002.

Digital Library

[33]

C. Qin, X. Bao, R. Roy Choudhury, and S. Nelakuditi. Tagsense: a smartphone-based approach to automatic image tagging. In MobiSys, pages 1--14. ACM, 2011.

Digital Library

[34]

J. Rekimoto and Y. Ayatsuka. Cybercode: designing augmented reality environments with visual tags. In Proceedings of DARE 2000 on Designing augmented reality environments, pages 1--10. ACM, 2000.

Digital Library

[35]

J. P. Rolland, L. Davis, and Y. Baillot. A survey of tracking technology for virtual environments. Fundamentals of wearable computers and augmented reality, 1:67--112, 2001.

[36]

M. Satyanarayanan, P. Bahl, R. Caceres, and N. Davies. The case for vm-based cloudlets in mobile computing. Pervasive Computing, IEEE, 8(4):14--23, 2009.

Digital Library

[37]

F. Schaffalitzky and A. Zisserman. Multi-view matching for unordered image sets, or "how do i organize my holiday snaps?". In Computer Vision - ECCV 2002, pages 414--431. Springer, 2002.

Digital Library

[38]

G. Takacs and C. et al. Outdoors augmented reality on mobile phone using loxel-based visual feature organization. In Proceedings of the 1st ACM international conference on Multimedia information retrieval, pages 427--434. ACM, 2008.

Digital Library

[39]

R. Tenmoku, M. Kanbara, and N. Yokoya. A wearable augmented reality system using positioning infrastructures and a pedometer. In ISWC, pages 110--110. IEEE Computer Society, 2003.

Digital Library

[40]

D. Wagner, G. Reitmayr, A. Mulloni, T. Drummond, and D. Schmalstieg. Pose tracking from natural features on mobile phones. In ISMAR, pages 125--134. IEEE Computer Society, 2008.

Digital Library

[41]

D. Wagner and D. Schmalstieg. First steps towards handheld augmented reality. In ISWC, pages 127--127. IEEE Computer Society, 2003.

Digital Library

[42]

D. Wagner and D. Schmalstieg. Making augmented reality practical on mobile phones, part 1. Computer Graphics and Applications, IEEE, 29(3):12--15, 2009.

Digital Library

[43]

T. Yan, D. Ganesan, and R. Manmatha. Distributed image search in camera sensor networks. In SenSys, pages 155--168. ACM, 2008.

Digital Library

[44]

S. You, U. Neumann, and R. Azuma. Hybrid inertial and vision tracking for augmented reality registration. In Virtual Reality, 1999. Proceedings., IEEE, pages 260--267. IEEE, 1999.

Digital Library

[45]

F. Zhou, H. B.-L. Duh, and M. Billinghurst. Trends in augmented reality tracking, interaction and display: A review of ten years of ismar. In ISMAR, pages 193--202. IEEE Computer Society, 2008.

Digital Library

Cited By

Jingru ZYahaya WSanmugam M(2024)A Concise Guide of AR Features, Types, Key Technologies, and the Dynamic Applications LandscapeIntegrating Cutting-Edge Technology Into the Classroom10.4018/979-8-3693-3124-8.ch001(1-14)Online publication date: 8-Mar-2024
https://doi.org/10.4018/979-8-3693-3124-8.ch001
Fang CLiu SZhou ZGuo BTang JMa KYu ZShu YLiu JTan RHe YChen J(2024)AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile EnvironmentsProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699339(295-308)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699339
Zhang ZJiang HSingh H(2024)NeuFlow: Real-time, High-accuracy Optical Flow Estimation on Robots Using Edge Devices2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS58592.2024.10802353(5048-5055)Online publication date: 14-Oct-2024
https://doi.org/10.1109/IROS58592.2024.10802353
Show More Cited By

Index Terms

OverLay: Practical Mobile Augmented Reality
1. Information systems
  1. Information retrieval
  2. Information storage systems

Recommendations

Development of an Inexpensive Augmented Reality (AR) Headset
CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems

We outline our work in developing an Augmented Reality (AR) headset with low purchase and maintenance costs. Similar to Google Cardboard, the headset uses a smartphone to provide the compute power, connectivity and display. Unlike Google Cardboard, our ...
Human or machine perspective? throwing "light" on optical see-through headsets in museums
UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct

Augmented Reality (AR) systems, particularly headsets with see-through displays, are increasingly available and particularly interesting in the museum field where they can support novel navigation through the collection. However, Optical See-Through (...
Reviving Aggersborg: Conveying lost heritage sites through indirect augmented reality
VRIC '17: Proceedings of the Virtual Reality International Conference - Laval Virtual 2017

This paper propose a simple and user friendly method for aligning an Indirect Augmented Reality application with the real world. Augmented Reality is often considered an innovative platform for conveying cultural heritage, as well as for many other ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MobiSys '15: Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services

May 2015

516 pages

ISBN:9781450334945

DOI:10.1145/2742647

General Chairs:
Gaetano Borriello
University of Washington, USA
,
Giovanni Pau
UPMC-LIP6, France / UCLA, USA
,
Program Chairs:
Marco Gruteser
Rutgers University, USA
,
Jason Hong
Carnegie Mellon University, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

In-Cooperation

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 May 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF
Google
Intel

Conference

MobiSys'15

Sponsor:

SIGMOBILE

MobiSys'15: The 13th Annual International Conference on Mobile Systems, Applications, and Services

May 18 - 22, 2015

Florence, Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

102
Total Citations
View Citations
1,934
Total Downloads

Downloads (Last 12 months)120
Downloads (Last 6 weeks)15

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Jingru ZYahaya WSanmugam M(2024)A Concise Guide of AR Features, Types, Key Technologies, and the Dynamic Applications LandscapeIntegrating Cutting-Edge Technology Into the Classroom10.4018/979-8-3693-3124-8.ch001(1-14)Online publication date: 8-Mar-2024
https://doi.org/10.4018/979-8-3693-3124-8.ch001
Fang CLiu SZhou ZGuo BTang JMa KYu ZShu YLiu JTan RHe YChen J(2024)AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile EnvironmentsProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699339(295-308)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699339
Zhang ZJiang HSingh H(2024)NeuFlow: Real-time, High-accuracy Optical Flow Estimation on Robots Using Edge Devices2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS58592.2024.10802353(5048-5055)Online publication date: 14-Oct-2024
https://doi.org/10.1109/IROS58592.2024.10802353
Liu KWu NHan BMontpetit MLeivadeas AUhlig SJaved M(2023)Demystifying Web-based Mobile Extended Reality Accelerated by WebAssemblyProceedings of the 2023 ACM on Internet Measurement Conference10.1145/3618257.3624833(145-153)Online publication date: 24-Oct-2023
https://dl.acm.org/doi/10.1145/3618257.3624833
Tsai YLuo JWang YChen JZimmermann RShirmohammadi SGriwodz CHefeeda MWang M(2023)The World is Too Big to Download: 3D Model Retrieval for World-Scale Augmented RealityProceedings of the 14th Conference on ACM Multimedia Systems10.1145/3587819.3590970(14-26)Online publication date: 7-Jun-2023
https://dl.acm.org/doi/10.1145/3587819.3590970
Ayala-Romero JGarcia-Saavedra ACosta-Perez XIosifidis G(2023)EdgeBOL: A Bayesian Learning Approach for the Joint Orchestration of vRANs and Mobile Edge AIIEEE/ACM Transactions on Networking10.1109/TNET.2023.326898131:6(2978-2993)Online publication date: Dec-2023
https://doi.org/10.1109/TNET.2023.3268981
Didar NBrocanelli M(2023)eAR: An Edge-Assisted and Energy-Efficient Mobile Augmented Reality FrameworkIEEE Transactions on Mobile Computing10.1109/TMC.2022.314487922:7(3898-3909)Online publication date: 1-Jul-2023
https://doi.org/10.1109/TMC.2022.3144879
Chi GXu JZhang JZhang QMa QYang Z(2023)Locate, Tell, and Guide: Enabling Public Cameras to Navigate the PublicIEEE Transactions on Mobile Computing10.1109/TMC.2021.309272522:2(1010-1024)Online publication date: 1-Feb-2023
https://doi.org/10.1109/TMC.2021.3092725
Deng CFang XWang X(2023)UAV-Enabled Mobile-Edge Computing for AI Applications: Joint Model Decision, Resource Allocation, and Trajectory OptimizationIEEE Internet of Things Journal10.1109/JIOT.2022.315161910:7(5662-5675)Online publication date: 1-Apr-2023
https://doi.org/10.1109/JIOT.2022.3151619
Cai XYang ZDong LMa QMiao XLiu Z(2023)DiTing: Edge Assisted Real-time ID-aware Visual Interaction for Multi-user Augmented Reality2022 IEEE 28th International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS56603.2022.00101(737-744)Online publication date: Jan-2023
https://doi.org/10.1109/ICPADS56603.2022.00101
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

EPUB

View this article in ePub.

Media

Figures

Other

Tables

View Table of Contents