skip to main content
10.1145/3126594.3126665acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

Reflector: Distance-Independent, Private Pointing on a Reflective Screen

Published: 20 October 2017 Publication History

Abstract

Reflector is a novel direct pointing method that utilizes hidden design space on reflective screens. By aligning a part of the user's onscreen reflection with objects rendered on the screen, Reflector enables (1) distance-independent and (2) private pointing on commodity screens. Reflector can be implemented easily in both desktop and mobile conditions through a single camera installed at the edge of the screen. Reflector's pointing performance was compared to today's major direct input devices: eye trackers and touchscreens. We demonstrate that Reflector allows the user to point more reliably, regardless of distance from the screen, compared to an eye tracker. Further, due to the private nature of an onscreen reflection, Reflector shows a shoulder surfing success rate 20 times lower than that of touchscreens for the task of entering a 4-digit PIN.

Supplementary Material

ZIP File (uistf4922-file4.zip)
suppl.mov (uistf4922-file3.mp4)
Supplemental video

References

[1]
Pär-Anders Albinsson and Shumin Zhai. 2003. High precision touch screen interaction. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 105--112.
[2]
Fraser Anderson, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2013. YouMove: enhancing movement training with an augmented reality mirror. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 311--320.
[3]
Amartya Banerjee, Jesse Burstyn, Audrey Girouard, and Roel Vertegaal. 2011. Pointable: an in-air pointing technique to manipulate out-of-reach targets on tabletops. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces. ACM, 11--20.
[4]
Florian Block, Daniel Wigdor, Brenda Caldwell Phillips, Michael S Horn, and Chia Shen. 2012. FlowBlocks: a multi-touch ui for crowd interaction. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 497--508.
[5]
Gary Bradski and Adrian Kaehler. 2008. Learning OpenCV: Computer vision with the OpenCV library. " O'Reilly Media, Inc.".
[6]
Kjell Brunnström, Börje Andrén, Zacharias Konstantinides, and Lukas Nordström. 2007. Visual ergonomic aspects of computer displays: glossy screens and angular dependence. Proc. of SPIE-IS&T Human Vision and Electronic Imaging XII 6492 (2007).
[7]
Kjell Brunnström, Katarina Josefsson, and Börje Andrén. 2008. The effects of glossy screens on the acceptance of flat-panel displays. Journal of the Society for Information Display 16, 10 (2008), 1041--1049.
[8]
Marcus Carter, Joshua Newn, Eduardo Velloso, and Frank Vetere. 2015. Remote gaze and gesture tracking on the Microsoft Kinect: Investigating the role of feedback. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction. ACM, 167--176.
[9]
Géry Casiez, Nicolas Roussel, and Daniel Vogel. 2012. 1C filter: a simple speed-based low-pass filter for noisy input in interactive systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2527--2530.
[10]
Ishan Chatterjee, Robert Xiao, and Chris Harrison. 2015. Gaze+ Gesture: Expressive, Precise and Targeted Free-Space Interactions. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM, 131--138.
[11]
Michael Chau and Margrit Betke. 2005. Real time eye tracking and blink detection with usb cameras.
[12]
Alexander De Luca, Marian Harbach, Emanuel von Zezschwitz, Max-Emanuel Maurer, Bernhard Ewald Slawik, Heinrich Hussmann, and Matthew Smith. 2014. Now you see me, now you don't: protecting smartphone authentication from shoulder surfers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2937--2946.
[13]
Alexander De Luca, Emanuel Von Zezschwitz, and Heinrich Hußmann. 2009. Vibrapass: secure authentication based on shared lies. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 913--916.
[14]
Alexander De Luca, Emanuel Von Zezschwitz, Ngo Dieu Huong Nguyen, Max-Emanuel Maurer, Elisa Rubegni, Marcello Paolo Scipioni, and Marc Langheinrich. 2013. Back-of-device authentication on smartphones. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2389--2398.
[15]
Paul Dietz and Darren Leigh. 2001. DiamondTouch: a multi-user touch technology. In Proceedings of the 14th annual ACM symposium on User interface software and technology. ACM, 219--226.
[16]
Sarah A Douglas, Arthur E Kirkpatrick, and I Scott MacKenzie. 1999. Testing pointing device performance and user assessment with the ISO 9241, Part 9 standard. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 215--222.
[17]
Augusto Esteves, Eduardo Velloso, Andreas Bulling, and Hans Gellersen. 2015. Orbits: Gaze interaction for smart watches using smooth pursuit eye movements. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 457--466.
[18]
Anna Maria Feit, Shane Williams, Arturo Toledo, Ann Paradiso, Harish S. Kulkarni, Shaun Kane, and Meredith Ringel Morris. 2017. Toward Everyday Gaze Input: Accuracy and Precision of Eye Tracking and Implications for Design. (May 2017).
[19]
Alexandre François, Elaine Kang, and Umberto Malesci. 2002. A handheld virtual mirror. In ACM SIGGRAPH 2002 conference abstracts and applications. ACM, 140--140.
[20]
Alexandre RJ François and E-YE Kang. 2003. A handheld mirror simulation. In Multimedia and Expo, 2003. ICME'03. Proceedings. 2003 International Conference on, Vol. 2. IEEE, II--745.
[21]
Kaori Fujinami, Fahim Kawsar, and Tatsuo Nakajima. 2005. AwareMirror: a personalized display using a mirror. In International Conference on Pervasive Computing. Springer, 315--332.
[22]
Mayank Goel, Jacob Wobbrock, and Shwetak Patel. 2012. GripSense: using built-in sensors to detect hand posture and pressure on commodity mobile phones. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 545--554.
[23]
Alix Goguey, Daniel Vogel, Fanny Chevalier, Thomas Pietrzak, Nicolas Roussel, and Géry Casiez. 2017. Leveraging finger identification to integrate multi-touch command selection and parameter manipulation. International Journal of Human-Computer Studies 99 (2017), 21--36.
[24]
Martin Hachet, Benoit Bossavit, Aurélie Cohé, and Jean-Baptiste de la Rivière. 2011. Toucheo: multitouch and stereo combined in a seamless workspace. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 587--592.
[25]
Jaehyun Han and Geehyuk Lee. 2015. Push-push: A drag-like operation overlapped with a page transition operation on touch interfaces. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. acm, 313--322.
[26]
Jefferson Y Han. 2005. Low-cost multi-touch sensing through frustrated total internal reflection. In Proceedings of the 18th annual ACM symposium on User interface software and technology. ACM, 115--118.
[27]
Faizan Haque, Mathieu Nancel, and Daniel Vogel. 2015. Myopoint: Pointing and clicking using forearm mounted electromyography and inertial motion sensors. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3653--3656.
[28]
Chris Harrison and Scott Hudson. 2012. Using shear as a supplemental two-dimensional input channel for rich touchscreen interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3149--3152.
[29]
Seongkook Heo, Jiseong Gu, and Geehyuk Lee. 2014. Expanding touch input vocabulary by using consecutive distant taps. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2597--2606.
[30]
Luke Hespanhol and Martin Tomitsch. 2012. Designing for collective participation with media installations in public spaces. In Proceedings of the 4th Media Architecture Biennale Conference: Participation. ACM, 33--42.
[31]
Otmar Hilliges, David Kim, Shahram Izadi, Malte Weiss, and Andrew Wilson. 2012. HoloDesk: direct 3d interactions with a situated see-through display. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2421--2430.
[32]
Ken Hinckley and Hyunyoung Song. 2011. Sensor synaesthesia: touch in motion, and motion in touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 801--810.
[33]
Thomas E Hutchinson, K Preston White, Worthy N Martin, Kelly C Reichert, and Lisa A Frey. 1989. Human-computer interaction using eye-gaze input. IEEE Transactions on systems, man, and cybernetics 19, 6 (1989), 1527--1534.
[34]
Petra Isenberg, Sheelegh Carpendale, Anastasia Bezerianos, Nathalie Henry, and Jean-Daniel Fekete. 2009. Coconuttrix: Collaborative retrofitting for information visualization. IEEE Computer Graphics and Applications 29, 5 (2009), 44--57.
[35]
Robert JK Jacob. 1995. Eye tracking in advanced interface design. Virtual environments and advanced interface design (1995), 258--288.
[36]
Mikkel R Jakobsen and Kasper Hornbæk. 2016. Negotiating for Space?: Collaborative Work Using a Wall Display with Mouse and Touch Input. In CHI. 2050--2061.
[37]
Ricardo Jota, Miguel A Nacenta, Joaquim A Jorge, Sheelagh Carpendale, and Saul Greenberg. 2010. A comparison of ray pointing techniques for very large displays. In Proceedings of Graphics Interface 2010. Canadian Information Processing Society, 269--276.
[38]
Vahid Kazemi and Josephine Sullivan. 2014. One millisecond face alignment with an ensemble of regression trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1867--1874.
[39]
Mohamed Khamis, Florian Alt, Mariam Hassib, Emanuel von Zezschwitz, Regina Hasholzner, and Andreas Bulling. 2016. Gazetouchpass: Multimodal authentication using gaze and touch on mobile devices. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2156--2164.
[40]
Christian Lander, Sven Gehring, Antonio Krüger, Sebastian Boring, and Andreas Bulling. 2015. Gazeprojector: Accurate gaze estimation and seamless gaze interaction across multiple displays. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 395--404.
[41]
Byungjoo Lee and Hyunwoo Bang. 2013. A kinematic analysis of directional effects on mouse control. Ergonomics 56, 11 (2013), 1754--1765.
[42]
Byungjoo Lee and Hyunwoo Bang. 2015. A mouse with two optical sensors that eliminates coordinate disturbance during skilled strokes. Human--Computer Interaction 30, 2 (2015), 122--155.
[43]
Byungjoo Lee, Mathieu Nancel, and Antti Oulasvirta. 2016. AutoGain: Adapting Gain Functions by Optimizing Submovement Efficiency. arXiv preprint arXiv:1611.08154 (2016).
[44]
Byungjoo Lee and Antti Oulasvirta. 2016. Modelling error rates in temporal pointing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1857--1868.
[45]
Wing Ho Andy Li and Hongbo Fu. 2012. Augmented reflection of reality. In ACM SIGGRAPH 2012 Emerging Technologies. ACM, 3.
[46]
Chengdong Lu and Douglas Frye. 1992. Mastering the machine: A comparison of the mouse and touch screen for children's use of computers. Computer assisted learning (1992), 417--427.
[47]
Yuexing Luo and Daniel Vogel. 2015. Pin-and-cross: A unimanual multitouch technique combining static touches with crossing selection. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 323--332.
[48]
Päivi Majaranta, I Scott MacKenzie, Anne Aula, and Kari-Jouko Räihä. 2006. Effects of feedback and dwell time on eye typing speed and accuracy. Universal Access in the Information Society 5, 2 (2006), 199--208.
[49]
Päivi Majaranta and Kari-Jouko Räihä. 2002. Twenty years of eye typing: systems and design issues. In Proceedings of the 2002 symposium on Eye tracking research & applications. ACM, 15--22.
[50]
Diego Martinez Plasencia, Florent Berthaut, Abhijit Karnik, and Sriram Subramanian. 2014. Through the combining glass. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 341--350.
[51]
Luana Micallef, Gregorio Palmas, Antti Oulasvirta, and Tino Weinkauf. 2017. Towards Perceptual Optimization of the Visual Design of Scatterplots. IEEE Transactions on Visualization and Computer Graphics 23, 6 (2017), 1588--1599.
[52]
Carlos H Morimoto and Marcio RM Mimica. 2005. Eye gaze tracking techniques for interactive applications. Computer vision and image understanding 98, 1 (2005), 4--24.
[53]
Jurriaan D Mulder and BR Boscker. 2004. A modular system for collaborative desktop vr/ar with a shared workspace. In Virtual Reality, 2004. Proceedings. IEEE. IEEE, 75--280.
[54]
Mathieu Nancel, Emmanuel Pietriga, Olivier Chapuis, and Michel Beaudouin-Lafon. 2015. Mid-air pointing on ultra-walls. ACM Transactions on Computer-Human Interaction (TOCHI) 22, 5 (2015), 21.
[55]
Antti Oulasvirta. 2017. User interface design with combinatorial optimization. Computer 50, 1 (2017), 40--47.
[56]
Volker Roth, Kai Richter, and Rene Freidinger. 2004. A PIN-entry method resilient against shoulder surfing. In Proceedings of the 11th ACM conference on Computer and communications security. ACM, 236--245.
[57]
Hideaki Sato, Itaru Kitahara, and Yuichi Ohta. 2009. MR-mirror: a complex of real and virtual mirrors. In International Conference on Virtual and Mixed Reality. Springer, 482--491.
[58]
Florian Schaub, Ruben Deyhle, and Michael Weber. 2012. Password entry usability and shoulder surfing susceptibility on different smartphone platforms. In Proceedings of the 11th international conference on mobile and ubiquitous multimedia. ACM, 13.
[59]
Christopher Schmandt. 1983. Spatial input/display correspondence in a stereoscopic computer graphic work station. In ACM SIGGRAPH Computer Graphics, Vol. 17. ACM, 253--261.
[60]
Ju Shen, SC S Cheung, and Jian Zhao. 2012. Virtual mirror by fusing multiple RGB-D cameras. In Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific. IEEE, 1--9.
[61]
Ju Shen, Po-Chang Su, Sen-ching Samson Cheung, and Jian Zhao. 2013. Virtual mirror rendering with stationary rgb-d cameras and stored 3-d background. IEEE Transactions on Image Processing 22, 9 (2013), 3433--3448.
[62]
Matthias Straka, Stefan Hauswiesner, Matthias Rüther, and Horst Bischof. 2011. A free-viewpoint virtual mirror with marker-less user interaction. In Scandinavian Conference on Image Analysis. Springer, 635--645.
[63]
Yusuke Sugano, Yasuyuki Matsushita, and Yoichi Sato. 2013. Appearance-based gaze estimation using visual saliency. IEEE transactions on pattern analysis and machine intelligence 35, 2 (2013), 329--341.
[64]
Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016. Sketchplore: Sketch and explore with a layout optimiser. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, 543--555.
[65]
Keita Ushida, Yu Tanaka, Takeshi Naemura, and Hiroshi Harashima. 2002. i-mirror: An Interaction/Information Environment Based on a Mirror Metaphor Aiming to Install into Our Life Space. In Proceedings of the 12th International Conference on Artificial Reality and Telexistence (ICAT2002). 113--118.
[66]
Mélodie Vidal, Andreas Bulling, and Hans Gellersen. 2013. Pursuits: spontaneous interaction with displays based on smooth pursuit eye movement and moving targets. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. ACM, 439--448.
[67]
Daniel Vogel and Ravin Balakrishnan. 2004. Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. In Proceedings of the 17th annual ACM symposium on User interface software and technology. ACM, 137--146.
[68]
Daniel Vogel and Ravin Balakrishnan. 2005. Distant freehand pointing and clicking on very large, high resolution displays. In Proceedings of the 18th annual ACM symposium on User interface software and technology. ACM, 33--42.
[69]
Emanuel Von Zezschwitz, Alexander De Luca, Bruno Brunkow, and Heinrich Hussmann. 2015. Swipin: Fast and secure pin-entry on smartphones. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1403--1406.
[70]
Andrew D Wilson. 2004. TouchLight: an imaging touch screen and display for gesture-based interaction. In Proceedings of the 6th international conference on Multimodal interfaces. ACM, 69--76.
[71]
Xuan Zhang and I Scott MacKenzie. 2007. Evaluating eye tracking with ISO 9241-part 9. In International Conference on Human-Computer Interaction. Springer, 779--788.

Cited By

View all

Index Terms

  1. Reflector: Distance-Independent, Private Pointing on a Reflective Screen

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology
    October 2017
    870 pages
    ISBN:9781450349819
    DOI:10.1145/3126594
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 20 October 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. direct input
    2. distance independence
    3. eye tracking
    4. onscreen reflection
    5. private pointing
    6. shoulder surfing
    7. touch screen

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    UIST '17

    Acceptance Rates

    UIST '17 Paper Acceptance Rate 73 of 324 submissions, 23%;
    Overall Acceptance Rate 561 of 2,567 submissions, 22%

    Upcoming Conference

    UIST '25
    The 38th Annual ACM Symposium on User Interface Software and Technology
    September 28 - October 1, 2025
    Busan , Republic of Korea

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)18
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 03 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)GazeCast: Using Mobile Devices to Allow Gaze-based Interaction on Public DisplaysProceedings of the 2023 Symposium on Eye Tracking Research and Applications10.1145/3588015.3589663(1-8)Online publication date: 30-May-2023
    • (2022)Understanding and Creating Spatial Interactions with Distant Displays Enabled by Unmodified Off-The-Shelf SmartphonesMultimodal Technologies and Interaction10.3390/mti61000946:10(94)Online publication date: 19-Oct-2022
    • (2019)Privacy by Design and Cybersecurity for Safe, Effective and Reliable Home Health Care for Aging in PlaceInnovation in Information Systems and Technologies to Support Learning Research10.1007/978-3-030-36778-7_49(442-450)Online publication date: 1-Dec-2019
    • (2018)The past, present, and future of gaze-enabled handheld mobile devicesProceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services10.1145/3229434.3229452(1-17)Online publication date: 3-Sep-2018

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media