skip to main content
research-article
Open access

Robot Classification of Human Interruptibility and a Study of Its Effects

Published: 24 October 2018 Publication History

Abstract

As robots become increasingly prevalent in human environments, there will inevitably be times when the robot needs to interrupt a human to initiate an interaction. Our work introduces the first interruptibility-aware mobile-robot system, which uses social and contextual cues online to accurately determine when to interrupt a person. We evaluate multiple non-temporal and temporal models on the interruptibility classification task, and show that a variant of Conditional Random Fields (CRFs), the Latent-Dynamic CRF, is the most robust, accurate, and appropriate model for use on our system. Additionally, we evaluate different classification features and show that the observed demeanor of a person can help in interruptibility classification; but in the presence of detection noise, robust detection of object labels as a visual cue to the interruption context can improve interruptibility estimates. Finally, we deploy our system in a large-scale user study to understand the effects of interruptibility-awareness on human-task performance, robot-task performance, and on human interpretation of the robot’s social aptitude. Our results show that while participants are able to maintain task performance, even in the presence of interruptions, interruptibility-awareness improves the robot’s task performance and improves participant social perceptions of the robot.

References

[1]
Piotr D. Adamczyk and Brian P. Bailey. 2004. If not now, when? In Proceedings of the 2004 Conference on Human Factors in Computing Systems (CHI’04). ACM Press, New York, New York, 271--278.
[2]
Erik M. Altmann and J. Gregory Trafton. 2002. Memory for goals: An activation-based model. Cognitive Science 26, 1 (Jan. 2002), 39--83.
[3]
Siddhartha Banerjee and Sonia Chernova. 2017. Temporal models for robot classification of human interruptibility. In Int. Conf. on Autonomous Agents 8 Multiagent Systems. IFAAMAS, 1350--1359. http://www.aamas2017.org/proceedings/pdfs/p1350.pdf.
[4]
Siddhartha Banerjee, Andrew Silva, Karen Feigh, and Sonia Chernova. 2018. Effects of interruptibility-aware robot behavior. arXiv Preprint arXiv:1804.06383 (2018).
[5]
Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological) (1995), 289--300.
[6]
Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer-Verlag New York.
[7]
Dan Bohus and Eric Horvitz. 2009. Dialog in the open world. In Proceedings of the 2009 International Conference on Multimodal Interfaces (ICMI-MLMI’09). ACM Press, New York, New York, 31.
[8]
Leo Breiman. 2001. Random forests. Machine Learning 45, 1 (2001), 5--32.
[9]
Dražen Brščić, Tetsushi Ikeda, and Takayuki Kanda. 2017. Do you need help? A robot providing information to people who behave atypically. IEEE Transactions on Robotics 33, 2 (2017), 500--506.
[10]
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2016. Realtime multi-person 2D pose estimation using part affinity fields. arXiv Preprint arXiv:1611.08050 (2016).
[11]
Stuart K. Card, Thomas P. Moran, and Allen Newell. 1980. Computer text-editing: An information-processing analysis of a routine cognitive skill. Cognitive Psychology 12, 1 (Jan. 1980), 32--74.
[12]
Pierluigi Casale, Oriol Pujol, and Petia Radeva. 2011. Human activity recognition from accelerometer data using a wearable device. In Iberian Conference on Pattern Recognition and Image Analysis. Springer, 289--296.
[13]
Yi-Shiu Chiang, Ting-Sheng Chu, Chung Dial Lim, Tung-Yen Wu, Shih-Huan Tseng, and Li-Chen Fu. 2014. Personalizing robot behavior for interruption in social human-robot interaction. In 2014 IEEE International Workshop on Advanced Robotics and its Social Impacts. IEEE, 44--49.
[14]
Vivian Chu, Kalesha Bullard, and Andrea L. Thomaz. 2014. Multimodal real-time contingency detection for HRI. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 3327--3332.
[15]
Meng-Che Chuang, Raja Bala, Edgar A Bernal, Peter Paul, Aaron Burry, et al. 2014. Estimating gaze direction of vehicle drivers using a smartphone camera. In CVPR Workshops. 165--170.
[16]
Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning 20, 3 (01 Sept. 1995), 273--297.
[17]
Christian Dondrup, Nicola Bellotto, Ferdian Jovan, and Marc Hanheide. 2015. Real-time multisensor people tracking for human-robot spatial interaction. In Workshop on Machine Learning for Social Robotics at International Conference on Robotics and Automation (ICRA). ICRA/IEEE.
[18]
Daniel A. Epstein, Daniel Avrahami, and Jacob T. Biehl. 2016. Taking 5: Work-breaks, productivity, and opportunities for personal informatics for knowledge workers. In CHI’16. ACM Press, San Jose, CA.
[19]
Kerstin Fischer, Bianca Soto, Caroline Pantofaru, and Leila Takayama. 2014. Initiating interactions in order to get help: Effects of social framing on people’s responses to robots’ requests for assistance. In 2014 RO-MAN: The 23rd IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 999--1005.
[20]
James Fogarty, Scott E. Hudson, Christopher G. Atkeson, Daniel Avrahami, Jodi Forlizzi, Sara Kiesler, Johnny C. Lee, and Jie Yang. 2005. Predicting human interruptibility with sensors. ACM Transactions on Computer-Human Interaction 12, 1 (Mar. 2005), 119--146.
[21]
Mary Ellen Foster, Andre Gaschler, and Manuel Giuliani. 2017. Automatically classifying user engagement for dynamic multi-party human-robot interaction. International Journal of Social Robotics (July 2017).
[22]
A. Garrell, M. Villamizar, F. Moreno-Noguer, and A. Sanfeliu. 2017. Teaching robot’s proactive behavior using human assistance. International Journal of Social Robotics 9, 2 (Apr. 2017), 231--249.
[23]
T. Grundgeiger, D. Liu, P. M. Sanderson, S. Jenkins, and T. Leane. 2008. Effects of interruptions on prospective memory performance in anesthesiology. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 52, 12 (Sept. 2008), 808--812.
[24]
Tobias Grundgeiger and Penelope Sanderson. 2009. Interruptions in healthcare: Theoretical views. International Journal of Medical Informatics 78, 5 (May 2009), 293--307.
[25]
Edward Twichell Hall. 1969. The Hidden Dimension. Anchor Books.
[26]
Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (task load index): Results of empirical and theoretical research. 139--183.
[27]
Zhenyu He and Lianwen Jin. 2009. Activity recognition from acceleration data based on discrete consine transform and SVM. In IEEE International Conference on Systems, Man and Cybernetics. SMC 2009. IEEE, 5041--5044.
[28]
Geoffrey E. Hinton. 1990. Connectionist learning procedures. In Machine Learning, Volume III. Elsevier, 555--610.
[29]
Eric Horvitz and Johnson Apacible. 2003. Learning and reasoning about interruption. In Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI’03). ACM Press, New York, New York, 20.
[30]
Shamsi T. Iqbal and Brian P. Bailey. 2006. Leveraging characteristics of task structure to predict the cost of interruption. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’06). ACM Press, New York, New York, 741.
[31]
Yusuke Kato, Takayuki Kanda, and Hiroshi Ishiguro. 2015. May I help you? In Proceedings of the 10th Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI’15). ACM Press, New York, New York, 35--42.
[32]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv Preprint arXiv:1412.6980 (2014).
[33]
Ari Kolbeinsson, Peter Thorvald, and Jessica Lindblom. 2017. Coordinating the interruption of assembly workers in manufacturing. Applied Ergonomics 58 (Jan. 2017), 361--371.
[34]
John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning, ICML, Vol. 1. 282--289.
[35]
Byung Cheol Lee and Vincent G. Duffy. 2015. The effects of task interruption on human performance: A study of the systematic classification of human behavior and interruption frequency. Human Factors and Ergonomics in Manufacturing 8 Service Industries 25, 2 (Mar. 2015), 137--152.
[36]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European Conference on Computer Vision. Springer, 740--755.
[37]
Gloria Mark, Daniela Gudith, and Ulrich Klocke. 2008. The cost of interrupted work. In Proceeding of the 26th Annual CHI Conference on Human Factors in Computing Systems (CHI’08). ACM Press, New York, New York, 107.
[38]
Mark A. McDaniel and Gilles O. Einstein. 2000. Strategic and automatic processes in prospective memory retrieval: A multiprocess framework. Applied Cognitive Psychology 14, 7 (2000), S127--S144.
[39]
Daniel McFarlane and Kara Latorella. 2002. The scope and importance of human interruption in human-computer interaction design. Human-Computer Interaction 17, 1 (Mar. 2002), 1--61.
[40]
Daniel Craig McFarlane. 2002. Comparison of four primary methods for coordinating the interruption of people in human-computer interaction. Human-Computer Interaction 17, 1 (Mar. 2002), 63--139.
[41]
Yoshiro Miyata and Donald A. Norman. 1986. Psychological issues in support of multiple activities. User Centered System Design: New Perspectives on Human-Computer Interaction (1986), 265--284.
[42]
C. Mollaret, A. A. Mekonnen, F. Lerasle, I. Ferrané, J. Pinquier, B. Boudet, and P. Rumeau. 2016. A multi-modal perception based assistive robotic system for the elderly. Computer Vision and Image Understanding (Mar. 2016).
[43]
Christopher A. Monk, Deborah A. Boehm-Davis, George Mason, and J. Gregory Trafton. 2004. Recovering from interruptions: Implications for driver distraction research. Human Factors: The Journal of the Human Factors and Ergonomics Society 46, 4 (Dec. 2004), 650--663.
[44]
Louis-Philippe Morency, Ariadna Quattoni, and Trevor Darrell. 2007. Latent-dynamic discriminative models for continuous gesture recognition. In 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1--8.
[45]
Bilge Mutlu and Jodi Forlizzi. 2008. Robots in organizations. In Proceedings of the 3rd International Conference on Human Robot Interaction (HRI’08). ACM Press, New York, New York, 287.
[46]
Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10). 807--814.
[47]
Aastha Nigam and Laurel D. Riek. 2015. Social context perception for mobile robots. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 3621--3627.
[48]
Aude Oliva and Antonio Torralba. 2001. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision 42, 3 (2001), 145--175.
[49]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in python. Journal of Machine Learning Research 12 (Oct. 2011), 2825--2830.
[50]
L. R. Rabiner. 1989. A tutorial on Hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, 2 (1989), 257--286.
[51]
Joseph Redmon and Ali Farhadi. 2016. YOLO9000: Better, faster, stronger. arXiv Preprint arXiv:1612.08242 (2016).
[52]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems. 91--99.
[53]
A. Joy Rivera. 2014. A socio-technical systems approach to studying interruptions: Understanding the interrupter’s perspective. Applied Ergonomics 45, 3 (May 2014), 747--756.
[54]
Stephanie Rosenthal, Manuela M. Veloso, and Anind K. Dey. 2012. Is someone in this office available to help me?Journal of Intelligent 8 Robotic Systems 66, 1--2 (Apr. 2012), 205--221.
[55]
Penelope M. Sanderson and Tobias Grundgeiger. 2015. How do interruptions affect clinician performance in healthcare? Negotiating fidelity, control, and potential generalizability in the search for answers. International Journal of Human-Computer Studies 79 (July 2015), 85--96.
[56]
Nadine B. Sarter. 2013. Multimodal support for interruption management: Models, empirical findings, and design recommendations. Proc. IEEE 101, 9 (Sept. 2013), 2105--2112.
[57]
Satoru Satake, Takayuki Kanda, Dylan F. Glas, Michita Imai, Hiroshi Ishiguro, and Norihiro Hagita. 2009. How to approach humans? In Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction (HRI’09). ACM Press, New York, New York, 109.
[58]
Paul Saulnier, Ehud Sharlin, and Saul Greenberg. 2011. Exploring minimal nonverbal interruption in HRI. In 2011 RO-MAN. IEEE, 79--86.
[59]
Chao Shi, Masahiro Shiomi, Takayuki Kanda, Hiroshi Ishiguro, and Norihiro Hagita. 2015. Measuring communication participation to initiate conversation in human robot interaction. International Journal of Social Robotics 7, 5 (Nov. 2015), 889--910.
[60]
Elaine Schaertl Short, Mai Lee Chang, and Andrea Thomaz. 2018. Detecting contingency for HRI in open-world environments. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction (HRI’18). ACM Press, New York, New York, 425--433.
[61]
Cheri Speier, Joseph S. Valacich, and Iris Vessey. 1997. The effects of task interruption and information presentation on individual decision making. In Proceedings of the 18th International Conference on Information Systems. Association for Information Systems, 21--36. http://dl.acm.org/citation.cfm?id=353080
[62]
Hermann Stern, Viktoria Pammer, and Stefanie N. Lindstaedt. 2011. A preliminary study on interruptibility detection based on location and calendar information. Proc. CoSDEO 11 (2011).
[63]
Sy Bor Wang, A. Quattoni, L.-P. Morency, D. Demirdjian, and T. Darrell. 2006. Hidden conditional random fields for gesture recognition. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR’06), Vol. 2. IEEE, 1521--1527.
[64]
Edward R. Sykes. 2014. A cloud-based interaction management system architecture for mobile devices. Procedia Computer Science 34 (2014), 625--632.
[65]
Greg Trafton, Laura Hiatt, Anthony Harrison, Frank Tanborello, Sangeet Khemlani, and Alan Schultz. 2013. ACT-R/E: An embodied cognitive architecture for human-robot interaction. Journal of Human-Robot Interaction 2, 1 (Mar. 2013), 30--55.
[66]
J. Gregory Trafton, Allison Jacobs, and Anthony M. Harrison. 2012. Building and verifying a predictive model of interruption resumption. Proc. IEEE 100, 3 (Mar. 2012), 648--659.
[67]
Liam D. Turner, Stuart M. Allen, and Roger M. Whitaker. 2015. Interruptibility prediction for ubiquitous systems. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’15). ACM Press, New York, New York, 801--812.
[68]
Douglas L. Vail, Manuela M. Veloso, and John D. Lafferty. 2007. Conditional random fields for activity recognition. In Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS’07). ACM Press, New York, New York, 1.
[69]
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters 23, 10 (2016), 1499--1503.
[70]
Min-Ling Zhang and Zhi-Hua Zhou. 2007. ML-KNN: A lazy learning approach to multi-label learning. Pattern Recognition 40, 7 (2007), 2038--2048.

Cited By

View all

Index Terms

  1. Robot Classification of Human Interruptibility and a Study of Its Effects

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Human-Robot Interaction
    ACM Transactions on Human-Robot Interaction  Volume 7, Issue 2
    Special Issue on Artificial Intelligence and Human-Robot Interaction
    July 2018
    109 pages
    EISSN:2573-9522
    DOI:10.1145/3284682
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 24 October 2018
    Accepted: 01 August 2018
    Received: 01 April 2018
    Published in THRI Volume 7, Issue 2

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Interruptibility
    2. conditional random fields

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    • Early Career Faculty
    • NASA's Space Technology Research Grants Program

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)240
    • Downloads (Last 6 weeks)37
    Reflects downloads up to 28 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Full Access

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media