skip to main content
10.1145/3313831.3376131acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Public Access

If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills

Published: 23 April 2020 Publication History

Abstract

Interview chatbots engage users in a text-based conversation to draw out their views and opinions. It is, however, challenging to build effective interview chatbots that can handle user free-text responses to open-ended questions and deliver engaging user experience. As the first step, we are investigating the feasibility and effectiveness of using publicly available, practical AI technologies to build effective interview chatbots. To demonstrate feasibility, we built a prototype scoped to enable interview chatbots with a subset of active listening skills-the abilities to comprehend a user's input and respond properly. To evaluate the effectiveness of our prototype, we compared the performance of interview chatbots with or without active listening skills on four common interview topics in a live evaluation with 206 users. Our work presents practical design implications for building effective interview chatbots, hybrid chatbot platforms, and empathetic chatbots beyond interview tasks.

Supplementary Material

ZIP File (paper004aux.zip)
The ZIP file contains a short demo video of an interview chatbot created by our prototype.
MP4 File (paper004pv.mp4)
Preview video
MP4 File (a4-xiao-presentation.mp4)

References

[1]
Jun Araki, Dheeraj Rajagopal, Sreecharan Sankaranarayanan, Susan Holm, Yukari Yamakawa, and Teruko Mitamura. 2016. Generating questions and multiple-choice answers using semantic analysis of texts. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. 1125--1136.
[2]
Christine Bauer, Kathrin Figl, and Renate MotschnigPitrik. 2010. Introducing'Active Listening'to Instant Messaging and E-mail: Benefits and Limitations. IADIS International Journal on WWW/Internet 7, 2 (2010), 1--17.
[3]
Christine Bauer and Kathrin Figl. 2008. Active listening" in written online communication-a case study in a course on soft skills for computer scientists. In 2008 38th Annual Frontiers in Education Conference. IEEE, F2C--1.
[4]
Timothy Bickmore. 2010. Relational agents for chronic disease self management. Health Informatics: A Patient-Centered Approach to Diabetes (2010), 181--204.
[5]
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research 3, Jan (2003), 993--1022.
[6]
Antoine Bordes, Y-Lan Boureau, and Jason Weston, 2017. Learning end-to-end goal-oriented dialog. ICLR' 2017.
[7]
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, and others. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018).
[8]
Chatfuel. 2019. Retrieved from https://chatfuel.com/
[9]
Bert Decker., 1989. How to communicate effectively, Page, London, UK.
[10]
David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, and others. 2014. SimSensei Kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. International Foundation for Autonomous Agents and Multiagent Systems, 1061--1068.
[11]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 4171-- 4186.
[12]
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to Ask: Neural Question Generation for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1342--1352.
[13]
Dialogflow. 2019. Retrieved from https://dialogflow.com/
[14]
Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research 22 (2004), 457--479.
[15]
Jerome Friedman, Trevor Hastie, and Robert Tibshirani, 2001. The Elements of Statistical Learning (Vol. 1, No. 10). New York, NY, USA:: Springer series in statistics.
[16]
Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, and Gianluca Demartini. 2015. Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1631--1640.
[17]
Ryan J Gallagher, Kyle Reing, David Kale, and Greg Ver Steeg. 2017. Anchored correlation explanation: Topic modeling with minimal domain knowledge. Transactions of the Association for Computational Linguistics 5 (2017), 529--542.
[18]
Patrick Gebhard, Tobias Baur, Ionut Damian, Gregor Mehlmann, Johannes Wagner, and Elisabeth André. 2014. Exploring interaction strategies for virtual characters to induce stress in simulated job interviews. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. International Foundation for Autonomous Agents and Multiagent Systems, 661--668.
[19]
Thomas Gordon. 1977. Leader Effectiveness Training. New York: Wyden books. p. 57.
[20]
Sambhav Gupta, Krithika Jagannath, Nitin Aggarwal, Ramamurti Sridar, Shawn Wilde, and Yu Chen. 2019. Artificially Intelligent (AI) tutors in the classroom: A need assessment study of designing chatbots to sup-port student learning. In Proceedings of the 2019 PACIS Pacific Asia Conference on Information Systems. AIS.
[21]
Matt Grech. 2017. The Current State of Chatbots in 2017. (Apr 2017). https://getvoip.com/blog/2017/04/ 21/the- current- state- of- chatbots- in- 2017/
[22]
Jonathan Grudin and Richard Jacques. 2019. Chatbots, Humbots, and the Quest for Artificial General Intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 209.
[23]
Tianran Hu, Anbang Xu, Zhe Liu, Quanzeng You, Yufan Guo, Vibha Sinha, Jiebo Luo, and Rama Akkiraju. 2018. Touch Your Heart: A Tone-aware Chatbot for Customer Care on Social Media. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 415.
[24]
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. 2016. Harnessing deep neural networks with logic rules. arXiv preprint arXiv:1603.06318 (2016).
[25]
IBM Watson Assistant. 2019. Retrieved from https://www.ibm.com/cloud/watson-assistant/
[26]
Mohit Jain, Pratyush Kumar, Ishita Bhansali, Q Vera Liao, Khai Truong, and Shwetak Patel. 2018. FarmChat: A Conversational Agent to Answer Farmer Queries. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 4 (2018), 170.
[27]
Douglas Jones, 1979, Elementary Information Theory. Clarendon Press.
[28]
Juji document for chatbot designers. 2019 Retrieved from https://docs.juji.io/
[29]
Soomin Kim, Joonhwan Lee, and Gahgene Gweon. 2019. Comparing Data from Chatbot and Web Surveys: Effects of Platform and Conversational Style on Survey Response Quality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 86.
[30]
Minha Lee, Sander Ackermans, Nena van As, Hanwen Chang, Enzo LUniversity of Chinese Academy of Sciences, and Wijnand IJsselsteijn. 2019. Caring for Vincent: A Chatbot for Self-Compassion. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 702.
[31]
Terri Lee, Krithika Jagannath, Nitin Aggarwal, Ramamurti Sridar, Shawn Wilde, Timothy Hill, and Yu Chen. 2019b. Intelligent Career Advisers in Your Pocket? A Need Assessment Study of Chatbots for Student Career Advising. (2019).
[32]
Jingyi Li, Michelle X. Zhou, Huahai Yang, and Gloria Mark. 2017. Confiding in and listening to virtual agents. In Proceedings of the 22nd International Conference on Intelligent User Interfaces-IUI, Vol. 17.
[33]
Shixia Liu, Michelle X Zhou, Shimei Pan, Yangqiu Song, Weihong Qian, Weijia Cai, and Xiaoxiao Lian. 2012. Tiara: Interactive, topic-based visual text summarization and analysis. ACM Transactions on Intelligent Systems and Technology (TIST) 3, 2 (2012), 25.
[34]
Gale M LUniversity of Chinese Academy of Sciences, Jonathan Gratch, Aisha King, and Louis-Philippe Morency. 2014. It's only a computer: Virtual humans increase willingness to disclose. Computers in Human Behavior 37 (2014), 94--100.
[35]
Stephen Louw, R Watson Todd, and P Jimarkon. 2011. Active listening in qualitative research interviews. In Proceedings of the International Conference: Research in Applied Linguistics, April.
[36]
Ryan Thomas Lowe, Nissan Pow, Iulian Vlad Serban, Laurent Charlin, Chia-Wei Liu, and Joelle Pineau. 2017. Training end-to-end dialogue systems with the ubuntu dialogue corpus. Dialogue & Discourse 8, 1 (2017), 31--65.
[37]
Manychat. 2019 Retrieved from https://manychat.com/
[38]
Michael C McCord, J William Murdock, and Branimir K Boguraev. 2012. Deep parsing in Watson. IBM Journal of research and development 56, 3.4 (2012), 3--1.
[39]
Clifford Nass, Jonathan Steuer, and Ellen R Tauber. 1994. Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 72--78.
[40]
Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2015. A review of relational machine learning for knowledge graphs. Proc. IEEE 104, 1 (2015), 11--33.
[41]
Jay Nunamaker, Derrick Douglas, Elkins Aaron, Burgoon Judee, and Patton Mark.2011 Embodied conversational agent-based kiosk for automated interviewing. Journal of Management Information Systems 28.1: 17--48.
[42]
Ioannis Papaioannou, Christian Dondrup, Jekaterina Novikova, and Oliver Lemon. 2017. Hybrid chat and task dialogue for more engaging hri using reinforcement learning. In 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, 593--598.
[43]
Geoffrey Keppel. 1991 Design and analysis: A researcher's handbook. Prentice-Hall, Inc.
[44]
Radim Rehrek and Petr Sojka. 2011. Gensim-statistical semantics in python. statistical semantics; gensim; Python; LDA; SVD (2011).
[45]
Carl R Rogers and Richard E Farson. 1984. Active listening. Organizational Psychology, 4th Ed. Englewood Cliffs, NJ (1984), 255--266.
[46]
Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence.
[47]
Louis Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2016. Generating long and diverse responses with neural conversation models.
[48]
Yangqiu Song, Shimei Pan, Shixia Liu, Michelle X Zhou, and Weihong Qian. 2009. Topic and keyword re-ranking for LDA-based topic modeling. In Proceedings of the 18th ACM conference on Information and knowledge management. ACM, 1757--1760.
[49]
S. Shyam Sundar and Jingyoung Kim 2019. Machine Heuristic: When We Trust Computers More than Humans with Our Personal Information. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 538.
[50]
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 3319-- 3328.
[51]
Ella Tallyn, Hector Fried, Rory Gianni, Amy Isard, and Chris Speed. 2018. The Ethnobot: Gathering Ethnographies in the Age of IoT. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 604.
[52]
David Traum. 2017 Computational Approaches to Dialogue. The Routledge Handbook of Language and Dialogue, Chapter 9, Weigand, E. (Ed.), New York.
[53]
Debbe Thompson, Karen W Cullen, Maria J Redondo, and Barbara Anderson. 2016. Use of relational agents to improve family communication in type 1 diabetes: methods. JMIR research protocols 5, 3 (2016), e151.
[54]
James Vincent. 2016. Twitter taught Microsoft's friendly AI chatbot to be a racist asshole in less than a day. Retrieved from https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbotracist
[55]
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering 29, 12 (2017), 2724--2743.
[56]
Yansen Wang, Chenyi Liu, Minlie Huang, and Liqiang Nie. 2018. Learning to Ask Questions in Opendomain Conversational Systems with Typed Decoders. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2193--2203.
[57]
Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. 2018a. Joint embedding of words and labels for text classification. arXiv preprint arXiv:1805.04174 (2018).
[58]
Alice Watson, Timothy Bickmore, Abby Cange, Ambar Kulshreshtha, and Joseph Kvedar. 2012. An internet-based virtual coach to promote physical activity adherence in overweight adults: randomized controlled trial. Journal of medical Internet research 14, 1 (2012), e1.
[59]
Harry Weger Jr, Gina R Castle, and Melissa C Emmett. 2010. Active listening in peer interviews: The influence of message paraphrasing on perceptions of listening skill. The Intl. Journal of Listening 24, 1 (2010), 34--49.
[60]
Joseph Weizenbaum and others. 1966. ELIZA-a computer program for the study of natural language communication between man and machine. Commun. ACM 9, 1 (1966), 36--45.
[61]
Alex C Williams, Harmanpreet Kaur, Gloria Mark, Anne Loomis Thompson, Shamsi T Iqbal, and Jaime Teevan. 2018. Supporting workplace detachment and reattachment with conversational intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 88.
[62]
Jason D Williams, Kavosh Asadi, and Geoffrey Zweig. 2017. Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017
[63]
Ziang Xiao, Michelle X Zhou, and Wai-Tat Fu. 2019a. Who should be my teammates: Using a conversational agent to understand individuals and help teaming. In Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, 437--447.
[64]
Ziang Xiao, Michelle X Zhou, Q Vera Liao, Gloria Mark, Changyan Chi, Wenxi Chen, and Huahai Yang. 2019b. Tell Me About Yourself: Using an AIPowered Chatbot to Conduct Conversational Surveys. arXiv preprint arXiv:1905.10700 (2019).
[65]
Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In Thirty-First AAAI Conference on Artificial Intelligence.
[66]
Zhou Yu, Alan W Black, and Alexander I Rudnicky. 2017. Learning conversational systems that interleave task and non-task content. Proceedings of the 26th International Joint Conference on Artificial Intelligence. AAAI Press, 2017.
[67]
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 654--664.
[68]
Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018. Emotional chatting machine: Emotional conversation generation with internal and external memory. In Thirty-Second AAAI Conference on Artificial Intelligence.
[69]
Michelle X Zhou, Gloria Mark, Jingyi Li, and Huahai Yang. 2019. Trusting Virtual Agents: The Effect of Personality. ACM Transactions on Interactive Intelligent Systems (TiiS) 9, 2--3 (2019), 10.

Cited By

View all

Index Terms

  1. If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
      April 2020
      10688 pages
      ISBN:9781450367080
      DOI:10.1145/3313831
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 23 April 2020

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. active listening
      2. ai chatbot
      3. chatbot platform
      4. conversational agents
      5. deep learning
      6. interview chatbot

      Qualifiers

      • Research-article

      Funding Sources

      Conference

      CHI '20
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

      Upcoming Conference

      CHI '25
      CHI Conference on Human Factors in Computing Systems
      April 26 - May 1, 2025
      Yokohama , Japan

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)1,933
      • Downloads (Last 6 weeks)339
      Reflects downloads up to 06 Nov 2024

      Other Metrics

      Citations

      Cited By

      View all

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Get Access

      Login options

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media