research-article

Creating chatbots to talk with humans: HCI evaluations and perspectives

Authors:

Vinícius F Galvão,

Cristiano Maciel,

Ana Cristina Bicharra GarciaAuthors Info & Claims

IHC '19: Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems

Article No.: 20, Pages 1 - 11

https://doi.org/10.1145/3357155.3358460

Published: 22 October 2019 Publication History

Abstract

More softwares are with the power of Artificial Intelligence (AI) making their functions, simultaneously more dynamic and capable of attending each users' personal needs. Chatfuel an application that allows users to create chatbots, an AI that can answer messages autonomously. However we question it: How easy chatbot creation is, especially when dealing the complex AI operating in them? In this sense, this exploratory research seeks to understand, through our volunteers' perspectives, with different levels of computational expertise, if this creation process is really as simple as divulged. To conduct the experiments, usability methods were used such as Cognitive Walkthrough, Communicability Evaluation and Attrakdiff. With the study results, among many others, users's noticed certain difficulty while using. the tool and we problematize certain aspects connected to AI, in the context of a human-computer interaction evaluation.

References

[1]

S. L. dos Santos & F.G.Teixeira. (2010). Design de uma interface de interação tridimensional com foco na usabilidade e no desempenho gráfico. In: Design & tecnologia [recurso eletrônico]. Vol. 1, n. 1, p. 39--50. Porto Alegre, RS, Brazil.

[2]

N. Statt (2018) - The AI boom is happening all over the world, and it's accelerating quickly. The Verge 12 Dez 2018. Disponível em: https://www.theverge.com/2018/12/12/18136929/artificial-intelligence-ai-index-report-2018-machine-learning-global-progress-research > - Acesso em 25 de fev. 2019.

[3]

S. Etlinger. (2017) The Age of AI-How Artificial Intelligence is Transforming Organizations. Altimeter, São Francisco, USA,

[4]

D. Meyer. (2018) YouTube's AI is so good at finding offensive content it needs more staff to keep up. World Economic Forum, 25 Abr 2018. Disponível em: https://www.weforum.org/agenda/2018/04/ai-is-now-youtube-s-biggest-weapon-against-the-spread-of-offensive-videos/> - Acesso em 25 de fev. 2019.

[5]

M. Gavira. (2019) Here's How Netflix Uses AI and Data to Conquer the World. VibeProjects the power of purpose. Disponível em: https://vibeprojects.com/heres-how-netflix-uses-ai-and-data-to-conquer-the-world/ - Acesso em 25 de fev. 2019.

[6]

B. Marr. (2018) Machine Learning In Practice: How Does Amazon's Alexa Really Work? Forbes, 5 Out 2018. Disponível em: https://www.forbes.com/sites/bernardmarr/2018/10/05/how-does-amazons-alexa-really-work/#422cc34e1937> - Acesso em 25 de fev. 2019.

[7]

B. Marr. (2017). How Walmart is Using Machine Learning AI, IoT and Big Data to Boost Retail Performance. Forbes, 29 Ago 2017. Disponível em: https://www.forbes.com/sites/bernardmarr/2017/08/29/how-walmart-is-using-machine-learning-ai-iot-and-big-data-to-boost-retail-performance/#3aee77fa6cb1> - Acesso em 25 de fev. 2019.

[8]

A. J. Ask, M. Facemire & A. Hogan. (2016) The State of Chatbots-Pilot Chatbots as Part of your App + Mobile Strategy. Forrester Research. Disponível em: https://www.forrester.com/report/The+State+Of+Chatbots/-/E-RES136207> Acessado em: 5 Mar de 2019.

[9]

A. B. Shawar & E. Atwell. (2007). Chatbots: are they really useful?. In: Ldv forum. p. 29--49.

[10]

CHATFUEL. Site Oficial da Empresa Disponível em:: https://chatfuel.com/about-us.html> - Acessado em 25 de Fev de 2019.

[11]

J. F. Teixeira. (2014). Inteligência artificial. Pia Sociedade de São Paul, Editora Paulus.

[12]

S. Russell & P. Norvig. (2004). Inteligência Artificial. Ed. Campus. Rio de Janeiro, 2004.

[13]

L. M. Pereira. (1988) Inteligência Artificial: mito e ciência. In Revista Colóquio-Ciências, v. 3, p. 1--13.

[14]

S. Russell & P. Norvig. (2016) Artificial intelligence: a modern approach. Pearson Education Limited, Malaysia.

[15]

J. Gordon, E. Edward & H. Shortliffe. (1984) H. Rule-Based Expert Systems. Chapter The Dempster-Shafer Theory of Evidence.

[16]

E. Castillo, J. M. Gutierrez, M. Jose & A.S. Hadi. (2012). Expert systems and probabilistic network models. Springer Science & Business Media.

[17]

R. R. Yager & L. A. Zadeh (Ed.). (2012). An introduction to fuzzy logic applications in intelligent systems. Springer Science & Business Media.

[18]

F. Zambonelli, R. N. Jennings & M. Wooldridg. (2003) Developing multiagent systems: The Gaia methodology. In ACM Transactions on Software Engineering and Methodology (TOSEM), v. 12, n. 3, p. 317--370.

[19]

Y. Shoham & K. Leyton-Brown. (2008) Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press.

[20]

J. Kolodner. (2014) Case-based reasoning. Morgan Kaufmann.

[21]

Â. G. B. Lima. (2003) Interfaces entre a ciência da informação e a ciência cognitiva. In Ciência da Informação, v. 32, n. 1.

[22]

P. B. Brandtzaeg. & A. Følstad. (2017) Why people use chatbots. In: International Conference on Internet Science. Springer, Cham. p. 377--392.

Digital Library

[23]

C. F. De C. Júnior et al. (2018). Chatbot: uma visão geral sobre aplicações inteligentes. Revista Sítio Novo, v. 2, n. 2, p. 68--84.

[24]

Natural Language Processing: Why The Machines Still Struggle To Understand Us? - Intellectsoft, 3 Dez 2015. Disponível em: https://www.intellectsoft.net/blog/natural-language-processing-why-the-machines-still-struggle-to-understand-us/>, Acessado em: 25 de fev de 2019.

[25]

Types of Bots: Na Overview. Botnerds. Disponível em: http://botnerds.com/types-of-bots/> Acessado em 25 de fev de 2015.

[26]

S. R. Fussell et al. (2008). How people anthropomorphize robots. In: 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, p. 145--152.

Digital Library

[27]

How to have great sex with cleverbot? Yahoo! Answers Disponível em: https://answers.yahoo.com/question/index?qid=20111014235033AAqeTOY> Acessado em: 25 de fev 2015.

[28]

J. Brixey et al. (2017). Shihbot: A facebook chatbot for sexual health information on hiv/aids. In: Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue.. p. 370--373. Pathfinding Discourses of Self in Social Network Sites Ariza Geogalou.

[29]

H. S. de A. Souza. (2017) A robô Eurekka: desenvolvendo um chatbot de psicoeducação para o messenger do Facebook. In Automating power: Social bot interference in global politics. Samuel C. Woolley.

[30]

R. D. Vukovic & I. M. Dujlovic. (2016). Facebook messenger bots and their application for business. In: 2016 24th Telecommunications Forum (TELFOR). IEEE. p. 1--4.

[31]

Y. A. De Lima, A. C Chares & J. S. De Oliveira. (2018) Identificação de fake news: uma abordagem utilizando métodos de busca e chatbots. In: 7° Brazilian Workshop on Social Network Analysis and Mining (BraSNAM 2018). SBC.

[32]

M. A. Ruediger et al. (2017) Robôs, redes sociais e política no Brasil: estudo sobre interferências ilegítimas no debate público na web, riscos à democracia e processo eleitoral de 2018.

[33]

Crunch Base | ChatFuel Disponível: https://www.crunchbase.com/organization/chatfuel> Acessado em: 25 de fev de 2019.

[34]

P. Polson, C. Lewis, J. Rieman & C. Wharton. (1992). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. In: International Journal of ManMachine Studies, 36 (5). p. 741--773.

[35]

C. Wharton, J. Rieman, C. Lewis & P. Polson. (1994). The cognitive walkthrough method: A practitioner's guide. In Nielsen, J., and Mack, R. L. (Eds.), Usability Inspection Methods. John Wiley & Sons. 105--140. New York, USA.

[36]

Bevan, N. (1995). Usability is quality of use. In Advances in Human Factors/Ergonomics (Vol. 20, pp. 349--354). Elsevier.

[37]

-Nielsen, J. (1994). Usability inspection methods. In Conference companion on Human factors in computing systems (pp. 413--414). ACM. Boston. Massachusetts, USA.

[38]

D. Norman & J. Nielsen. (2016). Avaliando a Experiência do Usuário e a Usabilidade de um Aplicativo Web Móvel: um relato de experiência. In The definition of user experience (UX). Nielsen Norman Group Publication, v. 1.

[39]

AttrakDiff. Site Oficial. Disponível em: http://attrakdiff.de/index-en.html. Acessado em 25 de Fev de 2019

[40]

N. M. C. Valentim, W. Silva & T. Conte. (2015). Avaliando a Experiência do Usuário e a Usabilidade de um Aplicativo Web Móvel: um relato de experiência. In: XVIII Congresso Ibero-Americano em Engenharia de Software, 2015, Congresso Ibero-Americano em Engenharia de Software (CIbSE 2015). 2015. p. 788--801. Lima. Peru.

[41]

A. Macaulay. (2003) Don't Tell Me, Show Me: Directors Talk about Acting. Currency Press. Sydney.

[42]

J. Nielsen. (2012). How Many Test Users in a Usability Study? Nielsen Norman Group, 04 de Junho de 2012. Tópico: User Testing. Fremont, Califórnia, USA. Disponível em: https://www.nngroup.com/articles/how-many-test-users/> Acessado em: em 25 de Fev de 2019.

[43]

J. Vincent. (2019). The State of AI in 2019 The Verge, 28 de Janeiro de 2019. Manhattan, New York USA. Disponível em: https://www.nngroup.com/articles/how-many-test-users/> Acessado em: em 03 de Jun de 2019. P. B. Brandtzaeg. & A. Følstad. (2017) Chatbots and the New World of HCI. In: interactions, v. 24, n. 4, p. 38--42.

Digital Library

[44]

Winckler, M., & Pimenta, M. S. (2002). Avaliação de usabilidade de sites web. In: ESCOLA REGIONAL de Informática. SBC, p.1--54, Porto AlegreBrazil.

[45]

Kreitzberg C. (2017). The Intuitive Interface. User Experience Office. Princenton University. 3 Fev 2017. Disponível em: https://ux.princeton.edu/designing-experience/intuitive-interface> Acessado em: 1 de Maio de 2019.

[46]

Gasparini, I., Kimura, M. H., & Pimenta, M. S. (2013, October). Visualizando 15 anos de IHC. In Proceedings of the 12th Brazilian Symposium on Human Factors in Computing Systems (pp. 238--247). Brazilian Computer Society. Manaus. Brazil.

Digital Library

[47]

Pereira, R. (2013). IHC na Especialização em Engenharia de Software: discussões sobre uma experiência prática. In WEIHC - Workshop de Ensino de IHC. (pp. 1--6) Manaus, Brazil. .

[48]

Ferreira, A. L. S., de Miranda, L. C., & de Miranda, E. E. C. (2012). Interfaces cérebro-computador de sistemas interativos: estado da arte e desafios de IHC. In Proceedings of the 11th Brazilian Symposium on Human Factors in Computing Systems (pp. 239--248). Brazilian Computer Society. Cuiabá. Brazil

Digital Library

[49]

Ko, A. J., Myers, B. A., & Aung, H. H. (2004, September). Six learning barriers in end-user programming systems. In 2004 IEEE Symposium on Visual Languages-Human Centric Computing (pp. 199--206). IEEE. Rome. Italy

Digital Library

[50]

Barajas, M., & Owen, M. (2000). Implementing virtual learning environments: Looking for holistic approach. Journal of Educational Technology & Society, v.3, n(3) p39--53.

[51]

Teixeira, F. A. A. (2008). Chatbot para serviços bancários. Faculdade de Ciências, Universidade do Porto. Cidade do Porto, Portugal.

[52]

ValÈrio, F. A. M., Guimar, T. G., Prates, R. O., & Candello, H. (2018). Chatbots Explain Themselves: Designers' Strategies for Conveying Chatbot Features to Users. SBC Journal on Interactive Systems, 9(3), 61--79.

[53]

Smestad, T. L. (2018). Personality Matters! Improving The User Experience of Chatbot Interfaces-Personality provides a stable pattern to guide the design and behaviour of conversational agents (Master's thesis, NTNU).

[54]

Razaque, A., & Yang, G. (2018, November). Chatbot: Efficient and Utility-Based Platform. In Intelligent Computing: Proceedings of the 2018 Computing Conference (Vol. 1, p. 109). Springer.

[55]

Duijst, D. (2017). Can we improve the User Experience of Chatbots with Personalisation (Doctoral dissertation, M. Sc Thesis, University of Amsterdam).

[56]

Hung, V., Elvir, M., Gonzalez, A., & DeMara, R. (2009, October). Towards a method for evaluating naturalness in conversational dialog systems. In 2009 IEEE International Conference on Systems, Man and Cybernetics (pp. 1236--1241). IEEE.

[57]

Valério, F. A., Guimarães, T. G., Prates, R. O., & Candello, H. (2017, October). Here's What I Can Do: Chatbots' Strategies to Convey Their Features to Users. In Proceedings of the XVI Brazilian Symposium on Human Factors in Computing Systems (p. 28). ACM.

Digital Library

[58]

Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018, June). Evaluating and informing the design of chatbots. In Proceedings of the 2018 Designing Interactive Systems Conference (pp. 895--906). ACM.

Digital Library

[59]

Jylkäs, T., Aijälä, M., Vuorikari, T., & Rajab, V. (2018). AI Assistants as Non-human Actors in Service Design.

Cited By

de Souza PPires IMotti VCaseli HNeto JMartini Lde Almeida Neris VSantos CBento Villela MRios da Hora Rodrigues Kde Gois Ribeiro Darin T(2022)Design recommendations for chatbots to support people with depressionProceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems10.1145/3554364.3559119(1-11)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3554364.3559119

Index Terms

Creating chatbots to talk with humans: HCI evaluations and perspectives

Recommendations

Chatbots, Humbots, and the Quest for Artificial General Intelligence
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

What began as a quest for artificial general intelligence branched into several pursuits, including intelligent assistants developed by tech companies and task-oriented chatbots that deliver more information or services in specific domains. Progress ...
Modern software cybernetics

Classify software cybernetics as Software Cybernetics I and II.Identify the transition from Software Cybernetics I to Software Cybernetics II.Indicate that some new research areas are related to Software Cybernetics II.Highlight new research trends of ...
Chatbots at Work: A Taxonomy of the Use of Chatbots in the Workplace
Responsible AI and Analytics for an Ethical and Inclusive Digitized Society
Abstract
The adoption of Artificial Intelligence (AI) applications in organisations is growing rapidly. In this study, we focus on chatbots as one type of AI applications in organisations. As an AI-enabled application, Chatbots learn from their use ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

IHC '19: Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems

October 2019

679 pages

ISBN:9781450369718

DOI:10.1145/3357155

General Chairs:
Rodrigo Laiola Guimaraes
Universidade Federal do Espirito Santo (UFES)
,
Celso Alberto Saibel Santos
Universidade Federal do Espirito Santo (UFES)
,
Program Chairs:
Vagner Figueredo de Santana
IBM Research
,
Artur Kronbauer
Universidade Salvador (UNIFACS)

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SBC: Brazilian Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IHC '19

Sponsor:

SBC

IHC '19: XVIII Brazilian Symposium on Human Factors in Computing Systems

October 22 - 25, 2019

Espírito Santo, Vitória, Brazil

Acceptance Rates

IHC '19 Paper Acceptance Rate 56 of 165 submissions, 34%;

Overall Acceptance Rate 331 of 973 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
417
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

de Souza PPires IMotti VCaseli HNeto JMartini Lde Almeida Neris VSantos CBento Villela MRios da Hora Rodrigues Kde Gois Ribeiro Darin T(2022)Design recommendations for chatbots to support people with depressionProceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems10.1145/3554364.3559119(1-11)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3554364.3559119

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents