poster

BoTest: a Framework to Test the Quality of Conversational Agents Using Divergent Input Examples

Authors:

Julie Carson-Berndsen,

Anthony VentresqueAuthors Info & Claims

IUI '18 Companion: Companion Proceedings of the 23rd International Conference on Intelligent User Interfaces

Article No.: 64, Pages 1 - 2

https://doi.org/10.1145/3180308.3180373

Published: 05 March 2018 Publication History

Get Access

Abstract

Quality of conversational agents is important as users have high expectations. Consequently, poor interactions may lead to the user abandoning the system. In this paper, we propose a framework to test the quality of conversational agents. Our solution transforms working input that the conversational agent accurately recognises to generate divergent input examples that introduce complexity and stress the agent. As the divergent inputs are based on known utterances for which we have the 'normal' outputs, we can assess how robust the conversational agent is to variations in the input. To demonstrate our framework we built ChitChatBot, a simple conversational agent capable of making casual conversation.

References

[1]

Grand View Research Report: IVA Market Size Projected To Reach $12.28 Billion By 2024. Accessed: 2017-12-21.

Google Scholar

[2]

Cassell, J. Embodied conversational agents. MIT press, 2000.

Crossref

Google Scholar

[3]

Luger, E., and Sellen, A. Like having a really bad pa: the gulf between user expectation and experience of conversational agents. In CHI (2016), 5286--5297.

Digital Library

Google Scholar

[4]

Nguyen, M.-H. Business Insider: The latest market research, trends & landscape in the growing AI chatbot industry. Accessed: 2017-12-21.

Google Scholar

[5]

Weizenbaum, J. Eliza a computer program for the study of natural language communication between man and machine. Communications of the ACM 9, 1 (1966), 36--45.

Digital Library

Google Scholar

Cited By

View all

Urrico MClerissi DMariani LRoychoudhury APaiva AAbreu RStorey M(2024)MutaBot: A Mutation Testing Approach for ChatbotsProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3640032(79-83)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639478.3640032
Sharma KNguyen AHong Y(2024)Self‐regulation and shared regulation in collaborative learning in adaptive digital learning environments: A systematic review of empirical studiesBritish Journal of Educational Technology10.1111/bjet.13459Online publication date: 9-Apr-2024
https://doi.org/10.1111/bjet.13459
Gao JGarsole PAgarwal RLiu S(2024)AI Test Modeling and Analysis for Intelligent Chatbot Mobile App - A Case Study on Wysa2024 IEEE International Conference on Artificial Intelligence Testing (AITest)10.1109/AITest62860.2024.00024(132-141)Online publication date: 15-Jul-2024
https://doi.org/10.1109/AITest62860.2024.00024
Show More Cited By

Index Terms

BoTest: a Framework to Test the Quality of Conversational Agents Using Divergent Input Examples
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Evaluating and Informing the Design of Chatbots
DIS '18: Proceedings of the 2018 Designing Interactive Systems Conference

Text messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not ...
Small Talk Conversations and the Long-Term Use of Chatbots in Educational Settings – Experiences from a Field Study
Chatbot Research and Design
Abstract
In this paper, we analyze the use of small talk conversations based on a dialogue analysis of a long-term field study in which university students regularly interacted with a chatbot during a 3-month period of time in an educational setting. In ...
Single or Multiple Conversational Agents?: An Interactional Coherence Comparison
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Chatbots focusing on a narrow domain of expertise are in great rise. As several tasks require multiple expertise, a designer may integrate multiple chatbots in the background or include them as interlocutors in a conversation. We investigated both ...

Comments

Information & Contributors

Information

Published In

IUI '18 Companion: Companion Proceedings of the 23rd International Conference on Intelligent User Interfaces

March 2018

141 pages

ISBN:9781450355711

DOI:10.1145/3180308

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 March 2018

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Conference

IUI'18

Sponsor:

IUI'18: 23rd International Conference on Intelligent User Interfaces

March 7 - 11, 2018

Tokyo, Japan

Acceptance Rates

IUI '18 Companion Paper Acceptance Rate 63 of 127 submissions, 50%;

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
459
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)3

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Urrico MClerissi DMariani LRoychoudhury APaiva AAbreu RStorey M(2024)MutaBot: A Mutation Testing Approach for ChatbotsProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3640032(79-83)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639478.3640032
Sharma KNguyen AHong Y(2024)Self‐regulation and shared regulation in collaborative learning in adaptive digital learning environments: A systematic review of empirical studiesBritish Journal of Educational Technology10.1111/bjet.13459Online publication date: 9-Apr-2024
https://doi.org/10.1111/bjet.13459
Gao JGarsole PAgarwal RLiu S(2024)AI Test Modeling and Analysis for Intelligent Chatbot Mobile App - A Case Study on Wysa2024 IEEE International Conference on Artificial Intelligence Testing (AITest)10.1109/AITest62860.2024.00024(132-141)Online publication date: 15-Jul-2024
https://doi.org/10.1109/AITest62860.2024.00024
Feine JMorana SMaedche A(2022)Intrance: Designing an Interactive Enhancement System for the Development of QA ChatbotsProceedings of the ACM on Human-Computer Interaction10.1145/35551996:CSCW2(1-24)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555199
Paschoal LConte TSouza S(2022)On the Experimental Process in Evaluations of Brazilian Conversational Agents in EducationIEEE Revista Iberoamericana de Tecnologias del Aprendizaje10.1109/RITA.2022.314977617:1(99-107)Online publication date: Feb-2022
https://doi.org/10.1109/RITA.2022.3149776
Lopez-Morales JCanizares PPerez-Soler SGuerra Ede Lara J(2022)Asymob: a platform for measuring and clustering chatbots2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)10.1109/ICSE-Companion55297.2022.9793784(16-20)Online publication date: May-2022
https://doi.org/10.1109/ICSE-Companion55297.2022.9793784
Li XTao CGao JGuo H(2022)A Review of Quality Assurance Research of Dialogue Systems2022 IEEE International Conference On Artificial Intelligence Testing (AITest)10.1109/AITest55621.2022.00021(87-94)Online publication date: Aug-2022
https://doi.org/10.1109/AITest55621.2022.00021
Cabot JBurgueno LClariso RDaniel GPerianez-Pascual JRodriguez-Echeverria R(2021)Testing challenges for NLP-intensive bots2021 IEEE/ACM Third International Workshop on Bots in Software Engineering (BotSE)10.1109/BotSE52550.2021.00014(31-34)Online publication date: Jun-2021
https://doi.org/10.1109/BotSE52550.2021.00014
Santos MFurtado ANogueira SMoreira DCavalcante EDantas FBatista T(2020)OggyBugProceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing10.1145/3425174.3425230(79-87)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3425174.3425230
Guichard JRuane ESmith RBean DVentresque A(2019)Assessing the Robustness of Conversational Agents using Paraphrases2019 IEEE International Conference On Artificial Intelligence Testing (AITest)10.1109/AITest.2019.000-7(55-62)Online publication date: Apr-2019
https://doi.org/10.1109/AITest.2019.000-7

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Evaluating and Informing the Design of Chatbots

Small Talk Conversations and the Long-Term Use of Chatbots in Educational Settings – Experiences from a Field Study

Single or Multiple Conversational Agents?: An Interactional Coherence Comparison

Comments

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Evaluating and Informing the Design of Chatbots

Small Talk Conversations and the Long-Term Use of Chatbots in Educational Settings – Experiences from a Field Study

Single or Multiple Conversational Agents?: An Interactional Coherence Comparison

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations