research-article

Learning polite behavior with situation models

Authors:

Rémi Barraquand,

James L. CrowleyAuthors Info & Claims

HRI '08: Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction

Pages 209 - 216

https://doi.org/10.1145/1349822.1349850

Published: 12 March 2008 Publication History

Abstract

In this paper, we describe experiments with methods for learning the appropriateness of behaviors based on a model of the current social situation. We first review different approaches for social robotics, and present a new approach based on situation modeling. We then review algorithms for social learning and propose three modifications to the classical Q-Learning algorithm. We describe five experiments with progressively complex algorithms for learning the appropriateness of behaviors. The first three experiments illustrate how social factors can be used to improve learning by controlling learning rate. In the fourth experiment we demonstrate that proper credit assignment improves the effectiveness of reinforcement learning for social interaction. In our fifth experiment we show that analogy can be used to accelerate learning rates in contexts composed of many situations.

References

[1]

Adams, B. Breazeal, C. Brooks, R. A., Scassellati, B., "Humanoid robots: a new kind of tool," Intelligent Systems and Their Applications, IEEE {see also IEEE Intelligent Systems}, vol.15, no.4, pp.25--31, Jul/Aug 2000.

Digital Library

[2]

Bartlett, M., Littleworth, G., Fasel, I., and Movellan, J., Real Time Face Detection and Facial Expression Recognition: Development and Applications to Human Computer Interaction, Workshop on Computer Vision for HCI, CVPR 2003, Vancouver, Canada, 2003.

[3]

Brdiczka, O., Learning Situation Models for Context-Aware Services, Doctoral Dissertation, INPG, 2007.

[4]

Brdiczka, O., Maisonnasse, J., Reignier P., and Crowley, J. L., Learning individual roles from video in a smart home, International Conference on Intelligent Environments, 2006.

[5]

Breazeal C. and Aryananda, L., Recognition of Affective Communicative Intent in Robot-Directed Speech, Autonomous Robots, 12, 2002.

Digital Library

[6]

Breazeal, C., Designing Sociable Robots, MIT Press, Cambridge MA, 2002.

Digital Library

[7]

Brooks, R., Breazeal, C., Marjanovic, M., Scassellati, B., and Williamson, M., "The Cog Project: Building a Humanoid Robot". In Computation for metaphors, analogy, and agents, C. Nehaniv (ed), Lecture notes in artificial intelligence 1562. New York, Springer. 52--87, 1998.

Digital Library

[8]

Crowley, J. L., "Context Driven Observation of Human Activity", European Symposium on Ambient Intelligence, Amsterdam, 3-5 November 2003.

[9]

De Silva, L. C., and Pei Chi, N., Bimodal emotion recognition, FG 2000, Fourth IEEE Conference Automatic Face and Gesture Recognition, pp. 332--335, Grenoble, March 2000.

Digital Library

[10]

Even-Dar E. and Mansour, Y., Learning Rates for Q-Learning, 14th Annual Conference on Computational Learning Theory, EuroCOLT 2001, Amsterdam, The Netherlands, July 2001, Proceedings, 2111 (2001), pp. 589--604.

Digital Library

[11]

Fong, T., Nourbakhsh I., and Dautenhahn, K., A Survey of Socially Interactive Robots, Robotics and Autonomous Systems, 42, 2003.

[12]

Gockley, R., Bruce, A., Forlizzi, J., Michalowski, M., Mundell, A., Rosenthal, S., Sellner, B., Simmons, R., Snipes, K., Schultz A. and Wang, J., Designing robots for long-term social interaction, IROS 2005, International Conference on Intelligent Robots and Systems, 2005.

[13]

Isbell, C. L., Shelton, C. R., Kearns, M., Singh, S., and Stone, P., A social reinforcement learning agent, Proceedings of the fifth international conference on Autonomous agents, ACM Press, Montreal, Quebec, Canada, 2001.

Digital Library

[14]

Johnson-Laird, P. N., How We Reason. Oxford University Press (2006).

[15]

Johnson-Laird, P. N., Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Cambridge University Press; Cambridge, MA., 1983.

Digital Library

[16]

Kidd, C. D., and Breazeal, C., Designing a Sociable Robot System for Weight Maintenance, RO-MAN 2005,14th IEEE International Workshop on Robot and Human Interactive Communication, Nashville TN, Aug 2005.

[17]

Klopf, A. H., "Brain function and adaptive systems - A heterostatic theory", Technical Report AFCRL72-0164, Air Force Cambridge Research Laboratories, Bedford, MA, 1972.

[18]

Maisonnasse, J., Gourier, N., Brdiczka O., and Reignier, P., "Attentional Model for Perceiving Social Context in Intelligent Environments", 3rd IFIP Conference on Artificial Intelligence App22lications and Innovations (AIAI), pp171--178, June 2006.

[19]

Ormrod, J. E., Human Learning, Prentice Hall, 2003.

[20]

Padgett, C., and Cottrell, G., A simple neural network models categorical perception of facial expressions. In Proceedings of the 20th Annual Conference of the Cognitive Science Society, Lawerence Erlbaum, Hillsdale NJ, 1998.

[21]

Preux, P., Propagation of Q-values in Tabular TD(lambda), Proc. 13th European Conference on Machine Learning (ECML), 2430, pp. 369--380, 2002.

Digital Library

[22]

Reeves, B. and Nass, C. The Media Equation: how People Treat Computers, Television, and New Media Like Real People and Places. Cambridge University Press, 1998.

Digital Library

[23]

Shin, Y. S., A Neural Network Model for Classification of Facial Expressions Based on Dimension Model, Lecture Notes in Computer Science, Springer Berlin / Heidelberg, 2005.

Digital Library

[24]

Sutton, R. S. "Temporal Credit Assignment in Reinforcement Learning", Ph.D. dissertation, University of Massachusetts, Department of Computer and Information Science, 1984.

Digital Library

[25]

Sutton, R. S., and Barto, A. G., Reinforcement Learning: An Introduction, MIT press, 1998.

Digital Library

[26]

Thomaz, A. L. and Breazeal, C. Reinforcement Learning with Human Teachers: Evidence of Feedback and Guidance with Implications for Learning Performance, Proc. of the 21st National Conference on Artificial Intelligence, AAAI '06, Boston, Mass, Vol 21, Part 1, pp 1000--1005, 2006.

Digital Library

[27]

Thomaz, A. L., Hoffman G., and Breazeal, C., Reinforcement Learning with Human Teachers: Understanding How People Want to Teach Robots, The 15th IEEE International Symposium on Robot and Human Interactive Communication, pp. 352--357, University of Hertfordshire, Hatfield, Sept 2006.

[28]

Thomaz, A. L., "Socially Guided Machine Learning." MIT Ph.D. Thesis, June 2006

Digital Library

[29]

Watkins, C. J. C. H., Learning from Delayed Rewards, Doctoral Thesis, Cambridge University, 1989.

Cited By

Wullenkord RBellon JGransche BNähr-Wagener SEyssel F(2023)Social appropriateness in HMIInteraction Studies. Social Behaviour and Communication in Biological and Artificial SystemsInteraction Studies / Social Behaviour and Communication in Biological and Artificial SystemsInteraction Studies10.1075/is.22017.wul23:3(360-390)Online publication date: 21-Apr-2023
https://doi.org/10.1075/is.22017.wul
Janowski KRitschel HAndré E(2022)Adaptive Artificial PersonalitiesThe Handbook on Socially Interactive Agents10.1145/3563659.3563666(155-194)Online publication date: 27-Oct-2022
https://dl.acm.org/doi/10.1145/3563659.3563666
Lee HCheon ELim CFischer K(2022)Configuring Humans: What Roles Humans Play in HRI Research2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)10.1109/HRI53351.2022.9889496(478-492)Online publication date: 7-Mar-2022
https://doi.org/10.1109/HRI53351.2022.9889496
Show More Cited By

Index Terms

Learning polite behavior with situation models
1. Computing methodologies
  1. Machine learning

Recommendations

Backward Q-learning: The combination of Sarsa algorithm and Q-learning

Reinforcement learning (RL) has been applied to many fields and applications, but there are still some dilemmas between exploration and exploitation strategy for action selection policy. The well-known areas of reinforcement learning are the Q-learning ...
Proposal and evaluation of deep exploitation-oriented learning under multiple reward environment
Abstract
Recently, deep reinforcement learning (DRL) has attracted considerable attention. The well-known deep Q-network (DQN) architecture successfully combines deep learning and Q-learning which is a representative reinforcement learning (RL) ...
Social Learning in Networks of Friends versus Strangers

Networks and the embedded relationships are critical determinants of how people communicate and form beliefs. The explosion of social media has significantly increased the scope and impact of social learning among consumers. This paper studies ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

HRI '08: Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction

March 2008

402 pages

ISBN:9781605580173

DOI:10.1145/1349822

General Chairs:
Terry Fong
NASA Ames Research Center, USA
,
Kerstin Dautenhahn
University of Hertfordshire, UK
,
Program Chairs:
Matthias Scheutz
Indiana University Bloomington, USA
,
Yiannis Demiris
Imperial College London, UK

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 March 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

HRI '08

Sponsor:

HRI '08: International Conference on Human Robot Interaction

March 12 - 15, 2008

Amsterdam, The Netherlands

Acceptance Rates

Overall Acceptance Rate 268 of 1,124 submissions, 24%

Upcoming Conference

HRI '25

Sponsor:
sigai
sigai

ACM/IEEE International Conference on Human-Robot Interaction

March 4 - 6, 2025

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
463
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wullenkord RBellon JGransche BNähr-Wagener SEyssel F(2023)Social appropriateness in HMIInteraction Studies. Social Behaviour and Communication in Biological and Artificial SystemsInteraction Studies / Social Behaviour and Communication in Biological and Artificial SystemsInteraction Studies10.1075/is.22017.wul23:3(360-390)Online publication date: 21-Apr-2023
Janowski KRitschel HAndré E(2022)Adaptive Artificial PersonalitiesThe Handbook on Socially Interactive Agents10.1145/3563659.3563666(155-194)Online publication date: 27-Oct-2022
Lee HCheon ELim CFischer K(2022)Configuring Humans: What Roles Humans Play in HRI Research2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)10.1109/HRI53351.2022.9889496(478-492)Online publication date: 7-Mar-2022
(2022)The Handbook on Socially Interactive AgentsundefinedOnline publication date: 27-Oct-2022
Akalin NLoutfi A(2021)Reinforcement Learning Approaches in Social RoboticsSensors10.3390/s2104129221:4(1292)Online publication date: 11-Feb-2021
Kiderle TRitschel HJanowski KMertes SLingenfelser FAndre E(2021)Socially-Aware Personality Adaptation2021 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)10.1109/ACIIW52867.2021.9666197(1-8)Online publication date: 28-Sep-2021
Jackson RKim JTapus ASirkin DJung MKwak S(2019)Toward morally sensitive robotic communicationProceedings of the 14th ACM/IEEE International Conference on Human-Robot Interaction10.5555/3378680.3378867(715-717)Online publication date: 11-Mar-2019
Jackson RConitzer VHadfield GVallor S(2019)Generating Appropriate Responses to Inappropriate Robot CommandsProceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society10.1145/3306618.3314306(523-524)Online publication date: 27-Jan-2019
Jackson R(2019)Toward Morally Sensitive Robotic Communication2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI)10.1109/HRI.2019.8673209(715-717)Online publication date: Mar-2019
Ritschel HSeiderer AJanowski KAslan IAndré ENijholt AVelasco CObrist MOkajima KSpence C(2018)Drink-O-MenderProceedings of the 3rd International Workshop on Multisensory Approaches to Human-Food Interaction10.1145/3279954.3279957(1-8)Online publication date: 16-Oct-2018
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten