research-article

Public Access

"Do As I Say, Not As I Do": Challenges in Delegating Decisions to Automated Agents

Authors:

Celso M. de Melo,

Stacy Marsella,

Jonathan GratchAuthors Info & Claims

AAMAS '16: Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems

Pages 949 - 956

Published: 09 May 2016 Publication History

Abstract

There has been growing interest, across various domains, in computer agents that can decide on behalf of humans. These agents have the potential to save considerable time and help humans reach better decisions. One implicit assumption, however, is that, as long as the algorithms that simulate decision-making are correct and capture how humans make decisions, humans will treat these agents similarly to other humans. Here we show that interaction with agents that act on our behalf or on behalf of others is richer and more interesting than initially expected. Our results show that, on the one hand, people are more selfish with agents acting on behalf of others, than when interacting directly with others. We propose that agents increase the social distance with others which, subsequently, leads to increased demand. On the other hand, when people task an agent to interact with others, people show more concern for fairness than when interacting directly with others. In this case, higher psychological distance leads people to consider their social image and the long-term consequences of their actions and, thus, behave more fairly. To support these findings, we present an experiment where people engaged in the ultimatum game, either directly or via an agent, with others or agents representing others. We show that these patterns of behavior also occur in a variant of the ultimatum game -- the impunity game -- where others have minimal power over the final outcome. Finally, we study how social value orientation -- i.e., people's propensity for cooperation -- impact these effects. These results have important implications for our understanding of the psychological mechanisms underlying interaction with agents, as well as practical implications for the design of successful agents that act on our behalf or on behalf of others.

References

[1]

Lin, R., and Kraus, S. 2010. Can automated agents proficiently negotiate with humans? Comm. ACM 53, 78--88.

Digital Library

[2]

Jennings, N., Faratin, P., Lomuscio, A., Parsons S., Wooldridge, M., and Sierra, C. 2001. Automated negotiation: Prospects, methods and challenges. Group Dec. Negot. 10, 199--215.

[3]

Jonker, C., and Robu, V. 2007. An agent architecture for multi-attribute negotiation using incomplete preference information. In Proceedings of the Autonomous Agents and Multi-Agent Systems Conference (AAMAS'07).

Digital Library

[4]

Sycara, K., and Dai, T. 2010. Agent reasoning in negotiation. In Handbook of Group Decision and Negotiation, D. Kilgour, and C. Eden, Eds. Springer Netherlands, 437--451.

[5]

Lin, R., Kraus, S., Oshrat, Y., and Gal, Y. 2010. Facilitating the evaluation of automated negotiators using peer designed agents. In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI'10).

Digital Library

[6]

Davenport, T., and Harris, J. 2005. Automated decision making comes of age. MIT Sloan Manage. Rev. 46, 83--89.

[7]

Dresner, K., and Stone, P. 2007. Sharing the road: Autonomous vehicles meet human drivers. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI'07).

Digital Library

[8]

Gupte, S. 2012. A survey of quadrotor Unmanned Aerial Vehicles. In Proceedings of IEEE Southeastcon. IEEE, 1--6.

[9]

Nass, C., and Moon, Y. 2000. Machines and mindlessness: Social responses to computers. J. Soc. Issues 56, 81--103.

[10]

Nass, C., Fogg, B., and Moon, Y. 1996. Can computers be teammates? Int. J. Hum-Comput. St. 45, 669--678.

Digital Library

[11]

Nass, C., Isbister, K., and Lee, E.-J. 2000. Truth is beauty: Researching embodied conversational agents. In Embodied conversational agents, J. Cassell Ed. MIT Press, Cambridge, MA, 374--402.

Digital Library

[12]

Nass, C., Moon, Y., and Carney, P. 1999. Are people polite to computers? Responses to computer-based interviewing systems. J. App. Psychol. 29, 1093--1110.

[13]

Nass, C., Moon, Y., and Green, N. 1997. Are computers gender-neutral? Gender stereotypic responses to computers. J. App. Soc. Psychol. 27, 864--876.

[14]

Reeves, B., and Nass, C. 1996. The media equation: How people treat computers, television, and new media like real people and places. Cambridge University Press.

Digital Library

[15]

Gray, H., Gray, K., and Wegner, D. 2007. Dimensions of mind perception. Science 315, 619.

[16]

Blascovich, J., Loomis, J., Beall, A., Swinth, K., Hoyt, C., and Bailenson, J. 2002. Immersive virtual environment technology as a methodological tool for social psychology. Psychol. Inq. 13, 103--124.

[17]

Waytz, A., Gray, K., Epley, N., and Wegner, D. 2010. Causes and consequences of mind perception. Trends Cogn. Sci. 14, 383--388.

[18]

de Melo, C., and Gratch, J. 2015. People show envy, not guilt, when making decisions with machines. In Proceedings of the Affective Computing and Intelligent Interaction Conference (ACII) 2015.

Digital Library

[19]

Rilling, J., Gutman, D., Zeh, T., Pagnoni, G., Berns, G., and Kilts, C. 2002. A neural basis for social cooperation. Neuron 35, 395--405.

[20]

Krach, S., Hegel, F., Wrede, B., Sagerer, G., Binkofski, F., and Kircher, T. 2008. Can machines think? Interaction and perspective taking with robots investigated via fMRI. PLOS ONE 3, 1--11.

[21]

McCabe, K., Houser, D., Ryan, L., Smith, V., and Trouard, T. 2001. A functional imaging study of cooperation in two-person reciprocal exchange. Proc. Nat. Acad. Sci. 98, 11832--11835.

[22]

Sanfey, A., Rilling, J., Aronson, J., Nystrom, L., and Cohen, J. 2003. The neural basis of economic decision-making in the ultimatum game. Science 300, 1755--1758.

[23]

Kircher, T., Blümel, I., Marjoram, D., Lataster, T., Krabbendam, L., Weber, J., van Os, J., and Krach, S. 2009. Online mentalising investigated with functional MRI. Neurosci. Lett. 454, 176--181.

[24]

Gallagher, H., Anthony, J., Roepstorff, A., and Frith, C. 2002. Imaging the intentional stance in a competitive game. NeuroImage 16, 814--821.

[25]

Hoffman, E., McCabe, K., and Smith, V. 1996. Social distance and other-regarding behavior in dictator games. Am. Econ. Rev. 86, 653--660.

[26]

Hoffman, E., McCabe, K., Shachat, K., and Smith, V. 1994. Preferences, property rights, and anonymity in bargaining games. Games Econ. Behav. 7, 346--380.

[27]

Leider, S., Mobius, M., Rosenblat, T., and Do, Q-A. 2009. Directed altruism and enforced reciprocity in social networks. Q. J. Econ. 124, 1815--1851.

[28]

Goeree, J., McConnell, M., Mitchell, T., Tromp, T., and Yariv, L. 2010. The 1/d law of giving. Am. Econ. J. Microecon. 2, 183--203.

[29]

Pronin, E., Olivola, C., and Kennedy K. 2008. Doing unto future selves as you would do unto others: Psychological distance and decision making. Person. Soc. Psychol. Bull. 34, 224--236.

[30]

Nowak, M., and May, R. 1992. Evolutionary games and spatial chaos. Nature 359, 826--829.

[31]

Nowak, M., Tarnita, C., and Antal T. 2010. Evolutionary dynamics in structured populations. Philos. Trans. R. Soc. B: Biol. Sci. 365, 19--30.

[32]

Nowak, M., and Sigmund, K. 2005. Evolution of indirect reciprocity. Nature 437, 1291--1298.

[33]

Ohtsuki, H. and Iwasa, Y. 2006. The leading eight: social norms that can maintain cooperation by indirect reciprocity. J. Theor. Biol. 239, 435--444.

[34]

Andreoni, J. & Bernheim, B. 2009. Social image and the 50-50 norm. A theoretical and experimental analysis of audience effects. Econometrica 77, 1607--1636.

[35]

Trope, Y., and Nira, L. 2010. Construal-level theory of psychological distance. Psychol. Rev. 117, 440--463.

[36]

Agerström, J., and Björklund, F. (2009a). Moral concerns are greater for temporally distant events and are moderated by value strength. Soc. Cogn. 27, 261--282.

[37]

Agerström, J., and Björklund, F. (2009b). Temporal distance and moral concerns: Future morally questionable behavior is perceived as more wrong and evokes stronger prosocial intentions. Basic App. Soc. Psychol. 31, 49--59.

[38]

Sanna, L., Chang, E., Parks, C., and Kennedy, L. (2009). Construing collective concerns: Increasing cooperation by broadening construals in social dilemmas. Psychol. Sci. 20, 1319--1321.

[39]

Henderson, M., Trope, Y., and Carnevale, P. 2006. Negotiation from a near and distant time perspective. J. Pers. Soc. Psychol. 91, 712--729.

[40]

Giacomantonio, M., De Dreu, C., Shalvi, S., Sligte, D., and Leder, S. 2010. Psychological distance boosts value-behavior correspondence in ultimatum bargaining and integrative negotiation. J. Exp. Soc. Psychol. 46, 824--829.

[41]

Bacharach, S., and Lawler, E. 1981. Bargaining: Power, tactics, and outcomes. San Francisco: Jossey-Bass.

[42]

Kelley, H., and Thibaut, J. 1978. Interpersonal relations: A theory of interdependence. New York: Academic Press.

[43]

Pinkley, R., Neale, M., and Bennett, R. 1994. The impact of alternatives to settlement in dyadic negotiation. Organ. Behav. Human Dec. Proc. 57, 97--116.

[44]

De Dreu, C. 1995. Coercive power and concession making in bilateral negotiation. J. Confl. Resolut. 39, 646--670.

[45]

Lawler, E. 1992. Power processes in bargaining. Sociol. Q. 33, 17--34.

[46]

Messick, D., and McClintock, C. 1968. Motivational bases of choice in experimental games. J. Exp. Soc. Psychol. 4, 1--25.

[47]

Van Lange, P. 1999. The pursuit of joint outcomes and equality in outcomes: An integrative model of social value orientation. J. Pers. Soc. Psychol. 77, 337--349.

[48]

Bogaert, S., Boone, C., and Declerck, C. 2008. Social value orientation and cooperation in social dilemmas: A review and conceptual model. Brit. J. Soc. Psychol. 47, 453--480.

[49]

Balliet, D., Parks, C., and Joireman, J. 2009. Social value orientation and cooperation in social dilemmas: A meta-analysis. Group Process. Interg. 12, 533--547.

[50]

Güth, W., Schmittberger, R., and Schwarze, B. 1982. An experimental analysis of ultimatum bargaining. J. Econ. Behav. & Organ. 3, 367--388.

[51]

J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr, H. Gintis, and R. McElreath. 2001. In search of homo economicus: behavioral experiments in 15 small-scale societies. Am. Econ. Rev. 91, 73--78.

[52]

Camerer, C., and Thaler, R. 1995. Ultimatums, dictators, and manners. J. Econ. Perspect., 9, 209--219.

[53]

Yamagishi, T., Horita, H., Shinada, M., Tanida, S., and Cook, K. 2009. The private rejection of unfair offers and emotional commitment. Proc. Nat. Acad. Sci. 106, 11520--11523.

[54]

Bolton, G., and Zwick, R. 1995. Anonymity versus punishment in ultimatum bargaining. Games Econ. Behav. 10, 95--121.

[55]

Paolacci, G., Chandler, J., and Ipeirotis, P. 2010. Running experiments on Amazon Mechanical Turk. Judgm. Decis. Mak. 5, 411--419.

[56]

Murphy, R., Ackermann, K., and Handgraaf, M. 2011. Measuring social value orientation. Judgm. Decis. Mak. 6, 771--781.

[57]

de Melo, C., Carnevale, P., and Gratch, J. 2014. Social categorization and cooperation between humans and computers. In Proceedings of the Annual Meeting of the Cognitive Science Society (CogSci'14).

[58]

Simon, H. 1972. Theories of bounded rationality. In Decision and Organization, McGuire, C., and Radner, R., Eds. North-Holland Publishing Company, 161--176.

[59]

Rand, D., and Nowak, M. Human cooperation. 2013. Trends Cogn. Sci. 17, 413--425. In Proceedings of the Autonomous Agents and Multi-Agent Systems Conference (AAMAS'04).

[60]

Grosz, B., Kraus, S., and Talman, S. 2004. The influence of social dependencies on decision-making: Initial investigations with a new game. In

[61]

Breazeal, C. 2003. Toward sociable robots. Robotics and autonomous systems 42: 167--175.

[62]

Leite, I., Martinho, C., and Paiva, A. 2013. Social robots for long-term interaction. Int. J. Soc. Robotic. 5, 291--308.

[63]

Arkin, R. Ethical robots in warfare. IEEE Technol. Soc. Mag. 28, 30--33.

Cited By

Cohen RSchaekermann MLiu SCormier MElkind EVeloso MAgmon NTaylor M(2019)Trusted AI and the Contribution of Trust Modeling in Multiagent SystemsProceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems10.5555/3306127.3331890(1644-1648)Online publication date: 8-May-2019
https://dl.acm.org/doi/10.5555/3306127.3331890
Mell JLucas GGratch JAndre EKoenig SDastani MSukthankar G(2018)Welcome to the Real WorldProceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems10.5555/3237383.3237884(1250-1257)Online publication date: 9-Jul-2018
https://dl.acm.org/doi/10.5555/3237383.3237884
Mell JGratch JBaarslag TAydoğan RJonker CBogdanovych ARichards DSimoff SPelachaud CHeylen DTrescak T(2018)Results of the First Annual Human-Agent League of the Automated Negotiating Agents CompetitionProceedings of the 18th International Conference on Intelligent Virtual Agents10.1145/3267851.3267907(23-28)Online publication date: 5-Nov-2018
https://dl.acm.org/doi/10.1145/3267851.3267907
Show More Cited By

Index Terms

"Do As I Say, Not As I Do": Challenges in Delegating Decisions to Automated Agents
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
      1. Computer supported cooperative work

Recommendations

Increasing Fairness by Delegating Decisions to Autonomous Agents
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems

There has been growing interest in autonomous agents that act on our behalf, or represent us, across various domains such as negotiation, transportation, health, finance, and defense. As these agent representatives become immersed in society, it is ...
Automated agents' behavior in the trust-revenge game incomparison to other cultures
AAMAS '14: Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems

Agents that interact with humans are known to benefit from modeling them. Therefore, when designing agents intended for interaction with automated agents, it is crucial to model the other agents. However, little is known about how to model automated ...
Training with automated agents improves people's behavior in negotiation and coordination tasks

There is inconclusive evidence whether practicing tasks with computer agents improves people's performance on these tasks. This paper studies this question empirically using extensive experiments involving bilateral negotiation and three-player ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

AAMAS '16: Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems

May 2016

1580 pages

ISBN:9781450342391

General Chairs:
Catholijn M. Jonker
TU Delft, Netherlands
,
Stacy Marsella
University of Southern California, USA
,
Program Chairs:
John Thangarajah
RMIT University. Australia
,
Karl Tuyls
University of Liverpool, UK

Sponsors

IFAAMAS

In-Cooperation

SIGAI: ACM Special Interest Group on Artificial Intelligence

Publisher

International Foundation for Autonomous Agents and Multiagent Systems

Richland, SC

Publication History

Published: 09 May 2016

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Air Force Office of Scientific Research
National Science Foundation

Conference

AAMAS '16

Sponsor:

AAMAS '16: International Conference on Agents and Multiagent Systems

May 9 - 13, 2016

Singapore, Singapore

Acceptance Rates

AAMAS '16 Paper Acceptance Rate 137 of 550 submissions, 25%;

Overall Acceptance Rate 1,155 of 5,036 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
195
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)9

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cohen RSchaekermann MLiu SCormier MElkind EVeloso MAgmon NTaylor M(2019)Trusted AI and the Contribution of Trust Modeling in Multiagent SystemsProceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems10.5555/3306127.3331890(1644-1648)Online publication date: 8-May-2019
https://dl.acm.org/doi/10.5555/3306127.3331890
Mell JLucas GGratch JAndre EKoenig SDastani MSukthankar G(2018)Welcome to the Real WorldProceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems10.5555/3237383.3237884(1250-1257)Online publication date: 9-Jul-2018
https://dl.acm.org/doi/10.5555/3237383.3237884
Mell JGratch JBaarslag TAydoğan RJonker CBogdanovych ARichards DSimoff SPelachaud CHeylen DTrescak T(2018)Results of the First Annual Human-Agent League of the Automated Negotiating Agents CompetitionProceedings of the 18th International Conference on Intelligent Virtual Agents10.1145/3267851.3267907(23-28)Online publication date: 5-Nov-2018
https://dl.acm.org/doi/10.1145/3267851.3267907
Baarslag TKaisers MGerding EJonker CGratch J(2017)When will negotiation agents be able to represent us? the challenges and opportunities for autonomous negotiatorsProceedings of the 26th International Joint Conference on Artificial Intelligence10.5555/3171837.3171943(4684-4690)Online publication date: 19-Aug-2017
https://dl.acm.org/doi/10.5555/3171837.3171943
de Melo CMarsella SGratch JLarson KWinikoff MDas SDurfee E(2017)Increasing Fairness by Delegating Decisions to Autonomous AgentsProceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems10.5555/3091125.3091188(419-425)Online publication date: 8-May-2017
https://dl.acm.org/doi/10.5555/3091125.3091188

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents