Learning the IPA market with individual and social rewards
Pages 123 - 138
Abstract
Market-based mechanisms offer a promising approach for distributed resource allocation. In this paper we consider the Iterative Price Adjustment, a pricing mechanism that can be used in commodity-market resource allocation systems. We address the scenario where agents use utility functions to describe preferences in the allocation and learn demand functions optimized for the market by Reinforcement Learning. In particular, we investigate reward functions based on the individual utilities of the agents and the Social Welfare of the market. We also evaluate the quality of demand functions obtained throughout the learning process with the aim of analyzing its influence on the behavior of the agents and exploring how much learning is enough, so the amount required can be reduced. This investigation shows that both reward functions deliver similar results when the market consists of only learning agents. We further investigate this behavior and present its theoretical-experimental explanation.
References
[1]
S. Abdallah and V. Lesser, Learning the task allocation game, in: Proceedings of the Fifth International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS'06), New York, NY, USA, ACM Press, 2006, pp. 850-857.
[2]
A. Au Young, B. Chun, A. Snoeren and A. Vahdat, Resource allocation in federated distributed computing infrastructures, 2004.
[3]
M.H. Bowling and M.M. Veloso, Multiagent learning using a variable learning rate, Artificial Intelligence 136(2) (2002) 215-250.
[4]
R. Buyya, D. Abramson and S. Venugopal, The grid economy, in: Proceedings of the IEEE, M. Parashar and C. Lee, eds., volume 93 of Special Issue on Grid Computing, IEEE Press, New Jersey, USA, Mar 2005, pp. 698-714.
[5]
Y. Chevaleyre, P.E. Dunne, U. Endriss, J. Lang, M. Lemaître, N. Maudet, J. Padget, S. Phelps, J.A. Rodríguez-Aguilar and P. Sousa, Issues in multiagent resource allocation, Informatica 30 (2006) 3-31.
[6]
L. Chunlin and L. Layuan, Pricing and resource allocation in computational grid with utility functions, in: Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC'05), volume II, Washington, DC, USA, IEEE Computer Society, Apr 2005, pp. 175-180.
[7]
C. Claus and C. Boutilier, The dynamics of reinforcement learning in cooperative multiagent systems, in: Proceedings of the Fifteenth National Conference on Artificial Intelligence, Menlo Park, CA, USA, AAAI, 1998, pp. 746-752.
[8]
V. Conitzer and T. Sandholm, Self-interested automated mechanism design and implications for optimal combinatorial auctions, in: EC '04: Proceedings of the 5th ACM conference on Electronic commerce, New York, NY, USA, ACM, 2004, pp. 132-141.
[9]
B.C. Csáji and L. Monostori, Adaptive algorithms in distributed resource allocation, in: Proceedings of the 6th international workshop on emergent synthesis (IWES 06), 2006.
[10]
H. Everett, Generalized lagrange multiplier method for solving problems of optimum allocation of resources, Operations Research 11(3) (1963) 399-417.
[11]
A.M. Fink, Equilibrium in a stochastic n-person game, Journal of Science in Hiroshima University, Series A-I(28) (1964) 89- 93.
[12]
I. Foster and C. Kesselman, eds., The Grid: Blueprint for a Future Computing Infrastructure, Morgan-Kaufmann, Sao Mateo, USA, 1999.
[13]
N. Fulda and D. Ventura, Predicting and preventing coordination problems in cooperative q-learning systems, in: IJCAI, M.M. Veloso, ed., 2007, pp. 780-785.
[14]
A. Galstyan, K. Czajkowski and K. Lerman, Resource allocation in the grid using reinforcement learning, in: Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS'04), volume 3, Washington, DC, USA, IEEE Computer Society, 2004, pp. 1314-1315.
[15]
S. Gjerstad and J. Dickhaut, Price formation in double auctions, in: E-Commerce Agents, Marketplace Solutions, Security Issues, and Supply and Demand, London, UK, 2001, Springer-Verlag, pp. 106-134.
[16]
E.R. Gomes and R. Kowalczyk, Learning in market-based resource allocation, in: Proceedings of the 6th IEEE/ACIS International Conference on Computer and Information Science (ICIS 2007), IEEE Computer Society, 2007, pp. 475-482.
[17]
E.R. Gomes and R. Kowalczyk, Reinforcement learning with utility-aware agents for market-based resource allocation, in: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS'07), 2007.
[18]
D. Grosu and A. Das, Auctioning resources in grids: model and protocols, Concurrency and Computation: Practice and Experience 18(15) (2006) 1909-1927.
[19]
J. Hofbauer and K. Sigmund, Evolutionary Games and Population Dynamics, Cambridge University Press, 1998.
[20]
P. Jennergren, A price schedules decomposition algorithm for linear programming problems, Econometrica 41(5) (Sep 1973) 965-980.
[21]
J.O. Kephart and G. Tesauro, Pseudo-convergent q-learning by competitive pricebots, in: Proceedings of the Seventeenth International Conference on Machine Learning (ICML'00), San Francisco, CA, USA, Morgan Kaufmann Publishers Inc, 2000, pp. 463-470.
[22]
J.O. Kephart and G.J. Tesauro, Pseudo-convergent Q-learning by competitive pricebots, in: Proc. 17th International Conf. on Machine Learning, Morgan Kaufmann, San Francisco, CA, 2000, pp. 463-470.
[23]
V. Könönen, Dynamic pricing based on asymmetric multiagent reinforcement learning: Research articles, Int. J. Intell. Syst. 21(1) (2006) 73-98.
[24]
K. Lai, L. Rasmusson, E. Adar, L. Zhang and B.A. Huberman, Tycoon: An implementation of a distributed, market-based resource allocation system, Multiagent Grid Syst. 1(3) (2005) 169-182.
[25]
L. Panait and S. Luke, Cooperative multi-agent learning: The state of the art, Autonomous Agents and Multi-Agent Systems 11(3) (2005) 387-434.
[26]
L. Panait, K. Tuyls and S. Luke, Theoretical advantages of lenient learners: An evolutionary game theoretic perspective, Journal of Machine Learning Research 9(Mar) (2008) 423- 457.
[27]
D. Pardoe, P. Stone, M. Saar-Tsechansky and K. Tomak, Adaptive mechanism design: a metalearning approach, in: Proceedings of the 8th International Conference on Electronic Commerce, New York, NY, USA, ACM Press, 2006, pp. 92-102.
[28]
C. Preist, A. Byde and C. Bartolini, Economic dynamics of agents in multiple auctions, in: AGENTS '01: Proceedings of the fifth international conference on Autonomous agents, New York, NY, USA, ACM Press, 2001, pp. 545-551.
[29]
B. Schnizler, D. Neumann, D. Veit, M. Reinicke, W. Streitberger, T. Eymann, F. Freitag, I. Chao and P. Chacin, Catnets - wp 1: Theoretical and computational basis, 2005.
[30]
S. Sen and M. Sekaran, Individual learning of coordination knowledge, Journal of Experimental & Theoretical Artificial Intelligence 10(3) (1998) 333-356.
[31]
Y. Shoham, R. Powers and T. Grenager, Multi-agent reinforcement learning: a critical survey, 2003.
[32]
K. Subramoniam, M. Maheswaran and M. Toulouse, Towards a micro-economic model for resource allocation in grid computing systems, in: IEEE Canadian Conference on Electrical and Computer Engineering, CCECE2002, volume 2, 2002, pp. 782-785.
[33]
R.S. Sutton and A.G. Barto, Reinforcement Learning: An Introduction , MIT Press, Cambridge, MA, 1998.
[34]
G. Tesauro and J.O. Kephart, Pricing in agent economies using multi-agent q-learning, Autonomous Agents and Multi-Agent Systems 5(3) (2002) 289-304.
[35]
C.J.C.H. Watkins, Learning from Delayed Rewards, PhD thesis, King's College, Cambridge, UK, 1989.
[36]
R. Wolski, J.S. Plank, J. Brevik and T. Bryan, Analyzing market-based resource allocation strategies for the computational grid, International Journal of High Performance Computing Applications 15(10) (Aug 2001) 258-281.
[37]
T. Wu, N. Ye and D. Zhang, Comparison of distributed methods for resource allocation, International Journal of Production Research 43(3) (2005) 515-536.
[38]
C.S. Yeo and R. Buyya, A taxonomy of market-based resource management systems for utility-driven cluster computing, Softw. Pract. Exper. 36(13) (2006) 1381-1419.
- Learning the IPA market with individual and social rewards
Recommendations
Individual and Social Behaviour in the IPA Market with RL
SBIA '08: Proceedings of the 19th Brazilian Symposium on Artificial Intelligence: Advances in Artificial IntelligenceMarket-based mechanisms offer a promising approach for distributed resource allocation. Machine Learning has been proposed to influence and optimize market-based resource allocation. In particular, Reinforcement Learning (RL) has been used to improve ...
Non-symmetric Preferences in the IPA Market with Reinforcement Learning
WI-IAT '08: Proceedings of the 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology - Volume 02Machine Learning has been proposed to support and optimize market-based resource allocation. In particular, Reinforcement Learning (RL) has been used to improve the allocation in terms of the utility received by resource requesting agents in the ...
Learning the IPA Market with Individual and Social Rewards
IAT '07: Proceedings of the 2007 IEEE/WIC/ACM International Conference on Intelligent Agent TechnologyMarket-based mechanisms offer a promising approach for distributed allocation of resources without centralized control. One of those mechanisms is the Iterative Price Adjustment (IPA). Under standard assumptions, the IPA uses demand functions that do ...
Comments
Information & Contributors
Information
Published In
Publisher
IOS Press
Netherlands
Publication History
Published: 01 April 2009
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 17 Jan 2025