research-article

Multiagent Reinforcement Learning Methods to Resolve Demand Capacity Balance Problems

Authors:

Christos Spatharis,

Theocharis Kravaris,

George A. Vouros,

Konstantinos Blekas,

Georgios Chalkiadakis,

Jose Manuel Cordero Garcia,

Esther Calvo FernandezAuthors Info & Claims

SETN '18: Proceedings of the 10th Hellenic Conference on Artificial Intelligence

Article No.: 2, Pages 1 - 9

https://doi.org/10.1145/3200947.3201010

Published: 09 July 2018 Publication History

Abstract

In this article, we explore the computation of joint policies for autonomous agents to resolve congestions problems in the air traffic management (ATM) domain. Agents, representing flights, have limited information about others' payoffs and preferences, and need to coordinate to achieve their tasks while adhering to operational constraints. We formalize the problem as a multiagent Markov decision process (MDP) towards deciding flight delays to resolve demand and capacity balance (DCB) problems in ATM. To this end, we present multiagent reinforcement learning methods that allow agents to interact and form own policies in coordination with others. Experimental study on real-world cases, confirms the effectiveness of our approach in resolving the demand-capacity balance problem.

References

[1]

2011. Air Traffic Flow and Capacity Management (ATFCM). (2011). Eurocontrol, http://www.eurocontrol.int/articles/air-traffic-flow-and-capacity-management.

[2]

Adrian K Agogino and Kagan Tumer. 2012. A multiagent approach to managing air traffic flow. Autonomous Agents and Multi-Agent Systems 24, 1 (2012), 1--25.

Digital Library

[3]

Baraa Munqith Albaker and Nasrudin Abd Rahim. 2010. Unmanned aircraft collision avoidance system using cooperative agent-based negotiation approach. Int. J. Simulation, Syst. Sci. Technol 11, 4 (2010), 1--8.

[4]

Ana L. C. Bazzan, Joachim Wahle, and Franziska Klügl. 1999. Agents in Traffic Modelling - From Reactive to Social Behaviour. In KI-99: Advances in Artificial Intelligence, 23rd Annual German Conference on Artificial Intelligence, Bonn, Germany, September 13-15, 1999, Proceedings. 303--306.

Digital Library

[5]

Craig Boutilier. 1996. Planning, Learning and Coordination in Multiagent Decision Processes. In Proceedings of the Sixth Conference on Theoretical Aspects of Rationality and Knowledge, De Zeeuwse Stromen, The Netherlands, March 17-20 1996. 195--210.

Digital Library

[6]

Caroline Claus and Craig Boutilier. 1998. The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems. In Proceedings of the Fifteenth National/Tenth Conference on Artificial Intelligence/Innovative Applications of Artificial Intelligence (AAAI '98/IAAI '98). American Association for Artificial Intelligence, Menlo Park, CA, USA, 746--752. http://dl.acm.org/citation.cfm?id=295240.295800

Digital Library

[7]

Andrew J Cook and Graham Tanner. 2015. European airline delay cost reference values. (2015). http://www.eurocontrol.int/publications/european-airline-delay-cost-reference-values.

[8]

K. Dresner and P. Stone. 2004. Multiagent traffic management: A reservationbased intersection control mechanism. In Proceedings of the 3rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS '04). 530--537.

Digital Library

[9]

Carlos Guestrin, Michail Lagoudakis, and Ronald Parr. 2002. Coordinated Reinforcement Learning. In In Proceedings of the ICML-2002 The Nineteenth International Conference on Machine Learning. 227--234.

Digital Library

[10]

Carlos Ernesto Guestrin. 2003. Planning Under Uncertainty in Complex Structured Environments. Ph.D. Dissertation. Stanford, CA, USA. Advisor(s) Koller, Daphne. AAI3104233.

[11]

Jayesh K. Gupta, Maxim Egorov, and Mykel Kochenderfer. 2017. Cooperative Multiagent Control Using Deep Reinforcement Learning. In Adaptive Learning Agents at AAMAS-2017 (ALA2017) workshop.

[12]

Jelle R. Kok and Nikos Vlassis. 2006. Collaborative Multiagent Reinforcement Learning by Payoff Propagation. J. Mach. Learn. Res. 7 (Dec. 2006), 1789--1828. http://dl.acm.org/citation.cfm?id=1248547.1248612

Digital Library

[13]

Theocharis Kravaris, George A. Vouros, Christos Spatharis, Konstantinos Blekas, Georgios Chalkiadakis, and Jose Manuel Cordero Garcia. 2017. Learning Policies for Resolving Demand-Capacity Imbalances During Pre-tactical Air Traffic Management. In Multiagent System Technologies - 15th German Conference, MATES 2017, Leipzig, Germany, August 23-26, 2017, Proceedings. 238--255.

[14]

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2016. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR).

[15]

Kleanthis Malialis, Sam Devlin, and Daniel Kudenko. 2016. Resource Abstraction for Reinforcement Learning in Multiagent Congestion Problems. In Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS '16). 503--511.

Digital Library

[16]

C. Meyers. 2006. Network flow problems and congestion games: complexity and approximation results. Ph.D. Dissertation. Cambridge, MA, USA.

Digital Library

[17]

I. Milchtaich. 2004. Social Optimality and Cooperation in Nonatomic Congestion Games. Journal of Economic Theory 114 (2004), 56--87.

[18]

Judea Pearl. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.

Digital Library

[19]

Michal Penn, Maria Polukarov, and Moshe Tennenholtz. 2005. Congestion games with failures. In Proceedings 6th ACM Conference on Electronic Commerce (EC-2005), Vancouver, BC, Canada, June 5-8, 2005. 259--268.

Digital Library

[20]

R. W. Rosenthal. 1973. A Class of Games Possessing Pure-Strategy Nash Equilibria. International Journal of Game Theory 2 (1973), 65--67.

Digital Library

[21]

David Sislak, Přemysl Volf, and Michal Pechoucek. 2011. Agent-based cooperative decentralized airplane-collision avoidance. IEEE Transactions on Intelligent Transportation Systems 12, 1 (2011), 36--46.

Digital Library

[22]

Richard S Sutton and Andrew G Barto. 1998. Reinforcement learning: An introduction. MIT press Cambridge.

Digital Library

Cited By

Chen YZhao YFei FYang H(2024)Optimizing Large-Scale Demand and Capacity Balancing in Air Traffic Flow Management Using Deep Neural NetworksAerospace10.3390/aerospace1112096611:12(966)Online publication date: 25-Nov-2024
https://doi.org/10.3390/aerospace11120966
Nguyen NPham DDuong V(2024)A Hierarchical Multi-Agent Coordinated Decision-Making for Strategic Slot Allocation2024 AIAA DATC/IEEE 43rd Digital Avionics Systems Conference (DASC)10.1109/DASC62030.2024.10749348(1-10)Online publication date: 29-Sep-2024
https://doi.org/10.1109/DASC62030.2024.10749348
Razzaghi PTabrizian AGuo WChen STaye AThompson EBregeon ABaheri AWei P(2024)A survey on reinforcement learning in aviation applicationsEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108911136(108911)Online publication date: Oct-2024
https://doi.org/10.1016/j.engappai.2024.108911
Show More Cited By

Multiagent Reinforcement Learning Methods to Resolve Demand Capacity Balance Problems
1. Computing methodologies
  1. Artificial intelligence
    1. Distributed artificial intelligence

Recommendations

Hierarchical multiagent reinforcement learning schemes for air traffic management
Abstract
In this work we investigate the use of hierarchical multiagent reinforcement learning methods for the computation of policies to resolve congestion problems in the air traffic management domain. To address cases where the demand of airspace use ...
Resource Abstraction for Reinforcement Learning in Multiagent Congestion Problems
AAMAS '16: Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems

Real-world congestion problems (e.g. traffic congestion) are typically very complex and large-scale. Multiagent reinforcement learning (MARL) is a promising candidate for dealing with this emerging complexity by providing an autonomous and distributed ...
Mediated Multi-Agent Reinforcement Learning
AAMAS '23: Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems

The majority of Multi-Agent Reinforcement Learning (MARL) literature equates the cooperation of self-interested agents in mixed environments to the problem of social welfare maximization, allowing agents to arbitrarily share rewards and private ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SETN '18: Proceedings of the 10th Hellenic Conference on Artificial Intelligence

July 2018

339 pages

ISBN:9781450364331

DOI:10.1145/3200947

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

EETN: Hellenic Artificial Intelligence Society
UOP: University of Patras
University of Thessaly: University of Thessaly, Volos, Greece

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 July 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SETN '18

SETN '18: 10th Hellenic Conference on Artificial Intelligence

July 9 - 12, 2018

Patras, Greece

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
233
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)3

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen YZhao YFei FYang H(2024)Optimizing Large-Scale Demand and Capacity Balancing in Air Traffic Flow Management Using Deep Neural NetworksAerospace10.3390/aerospace1112096611:12(966)Online publication date: 25-Nov-2024
https://doi.org/10.3390/aerospace11120966
Nguyen NPham DDuong V(2024)A Hierarchical Multi-Agent Coordinated Decision-Making for Strategic Slot Allocation2024 AIAA DATC/IEEE 43rd Digital Avionics Systems Conference (DASC)10.1109/DASC62030.2024.10749348(1-10)Online publication date: 29-Sep-2024
https://doi.org/10.1109/DASC62030.2024.10749348
Razzaghi PTabrizian AGuo WChen STaye AThompson EBregeon ABaheri AWei P(2024)A survey on reinforcement learning in aviation applicationsEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108911136(108911)Online publication date: Oct-2024
https://doi.org/10.1016/j.engappai.2024.108911
Chen YXu YHu M(2023)General multi-agent reinforcement learning integrating heuristic-based delay priority strategy for demand and capacity balancingTransportation Research Part C: Emerging Technologies10.1016/j.trc.2023.104218153(104218)Online publication date: Aug-2023
https://doi.org/10.1016/j.trc.2023.104218
Huang CPetrunin ITsourdos A(2023)Strategic Conflict Management using Recurrent Multi-agent Reinforcement Learning for Urban Air Mobility Operations Considering UncertaintiesJournal of Intelligent & Robotic Systems10.1007/s10846-022-01784-0107:2Online publication date: 26-Jan-2023
https://doi.org/10.1007/s10846-022-01784-0
Mas-Pujol SSalamí EPastor E(2022)Image-Based Multi-Agent Reinforcement Learning for Demand–Capacity BalancingAerospace10.3390/aerospace91005999:10(599)Online publication date: 14-Oct-2022
https://doi.org/10.3390/aerospace9100599
Huang CPetrunin ITsourdos A(2022)Strategic Conflict Management for Performance-based Urban Air Mobility Operations with Multi-agent Reinforcement Learning2022 International Conference on Unmanned Aircraft Systems (ICUAS)10.1109/ICUAS54217.2022.9836139(442-451)Online publication date: 21-Jun-2022
https://doi.org/10.1109/ICUAS54217.2022.9836139
Tian HPresa-Reyes MTao YWang TPouyanfar SMiguel ALuis SShyu MChen SIyengar S(2021)Data Analytics for Air Travel Data: A Survey and New PerspectivesACM Computing Surveys10.1145/346902854:8(1-35)Online publication date: 4-Oct-2021
https://dl.acm.org/doi/10.1145/3469028
Huang CXu Y(2021)Integrated Frameworks of Unsupervised, Supervised and Reinforcement Learning for Solving Air Traffic Flow Management Problem2021 IEEE/AIAA 40th Digital Avionics Systems Conference (DASC)10.1109/DASC52595.2021.9594397(1-10)Online publication date: 3-Oct-2021
https://doi.org/10.1109/DASC52595.2021.9594397
Chen YXu YHu MYang L(2021)Demand and Capacity Balancing Technology Based on Multi-agent Reinforcement Learning2021 IEEE/AIAA 40th Digital Avionics Systems Conference (DASC)10.1109/DASC52595.2021.9594343(1-9)Online publication date: 3-Oct-2021
https://doi.org/10.1109/DASC52595.2021.9594343
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents