Abstract
The rapidly evolving urban air mobility (UAM) develops the heavy demand for public air transport tasks and poses great challenges to safe and efficient operation in low-altitude urban airspace. In this paper, the operation conflict is managed in the strategic phase with multi-agent reinforcement learning (MARL) in dynamic environments. To enable efficient operation, the aircraft flight performance is integrated into the process of multi-resolution airspace design, trajectory generation, conflict management, and MARL learning. The demand and capacity balancing (DCB) issue, separation conflict, and block unavailability introduced by wind turbulence are resolved by the proposed the multi-agent asynchronous advantage actor-critic (MAA3C) framework, in which the recurrent actor-critic networks allow the automatic action selection between ground delay, speed adjustment, and flight cancellation. The learned parameters in MAA3C are replaced with random values to compare the performance of trained models. Simulated training and test experiments performed on a small urban prototype and various combined use cases suggest the superiority of the MAA3C solution in resolving conflicts with complicated wind fields. And the generalization, scalability, and stability of the model are also demonstrated while applying the model to complex environments.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Availability of data and materials
The datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.
Code Availability
The code is available in https://github.com/ChengHuang-CH/uam_maa3c_wind_turbulence.git.
References
De l’aviation civile internationale, O.: Global Air Traffic Management Operational Concept (ICAO) (2005)
Unmanned aircraft system (uas) traffic management (utm) - concept of operations v2.0. Tech, Rep. 1-68 federal aviation administration (2020)
Kopardekar, P., etr al.: Unmanned Aircraft System Traffic Management (utm) Concept of Operations, pp 1–16. AIAA, Reston (2016)
Urban air mobility (uam) - concept of operations v1.0. Tech, Rep. 1-37 federal aviation administration (2020)
Hill, B.P., et al.: Uam vision concept of operations (conops) uam maturity level (uml) 4 (2020)
Drone dcb concept and process: Tech. Rep. 1-261, SESAR (2021)
Acevedo, J.J., Castaño, Á.R., Andrade-Pineda, J.L., Ollero, A.: A 4d Grid Based Approach for Efficient Conflict Detection in Large-Scale Multi-uav Scenarios, pp 18–23. IEEE, Cranfield, UK (2019)
Mohamed Salleh, M.F.B., et al.: In Preliminary concept of adaptive urban airspace management for unmanned aircraft operations 2260 (2018)
Pang, B., Dai, W., Ra, T., Low, K.H.: A Concept of Airspace Configuration and Operational Rules for uas in Current Airspace, pp 1–9. IEEE, San Antonio, TX, USA (2020)
Dai, W., Pang, B., Low, K.H.: Conflict-free four-dimensional path planning for urban air mobility considering airspace occupancy. Aerospace Science and Technology 107154 (2021)
Bertsimas, D., Patterson, S.S.: The air traffic flow management problem with enroute capacities. Oper. Res. 46(3), 406–422 (1998)
Zhang, Y., Su, R., Li, Q., Cassandras, C.G., Xie, L.: Distributed flight routing and scheduling for air traffic flow management. IEEE Trans. Intell. Transp. Syst. 18(10), 2681–2692 (2017)
Spatharis, C., et al.: Multiagent reinforcement learning methods to resolve demand capacity balance problems 1–9 (2018)
Huang, C., Xu, Y.: Integrated Frameworks of Unsupervised, Supervised and Reinforcement Learning for Solving Air Traffic Flow Management Problem, pp 1–10. IEEE, San Antonio, TX, USA (2021)
Spatharis, C., et al.: Hierarchical multiagent reinforcement learning schemes for air traffic management. Neural Comput. & Applic. 1–13 (2021)
Xie, Y., Gardi, A., Sabatini, R.: Reinforcement Learning-Based Flow Management Techniques for Urban Air Mobility and Dense Low-Altitude Air Traffic Operations, pp 1–10. IEEE, San Antonio, TX, USA (2021)
Pham, D. -T., Tran, N.P., Goh, S.K., Alam, S., Duong, V.: Reinforcement Learning for Two-Aircraft Conflict Resolution in the Presence of Uncertainty, pp 1–6. IEEE, Danang, Vietnam (2019)
Tran, P.N., Pham, D. -T., Goh, S.K., Alam, S., Duong, V.: An interactive conflict solver for learning air traffic conflict resolutions. Journal of Aerospace Information Systems 17(6), 271–277 (2020)
Abichandani, P., Lobo, D., Ford, G., Bucci, D., Kam, M.: Wind measurement and simulation techniques in multi-rotor small unmanned aerial vehicles. IEEE Access 8, 54910–54927 (2020)
Kadaverugu, R., Purohit, V., Matli, C., Biniwale, R.: Improving accuracy in simulation of urban wind flows by dynamic downscaling wrf with openfoam. Urban Climate 38, 100912 (2021)
Allison, S., Bai, H., Jayaraman, B.: In Modeling trajectory performance of quadrotors under wind disturbances. 1237 (2018)
Beard, R.W., McLain, T.W.: Small Unmanned Aircraft. Princeton University Press, Princeton (2012)
Zhang, K., Yang, Z., Başar, T.: Multi-agent reinforcement learning: a selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control 321–384 (2021)
Papoudakis, G., Christianos, F., Rahman, A., Albrecht, S.V.: Dealing with non-stationarity in multi-agent deep reinforcement learning. arXiv:1906.04737 (2019)
Mnih, V., et al.: Asynchronous Methods for Deep Reinforcement Learning, pp 1928–1937. PMLR, New York (2016)
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An open urban driving simulator. 1–16 (2017)
Jasak, H.: Openfoam: open source cfd in research and industry. International Journal of Naval Architecture and Ocean Engineering 1(2), 89–94 (2009)
Funding
This research was partially supported by grants from the Funds of China Scholarship Council (202008420248).
Author information
Authors and Affiliations
Contributions
Cheng Huang contributed to the algorithm design, implementation, and writing of this paper; Ivan Petrunin and Antonios Tsourdos contributed to the result analysis and revision of the manuscript.
Corresponding author
Ethics declarations
Ethics approval
No applicable as this study does not contain biological applications.
Consent to participate
All authors of this research paper have consented to participate in the research study.
Consent for Publication
All authors of this research paper have read and approved the submitted version.
Conflict of Interests
The authors have no relevant financial or nonfinancial interests to disclose.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Huang, C., Petrunin, I. & Tsourdos, A. Strategic Conflict Management using Recurrent Multi-agent Reinforcement Learning for Urban Air Mobility Operations Considering Uncertainties. J Intell Robot Syst 107, 20 (2023). https://doi.org/10.1007/s10846-022-01784-0
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-022-01784-0