Double Deep Q-Learning-Based Path Selection and Service Placement for Latency-Sensitive Beyond 5G Applications
Pages 5097 - 5110
Abstract
Nowadays, as the need for capacity continues to grow, entirely novel services are emerging. A solid cloud-network integrated infrastructure is necessary to supply these services in a real-time responsive, and scalable way. Due to their diverse characteristics and limited capacity, communication and computing resources must be collaboratively managed to unleash their full potential. Although several innovative methods have been proposed to orchestrate the resources, most ignored network resources or relaxed the network as a simple graph, focusing only on cloud resources. This paper fills the gap by studying the joint problem of communication and computing resource allocation, dubbed CCRA, including function placement and assignment, traffic prioritization, and path selection considering capacity constraints and quality requirements, to minimize total cost. We formulate the problem as a non-linear programming model and propose two approaches, dubbed B&B-CCRA and WF-CCRA, based on the Branch & Bound and Water-Filling algorithms to solve it when the system is fully known. Then, for partially known systems, a Double Deep Q-Learning (DDQL) architecture is designed. Numerical simulations show that B&B-CCRA optimally solves the problem, whereas WF-CCRA delivers near-optimal solutions in a substantially shorter time. Furthermore, it is demonstrated that DDQL-CCRA obtains near-optimal solutions in the absence of request-specific information.
References
[1]
T. Taleb et al., “Towards supporting XR services: Architecture and enablers,” IEEE Internet Things J., vol. 10, no. 4, pp. 3567–3586, Feb. 2023.
[2]
L. Corneo et al., “Surrounded by the clouds: A comprehensive cloud reachability study,” in Proc. Web Conf., New York, NY, USA: Association for Computing Machinery, 2021, pp. 295–304.
[3]
X. Yang, Z. Zho, and B. Huang, “URLLC key technologies and standardization for 6G power Internet of Things,” IEEE Commun. Standards Mag., vol. 5, no. 2, pp. 52–59, Sep./Oct. 2021.
[4]
T. Taleb, I. Afolabi, K. Samdanis, and F. Z. Yousaf, “On multi-domain network slicing orchestration architecture and federated resource control,” IEEE Netw., vol. 33, no. 5, pp. 242–252, Sep./Oct. 2019.
[5]
T. Taleb, P. A. Frangoudis, I. Benkacem, and A. Ksentini, “CDN slicing over a multi-domain edge cloud,” IEEE Trans. Mobile Comput., vol. 19, no. 9, pp. 2010–2027, Sep. 2020.
[6]
Y. Li, J. Huang, Q. Sun, T. Sun, and S. Wang, “Cognitive service architecture for 6G core network,” IEEE Trans. Ind. Inform., vol. 17, no. 10, pp. 7193–7203, Oct. 2021.
[7]
M. Emu, P. Yan, and S. Choudhury, “Latency aware VNF deployment at edge devices for IoT services: An artificial neural network based approach,” in Proc. IEEE Int. Conf. Commun. Workshops, 2020, pp. 1–6.
[8]
X. Vasilakos, M. Bunyakitanon, R. Nejabati, and D. Simeonidou, “Towards low-latent & load-balanced VNF placement with hierarchical reinforcement learning,” in Proc. IEEE Int. Mediterranean Conf. Commun. Netw., 2021, pp. 162–167.
[9]
H. Sami, A. Mourad, H. Otrok, and J. Bentahar, “Demand-driven deep reinforcement learning for scalable fog and service placement,” IEEE Trans. Serv. Comput., vol. 15, no. 5, pp. 2671–2684, Sep./Oct. 2021.
[10]
M. Liu and S. B. Alias, “Cost-efficient virtual network function placement in an industrial edge system: A proposed method,” IEEE Systems, Man, Cybern. Mag., vol. 9, no. 1, pp. 10–17, Jan. 2023.
[11]
N. He et al., “Leveraging deep reinforcement learning with attention mechanism for virtual network function placement and routing,” IEEE Trans. Parallel Distrib. Syst., vol. 34, no. 4, pp. 1186–1201, Apr. 2023.
[12]
M. Iwamoto, A. Suzuki, and M. Kobayashi, “Optimal VNF scheduling for minimizing duration of QoS degradation,” in Proc. IEEE 20th Consum. Commun. Netw. Conf., 2023, pp. 855–858.
[13]
D. H. P. Nguyen, Y.-H. Lien, B.-H. Liu, S.-I. Chu, and T. N. Nguyen, “Virtual network function placement for serving weighted services in NFV-Enabled networks,” IEEE Syst. J., early access, Mar. 30, 2023.
[14]
H. Xuan, Y. Zhou, X. Zhao, and Z. Liu, “Multi-agent deep reinforcement learning algorithm with self-adaption division strategy for VNF-SC deployment in SDN/NFV-Enabled networks,” Appl. Soft Comput., vol. 138, May 2023, Art. no.
[15]
T. Miyamura and A. Misawa, “Joint optimization of optical path provisioning and VNF placement in vCDN,” Opt. Switching Netw., vol. 49, May 2023, Art. no.
[16]
C. Yang, B. Hu, Y. Feng, H. Huang, H. Lai, and J. Tan, “An online service function chain orchestration method for profit maximization in edge computing networks,” Eng. Rep., 2023, Art. no.
[17]
T.-W. Kuo, B.-H. Liou, K. C.-J. Lin, and M.-J. Tsai, “Deploying chains of virtual network functions: On the relation between link and server usage,” IEEE/ACM Trans. Netw., vol. 26, no. 4, pp. 1562–1576, Aug. 2018.
[18]
B. E. Mada, M. Bagaa, T. Tale, and H. Flinck, “Latency-aware service placement and live migrations in 5G and beyond mobile systems,” in Proc. IEEE Int. Conf. Commun., 2020, pp. 1–6, Issn: 1938–1883.
[19]
Q. Zhang, F. Liu, and C. Zeng, “Adaptive interference-aware VNF placement for service-customized 5G network slices,” in Proc. IEEE Conf. Comput. Commun., 2019, pp. 2449–2457, Issn: 2641–9874.
[20]
Q. Yuan, X. Ji, H. Tang, and W. You, “Toward latency-optimal placement and autoscaling of monitoring functions in MEC,” IEEE Access, vol. 8, pp. 41 649–41 658, 2020.
[21]
T. Gao et al., “Cost-efficient VNF placement and scheduling in public cloud networks,” IEEE Trans. Commun., vol. 68, no. 8, pp. 4946–4959, Aug. 2020.
[22]
C. D. Alwis et al., “Survey on 6G frontiers: Trends, applications, requirements, technologies and future research,” IEEE Open J. Commun. Soc., vol. 2, pp. 836–886, 2021.
[23]
M. Shokrnezhad and T. Taleb, “Near-optimal cloud-network integrated resource allocation for latency-sensitive B5G,” in Proc. IEEE Glob. Commun. Conf., Rio De Janeiro, Brazil, 2022.
[24]
J. Lei, S. Deng, Z. Lu, Y. He, and X. Gao, “Energy-saving traffic scheduling in backbone networks with software-defined networks,” Cluster Comput., vol. 24, no. 1, pp. 279–292, Mar. 2021.
[25]
J. Specht and S. Samii, “Urgency-based scheduler for time-sensitive switched ethernet networks,” in Proc. 28th Euromicro Conf. Real-Time Syst., 2016, pp. 75–85, Issn: 2159–3833.
[26]
U. Arshad, M. Aleem, G. Srivastava, and J. C.-W. Lin, “Utilizing power consumption and SLA violations using dynamic VM consolidation in cloud data centers,” Renewable Sustain. Energy Rev., vol. 167, Oct. 2022, Art. no.
[27]
S. Kianpisheh and T. Taleb, “A survey on in-network computing: Programmable data plane and technology specific applications,” IEEE Commun. Surv. Tut., vol. 25, no. 1, pp. 701–761, First Quarter 2023.
[28]
J. R. Bhat and S. A. Alqahtani, “6G ecosystem: Current status and future perspective,” IEEE Access, vol. 9, pp. 43 134–43 167, 2021.
[29]
M. Giordani, M. Polese, M. Mezzavilla, S. Rangan, and M. Zorzi, “Toward 6G networks: Use cases and technologies,” IEEE Commun. Mag., vol. 58, no. 3, pp. 55–61, Mar. 2020.
[30]
U. Gustavsson et al., “Implementation challenges and opportunities in Beyond-5G and 6G communication,” IEEE J. Microw., vol. 1, no. 1, pp. 86–100, Jan. 2021.
[31]
N. H. Mahmood, G.- Y. Park, S.-K. Kim, C. Yoon, K. Anwar, and P. Seppänen, “Machine type communications: Key drivers and enablers towards the 6G era,” EURASIP J. Wireless Commun. Netw., vol. 2021, no. 1, Jun. 2021, Art. no.
[32]
H. Yu, C. Wang, T. Taleb, and J. Zhang, “Deep reinforcement learning based deterministic routing and scheduling for mixed-criticality flows,” IEEE Trans. Ind. Informat., vol. 19, no. 8, pp. 8806–8816, Aug. 2023.
[33]
Q. Guo, R. Gu, H. Yu, T. Taleb, and Y. Ji, “Probabilistic-assured resource provisioning with customizable hybrid isolation for vertical industrial slicing,” IEEE Trans. Netw. Service Manag., vol. 20, no. 2, pp. 1660–1675, Jun. 2023.
[34]
T. Taleb, I. Afolabi, and M. Bagaa, “Orchestrating 5G network slices to support industrial internet and to shape next-generation smart factories,” IEEE Netw., vol. 33, no. 4, pp. 146–154, Jul./Aug. 2019.
[35]
H. Kellerer, U. Pferschy, and D. Pisinger, “Multidimensional knapsack problems,” in Knapsack Problems, H. Kellerer, U. Pferschy, and D. Pisinger, Eds., Berlin, Germany: Springer, 2004, pp. 235–283.
[36]
G. Pataki, M. Tural, and E. B. Wong, “Basis reduction and the complexity of branch-and-bound,” in Proc. Annu. ACM-SIAM Symp. Discrete Algorithms, 2010, pp. 1254–1261.
[37]
H. V. Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” in Proc. AAAI Conf. Artif. Intell., 2016.
[38]
C. J. C. H. Watkins and P. Dayan, Technical Note in Reinforcement Learning, R. S. Sutton, Ed., Boston, MA, USA: Springer, 1992, pp. 55–68.
[39]
V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015.
[40]
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2014,.
[41]
C. J. Watkins and P. Dayan, “Q-learning,” Mach. Learn., vol. 8, pp. 279–292, 1992.
[42]
H. X. Nguyen, R. Trestian, D. To, and M. Tatipamula, “Digital twin for 5G and beyond,” IEEE Commun. Mag., vol. 59, no. 2, pp. 10–15, Feb. 2021.
Recommendations
Approximate Placement of Service-Based Applications in Hybrid Clouds
WETICE '12: Proceedings of the 2012 IEEE 21st International Workshop on Enabling Technologies: Infrastructure for Collaborative EnterprisesEnterprises are more and more using hybrid cloud environments for the deployment and execution of their applications. A hybrid cloud consists in private clouds that provides and manages some resources of an enterprise that uses others resources provided ...
Cloud-based load balancing using double Q-learning for improved Quality of Service
AbstractCloud computing improves the performance of software applications by providing on-demand usage, high availability, reliability, and agility. However, during peak traffic conditions the resources in cloud services can become over-utilized, ...
IBM deep learning service
Deep learning, driven by large neural network models, is overtaking traditional machine learning methods for understanding unstructured and perceptual data domains such as speech, text, and vision. At the same time, the "as-a-service"-based business ...
Comments
Information & Contributors
Information
Published In
© 2023 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
Publisher
IEEE Educational Activities Department
United States
Publication History
Published: 01 May 2024
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 23 Jan 2025