AraLive: Automatic Reward Adaption for Learning-based Live Video Streaming
Authors: 
Huanhuan Zhang, Liu zhuo, Haotian Li, Anfu Zhou, Chuanming Wang, Huadong MaAuthors Info & Claims
Huanhuan Zhang, Liu zhuo, Haotian Li, Anfu Zhou, Chuanming Wang, Huadong MaAuthors Info & ClaimsPages 11099 - 11108
Abstract
Optimizing user Quality of Experience (QoE) for live video streaming remains a long-standing challenge. The Bitrate Control Algorithm (BCA) plays a crucial role in shaping user QoE. Recent advancements have seen RL-based algorithms overtake traditional rule-based methods, promising enhanced QoE optimization. Nevertheless, our comprehensive study reveals a pressing issue: current RL-based BCAs are limited to the fixed and formulaic reward functions, rendering them ill-equipped to adapt to dynamic network environments and varied viewer preferences. In this work, we present AraLive, an automatically adaptive reward learning method designed for seamless integration with any existing learning-based approach in live streaming contexts. To achieve this goal, we have two main designs. First, we construct a dedicated user QoE assessment dataset for live streaming, which includes thousands of videos with millisecond-level metrics. Second, we custom-design an adversarial model that skillfully aligns human feedback with actual network scenarios. We have deployed AraLive in practical video streaming systems, in comparison to a series of state-of-the-art BCAs. The experimental results demonstrate that AraLive not only elevates overall QoE but also exhibits remarkable adaptability to varied user preferences.
References
[1]
2017. WebRTC source code. https://webrtc.googlesource.com/src/webrtc/.
[2]
2022. Traffic Control HOWTO v1.0.2. https://tldp.org/HOWTO/Traffic-Control- HOWTO/intro.html.
[3]
2024. Distribution of global monthly cellular data volume for mobile apps as of January 2024, by category. https://www.statista.com/statistics/383715/global-mobile-data-traffic-share/.
[4]
Soheil Abbasloo, Chen-Yu Yen, and H. Jonathan Chao. 2020. Classic Meets Modern: a Pragmatic Learning-Based Congestion Control for the Internet. In SIGCOMM '20: Proceedings of the 2020 Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, Virtual Event, USA, August 10-14, 2020, Henning Schulzrinne and Vishal Misra (Eds.). ACM, 632--647. https://doi.org/10.1145/3387514.3405892
[5]
Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. 2016. BBR: Congestion-Based Congestion Control. ACM Queue, Vol. 14, 5 (2016), 20--53. https://doi.org/10.1145/3012426.3022184
[6]
Gaetano Carlucci, Luca De Cicco, Stefan Holmer, and Saverio Mascolo. 2017. Congestion Control for Web Real-Time Communication. IEEE/ACM Trans. Netw., Vol. 25, 5 (2017), 2629--2642. https://doi.org/10.1109/TNET.2017.2703615
[7]
Mo Dong, Qingxi Li, Doron Zarchy, Philip Brighten Godfrey, and Michael Schapira. 2015. PCC: Re-architecting Congestion Control for Consistent High Performance. In 12th USENIX Symposium on Networked Systems Design and Implementation, NSDI 15, Oakland, CA, USA, May 4-6, 2015. USENIX Association, 395--408. https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/dong
[8]
Mo Dong, Tong Meng, Doron Zarchy, Engin Arslan, Yossi Gilad, Brighten Godfrey, and Michael Schapira. 2018. PCC Vivace: Online-Learning Congestion Control. In 15th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2018, Renton, WA, USA, April 9-11, 2018, Sujata Banerjee and Srinivasan Seshan (Eds.). USENIX Association, 343--356. https://www.usenix.org/conference/nsdi18/presentation/dong
[9]
Zhengfang Duanmu, Wentao Liu, Zhuoran Li, Diqi Chen, Zhou Wang, Yizhou Wang, and Wen Gao. 2020. Assessing the Quality-of-Experience of Adaptive Bitrate Video Streaming. CoRR, Vol. abs/2008.08804 (2020). [arXiv]2008.08804 https://arxiv.org/abs/2008.08804
[10]
Alfred Giessler, J. D. Haenle, Andreas König, and E. Pade. 1978. Free Buffer Allocation - An Investigation by Simulation. Comput. Networks, Vol. 2 (1978), 191--208. https://doi.org/10.1016/0376--5075(78)90028-4
[11]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems, Vol. 27 (2014).
[12]
Sangtae Ha, Injong Rhee, and Lisong Xu. 2008. CUBIC: a new TCP-friendly high-speed TCP variant. ACM SIGOPS Oper. Syst. Rev., Vol. 42, 5 (2008), 64--74. https://doi.org/10.1145/1400097.1400105
[13]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Comput., Vol. 9, 8 (1997), 1735--1780. https://doi.org/10.1162/NECO.1997.9.8.1735
[14]
Tianchi Huang, Rui-Xiao Zhang, Chenglei Wu, and Lifeng Sun. 2023. Optimizing Adaptive Video Streaming with Human Feedback. In Proceedings of the 31st ACM International Conference on Multimedia, MM 2023, Ottawa, ON, Canada, 29 October 2023- 3 November 2023, Abdulmotaleb El-Saddik, Tao Mei, Rita Cucchiara, Marco Bertini, Diana Patricia Tobon Vallejo, Pradeep K. Atrey, and M. Shamim Hossain (Eds.). ACM, 1707--1718. https://doi.org/10.1145/3581783.3611771
[15]
Tianchi Huang, Chao Zhou, Lianchen Jia, Rui-Xiao Zhang, and Lifeng Sun. 2022. Learned Internet Congestion Control for Short Video Uploading. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022, Jo ao Magalh aes, Alberto Del Bimbo, Shin'ichi Satoh, Nicu Sebe, Xavier Alameda-Pineda, Qin Jin, Vincent Oria, and Laura Toni (Eds.). ACM, 3064--3075. https://doi.org/10.1145/3503161.3548436
[16]
Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore. 1996. Reinforcement Learning: A Survey. J. Artif. Intell. Res., Vol. 4 (1996), 237--285. https://doi.org/10.1613/JAIR.301
[17]
Yiming Kong, Hui Zang, and Xiaoli Ma. 2018. Improving TCP Congestion Control with Machine Intelligence. In Proceedings of the 2018 Workshop on Network Meets AI & ML, NetAI@SIGCOMM 2018, Budapest, Hungary, August 24, 2018. ACM, 60--66. https://doi.org/10.1145/3229543.3229550
[18]
Insoo Lee, Seyeon Kim, Sandesh Sathyanarayana, Kyungmin Bin, Song Chong, Kyunghan Lee, Dirk Grunwald, and Sangtae Ha. 2022. R-FEC: RL-based FEC Adjustment for Better QoE in WebRTC. In Proceedings of the 30th ACM International Conference on Multimedia. 2948--2956.
[19]
Chaoyang Li, Rui-Xiao Zhang, Tianchi Huang, Lianchen Jia, and Lifeng Sun. 2023. Concerto: Client-server Orchestration for Real-Time Video Analytics. In Proceedings of the 31st ACM International Conference on Multimedia, MM 2023, Ottawa, ON, Canada, 29 October 2023- 3 November 2023, Abdulmotaleb El-Saddik, Tao Mei, Rita Cucchiara, Marco Bertini, Diana Patricia Tobon Vallejo, Pradeep K. Atrey, and M. Shamim Hossain (Eds.). ACM, 9215--9223. https://doi.org/10.1145/3581783.3611770
[20]
Zachary C Lipton, John Berkowitz, and Charles Elkan. 2015. A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv:1506.00019 (2015).
[21]
Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. 2017. Neural Adaptive Video Streaming with Pensieve. In SIGCOMM. ACM, 197--210.
[22]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. CoRR, Vol. abs/1707.06347 (2017). [arXiv]1707.06347 http://arxiv.org/abs/1707.06347
[23]
Henning Schulzrinne, Stephen L. Casner, Ron Frederick, and Van Jacobson. 2003. RTP: A Transport Protocol for Real-Time Applications. RFC, Vol. 3550 (2003), 1--104. https://doi.org/10.17487/RFC3550
[24]
Zhongze Tang, Xianglong Feng, Yi Xie, Huy Phan, Tian Guo, Bo Yuan, and Sheng Wei. 2020. VVSec: Securing Volumetric Video Streaming via Benign Use of Adversarial Perturbation. In MM '20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020, Chang Wen Chen, Rita Cucchiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann (Eds.). ACM, 3614--3623. https://doi.org/10.1145/3394171.3413639
[25]
Yizong Wang, Dong Zhao, Huanhuan Zhang, Chenghao Huang, Teng Gao, Zixuan Guo, Liming Pang, and Huadong Ma. 2023. Hermes: Leveraging Implicit Inter-Frame Correlation for Bandwidth-Efficient Mobile Volumetric Video Streaming. In Proceedings of the 31st ACM International Conference on Multimedia, MM 2023, Ottawa, ON, Canada, 29 October 2023- 3 November 2023, Abdulmotaleb El-Saddik, Tao Mei, Rita Cucchiara, Marco Bertini, Diana Patricia Tobon Vallejo, Pradeep K. Atrey, and M. Shamim Hossain (Eds.). ACM, 9185--9193. https://doi.org/10.1145/3581783.3613907
[26]
Keith Winstein and Hari Balakrishnan. 2013. TCP ex machina: computer-generated congestion control. In ACM SIGCOMM 2013 Conference, SIGCOMM 2013, Hong Kong, August 12-16, 2013, Dah Ming Chiu, Jia Wang, Paul Barford, and Srinivasan Seshan (Eds.). ACM, 123--134. https://doi.org/10.1145/2486001.2486020
[27]
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Hong Lin. 2023. AI-Generated Content (AIGC): A Survey. CoRR, Vol. abs/2304.06632 (2023). https://doi.org/10.48550/ARXIV.2304.06632 [arXiv]2304.06632
[28]
Francis Y. Yan, Hudson Ayers, Chenzhi Zhu, Sadjad Fouladi, James Hong, Keyi Zhang, Philip Alexander Levis, and Keith Winstein. 2020. Learning in situ: a randomized experiment in video streaming. In 17th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2020, Santa Clara, CA, USA, February 25-27, 2020, Ranjita Bhagwan and George Porter (Eds.). USENIX Association, 495--511. https://www.usenix.org/conference/nsdi20/presentation/yan
[29]
Francis Y. Yan, Jestin Ma, Greg D. Hill, Deepti Raghavan, Riad S. Wahby, Philip Alexander Levis, and Keith Winstein. 2018. Pantheon: the training ground for Internet congestion-control research. In 2018 USENIX Annual Technical Conference, USENIX ATC 2018, Boston, MA, USA, July 11-13, 2018, Haryadi S. Gunawi and Benjamin C. Reed (Eds.). USENIX Association, 731--743. https://www.usenix.org/conference/atc18/presentation/yan-francis
[30]
Xiaoqi Yin, Abhishek Jindal, Vyas Sekar, and Bruno Sinopoli. 2015. A Control-Theoretic Approach for Dynamic Adaptive Video Streaming over HTTP. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, SIGCOMM 2015, London, United Kingdom, August 17-21, 2015, Steve Uhlig, Olaf Maennel, Brad Karp, and Jitendra Padhye (Eds.). ACM, 325--338. https://doi.org/10.1145/2785956.2787486
[31]
Huanhuan Zhang, Congkai An, Anfu Zhou, Yifan Zhu, Weilin Sun, Yixuan Lu, Jiahao Chen, Liang Liu, Huadong Ma, and Aiguo Fei. 2024. Venus: Enhancing QoE of Crowdsourced Live Video Streaming by Exploiting Multiflow Viewer Assistance. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, ACM MobiCom 2024, Washington D.C., DC, USA, November 18-22, 2024, Weisong Shi, Deepak Ganesan, and Nicholas D. Lane (Eds.). ACM, 170--184. https://doi.org/10.1145/3636534.3649354
[32]
Huanhuan Zhang, Anfu Zhou, Yuhan Hu, Chaoyue Li, Guangping Wang, Xinyu Zhang, Huadong Ma, Leilei Wu, Aiyun Chen, and Changhui Wu. 2021. Loki: improving long tail performance of learning-based real-time video adaptation by fusing rule-based models. In MobiCom. 775--788.
[33]
Huanhuan Zhang, Anfu Zhou, Jiamin Lu, Ruoxuan Ma, Yuhan Hu, Cong Li, Xinyu Zhang, Huadong Ma, and Xiaojiang Chen. 2020. OnRL: improving mobile video telephony via online reinforcement learning. In MobiCom '20: The 26th Annual International Conference on Mobile Computing and Networking, London, United Kingdom, September 21-25, 2020. ACM, 29:1--29:14.
[34]
Huanhuan Zhang, Anfu Zhou, Ruoxuan Ma, Jiamin Lu, and Huadong Ma. 2021. Arsenal: Understanding Learning-Based Wireless Video Transport via In-Depth Evaluation. IEEE Trans. Veh. Technol., Vol. 70, 10 (2021), 10832--10844. https://doi.org/10.1109/TVT.2021.3105479
[35]
Ruixiao Zhang, Haiping Wang, Shu Shi, Xiaofei Pang, Yajie Peng, Zhichen Xue, and Jiangchuan Liu. 2024. Enhancing Resource Management of the World's Largest PCDN System for On-Demand Video Streaming. In Proceedings of the 2024 USENIX Annual Technical Conference, USENIX ATC 2024, Santa Clara, CA, USA, July 10-12, 2024, Saurabh Bagchi and Yiying Zhang (Eds.). USENIX Association, 951--965. https://www.usenix.org/conference/atc24/presentation/zhang-rui-xiao
[36]
Rui-Xiao Zhang, Changpeng Yang, Xiaochan Wang, Tianchi Huang, Chenglei Wu, Jiangchuan Liu, and Lifeng Sun. 2022. AggCast: Practical Cost-effective Scheduling for Large-scale Cloud-edge Crowdsourced Live Streaming. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022, Jo ao Magalh aes, Alberto Del Bimbo, Shin'ichi Satoh, Nicu Sebe, Xavier Alameda-Pineda, Qin Jin, Vincent Oria, and Laura Toni (Eds.). ACM, 3026--3034. https://doi.org/10.1145/3503161.3547807
[37]
Anfu Zhou, Huanhuan Zhang, Guangyuan Su, Leilei Wu, Ruoxuan Ma, Zhen Meng, Xinyu Zhang, Xiufeng Xie, Huadong Ma, and Xiaojiang Chen. 2019. Learning to Coordinate Video Codec with Transport Protocol for Mobile Video Telephony. In The 25th Annual International Conference on Mobile Computing and Networking, MobiCom 2019, Los Cabos, Mexico, October 21-25, 2019, Stephen A. Brewster, Geraldine Fitzpatrick, Anna L. Cox, and Vassilis Kostakos (Eds.). ACM, 29:1--29:16. https://doi.org/10.1145/3300061.3345430
Index Terms
- AraLive: Automatic Reward Adaption for Learning-based Live Video Streaming
Recommendations
Multi-camera Live Video Streaming over Wireless Network
Advances in Mobile Computing and Multimedia IntelligenceAbstractDue to the development of wireless communication technology, more and more streamers are using cameras mounted on mobile devices for live streaming in a wireless LAN environment. Conventional live streaming systems, which employ multiple images ...
Live Video Streaming Optimization Based on Deep Reinforcement Learning
ICMLC '20: Proceedings of the 2020 12th International Conference on Machine Learning and ComputingVideo players employ adaptive bitrate algorithms in video-on-demand (VoD) scenarios to improve user-perceived quality of experience (QoE), whereas performance will obviously decline in live video streaming scenarios. To this end, we propose a novel deep ...
A New Scheme for QoE Management of Live Video Streaming in Cloud Environment
Image and Video TechnologyAbstractLive video streaming process consumes very large data storage and takes very long time, so it requires big data storage and computing infrastructures for implementation. Accordingly, the use of cloud computing is becoming a common practice ...
Comments
Information & Contributors
Information
Published In
October 2024
11719 pages
ISBN:9798400706868
DOI:10.1145/3664647
- General Chairs:
- Jianfei Cai,
- Mohan Kankanhalli,
- Balakrishnan Prabhakaran,
- Susanne Boll,
- Program Chairs:
- Ramanathan Subramanian,
- Liang Zheng,
- Vivek K. Singh,
- Pablo Cesar,
- Lexing Xie,
- Dong Xu
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Innovation Research Group Project of NSFC
- Youth Top Talent Support Program
- NSFC Project
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Acceptance Rates
MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 103Total Downloads
- Downloads (Last 12 months)103
- Downloads (Last 6 weeks)44
Reflects downloads up to 04 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in