skip to main content
10.1145/3387514.3405856acmconferencesArticle/Chapter ViewAbstractPublication PagescommConference Proceedingsconference-collections
research-article

Neural-Enhanced Live Streaming: Improving Live Video Ingest via Online Learning

Published: 30 July 2020 Publication History

Abstract

Live video accounts for a significant volume of today's Internet video. Despite a large number of efforts to enhance user quality of experience (QoE) both at the ingest and distribution side of live video, the fundamental limitations are that streamer's upstream bandwidth and computational capacity limit the quality of experience of thousands of viewers.
To overcome this limitation, we design LiveNAS, a new live video ingest framework that enhances the origin stream's quality by leveraging computation at ingest servers. Our ingest server applies neural super-resolution on the original stream, while imposing minimal overhead on ingest clients. LiveNAS employs online learning to maximize the quality gain and dynamically adjusts the resource use to the real-time quality improvement. LiveNAS delivers high-quality live streams up to 4K resolution, outperforming WebRTC by 1.96 dB on average in Peak-Signal-to-Noise-Ratio on real video streams and network traces, which leads to 12%-69% QoE improvement for live stream viewers.

Supplementary Material

MP4 File (3387514.3405856.mp4)
20-minute talk about Neural-Enhanced Live Streaming: Improving Live Video Ingest via Online Learning presented at SIGCOMM' 20

References

[1]
Ant Media Server Official Website. https://antmedia.io/.
[2]
Apple HTTP Live Streaming Official Website. https://developer.apple.com/streaming/.
[3]
Apple Low-Latency HLS Specification. https://developer.apple.com/documentation/http_live_streaming/protocol_extension_for_low-latency_hls_preliminary_specification.
[4]
Chunked-encoded and Chunked-transferred CMAF Specification. https://www.akamai.com/us/en/multimedia/documents/white-paper/low-latency-streaming-cmaf-whitepaper.pdf.
[5]
Cisco Visual Networking Index Report. http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/complete-white-paper-c11-481360.pdf.
[6]
DASH Industry Forum Official Website. https://dashif.org/.
[7]
Encoding Video at the Edge with Intel® Xeon® Processors. https://builders.intel.com/docs/networkbuilders/encoding-video-at-the-edge-with-intel-xeon-processors.pdf.
[8]
FCC Broadband Bandwidth Measurement. https://www.fcc.gov/reports-research/reports/measuring-broadband-america/raw-data-measuring-broadband-america-eighth.
[9]
libWebRTC Official Github Repository. https://github.com/aisouard/libwebrtc.
[10]
Live Streaming Statistics about Twitch and Facebook. https://www.theverge.com/2020/1/9/21058907/twitch-youtube-mixer-facebook-live-streaming-numbers-growth-q4.
[11]
Real-Time Messaging Protocol (RTMP) Specification. https://wwwimages2.adobe.com/content/dam/acom/en/devnet/rtmp/pdf/rtmp_specification_license_1.0.pdf.
[12]
Real Time Streaming Protocol (RTSP) Specification. https://tools.ietf.org/html/rfc2326.
[13]
Real-time Transport Protocol (RTP) Specification. https://tools.ietf.org/html/rfc3550.
[14]
Streamlabs Q2 2019 Live Streaming Industry Report. https://blog.streamlabs.com/q2-2019-7e8039277b11.
[15]
Twitch Broadcasting Guidelines. https://stream.twitch.tv/encoding/.
[16]
Twitch Most Popular Game Category on February 2020. https://twitchtracker.com/games/rating. Last accessed Feb. 6, 2020.
[17]
Twitch Official Website. https://www.twitch.tv/.
[18]
Twitch Statistics about Streamer Channel (Saddummy). https://www.twitch.tv/saddummy.
[19]
Twitch Statistics Report by Influencer MarketingHub. https://influencermarketinghub.com/twitch-statistics/.
[20]
Twitch Stream Dataset (Escape from Tarkov). https://www.twitch.tv/stylishnoob4/.
[21]
Twitch Stream Dataset (Fornite). https://www.twitch.tv/nickmercs/videos.
[22]
Twitch Stream Dataset (Just Chatting). https://www.twitch.tv/pokimane/videos.
[23]
Twitch Stream Dataset (League of Legends). https://www.twitch.tv/riotgames/videos.
[24]
Twitch Stream Dataset (World of Warcraft). https://www.twitch.tv/method/.
[25]
WebM Official Website. https://www.webmproject.org/.
[26]
WebRTC Official Website. https://webrtc.org/.
[27]
Wowza 2019 Live-Streaming Advancements Report. https://www.wowza.com/blog/streaming-advancements-2019.
[28]
Wowza Live Streaming Guidebook. https://www.wowza.com/uploads/images/The-Complete-Guide-to-Live-Streaming-Wowza.pdf.
[29]
Wowza WebRTC Streaming Guidebook. https://www.wowza.com/low-latency/webrtc-streaming-software.
[30]
Youtube Live Encoding Guidelines. https://support.google.com/youtube/answer/2853702?hl=en.
[31]
Youtube Live Streaming Official Website. https://www.youtube.com/live.
[32]
Youtube Video Dataset (Food). https://www.youtube.com/watch?v=oLbRhd7p0Ts.
[33]
Youtube Video Dataset (Live Event). https://www.youtube.com/watch?v=aVZngNyEsjg.
[34]
Youtube Video Dataset (Pod Cast). https://www.youtube.com/watch?v=CpQZRLxJE5M&=&index=14.
[35]
Youtube Video Dataset (Sport). https://www.youtube.com/watch?v=ppb7hmQA6sU.
[36]
Eirikur Agustsson and Radu Timofte. 2017. NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
[37]
Ghufran Baig, Jian He, Mubashir Adnan Qureshi, Lili Qiu, Guohai Chen, Peng Chen, and Yinliang Hu. 2019. Jigsaw: Robust live 4k video streaming. In The 25th Annual International Conference on Mobile Computing and Networking. 1--16.
[38]
Gaetano Carlucci, Luca De Cicco, Stefan Holmer, and Saverio Mascolo. 2016. Analysis and Design of the Google Congestion Control for Web Real-Time Communication (WebRTC). In Proceedings of the 7th International Conference on Multimedia Systems (MMSys '16). Article Article 13, 12 pages.
[39]
M. Dasari, A. Bhattacharya, S. Vargas, P. Sahu, A. Balasubramanian, and S. R. Das. 2020. Streaming 360° Videos using Super-resolution. In Proceedings of the IEEE International Conference on Computer Communications (INFOCOM).
[40]
J. Deng, F. Cuadrado, G. Tyson, and S. Uhlig. 2015. Behind the game: Exploring the twitch streaming platform. In 2015 International Workshop on Network and Systems Support for Games (NetGames). 1--6.
[41]
Florin Dobrian, Vyas Sekar, Asad Awan, Ion Stoica, Dilip Joseph, Aditya Ganjam, Jibin Zhan, and Hui Zhang. 2011. Understanding the Impact of Video Quality on User Engagement. SIGCOMM Comput. Commun. Rev. 41, 4 (Aug. 2011), 362--373.
[42]
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a deep convolutional network for image super-resolution. In European conference on computer vision. Springer, 184--199.
[43]
Zhengfang Duanmu, Kai Zeng, Kede Ma, Abdul Rehman, and Zhou Wang. 2016. A quality-of-experience index for streaming video. IEEE Journal of Selected Topics in Signal Processing 11, 1 (2016), 154--166.
[44]
Sadjad Fouladi, John Emmons, Emre Orbay, Catherine Wu, Riad S. Wahby, and Keith Winstein. 2018. Salsify: Low-Latency Network Video through Tighter Integration between a Video Codec and a Transport Protocol. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18). Renton, WA, 267--282.
[45]
Sadjad Fouladi, Riad S. Wahby, Brennan Shacklett, Karthikeyan Vasuki Balasubramaniam, William Zeng, Rahul Bhalerao, Anirudh Sivaraman, George Porter, and Keith Winstein. 2017. Encoding, Fast and Slow: Low-Latency Video Processing Using Thousands of Tiny Threads. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). Boston, MA, 363--376.
[46]
Jeremy Fowers, Kalin Ovtcharov, Michael Papamichael, Todd Massengill, Ming Liu, Daniel Lo, Shlomi Alkalay, Michael Haselman, Logan Adams, Mahdi Ghandi, et al. 2018. A configurable cloud-scale DNN processor for real-time AI. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA). IEEE, 1--14.
[47]
F. Fund, C. Wang, Y. Liu, T. Korakis, M. Zink, and S. S. Panwar. 2013. Performance of DASH and WebRTC Video Services for Mobile Users. In 2013 20th International Packet Video Workshop. 1--8.
[48]
Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1994. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, Reading, Mass. http://www.worldcat.org/search?qt=worldcat_org_all&q=0201633612
[49]
Alain Hore and Djemel Ziou. 2010. Image quality metrics: PSNR vs. SSIM. In 2010 20th International Conference on Pattern Recognition. IEEE, 2366--2369.
[50]
Pan Hu, Rakesh Misra, and Sachin Katti. 2019. Dejavu: Enhancing Videoconferencing with Prior Knowledge. In Proceedings of the 20th International Workshop on Mobile Computing Systems and Applications (HotMobile '19). 63--68.
[51]
Mehdi Kaytoue, Arlei Silva, Loïc Cerf, Wagner Meira, and Chedy Raïssi. 2012. Watch Me Playing, I am a Professional: A First Study on Video Game Live Streaming. In Proceedings of the 21st International Conference on World Wide Web (WWW '12 Companion). 1181--1188.
[52]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image superresolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1646--1654.
[53]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[54]
Royson Lee, Stylianos I Venieris, Lukasz Dudziak, Sourav Bhattacharya, and Nicholas D Lane. 2019. MobiSR: Efficient On-Device Super-Resolution through Heterogeneous Mobile Processors. In The 25th Annual International Conference on Mobile Computing and Networking. 1--16.
[55]
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 136--144.
[56]
Siwei Ma, Wen Gao, and Yan Lu. 2005. Rate-distortion analysis for H. 264/AVC video coding and its application to rate control. IEEE transactions on circuits and systems for video technology 15, 12 (2005), 1533--1544.
[57]
Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. 2017. Neural Adaptive Video Streaming with Pensieve. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM '17). Association for Computing Machinery, 197--210.
[58]
Ravi Netravali, Anirudh Sivaraman, Somak Das, Ameesh Goyal, Keith Winstein, James Mickens, and Hari Balakrishnan. 2015. Mahimahi: Accurate Record-and-Replay for HTTP. In Proceedings of the USENIX Annual Technical Conference (ATC). 417--429.
[59]
KarinePires andGwendalSimon. 2014. Dashintwitch:Adaptive bitrate streaming in live game streaming platforms. In Proceedings of the 2014 Workshop on Design, Quality and Deployment of Adaptive Video Streaming. 13--18.
[60]
Karine Pires and Gwendal Simon. 2015. You Tube Live and Twitch: A Tour of User-Generated Live Streaming Systems. In Proceedings of the 6th ACM Multimedia Systems Conference (MMSys '15). 225--230.
[61]
Devdeep Ray, Jack Kosaian, K. V. Rashmi, and Srinivasan Seshan. 2019. Vantage: Optimizing Video Upload for Time-Shifted Viewing of Social Live Streams. In Proceedings of the ACM Special Interest Group on Data Communication (SIGCOMM '19). 380--393.
[62]
Haakon Riiser, Paul Vigmostad, Carsten Griwodz, and P~l Halvorsen. 2013. Commute path bandwidth traces from 3G networks: analysis and applications. In Proceedings of the 4th ACM Multimedia Systems Conference. ACM, 114--118.
[63]
Shai Shalev-Shwartz et al. 2012. Online learning and online convex optimization. Foundations and Trends® in Machine Learning 4, 2 (2012), 107--194.
[64]
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1874--1883.
[65]
Assaf Shocher, Nadav Cohen, and Michal Irani. 2018. "zero-shot" super-resolution using deep internal learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3118--3126.
[66]
Iraj Sodagar. 2011. The mpeg-dash standard for multimedia streaming over the internet. IEEE multimedia 18, 4 (2011), 62--67.
[67]
Mikko Uitto. 2016. Energy consumption evaluation of H. 264 and HEVC video encoders in high-resolution live streaming. In 2016 IEEE 12th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob). IEEE, 1--7.
[68]
Yiding Wang, Weiyan Wang, Junxue Zhang, Junchen Jiang, and Kai Chen. 2019. Bridging the Edge-Cloud Barrier for Real-time Advanced Vision Analytics. In 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19). USENIX Association, Renton, WA. https://www.usenix.org/conference/hotcloud19/presentation/wang
[69]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600--612.
[70]
David H Wolpert. 1996. The lack of a priori distinctions between learning algorithms. Neural computation 8, 7 (1996), 1341--1390.
[71]
Hyunho Yeo, Sunghyun Do, and Dongsu Han. 2017. How will Deep Learning Change Internet Video Delivery?. In Proceedings of the 16th ACM Workshop on Hot Topics in Networks. ACM, 57--64.
[72]
Hyunho Yeo, Youngmok Jung, Jaehong Kim, Jinwoo Shin, and Dongsu Han. 2018. Neural Adaptive Content-aware Internet Video Delivery. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). Carlsbad, CA, 645--661.
[73]
Xiaoqi Yin, Abhishek Jindal, Vyas Sekar, and Bruno Sinopoli. 2015. A control-theoretic approach for dynamic adaptive video streaming over HTTP. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. 325--338.
[74]
Cong Zhang and Jiangchuan Liu. 2015. On Crowdsourced Interactive Live Streaming: A Twitch.Tv-Based Measurement Study. In Proceedings of the 25th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV '15). 55--60.
[75]
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2472--2481.
[76]
Zhengdong Zhang and Vivienne Sze. 2017. FAST: A framework to accelerate super-resolution processing on compressed videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 19--28.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGCOMM '20: Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication
July 2020
814 pages
ISBN:9781450379557
DOI:10.1145/3387514
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 July 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. deep neural networks
  2. live streaming
  3. online learning
  4. super-resolution
  5. video delivery

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

SIGCOMM '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 462 of 3,389 submissions, 14%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)532
  • Downloads (Last 6 weeks)46
Reflects downloads up to 14 Sep 2024

Other Metrics

Citations

Cited By

View all

View Options

Get Access

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media