research-article

Design, Implementation, and Measurement of a Crowdsourcing-Based Content Distribution Platform

Authors:

Richard Tianbai MaAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 12, Issue 5s

Article No.: 80, Pages 1 - 23

https://doi.org/10.1145/2978655

Published: 08 November 2016 Publication History

Abstract

Content distribution, especially the distribution of video content, unavoidably consumes bandwidth resources heavily. Internet content providers invest heavily in purchasing content distribution network (CDN) services. By deploying tens of thousands of edge servers close to end users, CDN companies are able to distribute content efficiently and effectively, but at considerable cost. Thus, it is of great importance to develop a new system that distributes content at a lower cost but comparable service quality. In lieu of expensive CDN systems, we implement a crowdsourcing-based content distribution system, Thunder Crystal, by renting bandwidth for content upload/download and storage for content cache from agents. This is a large-scale system with tens of thousands of agents, whose resources significantly amplify Thunder Crystal’s content distribution capacity. The involved agents are either from ordinary Internet users or enterprises. Monetary rewards are paid to agents based on their upload traffic so as to motivate them to keep contributing resources. As far as we know, this is a novel system that has not been studied or implemented before. This article introduces the design principles and implementation details before presenting the measurement study. In summary, with the help of agent devices, Thunder Crystal is able to reduce the content distribution cost by one half and amplify the content distribution capacity by 11 to 15 times.

References

[1]

Henrik Abrahamsson and Mattias Nordmark. 2012. Program popularity and viewer behaviour in a large TV-on-Demand system. In Proceedings of the 2012 ACM Conference on Internet Measurement. ACM, New York, NY, 199--210.

Digital Library

[2]

Vijay Kumar Adhikari, Yang Guo, Fang Hao, Matteo Varvello, Volker Hilt, Moritz Steiner, and Zhi-Li Zhang. 2012. Unreeling Netflix: Understanding and improving multi-CDN movie delivery. In Proceedings of the 2012 Annual IEEE International Conference on Computer Communications. IEEE, New York, NY, 1620--1628.

[3]

Athula Balachandran, Vyas Sekar, Aditya Akella, and Srinivasan Seshan. 2013. Analyzing the potential benefits of CDN augmentation strategies for Internet video workloads. In Proceedings of the 2013 ACM Conference on Internet Measurement. ACM, New York, NY, 43--56.

Digital Library

[4]

Zachary S. Bischof, John S. Otto, Mario A. Sánchez, John P. Rula, David R. Choffnes, and Fabián E. Bustamante. 2011. Crowdsourcing ISP characterization to the network edge. In Proceedings of the 1st ACM SIGCOMM Workshop on Measurements Up the Stack. ACM, New York, NY, 61--66.

Digital Library

[5]

Meeyoung Cha, Haewoon Kwak, Pablo Rodriguez, Yong-Yeol Ahn, and Sue Moon. 2007. I tube, you tube, everybody tubes: Analyzing the world’s largest user generated content video system. In Proceedings of the 2007 ACM Conference on Internet Measurement. ACM, New York, NY, 1--14.

Digital Library

[6]

Liang Chen, Yipeng Zhou, and Dah Ming Chiu. 2014. A lifetime model of online video popularity. In Proceedings of the 2014 Conference on Computer Communication and Networks. IEEE, Los Alamitos, CA, 1--8.

[7]

Liang Chen, Yipeng Zhou, Mi Jing, and Richard T. B. Ma. 2015. Thunder Crystal: A novel crowdsourcing-based content distribution platform. In Proceedings of the 2015 ACM Workshop on Network and Operating Systems Support for Digital Audio and Video. ACM, New York, NY, 43--48.

Digital Library

[8]

Xiaowei Chen, Yixin Jiang, and Xiaowen Chu. 2010. Measurements, analysis and modeling of private trackers. In Proceedings of the 2010 IEEE 10th International Conference on Peer-to-Peer Computing. IEEE, Los Alamitos, CA, 1--10.

[9]

Cisco. 2012. Cisco Visual Networking Index: Forecast and Methodology, 2011--2016. White Paper. Cisco, San Jose, CA.

[10]

Bram Cohen. 2003. Incentives build robustness in BitTorrent. In Proceedings of the Workshop on Economics of Peer-to-Peer Systems, Vol. 6. 68--72.

[11]

Adriano Faggiani, Enrico Gregori, Luciano Lenzini, Valerio Luconi, and Alessio Vecchio. 2013. Network sensing through smartphone-based crowdsourcing. In Proceedings of the 2013 ACM Conference on Embedded Networked Sensor Systems. ACM, New York, NY, 1--2.

Digital Library

[12]

Bin Fan, Dah-Ming Chiu, and John Lui. 2006. The delicate tradeoffs in bittorrent-like file sharing protocol design. In Proceedings of the 2006 IEEE International Conference on Network Protocols. IEEE, Los Alamitos, CA, 239--248.

Digital Library

[13]

Tobias Hoßfeld, Michael Seufert, Matthias Hirth, Thomas Zinner, Phuoc Tran-Gia, and Raimund Schatz. 2011. Quantification of YouTube QoE via crowdsourcing. In Proceedings of the 2011 IEEE International Symposium on Multimedia. IEEE, Los Alamitos, CA, 494--499.

Digital Library

[14]

Cheng Huang, Angela Wang, Jin Li, and Keith W. Ross. 2008b. Measuring and evaluating large-scale CDNs. In Proceedings of the 2008 ACM Conference on Internet Measurement, Vol. 8. 15--29.

[15]

Yan Huang, Tom Z. J. Fu, Dah-Ming Chiu, John Lui, and Cheng Huang. 2008a. Challenges, design and analysis of a large-scale P2P-VoD system. ACM SIGCOMM Computer Communication Review 38, 4, 375--388.

Digital Library

[16]

Adele Lu Jia, Raziur Rahman, Tamas Vinko, Johan A. Pouwelse, and Dick H. J. Epema. 2013. Systemic risk and user-level performance in private P2P communities. IEEE Transactions on Parallel and Distributed Systems 24, 12, 2503--2512.

Digital Library

[17]

Lorenzo Keller, Anh Le, Blerim Cici, Hulya Seferoglu, Christina Fragouli, and Athina Markopoulou. 2012. MicroCast: Cooperative video streaming on smartphones. In Proceedings of the 2012 International Conference on Mobile Systems, Applications, and Services. ACM, New York, NY, 57--70.

Digital Library

[18]

Rupa Krishnan, Harsha V. Madhyastha, Sridhar Srinivasan, Sushant Jain, Arvind Krishnamurthy, Thomas Anderson, and Jie Gao. 2009. Moving beyond end-to-end path information to optimize CDN performance. In Proceedings of the 2009 ACM Conference on Internet Measurement. ACM, New York, NY, 190--201.

Digital Library

[19]

Zhengye Liu, Prithula Dhungel, Di Wu, Chao Zhang, and Keith W. Ross. 2010a. Understanding and improving ratio incentives in private communities. In Proceedings of the 2010 IEEE International Conference on Distributed Computing Systems. IEEE, Los Alamitos, CA, 610--621.

Digital Library

[20]

Zimu Liu, Chuan Wu, Baochun Li, and Shuqiao Zhao. 2010b. UUSee: Large-scale operational on-demand streaming with random network coding. In Proceedings of the 2010 Annual IEEE International Conference on Computer Communications. IEEE, Los Alamitos, CA, 1--9.

Digital Library

[21]

George Pallis and Athena Vakali. 2006. Insight and perspectives for content delivery networks. Communications of the ACM 49, 1, 101--106.

Digital Library

[22]

Andrea Passarella. 2012. A survey on content-centric technologies for the current Internet: CDN and P2P solutions. Computer Communications 35, 1, 1--32.

Digital Library

[23]

Rameez Rahman, Tamás Vinkó, David Hales, Johan Pouwelse, and Henk Sips. 2011. Design space analysis for modeling incentives in distributed systems. ACM SIGCOMM Computer Communication Review 41, 4, 182--193.

Digital Library

[24]

Bo Tan and Laurent Massoulié. 2013. Optimal content placement for peer-to-peer video-on-demand systems. IEEE/ACM Transactions on Networking 21, 2, 566--579.

Digital Library

[25]

Stefano Traverso, Mohamed Ahmed, Michele Garetto, Paolo Giaccone, Emilio Leonardi, and Saverio Niccolini. 2013. Temporal locality in today’s content caching: Why it matters and how to model it. ACM SIGCOMM Computer Communication Review 43, 5, 5--12.

Digital Library

[26]

Chen-Chi Wu, Kuan-Ta Chen, Yu-Chun Chang, and Chin-Laung Lei. 2013. Crowdsourcing multimedia QoE evaluation: A trusted framework. IEEE Transactions on Multimedia 15, 5, 1121--1137.

Digital Library

[27]

Di Wu, Chao Liang, Yong Liu, and Keith Ross. 2009. View-upload decoupling: A redesign of multi-channel P2P video systems. In Proceedings of the 2009 International Conference on Computer Communications. IEEE, Los Alamitos, CA, 2726--2730.

[28]

Weijie Wu, Richard T. B. Ma, and John C. S. Lui. 2014. Distributed caching via rewarding: An incentive scheme design in P2P-VoD systems. IEEE Transactions on Parallel and Distributed Systems 25, 3, 612--621.

Digital Library

[29]

Libin Yang and Wei Lou. 2012. Pricing, competition and innovation: A profitable business model to resolve the tussle involved in peer-to-peer streaming applications. In Proceedings of the 2012 IEEE 20th International Workshop on Quality of Service. 1--9.

Digital Library

[30]

Hao Yin, Xuening Liu, Tongyu Zhan, Vyas Sekar, Feng Qiu, Chuang Lin, Hui Zhang, and Bo Li. 2009. Design and deployment of a hybrid CDN-P2P system for live video streaming: Experiences with LiveSky. In Proceedings of the 2009 International Conference on Multimedia. ACM, New York, NY, 25--34.

Digital Library

[31]

Cong Zhang and Jiangchuan Liu. 2015. On crowdsourced interactive live streaming: A Twitch.TV-based measurement study. In Proceedings of the 2015 ACM Workshop on Network and Operating Systems Support for Digital Audio and Video. ACM, New York, NY, 55--60.

Digital Library

[32]

Bridge Qiao Zhao, John C. S. Lui, and Dah-Ming Chiu. 2012. A mathematical framework for analyzing adaptive incentive protocols in P2P networks. IEEE/ACM Transactions on Networking 20, 2, 367--380.

Digital Library

[33]

Yipeng Zhou, Liang Chen, Chunfeng Yang, and Dah Ming Chiu. 2015. Video popularity dynamics and its implication for replication. IEEE Transactions on Multimedia 17, 8, 1273--1285.

Digital Library

[34]

Yipeng Zhou, Tom Z. J. Fu, and Dah Ming Chiu. 2013. On replication algorithm in P2P VoD. IEEE/ACM Transactions on Networking 21, 1, 233--243.

Digital Library

[35]

Yipeng Zhou, Tom Z. J. Fu, and Dah Ming Chiu. 2015. A unifying model and analysis of P2P VoD replication and scheduling. IEEE/ACM Transactions on Networking 23, 4, 1163--1175.

Digital Library

Cited By

Shi ZZhou YWu DWang C(2022)PPVC: Online Learning Toward Optimized Video Content CachingIEEE/ACM Transactions on Networking10.1109/TNET.2021.313203830:3(1029-1044)Online publication date: Jun-2022
https://doi.org/10.1109/TNET.2021.3132038
Zhu GGu W(2022)User Mapping Strategy in Multi-CDN Streaming: A Data-Driven ApproachIEEE Internet of Things Journal10.1109/JIOT.2021.31125619:9(6638-6649)Online publication date: 1-May-2022
https://doi.org/10.1109/JIOT.2021.3112561
Sun XZhao KJiang WJin X(2021)Method of Analyzing and Managing Volleyball Action by Using Action Sensor of Mobile DeviceJournal of Sensors10.1155/2021/62329682021:1Online publication date: 21-Sep-2021
https://doi.org/10.1155/2021/6232968
Show More Cited By

Index Terms

Design, Implementation, and Measurement of a Crowdsourcing-Based Content Distribution Platform
1. Information systems
  1. Information systems applications
    1. Multimedia information systems
      1. Multimedia streaming
  2. World Wide Web
    1. Web applications
      1. Crowdsourcing
        Incentive schemes
2. Networks
  1. Network types
    1. Overlay and other logical network structures
      1. Peer-to-peer networks

Recommendations

RFC 6707: Content Distribution Network Interconnection (CDNI) Problem Statement
RFC 8006: Content Delivery Network Interconnection (CDNI) Metadata
Efficient content location algorithm for content distribution networks based on distributed construction of search tree from contents of proximal nodes
DBA'06: Proceedings of the 24th IASTED international conference on Database and applications

Content distribution networks (CDNs) are a type of distributed database using geographically dispersed servers to efficiently distribute large multimedia contents. Among various kinds of CDNs are those resembling peer-to-peer (P2P) networks in which all ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 12, Issue 5s

Special Section on Multimedia Big Data: Networking and Special Section on Best Papers From ACM MMSYS/NOSSDAV 2015

December 2016

288 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3001754

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2016

Accepted: 01 July 2016

Revised: 01 May 2016

Received: 01 August 2015

Published in TOMM Volume 12, Issue 5s

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Shenzhen Science and Technology Foundation
Foundation of Shenzhen City
Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
274
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)2

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Shi ZZhou YWu DWang C(2022)PPVC: Online Learning Toward Optimized Video Content CachingIEEE/ACM Transactions on Networking10.1109/TNET.2021.313203830:3(1029-1044)Online publication date: Jun-2022
https://doi.org/10.1109/TNET.2021.3132038
Zhu GGu W(2022)User Mapping Strategy in Multi-CDN Streaming: A Data-Driven ApproachIEEE Internet of Things Journal10.1109/JIOT.2021.31125619:9(6638-6649)Online publication date: 1-May-2022
https://doi.org/10.1109/JIOT.2021.3112561
Sun XZhao KJiang WJin X(2021)Method of Analyzing and Managing Volleyball Action by Using Action Sensor of Mobile DeviceJournal of Sensors10.1155/2021/62329682021:1Online publication date: 21-Sep-2021
https://doi.org/10.1155/2021/6232968
Zhou YChen JYe GWu DWang JChen M(2021)Collaboratively Replicating Encoded Content on RSUs to Enhance Video Services for VehiclesIEEE Transactions on Mobile Computing10.1109/TMC.2019.296002220:3(877-892)Online publication date: 1-Mar-2021
https://doi.org/10.1109/TMC.2019.2960022
Teja PMishra P(2021)Sealed Bid Single Price Auction Model (SBSPAM)-Based Resource Allocation for 5G NetworksWireless Personal Communications: An International Journal10.1007/s11277-020-07814-0116:3(2633-2650)Online publication date: 1-Feb-2021
https://dl.acm.org/doi/10.1007/s11277-020-07814-0
Wang LLi YPan BWu QYin JXu L(2020)Network Coding for Efficient Video Multicast in Device-to-Device CommunicationsSensors10.3390/s2008225420:8(2254)Online publication date: 16-Apr-2020
https://doi.org/10.3390/s20082254
Alsmirat MSarhan N(2018)Cross-layer optimization for many-to-one wireless video streaming systemsMultimedia Tools and Applications10.1007/s11042-018-5698-x77:19(24789-24811)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1007/s11042-018-5698-x
Xu YXiao ZFeng HYang THu BZhou Y(2017)Modeling Buffer Starvations of Video Streaming in Cellular Networks with Large-Scale Measurement of User BehaviorIEEE Transactions on Mobile Computing10.1109/TMC.2016.261640216:8(2228-2245)Online publication date: 1-Aug-2017
https://doi.org/10.1109/TMC.2016.2616402
Kim TKwon JKim E(2017)Categorization-based video streaming for traffic mitigation in content delivery servicesMultimedia Tools and Applications10.1007/s11042-017-4770-276:23(25495-25510)Online publication date: 1-Dec-2017
https://dl.acm.org/doi/10.1007/s11042-017-4770-2
Siekkinen MMasala EKämäräinen TGill PHeidemann JByers JGovindan R(2016)A First Look at Quality of Mobile Live Streaming ExperienceProceedings of the 2016 Internet Measurement Conference10.1145/2987443.2987472(477-483)Online publication date: 14-Nov-2016
https://dl.acm.org/doi/10.1145/2987443.2987472

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents