Article

A common substrate for cluster computing

Authors:

Benjamin Hindman,

Andy Konwinski,

Ion StoicaAuthors Info & Claims

HotCloud'09: Proceedings of the 2009 conference on Hot topics in cloud computing

Article No.: 19

Published: 15 June 2009 Publication History

Abstract

The success of MapReduce has sparked many efforts to design cluster computing frameworks. We argue that no single framework will be optimal for all applications, and that we should instead enable organizations to run multiple frameworks effciently in the same cloud. Furthermore, to ease development of new frameworks, it is critical to identify common abstractions and modularize their architectures. To achieve these goals, we propose Nexus, a low-level substrate that provides isolation and efficient resource sharing across frameworks running on the same cluster, while giving each framework freedom to implement its own programming model and fully control the execution of its jobs. Nexus fosters innovation in the cloud by letting organizations run new frameworks alongside existing ones and by letting framework developers focus on specific applications rather than building one-size-fits-all frameworks.

References

[1]

Amazon EC2. http://aws.amazon.com/ec2/.

[2]

distcc: a fast, free distributed c/c++ compiler. http: //distcc.samba.org/.

[3]

Hadoop. http://lucene.apache.org/hadoop.

[4]

Hadoop scheduling discussion. http://issues.apache. org/jira/browse/HADOOP-4667.

[5]

M. J. Accetta, R. V. Baron, W. J. Bolosky, D. B. Golub, R. F. Rashid, A. Tevanian, and M. Young. Mach: A new kernel foundation for unix development. In USENIX Summer, pages 93-113, 1986.

[6]

R. D. Blumofe, C. F. Joerg, B. C. Kuszmaul, C. E. Leiserson, K. H. Randall, and Y. Zhou. Cilk: An efficient multithreaded runtime system. In Journal of Parallel and Distributed Computing, pages 207-216, 1995.

Digital Library

[7]

V. G. Cerf, Robert, and E. Icahn. A protocol for packet network intercommunication. IEEE Transactions on Communications, 22:637-648, 1974.

[8]

J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, pages 137-150, 2004.

Digital Library

[9]

D. J. DeWitt, E. Paulson, E. Robinson, J. F. Naughton, J. Royalty, S. Shankar, and A. Krioukov. Clustera: an integrated computation and data management system. PVLDB, 1(1):28-41, 2008.

Digital Library

[10]

D. R. Engler, M. F. Kaashoek, and J. O'Toole. Exokernel: An operating system architecture for application-level resource management. In SOSP, pages 251-266, 1995.

Digital Library

[11]

E. C. Hendricks and T. C. Hartmann. Evolution of a virtual machine subsystem. IBM Systems Journal, 18(1):111-142, 1979.

Digital Library

[12]

M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. In EuroSys 07, pages 59-72, 2007.

Digital Library

[13]

D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, and D. Zagorodnov. The eucalyptus open-source cloud-computing system. In Proceedings of Cloud Computing and Its Applications {Online}, Chicago, Illinois, 10 2008.

[14]

C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: a not-so-foreign language for data processing. In SIGMOD Conference, pages 1099-1110, 2008.

Digital Library

[15]

R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the data: Parallel analysis with sawzall. Scientific Programming, 13(4):277-298, 2005.

Digital Library

[16]

J. Reinders. Intel Threading Building Blocks: Outfitting C++ for Mul ti-core PRocessor Parallelism. O'Reilly, 2007.

Digital Library

[17]

G. Staples. Torque - torque resource manager. In SC, page 8, 2006.

Digital Library

[18]

D. Thain, T. Tannenbaum, and M. Livny. Distributed computing in practice: the Condor experience. Concurrency and Computation - Practice and Experience, 17(2- 4):323-356, 2005.

Digital Library

Cited By

Midoglu CAvelino MGopalakrishnan SPham SHalvorsen PTimmerer CGrois DÇetinkaya ENguyen M(2022)Multimedia streaming analyticsProceedings of the 1st Mile-High Video Conference10.1145/3510450.3517321(62-69)Online publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1145/3510450.3517321
Celaya JArronategui U(2015)Fair scheduling of bag-of-tasks applications on large-scale platformsFuture Generation Computer Systems10.1016/j.future.2015.03.00249:C(28-44)Online publication date: 1-Aug-2015
https://dl.acm.org/doi/10.1016/j.future.2015.03.002
Ayuso LJordan HFahringer TKornberger BSchifko MHöckner BMoosbrugger SVerma K(2014)Parallelizing a CAD Model Processing Tool from the Automotive IndustryRevised Selected Papers, Part I, of the Euro-Par 2014 International Workshops on Parallel Processing - Volume 880510.1007/978-3-319-14325-5_3(24-35)Online publication date: 25-Aug-2014
https://dl.acm.org/doi/10.1007/978-3-319-14325-5_3
Show More Cited By

A common substrate for cluster computing

Recommendations

Cloud Computing and the Common Man

The cloud offers several advantages, but until some of its risks are better understood many major players might hold back.
Reusable mobile agents for cluster computing

Adopting mobile agent technology can eliminate the need for the administrator to manage cluster computing systems, e.g. installation and upgrading software; and auditing clusters and networks. However, creating mobile agent-based network management ...
Common LeSS Transformation Patterns
SAC '24: Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing

Various scaled agile frameworks have been developed to address the challenges of implementing agile methods in large-scale projects. Adoption of these frameworks is quite demanding. The paper aims to analyze selected case studies focused on the adoption ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

HotCloud'09: Proceedings of the 2009 conference on Hot topics in cloud computing

June 2009

22 pages

Publisher

USENIX Association

United States

Publication History

Published: 15 June 2009

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Midoglu CAvelino MGopalakrishnan SPham SHalvorsen PTimmerer CGrois DÇetinkaya ENguyen M(2022)Multimedia streaming analyticsProceedings of the 1st Mile-High Video Conference10.1145/3510450.3517321(62-69)Online publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1145/3510450.3517321
Celaya JArronategui U(2015)Fair scheduling of bag-of-tasks applications on large-scale platformsFuture Generation Computer Systems10.1016/j.future.2015.03.00249:C(28-44)Online publication date: 1-Aug-2015
https://dl.acm.org/doi/10.1016/j.future.2015.03.002
Ayuso LJordan HFahringer TKornberger BSchifko MHöckner BMoosbrugger SVerma K(2014)Parallelizing a CAD Model Processing Tool from the Automotive IndustryRevised Selected Papers, Part I, of the Euro-Par 2014 International Workshops on Parallel Processing - Volume 880510.1007/978-3-319-14325-5_3(24-35)Online publication date: 25-Aug-2014
https://dl.acm.org/doi/10.1007/978-3-319-14325-5_3
Sakr SLiu AFayoumi A(2013)The family of mapreduce and large-scale data processing systemsACM Computing Surveys10.1145/2522968.252297946:1(1-44)Online publication date: 11-Jul-2013
https://dl.acm.org/doi/10.1145/2522968.2522979
Ramakrishnan RCISL TGrossman RUthurusamy RDhillon IKoren Y(2013)Scale-out beyond map-reduceProceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining10.1145/2487575.2492151(1-1)Online publication date: 11-Aug-2013
https://dl.acm.org/doi/10.1145/2487575.2492151
Condie TMineiro PPolyzotis NWeimer MRoss KSrivastava DPapadias D(2013)Machine learning for big dataProceedings of the 2013 ACM SIGMOD International Conference on Management of Data10.1145/2463676.2465338(939-942)Online publication date: 22-Jun-2013
https://dl.acm.org/doi/10.1145/2463676.2465338
Low YBickson DGonzalez JGuestrin CKyrola AHellerstein J(2012)Distributed GraphLabProceedings of the VLDB Endowment10.14778/2212351.22123545:8(716-727)Online publication date: 1-Apr-2012
https://dl.acm.org/doi/10.14778/2212351.2212354
Huai YLee RZhang SXia CZhang XChase JAbbadi A(2011)DOTProceedings of the 2nd ACM Symposium on Cloud Computing10.1145/2038916.2038920(1-14)Online publication date: 26-Oct-2011
https://dl.acm.org/doi/10.1145/2038916.2038920
Zaharia MChowdhury MFranklin MShenker SStoica I(2010)SparkProceedings of the 2nd USENIX conference on Hot topics in cloud computing10.5555/1863103.1863113(10-10)Online publication date: 22-Jun-2010
https://dl.acm.org/doi/10.5555/1863103.1863113
Cafarella MRé CDong XNaumann F(2010)ManimalProcceedings of the 13th International Workshop on the Web and Databases10.1145/1859127.1859141(1-6)Online publication date: 6-Jun-2010
https://dl.acm.org/doi/10.1145/1859127.1859141
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents