demonstration

Online aggregation and continuous query support in MapReduce

Authors:

Tyson Condie,

Neil Conway,

Peter Alvaro,

Joseph M. Hellerstein,

John Gerth,

Justin Talbot,

Khaled Elmeleegy,

Russell SearsAuthors Info & Claims

SIGMOD '10: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data

Pages 1115 - 1118

https://doi.org/10.1145/1807167.1807295

Published: 06 June 2010 Publication History

Get Access

Abstract

MapReduce is a popular framework for data-intensive distributed computing of batch jobs. To simplify fault tolerance, the output of each MapReduce task and job is materialized to disk before it is consumed. In this demonstration, we describe a modified MapReduce architecture that allows data to be pipelined between operators. This extends the MapReduce programming model beyond batch processing, and can reduce completion times and improve system utilization for batch jobs as well. We demonstrate a modified version of the Hadoop MapReduce framework that supports online aggregation, which allows users to see "early returns" from a job as it is being computed. Our Hadoop Online Prototype (HOP) also supports continuous queries, which enable MapReduce programs to be written for applications such as event monitoring and stream processing. HOP retains the fault tolerance properties of Hadoop, and can run unmodified user-defined MapReduce programs.

References

[1]

R. Avnur and J. M. Hellerstein. Eddies: Continuously adaptive query processing. In SIGMOD, pages 261--272, 2000.

Digital Library

Google Scholar

[2]

M. Bostock and J. Heer. Protovis: A graphical toolkit for visualization. IEEE Transactions on Visualization and Computer Graphics. 15(6):1121--1128, 2009.

Digital Library

Google Scholar

[3]

T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears. Mapreduce online. In NSDI, 2010.

Digital Library

Google Scholar

[4]

J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, pages 137--150, 2004.

Digital Library

Google Scholar

[5]

J. M. Hellerstein, R. Avnur, A. Chou, C. Hidber, C. Olston, V. Raman, T. Roth, and P. J. Haas. Interactive data analysis with CONTROL. IEEE Computer, 32(8), Aug. 1999.

Digital Library

Google Scholar

[6]

J. M. Hellerstein, P. J. Haas, and H. J. Wang. Online aggregation. In SIGMOD, pages 171--182, 1997.

Digital Library

Google Scholar

[7]

C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: a not-so-foreign language for data processing. In SIGMOD, 2008.

Digital Library

Google Scholar

[8]

R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the data: Parallel analysis with Sawzall. Scientific Programming, 13(4):277--298, 2005.

Digital Library

Google Scholar

[9]

P. N. Skomoroch. Wikipedia page traffic statistics, 2009.

Google Scholar

[10]

A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive - a warehousing solution over a map-reduce framework. In VLDB, 2009.

Digital Library

Google Scholar

[11]

W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan. Detecting large-scale system problems by mining console logs. In SOSP, 2009.

Digital Library

Google Scholar

Cited By

View all

Ye SXu XWang YFu T(2023)Efficient Complex Aggregate Queries with Accuracy Guarantee Based on Execution Cost Model over Knowledge GraphsMathematics10.3390/math1118390811:18(3908)Online publication date: 14-Sep-2023
https://doi.org/10.3390/math11183908
Sheoran NChockchowwat SChheda AWang SVerma RPark Y(2023)A Step Toward Deep Online AggregationProceedings of the ACM on Management of Data10.1145/35892691:2(1-28)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1145/3589269
Usha DRaman R(2022)A Forensic Way to Find Solutions for Security Challenges in Cloudserver Through MapReduce TechniqueHandbook of Research on Technologies and Systems for E-Collaboration During Global Crises10.4018/978-1-7998-9640-1.ch021(330-338)Online publication date: 2022
https://doi.org/10.4018/978-1-7998-9640-1.ch021
Show More Cited By

Index Terms

Online aggregation and continuous query support in MapReduce
1. Information systems
  1. Information systems applications
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Abstraction, modeling and modularity

Recommendations

MapReduce online
NSDI'10: Proceedings of the 7th USENIX conference on Networked systems design and implementation

MapReduce is a popular framework for data-intensive distributed computing of batch jobs. To simplify fault tolerance, many implementations of MapReduce materialize the entire output of each map and reduce task before it can be consumed. In this paper, ...
MapReduce: Review and open challenges

The continuous increase in computational capacity over the past years has produced an overwhelming flow of data or big data, which exceeds the capabilities of conventional processing tools. Big data signify a new era in data exploration and utilization. ...
Survey on improving the performance of MapReduce in Hadoop
NISS '21: Proceedings of the 4th International Conference on Networking, Information Systems & Security

Hadoop has become the most popular and the most used platform in distributed data processing, Hadoop is also an open-source software that implements the MapReduce model for processing big data, it has taken a large part in scientific research in the ...

Comments

Information & Contributors

Information

Published In

SIGMOD '10: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data

June 2010

1286 pages

ISBN:9781450300322

DOI:10.1145/1807167

General Chair:
Ahmed Elmagarmid
Purdue University, USA
,
Program Chair:
Divyakant Agrawal
University of California at Santa Barbara, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 June 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tag

mapreduce

Qualifiers

Demonstration

Conference

SIGMOD/PODS '10

Sponsor:

SIGMOD

SIGMOD/PODS '10: International Conference on Management of Data

June 6 - 10, 2010

Indiana, Indianapolis, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

98
Total Citations
View Citations
1,391
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)7

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Ye SXu XWang YFu T(2023)Efficient Complex Aggregate Queries with Accuracy Guarantee Based on Execution Cost Model over Knowledge GraphsMathematics10.3390/math1118390811:18(3908)Online publication date: 14-Sep-2023
https://doi.org/10.3390/math11183908
Sheoran NChockchowwat SChheda AWang SVerma RPark Y(2023)A Step Toward Deep Online AggregationProceedings of the ACM on Management of Data10.1145/35892691:2(1-28)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1145/3589269
Usha DRaman R(2022)A Forensic Way to Find Solutions for Security Challenges in Cloudserver Through MapReduce TechniqueHandbook of Research on Technologies and Systems for E-Collaboration During Global Crises10.4018/978-1-7998-9640-1.ch021(330-338)Online publication date: 2022
https://doi.org/10.4018/978-1-7998-9640-1.ch021
Peng JDing BWang JZeng KZhou JIves ZBonifati AEl Abbadi A(2022)One Size Does Not Fit All: A Bandit-Based Sampler Combination Framework with Theoretical GuaranteesProceedings of the 2022 International Conference on Management of Data10.1145/3514221.3517900(531-544)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3517900
Wang YKhan AXu XJin JHong QFu T(2022)Aggregate Queries on Knowledge Graphs: Fast Approximation with Semantic-aware Sampling2022 IEEE 38th International Conference on Data Engineering (ICDE)10.1109/ICDE53745.2022.00263(2914-2927)Online publication date: May-2022
https://doi.org/10.1109/ICDE53745.2022.00263
Zhou ZWang YYu XMiao J(2022)A Targeted Privacy-Preserving Data Publishing Method Based on Bayesian NetworkIEEE Access10.1109/ACCESS.2022.320164110(89555-89567)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3201641
Guo DGuo D(2022)Collaborative Management of Correlated Incast TransferData Center Networking10.1007/978-981-16-9368-7_7(161-184)Online publication date: 24-Feb-2022
https://doi.org/10.1007/978-981-16-9368-7_7
Grinev MGrineva MHentschel MKossmann D(2020)Analytics for the real-time webProceedings of the VLDB Endowment10.14778/3402755.34027784:12(1391-1394)Online publication date: 3-Jun-2020
https://dl.acm.org/doi/10.14778/3402755.3402778
Zeitler ERisch T(2020)Massive scale-out of expensive continuous queriesProceedings of the VLDB Endowment10.14778/3402707.34027524:11(1181-1188)Online publication date: 3-Jun-2020
https://dl.acm.org/doi/10.14778/3402707.3402752
Wagan SJunaid MQureshi NShin DChoi K(2020)Comparative Survey on Big data Security Applications, A Blink on Interactive Security Mechanism in Apache Ozone2020 Global Conference on Wireless and Optical Technologies (GCWOT)10.1109/GCWOT49901.2020.9391610(1-6)Online publication date: 6-Oct-2020
https://doi.org/10.1109/GCWOT49901.2020.9391610
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

MapReduce online

MapReduce: Review and open challenges

Survey on improving the performance of MapReduce in Hadoop