research-article

Grand challenge: implementation by frequently emitting parallel windows and user-defined aggregate functions

Authors:

Tore RischAuthors Info & Claims

DEBS '13: Proceedings of the 7th ACM international conference on Distributed event-based systems

Pages 325 - 330

https://doi.org/10.1145/2488222.2488284

Published: 29 June 2013 Publication History

Get Access

Abstract

Our implementation of the DEBS 2013 Challenge is based on a scalable, parallel, and extensible DSMS, which is capable of processing general continuous queries over high volume data streams with low delays. A mechanism to provide user defined incremental aggregate functions over sliding windows of data streams provide real-time processing by emitting results continuously with low delays. To further eliminate delays caused by time critical operations, the system is extensible so that functions can be easily written in some external programming language. The query language provides user defined parallelization primitives where the user can express queries specifying how high volume data streams are split and reduced into lower volume parallel data streams. This enables expensive queries over data streams to be executed in parallel based on application knowledge. Our OS-independent implementation was tested on several computers and achieves the real-time requirement of the challenge on a regular PC.

References

[1]

Bai, Y., Thakkar, H., Wang, H., Luo, C., and Zaniolo, C.: A Data Stream Language and System Designed for Power and Extensibility. Proc. CIKM Conf., 2006.

Digital Library

Google Scholar

[2]

Botan, I., Derakhshan, R., Dindar, N., Haas, L., Miller, R. J. and Tatbul, N. SECRET: A Model for Analysis of the Execution Semantics of Stream Processing Systems. Proc. VLDB Conf., 2010.

Digital Library

Google Scholar

[3]

Botan, I., Fischer, P. M., Florescu, D., Kossmann, D., Kraska, T., and Tamosevicius, R. Extending XQuery with Window Functions. Proc. VLDB Conf., 2007.

Digital Library

Google Scholar

[4]

http://esper.codehaus.org/

Google Scholar

[5]

Law, Y-N, Wang, H., and Zaniolo, C.: Relational Languages and Data Models for Continuous Queries on Sequences and Data Streams. ACM TODS 36, 2, (May 2011).

Digital Library

Google Scholar

[6]

Li, J., Maier, D., Tufte, K., Papadimos,V., and Tucker, P. A. Semantics and evaluation techniques for window aggregates in data streams. Proc. SIGMOD Conf., pp. 311--322, 2005.

Digital Library

Google Scholar

[7]

Patroumpas, K. and Sellis, T. Window specification over data streams. Proc. EDBT Conf., 2006.

Digital Library

Google Scholar

[8]

Thakkar, H., Mozafari, B. and Zaniolo, C.: Designing an Inductive Data Stream Management System: the Stream Mill Experience. Proc. 2nd International Workshop on Scalable Stream Processing Systems, 2008.

Digital Library

Google Scholar

[9]

Zeitler, E. and Risch, T.: Massive scale-out of expensive continuous queries, Proc. of the VLDB Endowment, ISSN 2150--8097, Vol. 4, No. 11, pp.1181--1188, 2011

Google Scholar

Cited By

View all

Menconi SZiffer GBernardo AValle E(2023)Continuous Soccer Pass Detection: A Comparison between Traditional and Streaming Machine Learning Methods2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386458(2673-2682)Online publication date: 15-Dec-2023
https://doi.org/10.1109/BigData59044.2023.10386458
Probst LSchuldt HSeidenschwarz PRumo M(2020)StreamTeam-Football: Analyzing Football Matches in Real-Time on the Basis of Position Streams2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9377914(637-644)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9377914
Probst LRauschenbach FSchuldt HSeidenschwarz PRumo M(2018)Integrated Real-Time Data Stream Analysis and Sketch-Based Video Retrieval in Team Sports2018 IEEE International Conference on Big Data (Big Data)10.1109/BigData.2018.8622592(548-555)Online publication date: Dec-2018
https://doi.org/10.1109/BigData.2018.8622592
Show More Cited By

Index Terms

Grand challenge: implementation by frequently emitting parallel windows and user-defined aggregate functions
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Database query processing
      2. Parallel and distributed DBMSs
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Database theory
      1. Database query processing and optimization (theory)

Recommendations

MedSMan: a live multimedia stream querying system

Querying live media streams is a challenging problem that is becoming an essential requirement in a growing number of applications. Research in multimedia information systems has addressed and made good progress in dealing with archived data. Meanwhile, ...
The DEBS 2016 grand challenge
DEBS '16: Proceedings of the 10th ACM International Conference on Distributed and Event-based Systems

The DEBS Grand Challenge is a series of challenges which address problems in event stream processing. The focus of the Grand Challenge in 2016 is on processing of data streams that originate from social networks. Hence, the data represents an evolving ...
Sketching distributed sliding-window data streams

While traditional data management systems focus on evaluating single, ad hoc queries over static data sets in a centralized setting, several emerging applications require (possibly, continuous) answers to queries on dynamic data that is widely ...

Comments

Information & Contributors

Information

Published In

DEBS '13: Proceedings of the 7th ACM international conference on Distributed event-based systems

June 2013

360 pages

ISBN:9781450317580

DOI:10.1145/2488222

General Chairs:
Sharma Chakravarthy
The University of Texas at Arlington, USA
,
Susan D. Urban
Texas Tech University, USA
,
Program Chairs:
Peter Pietzuch
Imperial College, UK
,
Elke Rundensteiner
Worcester Polytechnic Institute, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 June 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

DEBS '13

Sponsor:

DEBS '13: The 7th ACM International Conference on Distributed Event-Based Systems

June 29 - July 3, 2013

Texas, Arlington, USA

Acceptance Rates

DEBS '13 Paper Acceptance Rate 16 of 58 submissions, 28%;

Overall Acceptance Rate 145 of 583 submissions, 25%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
180
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Menconi SZiffer GBernardo AValle E(2023)Continuous Soccer Pass Detection: A Comparison between Traditional and Streaming Machine Learning Methods2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386458(2673-2682)Online publication date: 15-Dec-2023
https://doi.org/10.1109/BigData59044.2023.10386458
Probst LSchuldt HSeidenschwarz PRumo M(2020)StreamTeam-Football: Analyzing Football Matches in Real-Time on the Basis of Position Streams2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9377914(637-644)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9377914
Probst LRauschenbach FSchuldt HSeidenschwarz PRumo M(2018)Integrated Real-Time Data Stream Analysis and Sketch-Based Video Retrieval in Team Sports2018 IEEE International Conference on Big Data (Big Data)10.1109/BigData.2018.8622592(548-555)Online publication date: Dec-2018
https://doi.org/10.1109/BigData.2018.8622592
Bauleo ECarnevale SCatarci TKimani SLeva MMecella M(2018)Design, realization and user evaluation of the SmartVortex Visual Query System for accessing data streams in industrial engineering applicationsJournal of Visual Languages and Computing10.1016/j.jvlc.2014.08.00225:5(577-601)Online publication date: 25-Dec-2018
https://dl.acm.org/doi/10.1016/j.jvlc.2014.08.002
Probst LBrix FSchuldt HRumo M(2017)Real-Time Football Analysis with StreamTeamProceedings of the 11th ACM International Conference on Distributed and Event-based Systems10.1145/3093742.3095089(319-322)Online publication date: 8-Jun-2017
https://dl.acm.org/doi/10.1145/3093742.3095089

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

MedSMan: a live multimedia stream querying system

The DEBS 2016 grand challenge

Sketching distributed sliding-window data streams