Article

Distributed streams algorithms for sliding windows

Authors:

Phillip B. Gibbons,

Srikanta TirthapuraAuthors Info & Claims

SPAA '02: Proceedings of the fourteenth annual ACM symposium on Parallel algorithms and architectures

Pages 63 - 72

https://doi.org/10.1145/564870.564880

Published: 10 August 2002 Publication History

Abstract

This paper presents algorithms for estimating aggregate functions over a "sliding window" of the N most recent data items in one or more streams. Our results include

For a single stream, we present the first ε-approximation scheme for the number of 1's in a sliding window that is optimal in both worst case time and space. We also present the first ε for the sum of integers in [0.R] in a sliding window that is optimal in both worst case time and space (assuming R is at most polynomial in N). Both algorithms are deterministic and use only logarithmic memory words.

In contrast, we show that an deterministic algorithm that estimates, to within a small constant relative error, the number of 1's (or the sum of integers) in a sliding window over the union of distributed streams requires Ω(N) space.

We present the first randomized (ε,δ)-approximation scheme for the number of 1's in a sliding window over the union of distributed streams that uses only logarithmic memory words. We also present the first (ε,δ)-approximation scheme for the number of distinct values in a sliding window over distributed streams that uses only logarithmic memory words.

Our results are obtained using a novel family of synopsis data structures.

References

[1]

N. Alon, Y. Matias, and M. Szegedy. The space complexity of approximating the frequency moments. J. of Computer and System Sciences, 58:137--147, 1999.]]

Digital Library

[2]

B. Babcock, M. Datar, and R. Motwani. Sampling from a moving window over streaming data. In Proc. 13th ACM-SIAM Symp. on Discrete Algorithms, pages 633--634, Jan. 2002. Short paper.]]

Digital Library

[3]

Z. Bar-Yossef, R. Kumar, and D. Sivakumar. Reductions in streaming algorithms, with an application to counting triangles in graphs. In Proc. 13th ACM-SIAM Symp. on Discrete Algorithms, pages 623--632, Jan. 2002.]]

Digital Library

[4]

M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows. In Proc. 13th ACM-SIAM Symp. on Discrete Algorithms, pages 635--644, Jan. 2002.]]

Digital Library

[5]

J. Feigenbaum, S. Kannan, M. Strauss, and M. Viswanathan. An approximate L1-difference algorithm for massive data streams. In Proc. 40th IEEE Symp. on Foundations of Computer Science, pages 501--511, Oct. 1999.]]

Digital Library

[6]

J. Feigenbaum, S. Kannan, M. Strauss, and M. Viswanathan. Testing and spot-checking of data streams. In Proc. 11th ACM-SIAM Symp. on Discrete Algorithms, pages 165--174, Jan. 2000.]]

Digital Library

[7]

P. Flajolet and G. N. Martin. Probabilistic counting algorithms for data base applications. J. Computer and System Sciences, 31:182--209, 1985.]]

Digital Library

[8]

J. Fong and M. Strauss. An approximate Lp-difference algorithm for massive data streams. In Proc. 17th Symp. on Theoretical Aspects of Computer Science, LNCS 1770, pages 193--204. Springer, Feb. 2000.]]

Digital Library

[9]

J. Gehrke, F. Korn, and D. Srivastava. On computing correlated aggregates over continual data streams. In Proc. ACM SIGMOD International Conf. on Management of Data, pages 13--24, May 2001.]]

Digital Library

[10]

P. B. Gibbons. Distinct sampling for highly-accurate answers to distinct values queries and event reports. In Proc. 27th International Conf. on Very Large Data Bases, pages 541--550, Sept. 2001.]]

Digital Library

[11]

P. B. Gibbons and Y. Matias. New sampling-based summary statistics for improving approximate query answers. In Proc. ACM SIGMOD International Conf. on Management of Data, pages 331--342, June 1998.]]

Digital Library

[12]

P. B. Gibbons and Y. Matias. Synopsis data structures for massive data sets. In J. M. Abello and J. S. Vitter, editors, External Memory Algorithms, pages 39--70. AMS, 1999. DIMACS: Series in Discrete Mathematics and Theoretical Computer Science, Vol. 50. A two page summary appeared as a short paper in SODA'99.]]

Digital Library

[13]

P. B. Gibbons and S. Tirthapura. Estimating simple functions on the union of data streams. In Proc. 13th ACM Symp. on Parallel Algorithms and Architectures, pages 281--290, July 2001.]]

Digital Library

[14]

A. C. Gilbert, Y. Kotidis, and M. J. Strauss. Surfing wavelets on streams: One-pass summaries for approximate aggregate queries. In Proc. 27th International Conf. on Very Large Data Bases, pages 79--88, Sept. 2001.]]

Digital Library

[15]

M. Greenwald and S. Khanna. Space-efficient online computation of quantile summaries. In Proc. ACM SIGMOD International Conf. on Management of Data, pages 58--66, May 2001.]]

Digital Library

[16]

S. Guha, N. Koudas, and K. Shim. Data-streams and histograms. In Proc. 33rd ACM Symp. on Theory of Computing, pages 471--475, July 2001.]]

Digital Library

[17]

S. Guha, N. Mishra, R. Motwani, and L. O'Callaghan. Clustering data streams. In Proc. 41st IEEE Symp. on Foundations of Computer Science, pages 359--366, Nov. 2000.]]

Digital Library

[18]

M. R. Henzinger, P. Raghavan, and S. Rajagopalan. Computing on data streams. Technical report, Digital Systems Research Center, Palo Alto, CA, May 1998.]]

[19]

P. Indyk. Stable distributions, pseudorandom generators, embeddings and data stream computation. In Proc. 41st IEEE Symp. on Foundations of Computer Science, pages 189--197, Nov. 2000.]]

Digital Library

[20]

P. Indyk. Personal communication, 2002.]]

[21]

I. Kremer, N. Nisan, and D. Ron. On randomized one-round communication complexity. Computational Complexity, 8(1):21--49, 1999.]]

Digital Library

[22]

E. Kushilevitz and N. Nisan. Communication Complexity. Cambridge University Press, Cambridge, UK, 1997.]]

Digital Library

[23]

I. Newman. Private vs. common random bits in communication complexity. Information Processing Letters, 39:67--71, 1991.]]

Digital Library

[24]

I. Newman and M. Szegedy. Public vs. private coin flips in one round communication games. In Proc. 28th ACM Symp. on the Theory of Computing, pages 561--570, May 1996.]]

Digital Library

[25]

L. Trevisan. A note on counting distinct elements in the streaming model. Unpublished manuscript, April 2001.]]

Cited By

Lam KZhao XZhu CKuo T(2024)MVLevelDB+: Meeting Relative Consistency Requirements of Temporal Queries in Sensor Stream DatabasesACM Transactions on Embedded Computing Systems10.1145/369478724:1(1-26)Online publication date: 4-Sep-2024
https://dl.acm.org/doi/10.1145/3694787
Jain SJain PSrivastava A(2024)An Efficient Anomaly Detection Approach Using Cube Sampling with Streaming DataPattern Recognition and Machine Intelligence10.1007/978-3-031-12700-7_51(498-505)Online publication date: 24-Jul-2024
https://doi.org/10.1007/978-3-031-12700-7_51
D PPrem M V(2023)A Novel Ensemble Classifier Framework to Preprocess, Learn and Predict Imbalanced Heterogeneous Drifted Data Stream2023 Second International Conference on Electrical, Electronics, Information and Communication Technologies (ICEEICT)10.1109/ICEEICT56924.2023.10157417(1-7)Online publication date: 5-Apr-2023
https://doi.org/10.1109/ICEEICT56924.2023.10157417
Show More Cited By

Index Terms

Distributed streams algorithms for sliding windows
1. Information systems
  1. Information systems applications
2. Theory of computation
  1. Logic
  2. Models of computation

Recommendations

Optimal sampling from sliding windows
PODS '09: Proceedings of the twenty-eighth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

A sliding windows model is an important case of the streaming model, where only the most "recent" elements remain active and the rest are discarded in a stream. The sliding windows model is important for many applications (see, e.g., Babcock, Babu, ...
Optimal sampling from sliding windows

A sliding windows model is an important case of the streaming model, where only the most ''recent'' elements remain active and the rest are discarded. The sliding windows model is important for many applications (see, e.g., Babcock, Babu, Datar, Motwani ...
Variance estimation over sliding windows
PODS '07: Proceedings of the twenty-sixth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

Capturing characteristics of large data streams has received considerable attention. The constraints in space and time restrict the data stream processing to only one pass (or a small number of passes). Processing data streams over sliding windows make ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SPAA '02: Proceedings of the fourteenth annual ACM symposium on Parallel algorithms and architectures

August 2002

302 pages

ISBN:1581135297

DOI:10.1145/564870

Conference Chair:
Arnold Rosenberg
University of Massachusetts
,
Program Chair:
Bruce Maggs
Carnegie Mellon University

Copyright © 2002 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 August 2002

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SPAA02

Sponsor:

SPAA02: Fourteenth ACM Symposium on Parallelism in Algorithms and Architectures

August 10 - 13, 2002

Manitoba, Winnipeg, Canada

Acceptance Rates

Overall Acceptance Rate 447 of 1,461 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

90
Total Citations
View Citations
934
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lam KZhao XZhu CKuo T(2024)MVLevelDB+: Meeting Relative Consistency Requirements of Temporal Queries in Sensor Stream DatabasesACM Transactions on Embedded Computing Systems10.1145/369478724:1(1-26)Online publication date: 4-Sep-2024
https://dl.acm.org/doi/10.1145/3694787
Jain SJain PSrivastava A(2024)An Efficient Anomaly Detection Approach Using Cube Sampling with Streaming DataPattern Recognition and Machine Intelligence10.1007/978-3-031-12700-7_51(498-505)Online publication date: 24-Jul-2024
https://doi.org/10.1007/978-3-031-12700-7_51
D PPrem M V(2023)A Novel Ensemble Classifier Framework to Preprocess, Learn and Predict Imbalanced Heterogeneous Drifted Data Stream2023 Second International Conference on Electrical, Electronics, Information and Communication Technologies (ICEEICT)10.1109/ICEEICT56924.2023.10157417(1-7)Online publication date: 5-Apr-2023
https://doi.org/10.1109/ICEEICT56924.2023.10157417
Shein AChrysanthis P(2022)Multi-Query Optimization of Incrementally Evaluated Sliding-Window AggregationsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2020.302977034:8(3899-3911)Online publication date: 1-Aug-2022
https://doi.org/10.1109/TKDE.2020.3029770
Jain PJain SZaiane OSrivastava A(2022)Anomaly Detection in Resource Constrained Environments With Streaming DataIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2021.30706606:3(649-659)Online publication date: Jun-2022
https://doi.org/10.1109/TETCI.2021.3070660
Wu GYun XWang YWang SLi BLiu Y(2022)A Sketching Approach for Obtaining Real-Time Statistics Over Data Streams in CloudIEEE Transactions on Cloud Computing10.1109/TCC.2020.298702310:2(1462-1475)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TCC.2020.2987023
William EChan M(2022)Low-Power Distinct Sum for Wireless Sensor Networks2022 18th International Conference on Distributed Computing in Sensor Systems (DCOSS)10.1109/DCOSS54816.2022.00022(65-68)Online publication date: May-2022
https://doi.org/10.1109/DCOSS54816.2022.00022
Basat RJo SSatti SUgare S(2021)Approximate Query Processing over Static Sets and Sliding WindowsTheoretical Computer Science10.1016/j.tcs.2021.06.015Online publication date: Jun-2021
https://doi.org/10.1016/j.tcs.2021.06.015
Mytilinis ITsoumakos DKoziris N(2021)Workload-aware wavelet synopses for sliding window aggregatesDistributed and Parallel Databases10.1007/s10619-020-07307-w39:2(445-482)Online publication date: 1-Jun-2021
https://dl.acm.org/doi/10.1007/s10619-020-07307-w
Nguyen TShih MSrivastava DTirthapura SXu B(2020)Stratified random sampling from streaming and stored dataDistributed and Parallel Databases10.1007/s10619-020-07315-wOnline publication date: 23-Oct-2020
https://doi.org/10.1007/s10619-020-07315-w
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents