research-article

Watermarks in stream processing systems: semantics and comparative analysis of Apache Flink and Google cloud dataflow

Authors:

Slava Chernyak,

Kathryn Knight,

Kenneth Knowles,

Dan SotolongoAuthors Info & Claims

Proceedings of the VLDB Endowment, Volume 14, Issue 12

Pages 3135 - 3147

https://doi.org/10.14778/3476311.3476389

Published: 01 July 2021 Publication History

Abstract

Streaming data processing is an exercise in taming disorder: from oftentimes huge torrents of information, we hope to extract powerful and timely analyses. But when dealing with streaming data, the unbounded and temporally disordered nature of real-world streams introduces a critical challenge: how does one reason about the completeness of a stream that never ends? In this paper, we present a comprehensive definition and analysis of watermarks, a key tool for reasoning about temporal completeness in infinite streams.

First, we describe what watermarks are and why they are important, highlighting how they address a suite of stream processing needs that are poorly served by eventually-consistent approaches:

• Computing a single correct answer, as in notifications.

• Reasoning about a lack of data, as in dip detection.

• Performing non-incremental processing over temporal subsets of an infinite stream, as in statistical anomaly detection with cubic spline models.

• Safely and punctually garbage collecting obsolete inputs and intermediate state.

• Surfacing a reliable signal of overall pipeline health.

Second, we describe, evaluate, and compare the semantically equivalent, but starkly different, watermark implementations in two modern stream processing engines: Apache Flink and Google Cloud Dataflow.

References

[1]

T. Akidau, A. Balikov, K. Bekiroğlu, S. Chernyak, J. Haberman, R. Lax, S. McVeety, D. Mills, P. Nordstrom, and S. Whittle. Millwheel: Fault-tolerant stream processing at internet scale. Proc. VLDB Endow., 6(11):1033--1044, Aug. 2013.

Digital Library

[2]

T. Akidau, R. Bradshaw, C. Chambers, S. Chernyak, R. J. Fernández-Moctezuma, R. Lax, S. McVeety, D. Mills, F. Perry, E. Schmidt, et al. The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proceedings of the VLDB Endowment, 8(12):1792--1803, 2015.

Digital Library

[3]

T. Akidau, S. Chernyak, and R. Lax. Streaming Systems. O'Reilly Media, Inc., 1st edition, 2018.

Digital Library

[4]

D. Anicic, P. Fodor, S. Rudolph, R. Stühmer, N. Stojanovic, and R. Studer. Etalis: Rule-based reasoning in event processing. In Reasoning in event-based distributed systems, pages 99--124. Springer, 2011.

[5]

A. Awad, J. Traub, and S. Sakr. Adaptive watermarks: A concept drift-based approach for predicting event-time progress in data streams. In EDBT, pages 622--625, 2019.

[6]

P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas. Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 36(4), 2015.

[7]

B. Chandramouli, J. Goldstein, M. Barnett, R. DeLine, D. Fisher, J. C. Platt, J. F. Terwilliger, and J. Wernsing. Trill: A high-performance incremental query processor for diverse analytics. Proceedings of the VLDB Endowment, 8(4):401--412, 2014.

Digital Library

[8]

T. Das. Event-time aggregation and watermarking in apache spark's structured streaming. https://databricks.com/blog/2017/05/08/event-time-aggregation-watermarking-apacmhe-sparks-structured-streaming.html, 2017. [Online; accessed 06-Feb-2021].

[9]

M. J. S. Eno Thereska, Michael Noll. Watermarks, tables, event time, and the dataflow model. https://www.confluent.io/blog/watermarks-tables-event-time-dataflow-model/, 2017. [Online; accessed 25-Jan-2021].

[10]

D. Gyllstrom, E. Wu, H.-J. Chae, Y. Diao, P. Stahlberg, and G. Anderson. Sase: Complex event processing over streams. arXiv preprint cs/0612128, 2006.

[11]

C. S. Jensen and R. Snodgrass. Temporal specialization and generalization. IEEE Transactions on Knowledge and Data Engineering, 6(6):954--974, 1994.

Digital Library

[12]

K. Kulkarni and J.-E. Michels. Temporal features in sql:2011. SIGMOD Rec., 41(3):34--43, Oct. 2012.

Digital Library

[13]

R. Lax. After lambda: Exactly-once processing in cloud dataflow, part 2 (ensuring low latency). https://cloud.google.com/blog/products/gcp/after-lambda-exactly-once-processing-in-cloud-dataflow-part-2-ensuring-low-latency, 2017. [Online; accessed 06-Feb-2021].

[14]

F. McSherry. Timelydataflow. https://github.com/TimelyDataflow/timely-dataflow, 2020.

[15]

F. McSherry, D. G. Murray, R. Isaacs, and M. Isard. Differential dataflow. In CIDR, 2013.

[16]

D. G. Murray, F. McSherry, R. Isaacs, M. Isard, P. Barham, and M. Abadi. Naiad: a timely dataflow system. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pages 439--455, 2013.

Digital Library

[17]

J. Roesler. Kafka streams' take on watermarks and triggers. https://www.confluent.io/blog/kafka-streams-take-on-watermarks-and-triggers/, 2019. [Online; accessed 25-Jan-2021].

[18]

U. Srivastava and J. Widom. Flexible time management in data stream systems. In Proceedings of the twenty-third ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 263--274, 2004.

Digital Library

[19]

J. Teich, L. Thiele, and E. A. Lee. Modeling and simulation of heterogeneous real-time systems based on a deterministic discrete event model. In Proceedings of the 8th international symposium on System synthesis, pages 156--161, 1995.

Digital Library

[20]

P. A. Tucker, D. Maier, T. Sheard, and L. Fegaras. Exploiting punctuation semantics in continuous data streams. IEEE Transactions on Knowledge and Data Engineering, 15(3):555--568, 2003.

Digital Library

[21]

A. Verma, L. Pedrosa, M. R. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes. Large-scale cluster management at Google with Borg. In Proceedings of the European Conference on Computer Systems (EuroSys), Bordeaux, France, 2015.

Digital Library

[22]

G. Wang, L. Chen, A. Dikshit, J. Gustafson, B. Chen, M. J. Sax, J. Roesler, S. Blee-Goldman, B. Cadonna, A. Mehta, V. Madan, and J. Rao. Consistency and completeness: Rethinking distributed stream processing in apache kafka. In Proceedings of the 2021 International Conference on Management of Data, SIGMOD '21, 2021.

Digital Library

[23]

E. Wu, Y. Diao, and S. Rizvi. High-performance complex event processing over streams. In Proceedings of the 2006 ACM SIGMOD international conference on Management of data, pages 407--418, 2006.

Digital Library

Cited By

Lam KZhao XZhu CKuo T(2024)MVLevelDB+: Meeting Relative Consistency Requirements of Temporal Queries in Sensor Stream DatabasesACM Transactions on Embedded Computing Systems10.1145/369478724:1(1-26)Online publication date: 4-Sep-2024
https://dl.acm.org/doi/10.1145/3694787
Kozar ADel Monte BZeuch SMarkl V(2024)Fault Tolerance Placement in the Internet of ThingsProceedings of the ACM on Management of Data10.1145/36549412:3(1-29)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654941
Akidau T(2024)Simplicity and Elegance in Stream Processing: a 5-Year OdysseyProceedings of the 18th ACM International Conference on Distributed and Event-based Systems10.1145/3629104.3674954(4-5)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3629104.3674954
Show More Cited By

Index Terms

Watermarks in stream processing systems: semantics and comparative analysis of Apache Flink and Google cloud dataflow

Index terms have been assigned to the content through auto-classification.

Recommendations

Stream processing with BigData: SSS-MapReduce
CLOUDCOM '12: Proceedings of the 2012 IEEE 4th International Conference on Cloud Computing Technology and Science (CloudCom)

We propose a Map Reduce based stream processing system, called SSS, which is capable of processing stream along with large scale static data. Unlike the existing stream processing systems that can work only on the relatively small on-memory data-set, ...
Dual-Paradigm Stream Processing
ICPP '18: Proceedings of the 47th International Conference on Parallel Processing

Existing stream processing frameworks operate either under data stream paradigm processing data record by record to favor low latency, or under operation stream paradigm processing data in micro-batches to desire high throughput. For complex and mutable ...
Generic windowing support for extensible stream processing systems

Stream processing applications process high volume, continuous feeds from live data sources, employ data-in-motion analytics to analyze these feeds, and produce near real-time insights with low latency. One of the fundamental characteristics of such ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment

Proceedings of the VLDB Endowment Volume 14, Issue 12

July 2021

587 pages

ISSN:2150-8097

Editors:
Xin Luna Dong
Amazon
,
Felix Naumann
HPI, University of Potsdam

Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 July 2021

Published in PVLDB Volume 14, Issue 12

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
173
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)7

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lam KZhao XZhu CKuo T(2024)MVLevelDB+: Meeting Relative Consistency Requirements of Temporal Queries in Sensor Stream DatabasesACM Transactions on Embedded Computing Systems10.1145/369478724:1(1-26)Online publication date: 4-Sep-2024
https://dl.acm.org/doi/10.1145/3694787
Kozar ADel Monte BZeuch SMarkl V(2024)Fault Tolerance Placement in the Internet of ThingsProceedings of the ACM on Management of Data10.1145/36549412:3(1-29)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654941
Akidau T(2024)Simplicity and Elegance in Stream Processing: a 5-Year OdysseyProceedings of the 18th ACM International Conference on Distributed and Event-based Systems10.1145/3629104.3674954(4-5)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3629104.3674954
Meldrum MCarbone P(2024)μWheel: Aggregate Management for Streams and QueriesProceedings of the 18th ACM International Conference on Distributed and Event-based Systems10.1145/3629104.3666031(54-65)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3629104.3666031
Henning SHasselbring W(2024)Benchmarking scalability of stream processing frameworks deployed as microservices in the cloudJournal of Systems and Software10.1016/j.jss.2023.111879208:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111879
Fragkoulis MCarbone PKalavri VKatsifodimos A(2024)A survey on the evolution of stream processing systemsThe VLDB Journal — The International Journal on Very Large Data Bases10.1007/s00778-023-00819-833:2(507-541)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s00778-023-00819-8
Wang ZChen SBai LGao JTao JBond RMulvenna M(2023)Reinforcement learning based task scheduling for environmentally sustainable federated cloud computingJournal of Cloud Computing: Advances, Systems and Applications10.1186/s13677-023-00553-012:1Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1186/s13677-023-00553-0
Kalogerakis SMagoutis KKemme BRiviere ESchiavoni VPasin M(2023)Discovery of breakout patterns in financial tick data via parallel stream processing with in-order guaranteesProceedings of the 17th ACM International Conference on Distributed and Event-based Systems10.1145/3583678.3596897(115-126)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3583678.3596897
Trofimov ASokolov NMarshalkin NKuralenok INovikov BZhou YChrysanthis PGulisano V(2022)Substream management in distributed streaming dataflowsProceedings of the 16th ACM International Conference on Distributed and Event-Based Systems10.1145/3524860.3539809(55-66)Online publication date: 27-Jun-2022
https://dl.acm.org/doi/10.1145/3524860.3539809
Cardellini VLo Presti FNardelli MRusso G(2022)Runtime Adaptation of Data Stream Processing Systems: The State of the ArtACM Computing Surveys10.1145/351449654:11s(1-36)Online publication date: 9-Sep-2022
https://dl.acm.org/doi/10.1145/3514496

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents