research-article

Dhalion: self-regulating stream processing in heron

Authors:

Avrilia Floratou,

Ashvin Agrawal,

Karthik RamasamyAuthors Info & Claims

Proceedings of the VLDB Endowment, Volume 10, Issue 12

Pages 1825 - 1836

https://doi.org/10.14778/3137765.3137786

Published: 01 August 2017 Publication History

Abstract

In recent years, there has been an explosion of large-scale real-time analytics needs and a plethora of streaming systems have been developed to support such applications. These systems are able to continue stream processing even when faced with hardware and software failures. However, these systems do not address some crucial challenges facing their operators: the manual, time-consuming and error-prone tasks of tuning various configuration knobs to achieve service level objectives (SLO) as well as the maintenance of SLOs in the face of sudden, unpredictable load variation and hardware or software performance degradation.

In this paper, we introduce the notion of self-regulating streaming systems and the key properties that they must satisfy. We then present the design and evaluation of Dhalion, a system that provides self-regulation capabilities to underlying streaming systems. We describe our implementation of the Dhalion framework on top of Twitter Heron, as well as a number of policies that automatically reconfigure Heron topologies to meet throughput SLOs, scaling resource consumption up and down as needed. We experimentally evaluate our Dhalion policies in a cloud environment and demonstrate their effectiveness. We are in the process of open-sourcing our Dhalion policies as part of the Heron project.

References

[1]

Apache Aurora. http://aurora.apache.org/.

[2]

Apache Flink. https://flink.apache.org/.

[3]

Apache Kafka. http://kafka.apache.org/.

[4]

Apache Samza. http://samza.apache.org/.

[5]

DataTorrent. https://www.datatorrent.com.

[6]

Distributed Log. http://distributedlog.incubator.apache.org/.

[7]

Heron Code Repository. https://github.com/twitter/heron.

[8]

Microsoft HDInsight. https://azure.microsoft.com/en-us/services/hdinsight/.

[9]

S4 Distributed Data Platform. http://incubator.apache.org/s4/.

[10]

Spark Streaming. http://spark.apache.org/streaming/.

[11]

D. J. Abadi et al. The Design of the Borealis Stream Processing Engine. In CIDR, pages 277--289, 2005.

[12]

T. Akidau et al. MillWheel: Fault-tolerant Stream Processing at Internet Scale. PVLDB, 6(11):1033--1044, Aug. 2013.

Digital Library

[13]

H. Balakrishnan et al. Retrospective on Aurora. The VLDB Journal, 13:2004, 2004.

Digital Library

[14]

S. Chaudhuri and V. Narasayya. Self-Tuning Database Systems: A Decade of Progress. In VLDB, September 2007.

Digital Library

[15]

J. Dean and L. A. Barroso. The Tail at Scale. Commun. ACM, 56(2):74--80, Feb. 2013.

Digital Library

[16]

T. Do et al. Limplock: Understanding the Impact of Limpware on Scale-out Cloud Systems. In SOCC '13, pages 14:1--14:14. ACM, 2013.

Digital Library

[17]

M. Fu et al. Twitter Heron: Towards Extensible Streaming Engines. In ICDE. IEEE, 2017.

[18]

T. Z. J. Fu et al. DRS: Dynamic Resource Scheduling for Real-Time Analytics over Fast Streams. In ICDCS, pages 411--420, 2015.

[19]

B. Gedik et al. Elastic Scaling for Data Stream Processing. IEEE Trans. Parallel Distrib. Syst., 25(6):1447--1463, June 2014.

Digital Library

[20]

T. S. Group. STREAM: The Stanford Stream Data Manager. Technical Report 2003-21, Stanford InfoLab, 2003.

[21]

H. Herodotou et al. Starfish: A Self-tuning System for Big Data Analytics. In CIDR, pages 261--272, 2011.

[22]

S. Kulkarni et al. Twitter Heron: Stream Processing at Scale. In ACM SIGMOD '15, pages 239--250, 2015.

Digital Library

[23]

A. Pavlo et al. Self-Driving Database Management Systems. In CIDR, 2017.

[24]

T. N. Pham, P. K. Chrysanthis, and A. Labrinidis. Avoiding Class Warfare: Managing Continuous Queries with Differentiated Classes of Service. The VLDB Journal, 25(2):197--221, Apr. 2016.

Digital Library

[25]

J. Schad, J. Dittrich, and J.-A. Quiané-Ruiz. Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance. PVLDB, 3(1-2):460--471, Sept. 2010.

Digital Library

[26]

A. Toshniwal et al. Storm@Twitter. In 2014 ACM SIGMOD.

Digital Library

[27]

V. K. Vavilapalli et al. Apache Hadoop YARN: Yet Another Resource Negotiator. In SOCC, pages 5:1--5:16. ACM, 2013.

Digital Library

Cited By

Ravikumar DYeo AZhu YLakra ANagulapalli HRavindran SSuh SDutta NFogarty APark YKhushalani STarafdar AParekh KKrishnan S(2024)Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud ServiceProceedings of the VLDB Endowment10.14778/3654621.365462917:7(1618-1627)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.14778/3654621.3654629
Chatziliadis XZacharatou EEracar AZeuch SMarkl V(2024)Efficient Placement of Decomposable Aggregation Functions for Stream Processing over Large Geo-Distributed TopologiesProceedings of the VLDB Endowment10.14778/3648160.364818617:6(1501-1514)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.14778/3648160.3648186
Justen DRitter DFraser CLamb ALee ABodner THaddad MZeuch SMarkl VBoehm M(2024)POLAR: Adaptive and Non-invasive Join Order Selection via Plans of Least ResistanceProceedings of the VLDB Endowment10.14778/3648160.364817517:6(1350-1363)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.14778/3648160.3648175
Show More Cited By

Dhalion: self-regulating stream processing in heron
1. Software and its engineering
  1. Software organization and properties
    1. Software system structures

Recommendations

Dhalion in action: automatic management of streaming applications

In a world where organizations are being inundated with data from various sources, analyzing data and gaining actionable insights in real-time has become a key service differentiator. Over the last few years, several stream processing frameworks have ...
Study On Purchase Intention In Different Live Streaming Scenarios Based On Experimental Approach
ICEBI '22: Proceedings of the 2022 6th International Conference on E-Business and Internet

Live streaming e-commerce has exploded recently. While the live streaming traffic is dominated by the top live streamers, merchants and ordinary live streamers attempt to establish self-operating live streaming, but the number of fans and sales ...
MedSMan: a live multimedia stream querying system

Querying live media streams is a challenging problem that is becoming an essential requirement in a growing number of applications. Research in multimedia information systems has addressed and made good progress in dealing with archived data. Meanwhile, ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment

Proceedings of the VLDB Endowment Volume 10, Issue 12

August 2017

427 pages

ISSN:2150-8097

Editors:
Peter Boncz
CWI
,
Ken Salem
University of Waterloo

Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 August 2017

Published in PVLDB Volume 10, Issue 12

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

67
Total Citations
View Citations
570
Total Downloads

Downloads (Last 12 months)55
Downloads (Last 6 weeks)2

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ravikumar DYeo AZhu YLakra ANagulapalli HRavindran SSuh SDutta NFogarty APark YKhushalani STarafdar AParekh KKrishnan S(2024)Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud ServiceProceedings of the VLDB Endowment10.14778/3654621.365462917:7(1618-1627)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.14778/3654621.3654629
Chatziliadis XZacharatou EEracar AZeuch SMarkl V(2024)Efficient Placement of Decomposable Aggregation Functions for Stream Processing over Large Geo-Distributed TopologiesProceedings of the VLDB Endowment10.14778/3648160.364818617:6(1501-1514)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.14778/3648160.3648186
Justen DRitter DFraser CLamb ALee ABodner THaddad MZeuch SMarkl VBoehm M(2024)POLAR: Adaptive and Non-invasive Join Order Selection via Plans of Least ResistanceProceedings of the VLDB Endowment10.14778/3648160.364817517:6(1350-1363)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.14778/3648160.3648175
Glaze NMcNeely TZhu YGleeson MSerr HBhopi RKrishnan S(2024)Lorentz: Learned SKU Recommendation Using Profile DataProceedings of the ACM on Management of Data10.1145/36549522:3(1-25)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654952
Geldenhuys MScheinert DKao OThamsen LBalsamo SKnottenbelt WAbad CShang W(2024)Demeter: Resource-Efficient Distributed Stream Processing under Dynamic Loads with Multi-Configuration OptimizationProceedings of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629526.3645048(142-153)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3629526.3645048
Pfister BScheinert DGeldenhuys MKao OBalsamo SKnottenbelt WAbad CShang W(2024)Daedalus: Self-Adaptive Horizontal Autoscaling for Resource Efficiency of Distributed Stream Processing SystemsProceedings of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629526.3645042(130-141)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3629526.3645042
Siachamis GChristodoulou GPsarakis KFragkoulis Mvan Deursen AKatsifodimos A(2024)Evaluating Stream Processing AutoscalersProceedings of the 18th ACM International Conference on Distributed and Event-based Systems10.1145/3629104.3666036(110-122)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3629104.3666036
Zhang ZJin PXie XWang XLiu RWan S(2024)Online Nonstop Task Management for Storm-Based Distributed Stream Processing EnginesJournal of Computer Science and Technology10.1007/s11390-021-1629-939:1(116-138)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1007/s11390-021-1629-9
Vogel ADanelutto MTorquati MGriebler DFernandes L(2024)Enhancing self-adaptation for efficient decision-making at run-time in streaming applications on multicoresThe Journal of Supercomputing10.1007/s11227-024-06191-w80:15(22213-22244)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1007/s11227-024-06191-w
Fragkoulis MCarbone PKalavri VKatsifodimos A(2024)A survey on the evolution of stream processing systemsThe VLDB Journal — The International Journal on Very Large Data Bases10.1007/s00778-023-00819-833:2(507-541)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s00778-023-00819-8
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents