research-article

Optimization of Data-intensive Flows: Is it Needed? Is it Solved?

Authors:

Georgia Kougka,

Anastasios GounarisAuthors Info & Claims

DOLAP '14: Proceedings of the 17th International Workshop on Data Warehousing and OLAP

Pages 95 - 98

https://doi.org/10.1145/2666158.2666174

Published: 07 November 2014 Publication History

Abstract

Modern data analysis is increasingly employing data-intensive flows for processing very large volumes of data. As the data flows become more and more complex and operate in a highly dynamic environment, we argue that we need to resort to automated cost-based optimization solutions rather than relying on efficient designs by human experts. We further demonstrate that the current state-of-the-art in flow optimizations needs to be extended and we propose a promising direction for optimizing flows at the logical level, and more specifically, for deciding the sequence of flow tasks.

References

[1]

D. Abadi et al. The beckman database research self-assessment meeting. Technical report, 2013.

[2]

S. Abrishami, M. Naghibzadeh, and D. H. Epema. Deadline-constrained workflow scheduling algorithms for infrastructure as a service clouds. Future Generation Computer Systems, 29(1):158 -- 169, 2013.

Digital Library

[3]

J. Burge, K. Munagala, and U. Srivastava. Ordering pipelined query operators with precedence constraints. Technical Report 2005-40, Stanford InfoLab, 2005.

[4]

S. Chaudhuri, U. Dayal, and V. Narasayya. An overview of business intelligence technology. Commun. ACM, 54:88--98, 2011.

Digital Library

[5]

R. Dewan, A. Seidmann, and Z. Walter. Workflow optimization through task redesign in business information processes. In HICSS, pages 240--252. IEEE Computer Society, 1998.

Digital Library

[6]

R. Halasipuram, P. M. Deshpande, and S. Padmanabhan. Determining essential statistics for cost based optimization of an etl workflow. In EDBT, pages 307--318, 2014.

[7]

S. Holl, O. Zimmermann, M. Palmblad, Y. Mohammed, and M. Hofmann-Apitius. A new optimization phase for scientific workflow management systems. Future Generation Comp. Syst., 36:352--362, 2014.

[8]

F. Hueske, M. Peters, M. Sax, A. Rheinlander, R. Bergmann, A. Krettek, and K. Tzoumas. Opening the black boxes in data flow optimization. PVLDB, 5(11):1256--1267, 2012.

Digital Library

[9]

G. Kougka and A. Gounaris. On optimizing work ows using query processing techniques. In SSDBM, pages 601--606, 2012.

Digital Library

[10]

G. Kougka and A. Gounaris. Declarative expression and optimization of data-intensive flows. In DaWaK, pages 13--25, 2013.

Digital Library

[11]

R. Krishnamurthy, H. Boral, and C. Zaniolo. Optimization of nonrecursive queries. In VLDB, pages 128--137, 1986.

Digital Library

[12]

N. Kumar and P. S. Kumar. An efficient heuristic for logical optimization of etl workflows. In BIRTE, volume 84 of Lecture Notes in Business Information Processing, pages 68--83. Springer, 2010.

[13]

E. S. Ogasawara, D. de Oliveira, P. Valduriez, J. Dias, F. Porto, and M. Mattoso. An algebraic approach for data-centric scientific workflows. PVLDB, 4:1328--1339, 2011.

Digital Library

[14]

A. Simitsis, P. Vassiliadis, and T. K. Sellis. State-space optimization of etl workflows. IEEE Trans. Knowl. Data Eng., 17(10):1404--1419, 2005.

Digital Library

[15]

Y. L. Varol and D. Rotem. An algorithm to generate all topological sorting arrangements. The Computer Journal, 24(1):83--84, 1981.

[16]

M. Vrhovnik, H. Schwarz, O. Suhre, B. Mitschang, V. Markl, A. Maier, and T. Kraft. An approach to optimize data processing in business processes. In VLDB, pages 615--626, 2007.

Digital Library

[17]

Z. Xiao, H. Chang, and Y. Yi. Optimization of workflow resources allocation with cost constraint. In Proc. of the 10th Int. Conf. on Computer supported cooperative work in design, pages 647--656, 2007.

Digital Library

Cited By

Cowley BKorpela J(2018)Computational Testing for Automated Preprocessing 2: Practical Demonstration of a System for Scientific Data-Processing Workflow Management for High-Volume EEGFrontiers in Neuroscience10.3389/fnins.2018.0023612Online publication date: 10-Apr-2018
https://doi.org/10.3389/fnins.2018.00236
Ritter DMay NForsberg FRinderle-Ma S(2018)Optimization Strategies for Integration Pattern CompositionsProceedings of the 12th ACM International Conference on Distributed and Event-based Systems10.1145/3210284.3210295(88-99)Online publication date: 25-Jun-2018
https://dl.acm.org/doi/10.1145/3210284.3210295
Kougka GGounaris ASimitsis A(2018)The many faces of data-centric workflow optimization: a surveyInternational Journal of Data Science and Analytics10.1007/s41060-018-0107-06:2(81-107)Online publication date: 6-Mar-2018
https://doi.org/10.1007/s41060-018-0107-0
Show More Cited By

Index Terms

Optimization of Data-intensive Flows: Is it Needed? Is it Solved?
1. Information systems
  1. Information systems applications

Recommendations

Declarative Expression and Optimization of Data-Intensive Flows
DaWaK 2013: Proceedings of the 15th International Conference on Data Warehousing and Knowledge Discovery - Volume 8057

Data-intensive analytic flows, such as populating a datawarehouse or analyzing a click stream at runtime, are very common in modern business intelligence scenarios. Current state-of-the-art data flow management techniques rely on the users to specify ...
SOFA

Recent years have seen an increased interest in large-scale analytical data flows on non-relational data. These data flows are compiled into execution graphs scheduled on large compute clusters. In many novel application areas the predominant building ...
Optimization of analytic data flows for next generation business intelligence applications
TPCTC'11: Proceedings of the Third TPC Technology conference on Topics in Performance Evaluation, Measurement and Characterization

This paper addresses the challenge of optimizing analytic data flows for modern business intelligence (BI) applications. We first describe the changing nature of BI in today's enterprises as it has evolved from batch-based processes, in which the back-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

DOLAP '14: Proceedings of the 17th International Workshop on Data Warehousing and OLAP

November 2014

110 pages

ISBN:9781450309998

DOI:10.1145/2666158

General Chair:
Il-Yeol Song
Drexel University, USA
,
Program Chairs:
Alkis Simitsis
HP Labs, USA
,
Alfredo Cuzzocrea
ICAR-CNR and University of Calabria, Italy

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM '14

Sponsor:

CIKM '14: 2014 ACM Conference on Information and Knowledge Management

November 7, 2014

Shanghai, China

Acceptance Rates

DOLAP '14 Paper Acceptance Rate 8 of 22 submissions, 36%;

Overall Acceptance Rate 29 of 79 submissions, 37%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
124
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cowley BKorpela J(2018)Computational Testing for Automated Preprocessing 2: Practical Demonstration of a System for Scientific Data-Processing Workflow Management for High-Volume EEGFrontiers in Neuroscience10.3389/fnins.2018.0023612Online publication date: 10-Apr-2018
https://doi.org/10.3389/fnins.2018.00236
Ritter DMay NForsberg FRinderle-Ma S(2018)Optimization Strategies for Integration Pattern CompositionsProceedings of the 12th ACM International Conference on Distributed and Event-based Systems10.1145/3210284.3210295(88-99)Online publication date: 25-Jun-2018
https://dl.acm.org/doi/10.1145/3210284.3210295
Kougka GGounaris ASimitsis A(2018)The many faces of data-centric workflow optimization: a surveyInternational Journal of Data Science and Analytics10.1007/s41060-018-0107-06:2(81-107)Online publication date: 6-Mar-2018
https://doi.org/10.1007/s41060-018-0107-0
Kougka GGounaris A(2018)Optimization of data flow execution in a parallel environmentDistributed and Parallel Databases10.1007/s10619-018-7243-3Online publication date: 22-Aug-2018
https://doi.org/10.1007/s10619-018-7243-3
Kougka GGounaris A(2017)Cost Optimization of Data Flows Based on Task Re-orderingTransactions on Large-Scale Data- and Knowledge-Centered Systems XXXIII10.1007/978-3-662-55696-2_4(113-145)Online publication date: 8-Aug-2017
https://doi.org/10.1007/978-3-662-55696-2_4
Kougka GGounaris A(2017)Optimal Task Ordering in Chain Data Flows: Exploring the Practicality of Non-scalable SolutionsBig Data Analytics and Knowledge Discovery10.1007/978-3-319-64283-3_2(19-32)Online publication date: 3-Aug-2017
https://doi.org/10.1007/978-3-319-64283-3_2
Kougka GGounaris ALeser U(2017)Modeling Data Flow Execution in a Parallel EnvironmentBig Data Analytics and Knowledge Discovery10.1007/978-3-319-64283-3_14(183-196)Online publication date: 3-Aug-2017
https://doi.org/10.1007/978-3-319-64283-3_14
Gounaris A(2016)Towards Automated Performance Optimization of BPMN Business ProcessesNew Trends in Databases and Information Systems10.1007/978-3-319-44066-8_2(19-28)Online publication date: 14-Aug-2016
https://doi.org/10.1007/978-3-319-44066-8_2

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents