research-article

Challenges and approaches for distributed workflow-driven analysis of large-scale biological data: vision paper

Authors:

Ilkay Altintas,

Weizhong LiAuthors Info & Claims

EDBT-ICDT '12: Proceedings of the 2012 Joint EDBT/ICDT Workshops

Pages 73 - 78

https://doi.org/10.1145/2320765.2320791

Published: 30 March 2012 Publication History

Abstract

Next-generation DNA sequencing machines are generating a very large amount of sequence data with applications in many scientific challenges and placing unprecedented demands on traditional single-processor bioinformatics algorithms. Middleware and technologies for scientific workflows and data-intensive computing promise new capabilities to enable rapid analysis of next-generation sequence data. Based on this motivation and our previous experiences in bioinformatics and distributed scientific workflows, we are creating a Kepler Scientific Workflow System module, called "bioKepler", that facilitates the development of Kepler workflows for integrated execution of bioinformatics applications in distributed environments. This vision paper discusses the challenges related to next-generation sequencing data, explains the approaches taken in bioKepler to help with analysis of such data, and presents preliminary results demonstrating these approaches.

References

[1]

I. Altintas, O. Barney, Z. Cheng, T. Critchlow, B. Ludaescher, S. Parker, A. Shoshani, and M. Vouk. Accelerating the scientific exploration process with scientific workflows. Journal of Physics: Conference Series, 46:468--478, 2006. SciDAC 2006.

[2]

I. Altintas, O. Barney, and E. Jaeger-Frank. Provenance collection support in the kepler scientific workflow system. In Proceedings of International Provenance and Annotation Workshop, pages 118--132, 2006.

Digital Library

[3]

I. Altintas, C. Berkley, E. Jaeger, M. Jones, B. Ludaescher, and S. Mock. Kepler: An extensible system for design and execution of scientific workflows. In Intl. Conference on Scientific and Statistical Database Management (SSDBM), Santorini Island, Greece, 2004.

Digital Library

[4]

S. F. Altschul, W. Gish, W. Miller, E. W. Myers, and D. J. Lipman. Basic Local Alignment Search Tool. Journal of Molecular Biology, 215(3):403--410, 1990.

[5]

D. Battré, S. Ewen, F. Hueske, O. Kao, V. Markl, and D. Warneke. Nephele/PACTs: A programming model and execution framework for web-scale analytical processing. In Proceedings of the 1st ACM symposium on Cloud computing, SoCC '10, pages 119--130, New York, NY, USA, 2010. ACM.

Digital Library

[6]

J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. Communications of the ACM, 51(1):107--113, 2008.

Digital Library

[7]

E. Deelman, G. Mehta, G. Singh, M.-H. Su, and K. Vahi. Pegasus: Mapping large-scale workflows to distributed resources. In I. J. Taylor, E. Deelman, D. B. Gannon, and M. Shields, editors, Workflows for e-Science, pages 376--394. Springer London, 2007.

[8]

T. Disz, M. Kubal, R. Olson, R. Overbeek, and R. Stevens. Challenges in large scale distributed computing: bioinformatics. In Proceedings of Challenges of Large Applications in Distributed Environments, 2005. CLADE 2005., pages 57--65. IEEE, 2005.

[9]

X. Fei, S. Lu, and C. Lin. A mapreduce-enabled scientific workflow composition framework. In ICWS '09: Proceedings of the 2009 IEEE International Conference on Web Services, pages 663--670, Washington, DC, USA, 2009. IEEE Computer Society.

Digital Library

[10]

A. Goderis, A. Brooks, I. Altintas, C. Goble, and E. Lee. Composing different models of computation in Kepler and Ptolemy II. Lecture Notes in Computer Science, III:182--190, 2007. Proc. 2nd International Workshop on Workflow systems in e-Science in conjunction with ICCS 2007.

Digital Library

[11]

A. Goderis, C. Brooks, I. Altintas, E. Lee, and C. Goble. Heterogeneous composition of models of computation. Future Generation Computer Systems, 25(5):552--560, 2009.

Digital Library

[12]

D. J. Goodman. Introduction and evaluation of martlet: a scientific workflow language for abstracted parallelisation. In WWW '07: Proceedings of the 16th international conference on World Wide Web, pages 983--992, New York, NY, USA, 2007. ACM.

Digital Library

[13]

I. Gorton, P. Greenfield, A. S. Szalay, and R. Williams. Data-intensive computing in the 21st century. IEEE Computer, 41(4):30--32, 2008.

Digital Library

[14]

B. Langmead, M. C. Schatz, J. Lin, M. Pop, and S. L. Salzberg. Searching for snps with cloud computing. Genome Biology, 10(134), November 2009.

[15]

H. Li and Z. Durbin. Fast and accurate short read alignment with burrows-wheeler transform. Bioinformatics, 25(14):1754--1760, 2009.

Digital Library

[16]

B. Ludaescher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger-Frank, J. Jones, M. and Lee, and Y. Zhao. Scientific workflow management and the Kepler system. Concurrency and Computation: Practice & Experience, Special Issue on Scientific Workflows, 18(10):1039--1065, 2006.

Digital Library

[17]

M. Margulies, M. Egholm, W. Altman, S. Attiya, J. Bader, L. Bemben, J. Berka, M. Braverman, Y. Chen, Z. Chen, S. Dewell, L. Du, J. Fierro, X. Gomes, B. Godwin, W. He, S. Helgesen, C. Ho, G. Irzyk, S. Jando, M. Alenquer, T. Jarvie, K. Jirage, J. Kim, J. Knight, J. Lanza, J. Leamon, S. Lefkowitz, M. Lei, J. Li, K. Lohman, H. Lu, V. Makhijani, K. McDade, M. McKenna, E. Myers, E. Nickerson, J. Nobile, R. Plant, B. Puc, M. Ronan, G. Roth, G. Sarkis, J. Simons, J. Simpson, M. Srinivasan, K. Tartaro, A. Tomasz, K. Vogt, G. Volkmer, S. Wang, Y. Wang, M. Weiner, P. Yu, R. Begley, and J. Rothberg. Genome sequencing in microfabricated high-density picolitre reactors. Nature, 437(7057):376--380, September 2005.

[18]

C. Moretti, H. Bui, K. Hollingsworth, B. Rich, P. Flynn, and D. Thain. All-pairs: An abstraction for data-intensive computing on campus grids. IEEE Transactions on Parallel and Distributed Systems, 21:33--46, 2010.

Digital Library

[19]

P. Mouallem, D. Crawl, I. Altintas, M. A. Vouk, and U. Yildiz. A fault-tolerance architecture for kepler-based distributed scientific workflows. In Proceedings of Scientific and Statistical Database Management, 22nd International Conference (SSDBM 2010), volume 6187 of Lecture Notes in Computer Science, pages 452--460, Berlin, Heidelberg, 2010. Springer.

Digital Library

[20]

T. Oinn, M. Addis, J. Ferris, D. Marvin, M. Senger, M. Greenwood, T. Carver, K. Glover, M. R. Pocock, A. Wipat, and P. Li. Taverna: a tool for the composition and enactment of bioinformatics workflows. in. Bioinformatics, Oxford University Press, London, UK, 20(17):3045--3054, 2004.

Digital Library

[21]

J. Qin and T. Fahringer. Advanced data flow support for scientific grid workflow applications. In SC '07: Proceedings of the 2007 ACM/IEEE conference on Supercomputing, pages 1--12, New York, NY, USA, 2007. ACM.

Digital Library

[22]

M. Schatz. Cloudburst: Highly sensitive read mapping with mapreduce. Bioinformatics, 25(11):1363--1369, April 2009.

Digital Library

[23]

J. Shendure and H. Ji. Next generation-dna sequencing. Nature Biotechnology, 26(10):1135--1145, 2008.

[24]

A. D. Smith, Z. Xuan, and M. Q. Zhang. Using quality scores and longer reads improves accuracy of solexa read mapping. BMC Bioinformatics, 9(128), February 2008.

[25]

L. D. Stein. The case for cloud computing in genome informatics. Genome Biology, 11(5):207, 2010.

[26]

I. Taylor, M. Shields, I. Wang, and O. Rana. Triana applications within grid computing and peer to peer environments. Journal of Grid Computing, 1, 2003.

[27]

I. J. Taylor, E. Deelman, D. B. Gannon, and M. Shields, editors. Workflows for e-Science. Springer, 2007.

[28]

J. Wang, I. Altintas, P. R. Hosseini, D. Barseghian, D. Crawl, C. Berkley, and M. B. Jones. Accelerating parameter sweep workflows by utilizing ad-hoc network computing resources: An ecological example. In Services, IEEE Congress on, pages 267--274. IEEE Computer Society, 2009.

Digital Library

[29]

J. Wang, D. Crawl, and I. Altintas. Kepler + Hadoop: A general architecture facilitating data-intensive applications in scientific workflow systems. In WORKS '09: Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, pages 1--8, Portland, Oregon, 2009. ACM New York, NY, USA.

Digital Library

[30]

D. Warneke and O. Kao. Exploiting dynamic resource allocation for efficient parallel data processing in the cloud. Parallel and Distributed Systems, IEEE Transactions on, 22(6):985--997, June 2011.

Digital Library

[31]

Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. V. Laszewski, I. Raicu, T. Stef-praun, and M. Wilde. Swift: Fast, reliable, loosely coupled parallel computation. In Services, 2007 IEEE Congress on, pages 199--206. IEEE Press, 2007.

Cited By

Kim YDonovan RRen YBian SWu TPurawat SManzo AAltintas ILi BLi G(2022)Smart connected worker edge platform for smart manufacturing: Part 1—Architecture and platform designJournal of Advanced Manufacturing and Processing10.1002/amp2.101294:4Online publication date: 13-Jun-2022
https://doi.org/10.1002/amp2.10129
Behrens HCandan KChen XGarg YLi MLi XLiu SSapino MShadab MTurner DVijayakumaren M(2021)DataStorm: Coupled, Continuous Simulations for Complex Urban EnvironmentsACM/IMS Transactions on Data Science10.1145/34475722:3(1-37)Online publication date: 21-Jul-2021
https://dl.acm.org/doi/10.1145/3447572
Purawat SDasgupta SBurbidge LZuo JWilson SGupta AAltintas I(2021)Quantum Data Hub: A Collaborative Data and Analysis Platform for Quantum Material ScienceComputational Science – ICCS 202110.1007/978-3-030-77961-0_52(656-670)Online publication date: 9-Jun-2021
https://doi.org/10.1007/978-3-030-77961-0_52
Show More Cited By

Index Terms

Challenges and approaches for distributed workflow-driven analysis of large-scale biological data: vision paper
1. Information systems
  1. Information systems applications

Recommendations

Distributed workflow-driven analysis of large-scale biological data using biokepler
PDAC '11: Proceedings of the 2nd international workshop on Petascal data analytics: challenges and opportunities

Next-generation DNA sequencing machines are generating a very large amount of sequence data with applications in many scientific challenges, placing unprecedented demands on traditional single-processor bioinformatics algorithms. Technologies like ...
RECOMBFLOW: a scientific workflow environment for Intragenomic Gene Conversion analysis in bacterial genomes, including the pathogen Streptococcus pyogenes

Intragenomic Gene Conversion (IGC) is important in the evolution of bacteria but has only been analysed computationally in a few strains of Escherichia coli. This paper describes a scientific workflow system, called RECOMBFLOW, that automates this ...
Data parallelism in bioinformatics workflows using Hydra
HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Large scale bioinformatics experiments are usually composed by a set of data flows generated by a chain of activities (programs or services) that may be modeled as scientific workflows. Current Scientific Workflow Management Systems (SWfMS) are used to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EDBT-ICDT '12: Proceedings of the 2012 Joint EDBT/ICDT Workshops

March 2012

265 pages

ISBN:9781450311434

DOI:10.1145/2320765

Editors:
Divesh Srivastava
AT&T Labs-Research
,
Ismail Ari
Ozyegin University, Turkey

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 March 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

ICDT '12

ICDT '12: 15th International Conference on Database Theory

March 30, 2012

Berlin, Germany

Acceptance Rates

Overall Acceptance Rate 7 of 10 submissions, 70%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
335
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kim YDonovan RRen YBian SWu TPurawat SManzo AAltintas ILi BLi G(2022)Smart connected worker edge platform for smart manufacturing: Part 1—Architecture and platform designJournal of Advanced Manufacturing and Processing10.1002/amp2.101294:4Online publication date: 13-Jun-2022
https://doi.org/10.1002/amp2.10129
Behrens HCandan KChen XGarg YLi MLi XLiu SSapino MShadab MTurner DVijayakumaren M(2021)DataStorm: Coupled, Continuous Simulations for Complex Urban EnvironmentsACM/IMS Transactions on Data Science10.1145/34475722:3(1-37)Online publication date: 21-Jul-2021
https://dl.acm.org/doi/10.1145/3447572
Purawat SDasgupta SBurbidge LZuo JWilson SGupta AAltintas I(2021)Quantum Data Hub: A Collaborative Data and Analysis Platform for Quantum Material ScienceComputational Science – ICCS 202110.1007/978-3-030-77961-0_52(656-670)Online publication date: 9-Jun-2021
https://doi.org/10.1007/978-3-030-77961-0_52
Singh APurawat SRao AAltintas I(2020)Modular performance prediction for scientific workflows using Machine LearningFuture Generation Computer Systems10.1016/j.future.2020.04.048Online publication date: May-2020
https://doi.org/10.1016/j.future.2020.04.048
Purawat SOlschanowsky CCondon LMaxwell RAltintas I(2020)Scalable Workflow-Driven Hydrologic Analysis in HydroFrameComputational Science – ICCS 202010.1007/978-3-030-50371-0_20(276-289)Online publication date: 15-Jun-2020
https://doi.org/10.1007/978-3-030-50371-0_20
Lluch Lafuente A(2019)A Framework for Provenance-Preserving History Distribution and Incremental ReductionModels, Languages, and Tools for Concurrent and Distributed Programming10.1007/978-3-030-21485-2_26(471-486)Online publication date: 3-Jul-2019
https://doi.org/10.1007/978-3-030-21485-2_26
Singh ARao APurawat SAltintas IMontagnat JTaylor IGesing SSakellariou R(2017)A machine learning approach for modular workflow performance predictionProceedings of the 12th Workshop on Workflows in Support of Large-Scale Science10.1145/3150994.3150998(1-11)Online publication date: 12-Nov-2017
https://dl.acm.org/doi/10.1145/3150994.3150998
Goyal ASingh ABhargava SCrawl DAltintas IHsu C(2016)Natural Language Processing Using Kepler Workflow SystemProcedia Computer Science10.1016/j.procs.2016.05.35880:C(712-721)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1016/j.procs.2016.05.358
Miller MSchwartz TPickett BHe SKlem EScheuermann RPassarotti MKaufman SO'Leary M(2015)A RESTful API for Access to Phylogenetic Tools via the CIPRES Science GatewayEvolutionary Bioinformatics10.4137/EBO.S2150111(EBO.S21501)Online publication date: 16-Mar-2015
https://doi.org/10.4137/EBO.S21501
Qian YKim HPurawat SWang JStanton RLee AXu WAltintas ISinkovits RScheuermann RPeterson G(2015)FlowGateProceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure10.1145/2792745.2792750(1-8)Online publication date: 26-Jul-2015
https://dl.acm.org/doi/10.1145/2792745.2792750
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents