An autonomous agent approach to query optimization in stream grids
Article No.: 9, Pages 57 - 64
Abstract
Stream grids are wide-area grid computing environments that are fed by a set of stream data sources. Queries arrive at the grid from users and applications external to the system. The kind of queries considered in this work are long-running continuous (LRC) queries, that we also term as "open-world" queries. These queries are neither short-lived nor infinitely long lived. They live long enough to make the prospect of multi-query optimization meaningful. But queries may also terminate at any time, requiring re-optimization of the query plans. The queries are "open" from the grid perspective as the grid cannot control or predict: (1) arrival of a query with time, location, required data and, (2) query revocations. Query optimization in such an environment has two major challenges: (a) optimizing in a multi-query environment and (b) continuous optimization due to new query arrivals and revocations. As generating a globally optimal query plan is an intractable problem, this work explores the idea of emergent optimization, where globally optimal query plans emerge as a result of local autonomous decisions taken by the grid nodes. Drawing concepts from evolutionary game theory, grid nodes are modeled as autonomous agents that seek to maximize a self-interest function using one of a set of different strategies. Grid nodes change strategies in response to variations in query arrival and revocation patterns. Changing of strategies is also autonomously decided by each grid node based on how its strategy is faring with respect to other strategies in the grid.
References
[1]
D. Abadi, Y. Ahmad., M. Balazinska., U. C. etintemel, M. Cherniack, J. Hwang, W. Lindner, A. Maskey, A. Rasin, E. Ryvkina, N. Tatbul, Y. Xing, and S. Zdonik. The design of the borealis stream processing engine. In The Conf. on Innovative Data Systems Research, pages 277--289, 2005.
[2]
Y. Ahmad, U. Cetintemel, J. Jannotti, A. Zgolinski, and S. Zdonik. Network awareness in internet-scale stream processing. In IEEE Data Engineering Bulletin, 2005.
[3]
M. Alpdemir, A. Mukherjee, N. W. Paton, P. Watson, A. A. Fernandes, A. Gounaris, and J. Smith. Ogsa-dqp: A service-based distributed query processor for the grid. In Proceedings of UK e-Science All Hands Meeting Nottingham, 2003.
[4]
D. Anderson. Resilient overlay networks. In MS Thesis, Department of Electrical Engineering and Computer Science, MIT, 2001.
[5]
A. Arasu, B. Babcock, S. Babu, M. Datar, K. Ito, R. Motwani, I. Nishizawa, U. Srivastava, D. Thomas, R. Varma, and J. Widom. Stream: The stanford stream data manager. In IEEE Data Engineering Bulletin, pages 19--26, 2003.
[6]
R. Avnur and J. Hellerstein. Eddies: Continously adaptive query processing. In ACM SIGMOD Record, 2000.
[7]
R. Axelrod. The Evolution of Cooperation. Basic Books, 1984. ISBN 0-465-02122-2.
[8]
D. Carney. Monitoring streams - a new class of data management operations. In The 28th VLDB Conference, 2002.
[9]
S. Chandrasekaran, O. Cooper, A. Deshpande, M. Franklin, J. Hellerstein, W. Hong, S. Krishnamurthy, S. Madden, V. Raman, F. Reiss, and M. Shah. Telegraphcq: Continuous dataflow processing for an uncertain world. In The Conf. on Innovative Data Systems Research, 2003.
[10]
S. Chandrasekaran and M. J. Franklin. Streaming queries over streaming data. In The 28th VLDB Conference, 2002.
[11]
K. M. Chandy and L. Lamport. Distributed snapshots: determining global states of distributed systems. In ACM Transactions on Computer Systems (TOCS), pages 63--75, February 1985.
[12]
G. Cormode and M. Garofalakis. Streaming in a connected world: quering and tracking distributed data streams. In International Conf. in Data Engineering, 2007.
[13]
K. Gorman, D. Agarwal, and A. E. Abbadi. Multiple query optimization by cache-aware middleware using query teamwork. In The 18th Intl Conf. on Data Engineering, 2002.
[14]
A. Gupta, S. Sudarshan, and S. Viswanathan. Query scheduling in multi query optimization. In The Intl. Symposium on Database Engineering and Applications, pages 11--19, 2001.
[15]
R. Kuntschke, B. Stegmaier, A. Kemper, and A. Reiser. Streamglobe: Processing and sharing data streams in grid-based p2p infrastructures. In The 31st VLDB Conference, pages 1259--1262, 2005.
[16]
S. Madden and M. Franklin. Fjording the stream: An architecture for queries over streaming sensor data. In The Intl Conf on Data Engineering, 2002.
[17]
S. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong. The design of an acquisitional query processor for sensor networks. In SIGMOD, 2003.
[18]
S. Madden, M. Shah, J. Hellerstein, and V. Raman. Continuously adaptive continuous queries over streams. In The ACM SIGMOD Intl. Conf on Management of Data, pages 49--60, 2002.
[19]
S. Mukherjee, S. Srinivasa, and S. Patil. Emergent (re)optimization for stream queries in grids. In IEEE Congress on Evolutionary Computation, pages 729--735, September 2007.
[20]
P. Pietzuch, J. Ledlie, J. Shneidman, M. Roussopoulos, M. Welsh, and M. Seltzer. Network-aware operator placement for stream-processing systems. In International Conference on Data Engineering, 2006.
[21]
V. Raman, A. Deshpande, and J. Hellerstein. Using state modules for adaptive query processing. In The Intl. Conf. on Data Engineering, 2003.
[22]
P. Roy, A. Seshadri, A. Sudarshan, and S. Bhobhe. Efficien and extensible algorithms for multi query optimization. In ACM SIGMOD Conf. on Management of Data, pages 249--260, 2000.
[23]
T. Sellis. Multiple-query optimization. In ACM Trans. on Database Systems, pages 23--52, 1988.
[24]
M. Shah, J. Hellerstein, S. Chandrasekharan, and M. Franklin. Flux: An adaptive reparitioning operator for continuous query systems. In The Intl. Conf. on Data Engineering, 2003.
[25]
S. Shah, K. Ramamritham, and P. Shenoy. Resilient and coherence preserving dissemination of dynamic data using cooperating peers. In IEEE Transactions on Knowledge and Data Engineering, 2004.
[26]
J. Smith, P. Watson, A. Gounaris, N. W. Paton, A. Fernandes, and R. Sakellariou. Distributed query processing on the grid. In The Intl. Journal of High Performance Computing Applications, pages 353--367, 2003.
[27]
B. Stegmaier, R. Kuntschke, and A. Kemper. Streamglobe: Adaptive query processing and optimization in streaming p2p environments. In Intl. Workshop on Data Management for Sensor Networks, pages 88--97, 2004.
[28]
Y. Yao and J. Gehrke. The cougar approach to in-network query processing in sensor networks. In ACM SIGMOD Record 31(3), pages 9--18, 2002.
[29]
S. Zdonik. The aurora and medusa projects. In IEEE Data Engineering Bulletin, 2003.
Index Terms
- An autonomous agent approach to query optimization in stream grids
Recommendations
An Autonomous Agent Approach to Query Optimization in Stream Grids
Stream grids are wide-area grid computing environments that are fed by a set of stream data sources, and Queries arrive at the grid from users and applications external to the system. The kind of queries considered in this work is long-running ...
Query processing of multi-way stream window joins
This paper introduces a class of join algorithms, termed W-join, for joining multiple infinite data streams. W-join addresses the infinite nature of the data streams by joining stream data items that lie within a sliding window and that match a certain ...
Massively multi-query join processing in publish/subscribe systems
SIGMOD '07: Proceedings of the 2007 ACM SIGMOD international conference on Management of dataThere has been much recent interest in XML publish/subscribe systems. Some systems scale to thousands of concurrent queries, but support a limited query language (usually a fragment of XPath 1.0). Other systems support more expressive languages, but do ...
Comments
Information & Contributors
Information
Published In
October 2009
525 pages
ISBN:9781605588292
DOI:10.1145/1643823
- General Chair:
- Nicolas Spyratos,
- Program Chairs:
- Epaminondas Kapetanios,
- Agma Traina,
- Publications Chairs:
- Richard Chbeir,
- Youakim Badr
Copyright © 2009 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
- The French Chapter of ACM Special Interest Group on Applied Computing
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 October 2009
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
MEDES '09
Sponsor:
MEDES '09: The International Conference on Management of Emergent Digital EcoSystems 2009
October 27 - 30, 2009
France
Acceptance Rates
Overall Acceptance Rate 267 of 682 submissions, 39%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 13Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)0
Reflects downloads up to 14 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in