research-article

Intelligent Scaling in Amazon Redshift

Authors:

Balakrishnan Narayanaswamy,

Tim KraskaAuthors Info & Claims

SIGMOD/PODS '24: Companion of the 2024 International Conference on Management of Data

Pages 269 - 279

https://doi.org/10.1145/3626246.3653394

Published: 09 June 2024 Publication History

Get Access

Abstract

Cloud-based data warehouses are built to be easy to use, requiring minimal intervention from customers as their workloads scale. However, there are still many dimensions of a workload that they do not scale with automatically. For example, in cloud-managed clusters, large ad-hoc queries and ETL workloads must use the same cluster size provisioned for the rest of the workload, and warehouse size does not automatically grow as the underlying data grows in size, causing queries to slow down. In this paper, we describe RAIS, the latest collection of AI-powered scaling and optimization techniques in Amazon Redshift, released in preview at re:Invent 2023, which enable it to scale both vertically and horizontally to adapt to all types of workload variability. RAIS dynamically provisions compute resources to run heavy queries efficiently and automatically optimizes warehouse size for the customer's workload, even as it shifts over time. We show that, depending on the workload, RAIS improves either cost or average query execution time by up to 7.6× and 14.2×, respectively, over existing baselines.

References

[1]

Divyakant Agrawal, Amr El Abbadi, Sudipto Das, and Aaron J Elmore. 2011. Database scalability, elasticity, and autonomy in the cloud. In International conference on database systems for advanced applications. Springer, 2--15.

Crossref

Google Scholar

[2]

AWS Amazon. 2023. Amazon Redshift pricing. https://aws.amazon.com/redshift/pricing/.

Google Scholar

[3]

Nikos Armenatzoglou, Sanuj Basu, Naga Bhanoori, Mengchu Cai, Naresh Chainani, Kiran Chinta, Venkatraman Govindaraju, Todd J Green, Monish Gupta, Sebastian Hillig, et al. 2022. Amazon Redshift re-invented. In Proceedings of the 2022 International Conference on Management of Data. 2205--2217.

Digital Library

Google Scholar

[4]

Yun Chi, Hyun Jin Moon, and Hakan Hacigümücs. 2011a. ICBS: Incremental Cost-Based Scheduling under Piecewise Linear SLAs. Proc. VLDB Endow., Vol. 4, 9, 563--574. https://doi.org/10.14778/2002938.2002942

Digital Library

Google Scholar

[5]

Yun Chi, Hyun Jin Moon, Hakan Hacigümücs, and Junichi Tatemura. 2011b. SLA-Tree: A Framework for Efficiently Supporting SLA-Based Decisions in Cloud Computing. In Proceedings of the 14th International Conference on Extending Database Technology (Uppsala, Sweden) (EDBT/ICDT '11). Association for Computing Machinery, New York, NY, USA, 129--140. https://doi.org/10.1145/1951365.1951383

Digital Library

Google Scholar

[6]

Google Cloud. 2023. What is BI Engine? https://cloud.google.com/bigquery/docs/bi-engine-intro.

Google Scholar

[7]

Songyun Duan, Vamsidhar Thummala, and Shivnath Babu. 2009. Tuning Database Configuration Parameters with ITuned. Proc. VLDB Endow., Vol. 2, 1, 1246--1257. https://doi.org/10.14778/1687627.1687767

Digital Library

Google Scholar

[8]

Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zili Meng, and Mohammad Alizadeh. 2019. Learning Scheduling Algorithms for Data Processing Clusters. In Proceedings of the ACM Special Interest Group on Data Communication (Beijing, China) (SIGCOMM '19). Association for Computing Machinery, New York, NY, USA, 270--288. https://doi.org/10.1145/3341302.3342080

Digital Library

Google Scholar

[9]

Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Alizadeh, and Tim Kraska. 2021. Bao: Making learned query optimization practical. In Proceedings of the 2021 International Conference on Management of Data. 1275--1288.

Digital Library

Google Scholar

[10]

Ryan Marcus and Olga Papaemmanouil. 2016. WiSeDB: A Learning-Based Workload Management Advisor for Cloud Databases. Proc. VLDB Endow., Vol. 9, 10, 780--791. https://doi.org/10.14778/2977797.2977804

Digital Library

Google Scholar

[11]

Ibrahim Sabek, Tenzin Samten Ukyab, and Tim Kraska. 2022. LSched: A Workload-Aware Learned Query Scheduler for Analytical Database Systems. In Proceedings of the 2022 International Conference on Management of Data (Philadelphia, PA, USA) (SIGMOD '22). Association for Computing Machinery, New York, NY, USA, 1228--1242. https://doi.org/10.1145/3514221.3526158

Digital Library

Google Scholar

[12]

Gaurav Saxena, Mohammad Rahman, Naresh Chainani, Chunbin Lin, George Caragea, Fahim Chowdhury, Ryan Marcus, Tim Kraska, Ippokratis Pandis, and Balakrishnan Narayanaswamy. 2023. Auto-WLM: Machine learning enhanced workload management in Amazon Redshift. In Companion of the 2023 International Conference on Management of Data. 225--237.

Digital Library

Google Scholar

[13]

Junjay Tan, Thanaa Ghanem, Matthew Perron, Xiangyao Yu, Michael Stonebraker, David DeWitt, Marco Serafini, Ashraf Aboulnaga, and Tim Kraska. 2019. Choosing a Cloud DBMS: Architectures and Tradeoffs. Proc. VLDB Endow., Vol. 12, 12, 2170--2182. https://doi.org/10.14778/3352063.3352133

Digital Library

Google Scholar

[14]

Dana Van Aken, Andrew Pavlo, Geoffrey J. Gordon, and Bohan Zhang. 2017. Automatic Database Management System Tuning Through Large-Scale Machine Learning. In Proceedings of the 2017 ACM International Conference on Management of Data (Chicago, Illinois, USA) (SIGMOD '17). Association for Computing Machinery, New York, NY, USA, 1009--1024. https://doi.org/10.1145/3035918.3064029

Digital Library

Google Scholar

Cited By

View all

van Renen AHorn DPfeil PVaidya KDong WNarayanaswamy MLiu ZSaxena GKipf AKraska T(2024)Why TPC is Not Enough: An Analysis of the Amazon Redshift FleetProceedings of the VLDB Endowment10.14778/3681954.368203117:11(3694-3706)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.14778/3681954.3682031
Diao YHorn DKipf AShchur OBenito IDong WPagano DPfeil PNathan VNarayanaswamy BKraska T(2024)Forecasting Algorithms for Intelligent Resource Scaling: An Experimental AnalysisProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698564(126-143)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698564

Index Terms

Intelligent Scaling in Amazon Redshift
1. Information systems
  1. Data management systems
    1. Database administration
      1. Autonomous database administration
    2. Database management system engines

Recommendations

Amazon Redshift and the Case for Simpler Data Warehouses
SIGMOD '15: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data

Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse solution that makes it simple and cost-effective to efficiently analyze large volumes of data using existing business intelligence tools. Since launching in February 2013, it has ...
Amazon Redshift Re-invented
SIGMOD '22: Proceedings of the 2022 International Conference on Management of Data

In 2013, AmazonWeb Services revolutionized the data warehousing industry by launching Amazon Redshift, the first fully-managed, petabyte-scale, enterprise-grade cloud data warehouse. Amazon Redshift made it simple and cost-effective to efficiently ...
A tenant-based resource allocation model for scaling Software-as-a-Service applications over cloud computing infrastructures

Cloud computing provides on-demand access to computational resources which together with pay-per-use business models, enable application providers seamlessly scaling their services. Cloud computing infrastructures allow creating a variable number of ...

Comments

Information & Contributors

Information

Published In

SIGMOD/PODS '24: Companion of the 2024 International Conference on Management of Data

June 2024

694 pages

ISBN:9798400704222

DOI:10.1145/3626246

General Chairs:
Pablo Barcelo
Universidad Catolica, Chile
,
Nayat Sanchez-Pi
INRIA Chile
,
Program Chairs:
Alexandra Meliou
University of Massachusetts Amherst, USA
,
S. Sudarshan
Indian Institute of Technology Bombay

This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 June 2024

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS '24

Sponsor:

SIGMOD

SIGMOD/PODS '24: International Conference on Management of Data

June 9 - 15, 2024

Santiago AA, Chile

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
164
Total Downloads

Downloads (Last 12 months)164
Downloads (Last 6 weeks)18

Reflects downloads up to 04 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

van Renen AHorn DPfeil PVaidya KDong WNarayanaswamy MLiu ZSaxena GKipf AKraska T(2024)Why TPC is Not Enough: An Analysis of the Amazon Redshift FleetProceedings of the VLDB Endowment10.14778/3681954.368203117:11(3694-3706)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.14778/3681954.3682031
Diao YHorn DKipf AShchur OBenito IDong WPagano DPfeil PNathan VNarayanaswamy BKraska T(2024)Forecasting Algorithms for Intelligent Resource Scaling: An Experimental AnalysisProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698564(126-143)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698564

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Amazon Redshift and the Case for Simpler Data Warehouses

Amazon Redshift Re-invented

A tenant-based resource allocation model for scaling Software-as-a-Service applications over cloud computing infrastructures

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations