Positive change for BigQuery partitioned tables this week: the previous limit of 4000 partitions per table is now raised to 10000 🎉 . From a practical point of view, 4000 partitions is almost 11 years of daily partitions, and 10000 is 27 years of daily partitions. I had in my head that partitions were introduced in BigQuery around 2014 so when I initially read the news, I thought to myself "Right on time to save those who started using daily partitions from the start" I wanted to double-check when partitions were introduced to BigQuery so like it's custom nowadays I asked AI: - Gemini was off, replying that a public date is not available but from Stackoverflow it estimated that it was before July 2017 - Chat GPT 4 pointed to 19th September 2016 and the official blog post as a source. However, when I asked to print the URL, it wasn't able to do so, i.e. it printed a blank URL, when I tried to get the real URL it changed the announcement date to 2nd June 2016 but still without a concrete URL So I got all the articles from the Google Cloud blog website to get to the post that introduced partitions and indeed it was 2nd June 2016. https://lnkd.in/ervusxVx (side note of scraping: there are in total 8230 articles on the official blog website, the first one published on 16th June 2010) However, looking at the BigQuery release page (I should have looked here in the first place), it looks like the initial max partition number was 2500 and on 4th May it was raised to 4000. In other words, they raised the buffer, so now they have some 20 years to figure out how to increase partitions to accommodate the period afterward. Of course, I'm ignoring the usage of partitioned historical data 😀 #rainysaturday #nonsence #googlecloud p.s. I started with my newsletter GCP Weekly on 3rd October 2016 so the post wasn't in the archives, of course that was the first place I went 👍
Zdenko Hrček’s Post
More Relevant Posts
-
Buckle up! We're headed to the BigQuery for ML course on Google Cloud Skills Boost for the third stop on our summer learning road trip. Learn how to create, train, evaluate, and predict with ML using existing SQL tools and skills on #BigQuery → https://goo.gle/3W6hTR5
-
-
Unleash the ML power right at the heart of your Data warehouse in BigQuery
Buckle up! We're headed to the BigQuery for ML course on Google Cloud Skills Boost for the third stop on our summer learning road trip. Learn how to create, train, evaluate, and predict with ML using existing SQL tools and skills on #BigQuery → https://goo.gle/3W6hTR5
-
-
We're delighted to be recognized as Google Cloud Ready - BigQuery. Our BigQuery integration means that with Harbr, you can now: 🔗 Securely connect to BigQuery instances to access, use, and distribute data. 🔎 Query and analyze data stored in BigQuery using SQL or natural language. 📊 Use industry-leading BI and analytics tools with data stored in BigQuery — without needing to move or copy data. 🔐 Maintain governance and strict access controls over how data stored in BigQuery is used. Check out the full announcement in the comments for more details. #googlecloud #googlecloudpartners
-
-
As a Sales Engineer specialist in Data Analytics & AI/ML (predictive & generative), I accompany enterprise customers to design and implement innovative solutions on Google Cloud
BigQuery vector search is now generally available! 🎉 This exciting feature allows for efficient similarity searches within your data. Learn more about how it works and its potential applications in this informative Cloud Blog post. 👇 https://lnkd.in/e6F4WTiY #BigQuery #VectorSearch #DataAnalytics #GoogleCloud
BigQuery vector search is now GA | Google Cloud Blog
cloud.google.com
-
Google Cloud #Research innovates with ... #BigLake: BigQuery’s Evolution toward a Multi-Cloud Lakehouse. 🥇⚡🔥 Check out our Research Paper here, to be presented at next month's #SIGMOD2024 event: https://lnkd.in/gKtHRvXG Innovations include: → #BigLake tables, making open-source table formats (e.g., Apache Parquet, Iceberg and Delta) first class citizens → Fine-grained #governance enforcement and performance acceleration for BigQuery and other open-source analytics engines → Design and implementation of BigLake #Object tables to integrate AI/ML for inferencing and processing over unstructured data → #Omni, a platform for deploying BigQuery on non-GCP clouds, enabling an enterprise lakehouse across cloud providers
BigLake: BigQuery’s Evolution toward a Multi-Cloud Lakehouse
-
Google Cloud #Research innovates with ... #BigLake: BigQuery’s Evolution toward a Multi-Cloud Lakehouse. 🥇⚡🔥 Check out our Research Paper here, to be presented at next month's #SIGMOD2024 event: https://lnkd.in/edzEH7z7 Innovations include: → #BigLake tables, making open-source table formats (e.g., Apache Parquet, Iceberg and Delta) first class citizens → Fine-grained #governance enforcement and performance acceleration for BigQuery and other open-source analytics engines → Design and implementation of BigLake #Object tables to integrate AI/ML for inferencing and processing over unstructured data → #Omni, a platform for deploying BigQuery on non-GCP clouds, enabling an enterprise lakehouse across cloud providers
BigLake: BigQuery’s Evolution toward a Multi-Cloud Lakehouse
-
Google Cloud #Research innovates with ... #BigLake: BigQuery’s Evolution toward a Multi-Cloud Lakehouse. 🥇⚡🔥 Check out our Research Paper here, to be presented at next month's #SIGMOD2024 event: https://lnkd.in/guqmN_Dh Innovations include: → #BigLake tables, making open-source table formats (e.g., Apache Parquet, Iceberg and Delta) first class citizens → Fine-grained #governance enforcement and performance acceleration for BigQuery and other open-source analytics engines → Design and implementation of BigLake #Object tables to integrate AI/ML for inferencing and processing over unstructured data → #Omni, a platform for deploying BigQuery on non-GCP clouds, enabling an enterprise lakehouse across cloud providers
BigLake: BigQuery’s Evolution toward a Multi-Cloud Lakehouse
-
Top content on Google Cloud Medium publication (https://lnkd.in/dQy5B3Uj) for month of April 2024. There are some fantastic articles in this list and not ranked in order of views, etc. An interesting trend on Google Cloud Medium publication in the months of April, May, June 2024 has been the rise of Generative AI articles, with almost 40-50% of the articles being on that topic. More on that in a later post. 📘 GCP Data Engineering Project: Streaming Data Pipeline with Pub/Sub and Apache Beam/Dataflow by Jana Polianskaja: [https://bit.ly/4eRixt9] 📘 Level Up your RAG: Tuning Embeddings on Vertex AI by Ivan Nardini: [https://bit.ly/4cKSgL1] 📘 High-performance Stable Diffusion XL Inference on GKE and TPU v5e with MaxDiffusion by Rick(Rugui) Chen: [https://bit.ly/3zs7VAF] 📘 Configuring Data Pipeline Environments in Dataform by Alex Feldman : [https://bit.ly/4eYRlJ0] 📘 Forwarding over 100 Mpps with FD.io VPP on x86 by Federico Iezzi : [https://bit.ly/3zsTtIz] 📘 Platform Engineering in action: Deploy the Online Boutique sample apps with Score and Humanitec by Mathieu Benoit : [https://bit.ly/3XO8qz3] 📘 Master Data Management Simplified: Match & Merge with Generative AI! by Abirami Sukumaran : [https://bit.ly/4cs5Tit] 📘 Hidden Gems of BigQuery by Artem Nikulchenko: [https://bit.ly/3xLVjUr] 📘 Fine Tuning Large Language Models: How Vertex AI Takes LLMs to the Next Level by Abirami Sukumaran : [https://bit.ly/3zpSAAy] 📘 Build Infrastructure on Google Cloud with Terraform — Google Challenge Lab Walkthrough by Darren Lester : [https://bit.ly/3Y71WM3] 📘 Making sense of Vector Search and Embeddings across GCP products by Steve Loh: [https://bit.ly/4coaYsg] 📘 Parsing Invoices using Gemini 1.5 API with Google Apps Script by Kanshi Tanaike : [https://bit.ly/4cCakae] 📘 Design your Landing Zone — Design Considerations Part 4— IaC, GitOps and CI/CD (Google Cloud Adoption Series) by Darren Lester : [https://bit.ly/4cOfBeH] Thankful to everyone who has joined the community and shared their Google Cloud knowledge with us. We are 500+ articles strong in 2024 alone. If you'd like to contribute, please reach out 1:1 and we'd love to bring you onboard as a writer. #GoogleCloud #Community #Articles
Google Cloud - Community – Medium
medium.com
-
Here’s the top medium articles for your weekend read about Google Cloud. Glad to have contritbuted to the list 🙏
Top content on Google Cloud Medium publication (https://lnkd.in/dQy5B3Uj) for month of April 2024. There are some fantastic articles in this list and not ranked in order of views, etc. An interesting trend on Google Cloud Medium publication in the months of April, May, June 2024 has been the rise of Generative AI articles, with almost 40-50% of the articles being on that topic. More on that in a later post. 📘 GCP Data Engineering Project: Streaming Data Pipeline with Pub/Sub and Apache Beam/Dataflow by Jana Polianskaja: [https://bit.ly/4eRixt9] 📘 Level Up your RAG: Tuning Embeddings on Vertex AI by Ivan Nardini: [https://bit.ly/4cKSgL1] 📘 High-performance Stable Diffusion XL Inference on GKE and TPU v5e with MaxDiffusion by Rick(Rugui) Chen: [https://bit.ly/3zs7VAF] 📘 Configuring Data Pipeline Environments in Dataform by Alex Feldman : [https://bit.ly/4eYRlJ0] 📘 Forwarding over 100 Mpps with FD.io VPP on x86 by Federico Iezzi : [https://bit.ly/3zsTtIz] 📘 Platform Engineering in action: Deploy the Online Boutique sample apps with Score and Humanitec by Mathieu Benoit : [https://bit.ly/3XO8qz3] 📘 Master Data Management Simplified: Match & Merge with Generative AI! by Abirami Sukumaran : [https://bit.ly/4cs5Tit] 📘 Hidden Gems of BigQuery by Artem Nikulchenko: [https://bit.ly/3xLVjUr] 📘 Fine Tuning Large Language Models: How Vertex AI Takes LLMs to the Next Level by Abirami Sukumaran : [https://bit.ly/3zpSAAy] 📘 Build Infrastructure on Google Cloud with Terraform — Google Challenge Lab Walkthrough by Darren Lester : [https://bit.ly/3Y71WM3] 📘 Making sense of Vector Search and Embeddings across GCP products by Steve Loh: [https://bit.ly/4coaYsg] 📘 Parsing Invoices using Gemini 1.5 API with Google Apps Script by Kanshi Tanaike : [https://bit.ly/4cCakae] 📘 Design your Landing Zone — Design Considerations Part 4— IaC, GitOps and CI/CD (Google Cloud Adoption Series) by Darren Lester : [https://bit.ly/4cOfBeH] Thankful to everyone who has joined the community and shared their Google Cloud knowledge with us. We are 500+ articles strong in 2024 alone. If you'd like to contribute, please reach out 1:1 and we'd love to bring you onboard as a writer. #GoogleCloud #Community #Articles
Google Cloud - Community – Medium
medium.com
-
I'm a great believer in growth mindset and value-driven strategy | Technology Strategist | Data Expert | Public Speaker | Solutions Architect | Software Engineer
For everyone who ever asked for #Nifi as a service floating in the cloud at extreme focus, I'll just put this here 😉 Also would recommend chatting with Dan Cohen if you're looking to explore further into #Datavolo
The latest Datavolo Cloud now live! https://lnkd.in/gCXmaa9Y You can now leverage Snowflake Cortex AI for LLM/prompting in the flows. Document parsing for PDFs has continued to get more powerful and now includes Microsoft Office Word, PPT, Excel, and other OpenDocument formats. Visualizations for understanding chunking and annotating parsed documents to understand the structure, bounding boxes, and types detected make parsing at scale easier than ever. We're still technically in a private beta but soon we'll open it up officially. Give us a shout at Datavolo if you're ready to parse documents at scale - or if you simply want the best way to run elastically scaling Apache NiFi workloads whether in the cloud or on-prem.
-
Head Of Technology at Modigie, Inc.
4moYou walked the extra mile. 😆 Just a few weeks ago I had this same discussion I already had a couple of times: Until the 4,000 become a problem they will figure something out, or somebody on our side, but it's so far out, I don't even feel bad. Even more so now with 10,000.