Vivek Supnekar

Someone rightly said : Snowflake is for rich, low to no-tech departments like Finance. Databricks is for data engineers. In the end, both are successful, multi-billion dollar companies. Choice in the end lies in defining a clear customer segment. #snowflake #databricks #dataengineering #customersegment

41

3 Comments

One for the data architects / data engineers & professionals who work with Snowflake - which approach do you think is better for “moving data” from a 3rd party who uses Snowflake to your Snowflake instance: 1) via API using an integration tool like ADF, Informatica, Matillion etc 2) or using Snowflake direct share There are considerations for costs, risks, security and efficiency. My personal opinion is Direct share as it is so much more simpler and more secure - point to point, plus zero costs for moving data and mostly zero development effort on the subscriber side. But there would be advantages of api as well that I am not across. Would appreciate your comments and thoughts.

16

10 Comments

You can easily deploy any John Snow Labs models within Snowflake's #Snowpark Container Services Ecosystem. This includes 2,500 #Healthcare-specific #NLP and #LLM models, nlp.deploy_model_as_snowflake_udf() will build, tag & push a John Snow Labs model server to your Snowflake image repository and finally create a service & udf from the model and test it. Everything runs within your infrastructure - no data ever leaves your security perimeter and no external API call is made. #ai #generativeai #healthcareai #healthai #datascience #nlproc #textmining

18

Top insights from #theCUBE’s real-time coverage at Snowflake’s #DataCloudSummit ❄️ https://lnkd.in/gcFpgcJm Under new CEO Sridhar Ramaswamy, Snowflake is shifting toward an AI-driven strategy, enhancing support for Apache Iceberg, and simplifying AI applications with Cortex. The event highlighted how Snowflake is evolving to meet customer demands for data simplicity and business value. “We use this term, the sixth data platform, so [there were] mainframes, then you had Oracle, you had enterprise data warehouse, you had Hadoop, and then Snowflake separated compute from storage, they put everything in the cloud, made it infinitely scalable and everything was integrated,” said theCUBE Research Chief Analyst David Vellante in his on-site report. “But not any compute can access any data, and we’re seeing a demand for that. Of course, the value proposition of Snowflake is ‘iPhone,’ all totally integrated. This is the nature of the disruption that we’re witnessing today.” These discussions underscore the growing importance of Snowflake's role in data management across various industries. 🌐 From media to retail, Snowflake is revolutionizing how enterprises leverage data for actionable insights. 📺Catch all the insights and watch the interviews now! #DataCloudSummit #theCUBE #AI #TechNews #CloudComputing #DataSimplicity

4

"Databricks Ushers In New Era for Open #Data And #AI Governance With Unity Catalog", click to get the full story via the News Desk over on Elnion. https://lnkd.in/gDnArSJ5 #dataaisummit #data #datalake #ai #ml #gpt #genai #cloud #analytics #bi #datascience #datascientist

1

Thrilled to share an insightful blog post by Community Member Harshal Pagar on Running Databricks Notebook with Airflow. This article delves into the integration of Databricks Notebooks with Apache Airflow, offering a step-by-step guide on how to leverage this powerful combination for your data workflows. Whether you're looking to streamline your data processing or explore new capabilities, this post is a must-read! Kudos to the author for providing such valuable content. Let’s dive in, learn, and continue to innovate together! #ApacheAirflow #Databricks #DataEngineering #DataScience #BigData #DataPipeline

146

1 Comment

Snowflake Optimization Tip of the Day #6 ➡ Identify and remove failing workloads. Even in mid-sized data teams, there’s often a chunk of money that’s going towards workloads that are just failing at the end due to various reasons. These workloads get no work done but consume compute credits. Simply identifying failing workloads can help reduce Snowflake costs. The tricky part is to define what exactly would be a failing workload. In the first comment, you’ll find a blog by Sahil Singla, my co-founder that helps define parameters for defining a workload as failing. Remediation is generally finding the user responsible for the workload and asking him/her to fix it or remove it. Apart from saving money, a secondary advantage for data teams is to keep track of what data is relevant for business users and what’s not.

13

2 Comments

Exciting news in the world of data analytics! 🌟 Databricks, a leading software maker, has acquired Tabular, a startup specializing in optimizing cloud-stored data. This strategic move will help Databricks develop new products faster and stay competitive against rivals like Snowflake. The acquisition, valued at over $1 billion, showcases Databricks' commitment to growth. With strong financial backing and recent acquisitions, Databricks is solidifying its position in the market. Tabular's expertise in enhancing the open-source format Apache Iceberg will benefit Databricks' efforts to attract Snowflake clients. This acquisition marks a significant step for Databricks in shaping the future of cloud data optimization. Stay tuned for more updates! #Databricks #Data #Cloud #Datanised #Tabular https://lnkd.in/eFniN9t3

4

For those that are setting up Databricks with Unity catalog on Amazon Web Services (AWS) , this is a fantastic video which covers step by step to setup AWS roles, policy and identity management to Databricks. https://lnkd.in/g2pwymdj

4

Snowflake vs. Databricks: An Honest Comparison in 2024 I recently stumbled upon this informative and timely post on select.dev while looking for something else. Given my current focus on this topic, I read the article thoroughly and listened to the interview. My key takeaway: Apache Spark may have its niche uses, but it isn't ideal for building a comprehensive data warehouse program. Ian Whitestone and Jeff Chou were careful to call out their individual bias' and so I will follow their lead. I'm am a huge advocate for Snowflake. I have been building data warehouses for 25 years and since I found Snowflake, and came to understand its power, I have built 18 data warehouses in Snowflake in the last 5 or 6 years. I love Snowflake for many reasons, including its ease of use, its complete separation of storage and compute, and the ability to seamlessly add either to near infinite scale. Interestingly, I've found that some Databricks users pull their data from Snowflake and return the results back to Snowflake for further dissemination. Another contact in my network shared that a recent POC confirmed that Databricks would not reduce their considerable Snowflake spend. But don't take my word for it: Read more about Snowflake vs. Databricks here. #DataAnalytics #DataWarehouse #Snowflake #Databricks #CloudDataConsulting, #ApacheSpark

80

5 Comments

📓 Quick Read: In this post, Jon Alkan provides an overview of 3 Snowflake performance features: (1) Search Optimization Service, (2) Query Acceleration Service, and (3) Cache.

25

🚀 Exciting News from Databricks at Data + AI Summit 2024! 🌟 The latest from Databricks: Delta Lake 4.0 has arrived with groundbreaking innovations in data management. 🔹 Liquid Clustering: Boosting performance up to 7x for writes and 12x for reads, this new data layout strategy simplifies schema evolution and resolves partitioning challenges without data rewriting. 🔹 Variant Data Type: Introducing Variant for efficient storage of semi-structured data, Delta 4.0 offers flexibility, performance, and an open format to meet diverse data management needs. Already benefiting over 1,500 customers, Delta Lake 4.0 integrates Variant in Databricks Runtime 15.3, ensuring seamless compatibility with Apache Spark and establishing it as a leading solution for modern data workflows. Discover how Delta 4.0 can empower your organization with faster, more flexible data management. Explore the future of data innovation with Databricks! Elevate your team's skills in Databricks, Apache Spark, and related technologies with DataCouch's expert trainers and consultants. Contact us today at [email protected] to get started! #DataManagement #DeltaLake #DataInnovation #Databricks #AI #DataScience #TechNews #DataEngineering #DataCouch

3

What are User-Defined Functions (UDFs) in Databricks (Spark) and which type is most efficient? UDFs allow custom logic to be reused in SQL or Spark environments. They are beneficial for ad hoc queries and small to medium datasets but can introduce inefficiencies in large-scale ETL jobs. Efficiency Rank for Various Databricks UDF Types: 1. Built-in Databricks functions 2. JVM UDFs (Scala, Java, Hive UDFs) 3. Pandas UDFs 4. Python UDFs Python Scalar UDFs are the least efficient. Note that some Python Scalar/Pandas UDFs are unsupported with Unity Catalog, depending on the runtime. Leverage Photon-enabled clusters for optimized SQL UDFs. For more details, check the full documentation https://lnkd.in/gUts-YnK #databricks #dataengineer #spark

4

If you had invested $700 in Snowflake stock the day shares went public, your stake would now be worth just $454 (a 35% drop). The S&P 500 index, for comparison, increased in value by 57% over that time. Consider when Nvidia made this list on April 15, 2005... if you invested $1,000 at that time, you’d have $566,624! https://lnkd.in/gABZNqDw #techstocks

7

Great to see Qlik investing in GenAI. Self-serve data/analytics/bi has been the narrative for a while now, but skillset hurdles still exist. LLMs on top of data feels like the natural lowering of the bar. Combine increased speed and quality of output, alongside improved data literacy, and this could really be something. Looking forward to seeing the roadmap. #QlikAnswers #AI #GenAI #Qlik #data #bi #analytics #QlikLuminary

17

Tableau introduces an interesting feature 'Multi-fact Relationships' with Tableau v2024.2 which allows to explore correlations and answer more sophisticated analytical questions. Schemas based on fact tables are often structured as a star or snowflake, depending on how the dimension tables are organized. When analysis needs to be performed across fact tables, this is called multi-fact analysis. Analysis is done in the context of the common dimension tables, known as shared dimensions or conformed dimensions. Before Multi-fact Relationships, a Tableau data model built with Relationships could support branching out downstream from a base table, but could not relate a table back to more than one upstream table. with this new feature, it unlocks new range of possibilities and data exploration scenarios. further details on link https://lnkd.in/gzmVSbp8

6

Are you ever stumped on which chart to use in Tableau to best convey your data story? Next time, take a look at this chart guide made by Damola Ladipo and featured on Andy Kriebel's blog. This handy tool will simplify your chart decision-making process in no time! https://lnkd.in/eNX2iHCs #TableauCharts #TableauTips #TableauTraining

6

Join us this Friday Nov 15th, 10 – 10:30 AM CT for an exciting session with our CTO Patrick Vinton and Databricks Sr. Solutions Architect Tristan Nixon. They're going to discuss specific ways the Databricks platform is helping companies move from AI experimentation to making AI a very real and transformative aspect of their data strategy. Come to learn how you can consolidate tools, reduce AI/ML complexity, and give your data scientists and engineers an interface they are familiar with! 🔗 Register here: https://lnkd.in/gAhHeMBa #AI #MachineLearning #Databricks #AIInnovation #TechTalk #Webinar

16

1 Comment