Mosha Pasumansky

Great blog on Spark AQE!

12

Imagine seamlessly ingesting incremental data from *any* API right into #DeltaLake via Structured #streaming. This article by Irfan Elahi goes deep into the new Custom Data Source API in #apachespark, showing how you can create your own data source for streaming - with checkpointing capabilities to achieve exactly-once guarantees! https://lnkd.in/gEvr5kQJ

17

Thank you David Linthicum for this article. I couldn't agree more. Enterprise Gen-AI is going to hit a wall. The ideal Enterprise Gen-AI strategy will be multi-cloud, multi-model, and multi-architecture. https://lnkd.in/dCiruGy2

16

4 Comments

If you'll be at the #DataAISummit, join Ashok Singamaneni and myself (Scott Haines) as we teach you the ropes of building "Effective Lakehouse Streaming with Delta Lake and Friends". https://lnkd.in/gqmcJmRC We'll be teaching lessons learned (the hard way), and showing how streaming-first is the correct fit for "most" if not all your workflows. Learn to leverage #DeltaLake like an expert, and we'll introduce you to some effortless patterns for making your streaming journey a breeze. You'll also hear from Ashok Singamaneni as we talk about how to leverage painless workflow orchestration with #brickflow, and how to build data trust-networks using #sparkexpectations. #spark #deltalake #sparkexpectations #brickflow #nike #oss #dataai

91

11 Comments

This is a huge step towards aligning the open data lakehouse formats. Recently Microsoft announced compatibilty with Snowflake for their fabric OneLake which was already compatible with Databricks. The future of data lakehouses is open and interoperable… your storage format should not constraint you from using your favorite compute engine..

14

Batch-based, multi-hop architectures tend to be slow, expensive, and brittle. In his latest video, Adam Bellemare explains how a #ShiftLeft Strategy vastly improves your data architecture's latency, cost, and consistency metrics by moving data preparation and cleaning work from its usual place in the analytical plane leftwards into the operational plane. Learn more about how to implement a Shift Left Strategy here ➡️ https://lnkd.in/exV3NDyD

11

Lately, we have observed a transition from Cloud Databases(Snowflake etc) to Iceberg OpenLakehouse format. Ambaricloud team is working on Snowflake to Iceberg Automation and end-user training. Why: Low-cost computing. SF is still a favored engine for Gold layer data. Less stickiness. More engine options to run on Iceberg tables. Cloud storage Parquet files Iceberg table format Catalog Status: Polaris -- Started Walking. Hopefully, it will Jog or Run soon! UnitCatalog -- Infant Glue -- Safe bet for now. Comments? Snowflake Cloudera Dremio Upsolver Databricks

8

1 Comment

It gives us great pleasure to announce that Apache Spark in BigQuery is now officially available in Private Preview. This feature expands BigQuery’s processing capabilities by offering Apache Spark as an OSS processing engine under a unified control plane (admin, monitoring, billing, security, governance). It is the first step in delivering on the Unified Data Platform vision and was a key announcement at Next ‘24 #googlecloud #bigquery #spark

38

2 Comments

3 days ago I raved about the team adding Fine Grain Access Control (FGAC) in EMR Spark, making it the most secure Apache Spark engine in the market. Here is the VLDB paper to back that claim up: https://lnkd.in/g7GMRNWh Spark is known for its extensive API, which makes it a very powerful tool but notoriously porous and hard to secure. The most common way customer secure it is by using Apache Ranger, but the technology does not meet modern security standards which renders it unsuitable for scenarios like cross user/account sharing. The approach taken by more sophisticated DBMS engines is to sandbox UDF and other user code artifacts in process-based containers. Although the approach works, we believe it is not well suited for large scale analytics processing: it suffers from performance problems in data marshaling, it doesn't solve problems with SQL-side channel attack on complex policies and in general, process-based isolation is not the gold-standard compared to full VM isolation. Our approach leverages Spark's use of exchange materialization between operators as a natural security boundary and uses the optimizer to rearrange operators to evaluate the constraints on both performance and security. The paper describes some of this in more detail as well as the challenges implementing it in Spark. where customers are free to write SQL, DataFrames and low-level RDDs. It has been amazing ideating this project with the team and supporting them make to make it a reality. Andrei Paduroiu, Sungheun Wi and Yan Yan will be presenting on VLDB if you are interested to learn more.

166

4 Comments

Great new article by Li (Luke) Yu on how to use #ray to perform feature extraction for #llms on #databricks: https://lnkd.in/d5qv3Xmk Key insights include: 🔧 How to setup and configure a Ray cluster on Databricks with #GPUs 📘 How to overcome tight context windows with various summarization approaches

50

4 Comments

Team DataStax is at #MSBuild! Stop by Booth FP26 to learn about building GenAI apps using Langflow, Astra DB Vector and our just-announced GitHub Co-Pilot extension 👩🏾‍💻 #DataStax #GenAI #VectorSearch #MicrosoftPartner #AzureOpenAI

56

🚨 Databricks Game Changer 🚨 Row and Column Level Security for Unity Catalog -- For the first time on any platform, you can apply fine-grained access controls across Apache Spark, data warehousing, AI, and real-time processing—all within a single platform.

11

I agree. Flattening JSON for analysis is a never-ending problem with relational schema. We just introduced #Couchbase columnar for faster insight on JSON. Simply import, ingest or stream JSON/BSON/etc and start issuing SQL++ queries. See here for details: https://lnkd.in/gaddvXn9

26

News: #snowflakeSummit. Open Data Revolution Continues My Analysis: Snowflake’s strategy around Iceberg is a nuanced topic with significant implications. While the simplicity of Snowflake’s compute services is enhanced by Iceberg, the real value lies in how Snowflake chooses to handle interoperability and external engine support. The Polaris news might crucial in defining its position in the ever-evolving data platform market. Polaris is another #opensource gift from Netflix. Implications: Many including us were asking..will Snowflake choose to embrace Iceberg and its open data potential, even if it means disrupting Snowflake’s current model? Looks like the team is going to disrupt themselves to bring in a new era in open data. The technology industry has a saying, especially true during major market shifts: “Better to eat your own before someone else does.” This means that companies should cannibalize their own advantages before competitors do. For Snowflake, this is happening as the fully embracing Iceberg and its open data capabilities before competitors leverage these advantages against them.

4

What keeps Postgres thriving after three decades? The strength of its open source community, the spirit of innovation, and its ability to evolve with each new data challenge. In our latest blog, EDB Vice President and Postgres Evangelist Bruce Momjian reflects on the Postgres journey and how its extensibility continues to shape the future of data. Postgres isn’t just a database; it’s a way of life. From its vibrant community of contributors to the continuous advancements in data handling, Postgres embodies the power of collaboration. If you’re passionate about technology and want to make an impact, then the Postgres community is where you want to be. Read the blog and see how you can be part of this journey ➡️ https://lnkd.in/e3shY6yw #Postgres #PostgreSQL #justsolveitwithPostgres #EDBPostgresAI #data #AI #database #DBA #opensource

6

Curious about integrating LLMs into your apps? This new blog from the Go team dives into building a powerful RAG server for Q&A using Go. From the blog: "LLM-powered applications are a lot like other modern cloud-native applications: they require excellent support for REST and RPC protocols, concurrency and performance. These just so happen to be the areas where Go excels, making it a fantastic language for writing LLM-powered applications." The blog will cover examples in LangChainGo and my teams new AI framework, Firebase Genkit. https://lnkd.in/eCkw36mY

21

Welcome Howso, a new #validated partner of Databricks! I've been posting about many synthetic data partners in the last few weeks, but Howso is different. Howso’s main differentiator is its focus on #AI_explainability. In industries where trust and compliance are critical (like finance, healthcare, and government), being able to explain how an AI model makes decisions is extremely valuable. In addition, Howso is also focused on #Causal_AI. This helps organizations uncover cause-and-effect relationships, not just correlations, which improves decision-making in complex environments. They uncover the truth about your data. Incredibly happy to have Howso, "The Understandable AI® Platform" as part of the Databricks AI partner ecosystem! Read more here: https://lnkd.in/e-SacuJH John Emmert

44

7 Comments

The subject of support for incremental reads of mutable tables and generating CDC streams in the four table formats I'm covering is far more nuanced than I expected. What was going to be one post is going to have to be a 5 parter to keep it manageable: 1. Introduction to the concepts 2-5. Apache Iceberg, Delta Lake, Apache Hudi, Apache Paimon

15

2 Comments

Snowflake Polaris: POC Status 2. Branching, tagging, branch fast-forwarding, expiring snapshots, and all other maintenance procedures are working smoothly without any issues. I’m hopeful that Snowflake will assist with the issues I’m experiencing related to add_files and register_table procedures. Snowflake Dremio ambariCloud Today's Quote: Arrogance has toppled empires.

7