Pantomath’s Post

🔎 LUPA ML: An interesting case on how uniformization amongst data teams can make a huge difference in generating business value. Main video: https://lnkd.in/dR3CeRa2 The main idea is based upon data mesh, a decentralised system to manage data for speed and scale developed by Zhamak Dehghani. The simplest, and maybe simplistic 😅, way to start exploring the concept of data mesh is to look at the four foundational points: 1. Domain ownership: decentralize the ownership and responsibility of analytical data to business domains closest to the data, while guaranteeing the defined Service Level Objectives and satisfying their data consumers. 2. Data as a product: these teams should think of their data as a product for other users instead of exclusively for themselves. 3. Self-service infrastructure platform: streamlines the experience of data users to discover, access, and use data products. It streamlines the experience of data providers to build, deploy, and maintain data products. This means it is domain-agnostic and managed by a platform team, standardizing procedures in the company. 4. Federated computational governance: there is a cross-functional board of representatives from domains (data product owners), Platform representatives, Subject matter experts, and Managers to adhere to governance (a set of rules that company data should follow: security, quality, availability, compliance, etc). It should allow adequate autonomy for the domains. This is a principle that needs a more in-depth analysis, so I would recommend you to check Colleen Tartow, Ph.D. and Andy Mott article https://lnkd.in/dZDWvt9x To understand a bit more about data mesh, I would recommend - Reading the following article by Jochen Christ, Dr. Larysa Visengeriyeva, and Dr. Simon Harrer: https://lnkd.in/ddE-FmiW - Watch the video version, hosted by Erik Wilde: https://lnkd.in/dZmxAKvT - A talk by Zhamak herself: https://lnkd.in/d6hygYq8 LUPA ML was born as a tool to implement this idea of data mesh. This 25-min talk by André Godinho and Gustavo Emanuel Santos Pereira explains how LUPA ML, a solution built around the concept of data mesh, has helped streamline processes at NOS SGPS : https://lnkd.in/dR3CeRa2

🔍 How can you ensure that your Data Governance team uses the latest technology? 🔎 LinkedIn's AI-assisted collaborative articles that begin with a GenAI question and add GenAI advice is getting better and better. Contributions from community experts is incrementally improving the whole concept. This particular topic of tools and latest tech is ALWAYS a hot topic that I love to debate. DG isn't a tools-thing, it's a process-thing. https://lnkd.in/ejWXb2Rc #Data #DataMastery #DataGovernance #Metadata #DataQuality #DQ #DataCatalog #DataInventory #DataAsAnAsset #ThePathIsTheGoal #ThinkLearnDo #ideaOutput

The data landscape is evolving fast! Key trends for Business Analysts: GenAI impacts strategy and customer service, emphasizing data ethics. Hybrid/multi-cloud environments drive data mesh and fabric architectures for better accessibility. Check out the DBTA 100 for top data management innovators. Stay ahead in the data-driven world! Link: https://lnkd.in/gbpMB4Jr #sceds #econlib #businessanalytics #data #datamanagement #bigdata #daf #nsbmgreenuniversity

💰 Turning data into dollars: Unraveling the impact of Sigmoid's data mesh approach! 📊💸 📊Dive into the world of #DataMesh, with insights from industry leader Sigmoid, and understand why organisations are embracing this approach to successfully monetise their data. Find out how this decentralised approach is breaking down data silos, enhancing data fluency, and creating custom data products that deliver actionable insights.💰 Read the full article now! 📖💡 #DataMesh #BusinessTransformation #DataEngineering #DataAnalytics #DataMonetisation

OpenMetadata is diving into General AI to transform metadata documentation. Join us on February 1st to explore the workings of Metapilot. #ai #genai

Data's exploding, but the value is trapped in silos! 😰 Time to break them down with DQLabs’ data semantics approach 😇 Learn more 👉 https://lnkd.in/geGMfNJD #DQLabs #moderndataquality #dataquality #dataobservability #datagovernance #datasemantics #GenAI

Change Data Merge is one of the critical components of any ETL pipeline. The needs and challenges grow with growth in data. If you apply this to a multi-tenant pipeline, the data growth patterns show variations and bring the efficiency of the Data merge down. Akshay Kumar shares his views on the scaling challenges along with how we solved it at Capillary Technologies and what were the pros and cons of various choices. Prakhar Verma Pankaj Meena Rajat Kothari Nayan Kumar Rammohan Tangirala Satish Tvv Lakshminarayana Potukuchi Nalin Bhan #datawarehouse #etl-pipeline #change-data-merge #architecture #multi-tenant

Using a high cardinality column (like auto increment id) for partitioning data?? yes we did and this helped our small centralised data team manage/scale variations in numerous datasets. **At capillary we democratise data but not the problems and technical complexity. With this philosophy the central data team had to cater to the requirements of various products/teams/tenants and thus various operational data patterns. Here is one of the problems we faced and how we solved. Akshay Kumar Nalin Bhan Thrivikram G L Lakshminarayana Potukuchi Suhani Porwal #CDC with Upsert #ETL #RangePartitioning #DeltaLake

Stay updated with the latest in data management! Solutions Review has compiled news highlights from the week, featuring updates from Atlan, BigID, Stibo Systems, and more. Whether you're in tech or interested in data trends, this article offers insights into innovations shaping the field. Explore the link to learn about the newest developments in data management technologies. https://lnkd.in/gr3ieErE #DataManagement #TechNews #SolutionsReview #Atlan #BigID #StiboSystems #DataInnovation #DataPrivacy #DataSecurity #DigitalTransformation #TechUpdates #UnderstandingEnterpriseTech #EnterpriseTechnologyNow #EnterpriseTechnologyToday