Tomislav Bodrozic

Unlock the future of data warehousing with Snowflake and AWS: Drive into our comprehensive guide to scalability, performance and integration. Thankyou PCG Germany and Carsten Riggelsen for giving me the opportunity to publish my thoughts! #DataWarehousing #AI #ML #Snowflake #AWS #PCG Hannes Novak Raphael Bögel Julius Otto Mathias Rehberg David Lewenko

6

📡 As a seasoned data engineer, I have vast experience designing robust pipelines for seamless data movement, leveraging Apache Kafka 🐘 and integrating with Snowflake ❄️. I architect end-to-end ingestion solutions harnessing Kafka's power to ingest real-time streams from diverse sources 💻🌐, ensuring reliable and efficient ingestion, even with massive volumes and high velocity 🚀. ⛄️ I seamlessly integrate Snowflake, utilizing its powerful processing, scalability, and semi-structured data support. Through proficiency in Snowflake's ingestion mechanisms like Snowpipe and Kafka Connector, I facilitate continuous loading of data from Kafka topics into Snowflake tables, enabling real-time analytics and data-driven decisions 📊. 🔄 I implement advanced transformation and enrichment techniques within the Kafka ecosystem, using tools like Kafka Streams and ksqlDB. This allows cleansing, enrichment, and transformation of data streams before loading into Snowflake, ensuring quality and consistency 🧼. ⚙️ I prioritize automation, monitoring, and scalability, leveraging Apache Airflow 🪰, Kubernetes 🐳, and cloud-native services for orchestrating and managing pipelines. This enables efficient and reliable data movement while minimizing manual intervention and ensuring high availability ☁️. 🚀 With expertise in Kafka, Snowflake, and related technologies, I deliver solutions empowering organizations to unlock data's potential, enabling real-time analytics, data-driven decisions, and new business insights 💡. My deep understanding of these technologies and their integration allows me to architect and implement robust, scalable, and efficient pipelines tailored to business needs 🎯. #DataEngineer #DataAnalyst #DataScience #C2C #C2H #ApacheKafka #Apacheairflow #Snowflake #Datawarehouse #data C2C JOBS #C2Crequirements #C2Cjobposts #C2C #AWS #AZURE #PYSPARK #PYTHON #ETL #TABLEAU #POWERBI #aws services #azure services

4

1 Kommentar

SQLAlchemy combines the robustness of SQL with Python's flexibility, making database management not just easier, but kinda fun too! https://lnkd.in/eFJzyTxG #Python #Libraries #Development #Automation #Data #Database

2

If you have large non-partitioned tables in your data lake, at a minimum partition the dataset and update your ETL to target specific partitions. You will save a lot of I/O and compute with that approach. For situations where you can’t find a good partition column in your table, make one as a load/create date at a minimum. You should be able to get creative enough then when loading that data to work with a rolling x amount of days to make changes against on the target dataset. On dimensional data, this is a bit more tricky since that data is sometimes not time-series based and might require full rewrite. I know I just spoke in high level terms here and some will disagree but as I’ve stressed many times over, “partitions are your friend on big data”. They are one of the “cheapest” ways to make your data loads and reads fast without the overhead of having to do things like build and maintain an index. #dataengineering

61

28 Kommentare

#Technology like history tend to repeat itself. #Llama is doing an #opensource like what #redhat did decades ago. There's many nuances though ✅ #meta is a giant compared to #redhat ✅ unlike an #os that can run on any low cost #pc hardware, these #genai models require hundreds to thousands of super expensive specialized #gpu to produce outcomes. heck, even #gpu is a misnomer as the tech evolves, they'll be specific hardware for each of the major providers (#openai, #google) just like #crypto #mining cards. ✅ it'll be a large #corporate and #government play as a vendor because smallco won't be able to afford training their own models and owning those cards to optimize #training ✅ They'll be secondary market play via specialist smaller #ai companies that ride on the #platforms and existing models provided by #microsoft and others. in which they then offer services to smaller companies. ✅ what is still in immaturity, is building the #genai #usecase. While creative outputs like videos and drawings are entertaining, outputs that are beneficial to the world are still rare. I'll share in a different post what I'm looking forward to.

8

When working with Spark SQL, I sometimes find a very nifty function that I want to externalize later in the dataframe syntax, but I notice that the function I'm after is not available as an import. So how do you still get to it? You can use the "expr" dataframe function to get to these functions. In the example below, I'm using the "expr" function to use the Spark SQL "uuid()" function to generate a guid. It's simple, fast, and scales well. And it will almost always outperform a user defined function. #dataengineering

229

21 Kommentare

If you're diving into the world of Infrastructure as Code (IaC) and looking to enhance your security and compliance measures, find my new blog post on #devto and learn how MRH Trowe is using Amazon Web Services (AWS) via IaC! 🔍 Key Highlights: - Introduction to #CDK Nag: Learn how CDK Nag integrates with #AWS Cloud Development Kit (CDK) to enforce best practices and security policies. - Policy as Code: Understand how to implement and manage policies programmatically, ensuring your infrastructure adheres to organizational standards. - Practical Examples: The post provides hands-on examples and code snippets to get you started quickly. - Security and Compliance: Discover how CDK Nag can help you automate security checks and compliance validations, making your cloud deployments more robust and secure. 💡 Whether you're a seasoned DevOps engineer or just starting with AWS CDK, this article offers valuable insights into making your infrastructure more secure and compliant. Check out the full blog post here: https://lnkd.in/e6CCuNFG #AWS #CDK #PolicyAsCode #DevOps #CloudSecurity #InfrastructureAsCode #IaC

4

When an SQL alternative claims to be "more intuitive" due to its syntax, it's important to remember that intuitiveness is subjective. Just as natural languages vary—some placing verbs before subjects and others the reverse—what feels intuitive to one person may not to another. Preferences for syntax depend on individual backgrounds and cognitive styles. Some may find SQL's declarative nature aligns with their thinking, while others might prefer a different approach. Ultimately, the best query language is the one that fits the context and resonates with the user's way of problem-solving.

4

AWS is "closing new customer access to AWS CodeCommit" Code Commit is the AWS managed version of source control management, like Github and Gitlab. From experience, Code Commit integrates well with the AWS CI/CD pipeline, in particular Code Pipeline. This allows developers to maintain both cloud infrastructure and application code with AWS managed builds and deployments. They are basically saying you should use another service going forward. This will definitely be annoying for those who have chosen to go with Code Commit for source code management. https://lnkd.in/eZyiKMvD #aws #devops

1