Balaji Doppalapudi

#Azure #Cloud #DataBricks #DataBrick #C2C #W2 Immediate Need  Azure Cloud Engineer with Databricks Admin (Onsite)  Azure Cloud Engineer with Databricks Admin (Onsite)    Location: Daytona Beach, FL  Need to work from the Daytona office.    Scope of Work  ·    Deep understanding of Azure Cloud Platform and Engineering, Infrastructure setup and configuration, tools and services, Data access and sharing, and hands-on work.  ·    Key objectives Include tasks such as designing and maintaining Databricks infrastructure, optimizing data engineering workflows, and ensuring data quality and governance.  ·    Deep understanding of Databricks tools and products like Delta Tables, Delta Like Tables, Unity Catalog, Serverless execution and setup and configuring experience.  ·    The technical skills and expertise required for the role, such as proficiency in Databricks, Apache Spark, cloud platforms (Azure), programming languages (Python, Scala, Java), data modeling, and database design.  ·    Collaboration with other team members, such as data scientists, data engineers, and business stakeholders. Emphasize the need for effective communication, documentation, and knowledge sharing within the team.  ·    The expected outcomes and performance indicators for the Databricks Engineer. This could include metrics such as data processing efficiency, data quality improvements, and successful project completion.     Skill Set  ·    3-4 Years of working experience on Azure Platform – Infra, tools and services  ·    2-3 years of experience on optimize and tune Databricks clusters for performance, scalability, and reliability for data engineering workloads.  ·    Develop and maintain ETL pipelines to extract, transform, and load data from various sources into Databricks.  ·    Hands-on working experience on Python/PySpark using Databricks Notebooks and Workspace.  ·    Experience in troubleshoot and resolve Databricks-related issues and provide technical support to users.  ·    Architect and Design, Databricks deployment strategy and experience on data engineering tasks, including data processing, transformation, and integration.  ·    Implement Azure data engineering best practices, including data quality checks, data lineage, and data governance, within the Databricks environment.  ·    Leverage Azure DevOps and CI/CD best practices to automate the deployment and management of data pipelines and infrastructure Thanks & Regards,  Battu Raghu [email protected]

9

3 Comments

Snowflake question: on Netezza, you had to be connected to a database to create an object, eg a procedure. on Snowflake, you can compile a proc to a database you are not currently using if the procedure name is qualified with DB and schema. is it possible to prevent this and reciprocate Netezza behaviour which I believe was safer?

1

I recently received couple of requests to share prepration tips/ticks and important topics for SnowPro Advanced Data Engineer exam. I have compiled these on below article on medium. #snowflake #snowproadvanved #certificationtips #datalake #fidelity

21

1 Comment

ETL Vs ELT Datawarehouse Design Pattern. Designing effective data pipelines is crucial for efficiently managing and processing data in various environments, from simple ETL (Extract, Transform, Load) tasks to complex real-time analytics. Common data pipeline design patterns are Batch Processing Stream Processing Lambda Architecture Data Lake Architecture ETL (Extract, Transform, Load) ELT (Extract, Load, Transform) Data Mesh Architecture Data Pipeline Orchestration Data Replication and Synchronization ETL (Extract, Transform, Load) Description: Extracts data from sources, loads it into a destination, and then transforms it. Use Cases: Useful when dealing with large volumes of data or when leveraging the processing power of modern data warehouses. Good design pattern for designing DataLakes. Foundation for Change Data Capture. Tools: Google BigQuery, Snowflake, Amazon Redshift. ELT (Extract, Load, Transform) Description: Extracts data from sources, transforms it, and loads it into a destination, such as a data warehouse. Use Cases: Common in data warehousing and integration scenarios. Tools: Talend, Informatica, AWS Glue, Apache NiFi. In both the above scenarios the notable point is we don't want to disturb the operational systems hence we move the data to a staging source.

3