Azure Data Engineer Lead

  • San Francisco, CA
  • Posted 27 days ago | Updated 1 day ago

Overview

On Site
Depends on Experience
Contract - Independent
Contract - W2

Skills

Python
PySpark
azure
data lake

Job Details

Job Description:

We are seeking a highly skilled Senior Data Engineer with extensive experience in Python, PySpark, SQL, and Microsoft Azure Cloud Services. The ideal candidate will have a strong background in data warehousing, big data technologies, and cloud data engineering, along with a proven ability to lead development teams and drive high-quality project deliverables.

Key Responsibilities:

  • Develop and maintain scalable data pipelines using Python, PySpark, and SQL.

  • Design, implement, and optimize data warehouses and data lake architectures on Azure Cloud.

  • Build and manage ETL/ELT workflows using tools like Azure Data Factory (ADF), Informatica, or Talend.

  • Work with Azure Synapse, Azure SQL, Amazon Redshift, Snowflake, or Google BigQuery to support data analytics and reporting.

  • Develop dimensional data models and schema designs for optimized query performance.

  • Leverage Databricks (preferable) or similar platforms for big data processing.

  • Implement best practices in Medallion Architecture for data organization and management.

  • Utilize Azure Data Lake Storage, Cosmos DB, and Blob Storage for efficient data storage solutions.

  • Collaborate with cross-functional teams to solve complex data challenges and drive insights.

  • Provide technical leadership, mentor junior engineers, and ensure project success.

Required Skills & Experience:

  • 12+ years of experience in SQL, schema design, and dimensional data modeling.

  • 5+ years of experience in modern data engineering, data warehousing, and Azure cloud technologies.

  • Strong hands-on expertise in Python, PySpark, and SQL.

  • Proficiency in Azure Cloud Services, including Azure Synapse, Azure SQL Database, Azure Data Factory, and Azure Data Lake Storage.

  • Experience with big data platforms (Databricks preferred).

  • Solid understanding of ETL/ELT processes and tools such as ADF, Informatica, Talend.

  • Excellent problem-solving, analytical skills, and leadership abilities.

Preferred Qualifications:

  • Experience implementing Medallion Architecture for data organization.

  • Familiarity with Azure Cosmos DB and Blob Storage.

  • Strong ability to lead development teams and drive project execution.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.