Airflow Implementation Lead

Overview

Remote
Depends on Experience
Contract - Independent
Contract - 6 Month(s)
No Travel Required

Skills

Azure Data Factory
Azure Data Lake
Azure Databricks
Azure Synapse
Microsoft Purview
Azure Cloud
PowerShell
Python
RESTful APIs
NoSQL
Cassandra
MS SQL Server
T-SQL
Azure SQL Managed Instance
Shell Scripting
Microsoft Power BI
Azure Pipelines
EDW
Microservices
CI/CD
Airflow

Job Details

Requirement:

  • 5+ years of experience with Airflow implementation and optimization with Azure Data Engineering.
  • Proficiency in Azure cloud services, including Key Vault and Blob Storage.
  • Experience in Azure DevOps and GitHub for version control and configurations.
  • Experience with data migration, particularly from Databricks to Airflow.
  • Experience in setting up DevOps pipelines, such as Azure Pipelines and GitHub Actions.
  • Experience in monitoring tools like Sumo Logic for performance insights.
  • Excellent verbal and written communication skills.

Responsibilities:

  • Lead the implementation of Airflow.
  • Migrate orchestration jobs from Databricks to Airflow.
  • Create cost-optimized schedules and workflows for resource management.
  • Write and maintain comprehensive documentation throughout the process.
  • Manage authentication and secure access using Azure Key Vault.
  • Collaborate with data engineers to ensure data integration with Snowflake.
  • Monitor workflows and system health.

 

Founded in 1999, Resourcesoft, Inc. is a leading Technology Consulting and Professional Services organization. Headquartered in Marlborough, MA, the company serves the technology needs of its clients nationwide. Resourcesoft has often been recognized by prominent rating agencies for its exemplary growth and stability. With over two decades of industry experience, the Company has evolved as a front runner in enabling project optimization. We partner with leading organizations to provide technology solutions within the financial, insurance, education, government, publishing, healthcare and pharmaceutical domains. We take pride in mentoring a workforce that is well positioned to respond to the emerging IT trends and needs. With the employees at the crux of every business endeavor, our success is driven by our expertise in pairing the right talent with the best jobs in the technology sector. We forge long term, personalized relationships with our employees to advance their career to the next level. We engage them in technology centric client projects that provide opportunities for them to evolve, innovate and deliver world class products and services. Our career opportunities offer challenging assignments and exposure to emerging and cutting-edge technologies. We are committed to providing our employees with the tools necessary to accelerate their career progression, while maintaining a healthy, work-life balance. We are deeply committed to providing a workplace ambience that is both challenging and fulfilling.

Resourcesoft is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.

For more information about Resourcesoft and our services, please visit our website.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.