Sr Data Engineer - Major Streaming Service - NO H-1B or C2C

About the Role:

We are seeking a Senior Data Engineer to join our data engineering team and play a key role in maintaining, enhancing, and expanding our data pipelines. This role requires strong expertise in Python, Spark, and Scala, as well as experience working with modern data technologies such as Airflow, Databricks, Delta Lake, and Snowflake. You will collaborate with cross-functional teams to drive the success of Product Performance Data, ensuring data quality, operational efficiency, and scalability.

Key Responsibilities:

• Develop, maintain, and optimize large-scale data pipelines using Python and Spark, ensuring strict uptime SLAs.
• Design and implement shared libraries in Scala and Python to abstract complex business logic and maintain consistency across pipelines.
• Work with Airflow, Databricks, Delta Lake, and Snowflake as part of our core data stack.
• Partner with product managers, architects, and engineers to drive data-driven decision-making for key business stakeholders.
• Establish and document pipeline standards, including configurations, naming conventions, and partitioning strategies.
• Ensure data quality, reliability, and operational efficiency, meeting SLAs for engineering, data science, operations, and analytics teams.
• Actively participate in Agile/Scrum ceremonies to improve team processes and collaboration.
• Engage with customers and stakeholders to understand their needs and prioritize enhancements to our data platform.
• Maintain comprehensive documentation to support data governance and compliance.

Basic Qualifications:

• 5+ years of experience in data engineering, developing and maintaining large-scale data pipelines.
• Strong algorithmic problem-solving skills and Python programming expertise.
• Proficiency in SQL, with the ability to analyze complex datasets.
• Hands-on experience with distributed processing systems like Spark in a production environment.
• Experience with data pipeline orchestration tools, such as Airflow.
• Familiarity with AWS (or another cloud provider) and core resources like S3.
• Exposure to scripting languages for automation.
• Self-motivated, detail-oriented, and able to learn and adapt quickly in a fast-paced environment.

Preferred Qualifications:

• Experience working with Massively Parallel Processing (MPP) databases such as Snowflake, Redshift, or BigQuery.
• Hands-on experience in Scala or Java.
• Strong understanding of AWS or other cloud infrastructure, with experience in Infrastructure as Code.
• Knowledge of data modeling techniques and data warehousing best practices.
• Familiarity with test-driven development and maintaining a test-first mentality.
• Experience working in Agile/Scrum environments.

Education:

• Bachelor s or Master s degree in Computer Science, Information Systems, or a related field.