Big Data Engineer

  • Boston, MA
  • Posted 10 days ago | Updated 10 days ago

Overview

Hybrid
Depends on Experience
Accepts corp to corp applications
Contract - Independent
Contract - W2
Contract - 12 Month(s)

Skills

Big Data
Spark
Scala
Hive
Kafka

Job Details

Role: Big Data Engineer or Tester

Location: Boston, MA (3 Days/Week Hybrid)
Duration: Long Term Contract

Objectives of this role

  • Develop and implement pipelines that extract, transform, and load data into an information product that helps the organization reach its strategic goals
  • Focus on ingesting, storing, processing, and analyzing large datasets
  • Create scalable, high-performance web services for tracking data
  • Translate complex technical and functional requirements into detailed designs
  • Investigate alternatives for data storing and processing to ensure implementation of the most streamlined solutions
  • Serve as a mentor for junior staff members by conducting technical training sessions and reviewing project outputs

Responsibilities

  • Develop and maintain data pipelines using ETL processes
  • Take responsibility for Apache Hadoop development and implementation
  • Work closely with data science team to implement data analytics pipelines
  • Help define data governance policies and support data-versioning processes
  • Maintain security and data privacy, working closely with data protection officer
  • Analyze vast number of data stores to uncover insights

Required skills and qualifications

  • Experience with Python, Spark, and Hive
  • Understanding of data-warehousing and data-modeling techniques
  • Knowledge of industry-wide visualization and analytics tools (ex: Tableau, R)
  • Strong data engineering skills with Azure cloud platform
  • Experience with streaming frameworks such as Kafka
  • Knowledge of Core Java, Linux, SQL, and any scripting language
  • Good interpersonal skills and positive attitude

Preferred skills and qualifications

  • Degree in computer science, mathematics, or engineering
  • Expertise in ETL methodology for corporate-wide solution design using DataStage

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.