ETL Developer (Pentaho / Microsoft ADF)

ETL Developer (Pentaho / Microsoft ADF)

Location: White Plains, NY 10601

Start Date: Immediate

Duration: 6+ month(s)

Project Overview

The details of the different ETL development that will be enabled over the duration of the project are outlined below.

1. a) Analyze the ETL mapping sheet of customer data and create hierarchy for Organizations, Facilities, Accounts and meters
2. b) Analyze the existing hierarchy and propose changes to improve the overall data quality.
3. c) Create detailed design documents for development team use on new or existing customer integrations
4. d) Update the DevOps tasks and assist business in prioritizing the tasks in sprints
5. e) Develop test cases and test the data in QA environment
6. f) Develop ETL flows to ingest data into NYEM platform by extracting, transforming, loading from Energy Cap, BSNY, SAP and other etc.,

Job Functions & Responsibilities

1. ETL Development: - Design, build, and maintain efficient ETL workflows to process and integrate solar, Interval, etc., data from multiple sources. - Optimize ETL processes for performance and scalability.
2. Data Integration: - Collaborate with data providers and stakeholders to ensure accurate data ingestion and transformation. - Develop mechanisms for real-time and batch solar data processing.
3. 3. Data Quality & Validation: - Implement robust data validation and cleansing techniques. - Ensure data accuracy and consistency.
4. Collaboration: - Work closely with the business, and analysts. - Coordinate with IT teams to deploy ETL pipelines in production environments.
5. Documentation: - Document ETL processes, workflows, and best practices. - Maintain clear and concise technical documentation for stakeholders.

Technical Skills:

• 5+ years of experience as a Data Engineer or ETL Developer.
• Proven experience in integrating energy, and interval data, is preferred.
• Proficiency in ETL tools such as Pentaho/Microsoft ADF.
• Strong SQL and database management skills (e.g., SQL Server).
• Experience with cloud platforms (e.g., AWS, Google Cloud Platform, Azure).
• Knowledge of Python, Spark, or other programming languages for data processing.

Preferred Skills:

- Familiarity with energy management systems or renewable energy analytics.

- Experience with REST APIs and integrating IoT data, timeseries and Utility Data.

Education:

- Bachelor s degree in Computer Science, Information Systems, Engineering, or a related field.