Senior Data Engineer

工作内容

Do you value integrity and innovation? How about passion and caring? Great! Us too, and that’s why you’ll fit right in. Our intentional culture promotes trust and participation, encouraging you to bring your heart and mind to work every day.

In-Scope

Multiple Positions

Remote Work Opportunities may be considered

GENERAL ACCOUNTABILITY

The Senior Data Engineer is responsible for designing and implementing data pipelines, which includes transforming, aggregating, and integrating data from different sources and loading it into the Azure Data Platform as well as propagating the data through the environment. This role develops, manages, and monitors Azure data pipelines to ensure that data ingestion and data propagation perform as expected. The Senior Data Engineer works with other members of the Data Office team, as well as IT Developers, on data initiatives, and ensures a consistent approach to data ingestion and data propagation is achieved across disparate projects. This role is differentiated from the Data Engineer in handling the more complex work in the department. The Senior Data Engineer provides oversight and guidance to the Data Engineer classification.

Key Accountabilities

Note: This section is not intended to be an exhaustive list of duties and responsibilities – other duties and responsibilities may be assigned.

Data Engineering Design, Development, and Oversight

• Provides oversight and guidance to Data Engineers, including delegating, monitoring, and reviewing quality of work; providing direction, coaching, and motivating team members; and leading the development of work plans/approach for work led by the Senior Data Engineer.
• Provides expertise and guidance to deliver innovation, design, and analysis-related work within the scope of projects and outside of defined projects where research necessitates.
• Collaborates with the business to establish a clear scope of work and prioritization of activities.
• Collaborates with project team, change management, and business stakeholders to share change impacts and determine implications and business decisions for the project.
• Provides quality assurance to ensure that all programs operate error-free in accordance with the standards.
• In collaboration with members of the Data Office team, designs, develops, tests, and maintains data pipelines in the Azure Data Platform, implementing solutions for source data ingestion, reference data, data transformation, and data propagation across the Azure Data Platform.
• Designs, develops, tests, and maintains extract, transform, and load (ETL); orchestration (ADF); and/or custom-developed stored procedures.
• Designs, develops, tests, and maintains data validation, data transformations and calculations, and scheduling of the Data Platform load jobs.
• Designs system components that reconcile and audit the results of the extract/transform or conversion of data from source systems to the interim and target locations in the Azure Data Platform.
• Documents and maintains data validation routines, load processes, ETL, ADF orchestration, custom-developed stored procedures, and error handling procedures.
• Develops and maintains knowledge of data available from upstream sources and data within various operational systems.
• Facilitates the exposure and capture of metadata for ingestion into the enterprise metadata repository.
• Works with Data Scientists and Analytics Relationship Managers to create data assets and pipelines that improve the reporting and analytics process.
• Identifies ways to improve data reliability, efficiency, and quality.
• Identifies, designs, and implements internal process improvements (e.g., automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability).
• Structures and provisions data on an ad hoc basis to support modeling and data discovery, including filtering, tagging, joining, parsing, and normalizing/de-normalizing data.
  
  

Data Engineering Production Support

• Provides on-going production support to ensure the daily loads and propagation of data across the Azure Data Platform are meeting service-level agreements (SLAs).
• Assists in monitoring the overall performance and stability of the Azure Data Platform and data pipelines in production, along with other members of the Data Office Data Platform team, to ensure that the environment operates as planned, meeting respective SLAs.
• Helps all Data Office, IT development, and Testing teams resolve data issues as they arise.
• Proactively works with members of the Data Office team to identify and resolve any/all data issues.
• Engages with members of the Data Office team and the business to enable data governance, data quality, and metadata, aligning with SGI’s policies, standards, and frameworks.
  
  

High Performance Team & Culture

• Supports a culture of leadership and accountability to effectively meet the key accountabilities within the scope of the role.
• Displays leadership by committing to a culture of continuous learning/development of self and supports others by actively sharing knowledge, providing guidance, mentoring, training, and supporting developmental opportunities.
• Demonstrates that the Health, Safety and Emergency Management Policy is applied in area of responsibility for self and others.
• Actively applies knowledge to support transformation and strategic initiatives of the corporation, while participating and advocating change and applying a growth mindset.
  
  

Technical Knowledge & Skills

• Knowledge of cloud data platform technologies, including container technology, Spark processing, SQL pools, etc.
• Knowledge of data lake storage and data warehousing solution components (e.g., Azure Data Lake, Cosmos DB, Synapse, Databricks).
• Knowledge to seamlessly integrate data across relational and dimensional schemas.
• Skill in using enterprise ETL/ELT data integration tools.
• Skill in developing ETL/ELT strategies.
• Skill in using Azure Data Factory to build and optimize data pipelines.
• Skill in building processes for data transformation, data ingestion and loading, metadata ingestion, and data propagation.
• Skill in troubleshooting ADF and custom scripts to address production issues (e.g., performance tuning, scheduling and enhancement).
• Knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores.
• Skill in using common scripting languages (e.g., Python, Java, C++, Scala).
  
  

EDUCATION & CERTIFICATIONS

• Four-year degree from an accredited post-secondary education institution in a relevant field of study, such as Computer Science, Engineering, or Information Technology, or defined equivalency.
  
  

As per the above requirements, please ensure your education and/or course credentials (excluding driver’s licence) are updated in your Workday profile before you submit your application. If you make any changes to your education, please inform the Recruiter before the job posting closes. Failure to do so may result in your application not being considered.

Experience

• 5 – 7 years’ database experience in a data engineer, ETL developer, or equivalent technology role.
  
  

Pay Range:$82,368.00 - $103,956.00

Posting Close Date:

June 8, 2022

As you prepare to submit your application, and cover letter if applicable, please highlight the achievements that demonstrate why you’re a great candidate for this role.