Lead Data Engineer

Title: Lead Data Engineer

Location: Warwick, UK

Duration: 06 Months Possible extension

Work Type: Hybrid

Pay: £350 to £460 per day Umbrella (inside IR35)

Background

• Client is is maturing and embedding its existing Data & Insights platform, which is on the Azure stack. We have a combination of strategic Databricks and legacy Synapse engineering.
• We have an existing programme team which requires a skilled individual to lead the engineering resource. This person will need to embed our Databricks architecture, ensure correct configuration and documentation, whilst also overseeing the ingestion of new data sources.
• It is a live platform, and the role may also need to provide technical guidance to support engineers to ensure any production issues are understood and resolved efficiently.
• This is a UK based role. The team predominantly work across the UK (with some off-shore team members). Occasional onsite meeting attendance.

Responsibilities

• Design and delivery of modern Azure and Databricks Data platform aligning to agreed technical designs and best practice.
• Responsible for data engineering across all layers of our data platform. There is currently a mix of legacy synapse and strategic Databricks.
• Responsible for leading work on related DevOps - branch supervision, merging
• In depth Databricks DBA
• Responsible for promotion of data pipelines and associated notebooks through the various environments (development, fof, production)
• Responsible for the correct configuration and documentation of Databricks
• Enhancing existing monitoring & control framework
• Provide technical leadership & coaching for a small group of Data Engineers

Required skills:

• Extensive “ hands on” Databricks & Synapse experience
• Extensive Azure Dev Ops experience
• Ideally SQL and PySpark in a Databricks environment
• Experience of using Databricks in a federated cloud environment
• Leadership of small engineering teams
• Configuration management and documentation
• Certified Data Bricks Professional Engineer