Databricks Technical Lead

Databricks Technical Lead l Contract | 12 Months | Outside IR35 l Up to £1,000 per day | Remote first, occasional on-site (Midlands & South East)

If you are considering sending an application, make sure to hit the apply button below after reading through the entire description.
The role
Method Resourcing have been asked by one of our consulting partners to find a hands-on Databricks Technical Lead to design and deliver a new Azure Databricks platform for a major life sciences organisation replacing its enterprise data and analytics stack.
The programme uses AI and LLM tooling to reverse-engineer existing pipelines, business logic and reports into documented, tested code. You'll mentor three senior data engineers, own the technical architecture and stay hands-on in development.
What you'll do
Architect and build the platform on Azure Databricks (Delta Lake, Unity Catalog, medallion architecture)
Use AI agents and LLMs to reverse-engineer legacy reports, queries and transformations, and build repeatable AI-assisted processes for code generation, testing and migration
Build automated testing and reconciliation to validate migrated data against existing reporting
Integrate SAP, Azure Data Lake, Salesforce and other systems, and design gold-layer models for Power BI
Set engineering standards (Workflows, CI/CD, IaC, governance) and mentor the team through to UAT and cutover
What you'll need
Extensive hands-on experience delivering and technically leading enterprise solutions on Azure Databricks
Demonstrable use of AI, LLMs or agents to reverse-engineer data pipelines or reporting logic
Strong Python, PySpark and SQL
Deep knowledge of Delta Lake, Unity Catalog, Workflows and medallion architecture
Automated validation, reconciliation and testing experience
Azure data engineering (ADLS Gen2, ADF)
Databricks CI/CD (Terraform, Asset Bundles or Azure DevOps)
SAP ECC or S/4HANA integration experience
Dimensional modelling and Power BI semantic model familiarity
Proven team leadership while staying hands-on
Desirable
Databricks certification (Data Engineer Professional or Solutions Architect)
Platform migration, carve-out or separation experience
Reusable AI agent frameworks for data engineering
Strong governance and data quality experience
Consulting background
Contract
outside IR35
£900-£1,000 per day
Remote with some travel required
12 month contract
If this is you, send your CV to and if suitable, i'll be in touch. As you can appreciate, this role will receive a high volume of candidates so I will do my best to respond if possible. xwzovoh
Databricks Technical Lead l Contract | 12 Months | Up to £1,000 per day | Remote first, occasional on-site (Midlands & South East)
RSG Plc is acting as an Employment Business in relation to this vacancy.