Data Engineer - Databricks

Job Role

We are looking for a Data Engineer to design, build and operate scalable data pipelines and data platform capabilities within a modern lakehouse architecture. You will work in a collaborative agile engineering team delivering production‑grade data solutions using Databricks, Spark, Python and SQL in a cloud environment. The role focuses on developing robust data pipelines, ensuring high data quality and enabling analytics and data product delivery for enterprise clients.

Key Responsibilities

• Design and implement ELT/ETL pipelines using PySpark and Databricks
• Build scalable batch and streaming data pipelines using Spark and Kafka, develop optimised SQL and Python pipelines for data transformation and integration, and integrate external data sources using REST APIs and data ingestion frameworks
• Optimise Spark jobs and cluster performance for reliability and cost efficiency, implement data quality checks, validation and monitoring
• Apply CI/CD practices and version control for pipeline deployment
• Work closely with solution architects, DevOps engineers and business analysts to deliver data products, produce technical documentation, architecture diagrams and operational runbooks

Key Requirements

• 3+ years experience building production data pipelines
• Hands‑on experience with Databricks, Spark and PySpark; experience with Delta Lake, Unity Catalog and Databricks
• Working experience with data pipeline design, testing and deployment; familiarity with test‑driven development, CI/CD practices and Git‑based development
• Strong collaboration and communication skills
• Working knowledge of Hadoop REST APIs and Kafka experience

Nice to Have

• Cloud experience with AWS, Azure or GCP; certifications in Databricks, AWS or data engineering
• Infrastructure as Code (Terraform); orchestration with Airflow or AWS Glue; streaming technologies including Structured Streaming; observability tooling for data pipelines
• Experience working with regulated industries such as banking or financial services

Databricks experience is preferred; training will be offered to strong candidates.