DevOps Consultant - Databricks

Job Role

We are seeking a Databricks DevOps Engineer to design and operate secure, scalable and reliable Databricks-based data platforms on AWS. This role focuses on infrastructure automation, CI/CD implementation, platform governance and observability. You will help build the engineering foundations that enable teams to deliver reliable data products.

Key Responsibilities

• Design and implement ELT/ETL pipelines using PySpark and Databricks
• Build scalable batch and streaming data pipelines using Spark and Kafka
• Develop optimised SQL and Python pipelines for data transformation and integration
• Integrate external data sources using REST APIs and data ingestion frameworks
• Optimise Spark jobs and cluster performance for reliability and cost efficiency
• Implement data quality checks, validation and monitoring
• Apply CI/CD practices and version control for pipeline deployment
• Work closely with solution architects, DevOps engineers and business analysts to deliver data products
• Produce technical documentation, architecture diagrams and operational runbooks

Key Requirements

• Eligible and willing to pass relevant background checks
• 3+ years experience building production data pipelines
• Hands‑on experience with Databricks, Spark and PySpark
• Experience with Delta Lake, Unity Catalog and Databricks Workflows
• Experience with data pipeline design, testing and deployment
• Familiarity with test‑driven development, CI/CD practices and Git-based development
• Strong collaboration and communication skills
• Working knowledge of Hadoop
• REST APIs and Kafka experience

Nice to Have

• Cloud experience with AWS, Azure or GCP
• Certifications in Databricks, AWS or data engineering
• Infrastructure as Code (Terraform)
• Orchestration: Airflow or AWS Glue
• Streaming technologies including Structured Streaming
• Observability tooling for data pipelines
• Experience working with regulated industries such as banking or financial services