Senior Data Engineer (Databricks)
Salary £65,000
Type Full time, permanent
Hours 37.5 per week
Location Remote or Hybrid UK, with occasional Richmond office attendance (a couple of times per month)
Overview
We are looking for a Senior Data Engineer to join the Retail Insight Data and Engineering team, contributing to the design, development and day to day operation of our Databricks environment. You will build reliable, scalable pipelines and help shape engineering standards, working closely with the Lead Data Engineer and partnering with data science, analytics and product teams.
Key responsibilities
* Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake
* Develop and optimise batch and streaming workflows for reliability and performance
* Contribute to lakehouse architecture and medallion pattern implementation
* Implement data quality checks, monitoring and observability across pipelines
* Apply platform security, access control and governance standards
* Support code reviews and high engineering quality standards
* Identify cloud cost optimisation opportunities
* Translate business requirements into technical solutions
* Contribute to documentation and continuous improvement
Required experience
* Strong experience delivering production grade Databricks pipelines
* Strong Spark, Delta Lake and Databricks Workflows experience
* Python and SQL proficiency
* Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps, or equivalent on AWS or GCP
* Understanding of data modelling, ETL and ELT patterns, distributed processing
* Experience implementing data quality checks and pipeline monitoring
* Strong communication and stakeholder management
Desirable
* Databricks certification
* Unity Catalog and Databricks Asset Bundles
* Terraform or Bicep
* dbt or similar transformation frameworks
* Retail or CPG data exposure such as POS, inventory, supply chain
* Streaming technologies such as Kafka or Event Hubs
Education
* Degree in computer science, data engineering or related field, or equivalent practical experience
