Lead Data Engineer (Databricks)
Salary £65,000
Type Full time, permanent
Hours 37.5 per week
Location Remote or Hybrid UK, with occasional Richmond office attendance (a couple of times per month)
Overview
We are looking for a Lead Data Engineer to join the Retail Insight Data and Engineering team and own the architecture and development of our Databricks environment. This is a senior, hands on role where you will design and build scalable data infrastructure that underpins our products, while guiding a small team of engineers and defining platform standards, data architecture and engineering best practices supporting analytics, machine learning and AI across the business.
Key responsibilities
* Lead the design, development and optimisation of a Databricks based data platform, ensuring reliability, performance and scalability
* Design enterprise scale data architectures on Databricks and cloud platforms such as Azure, AWS or GCP
* Define lakehouse standards, data modelling approaches and governance frameworks
* Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies
* Establish security, access control, data quality and compliance standards
* Optimise performance, scalability and cloud cost management
* Lead and mentor a small team of data engineers, setting technical standards and reviewing code
* Partner with data science, analytics and product teams to ensure the platform supports their work
* Own data quality, governance and observability across pipelines
* Support AI and ML enablement including feature stores and model deployment patterns
* Translate business requirements into technical solutions
* Contribute to platform roadmap, tooling selection and continuous improvement
Required experience
* Strong experience as a senior or lead data engineer building and scaling production data platforms
* Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows
* Python and SQL proficiency
* Infrastructure as code experience using Terraform, Bicep or CloudFormation
* Cloud data platform experience, ideally Azure including Azure Data Factory, ADLS Gen2 and Azure DevOps
* Strong understanding of data modelling, ETL and ELT patterns, distributed processing
* Experience with security, governance and data platform operations
* Experience leading or mentoring engineers, with strong stakeholder management
* Rigorous approach to data quality and testing
* Databricks certification
Desirable
* Unity Catalog, MLflow and Databricks Asset Bundles
* dbt or similar transformation frameworks
* Retail or CPG data such as POS, inventory, supply chain
* Streaming technologies such as Kafka or Event Hubs
Education
* Degree in computer science, data engineering or related field, or equivalent practical experience
