Lead Data Engineer - Databricks

Lead Data Engineer (Databricks): Shaping and delivering an organization's modern Enterprise Data Platform, ensuring consistent engineering standards and guiding the design of scalable data solutions with an accent on technical leadership and hands-on engineering. Focus on high-performance data platforms at scale, mentoring engineers, and influencing the direction of the data platform.

Location: Remote UK or hybrid / onsite at client offices across England

Company

ARCA Resourcing is partnered with an innovative and leading retailer that is investing heavily in its data capabilities to drive smarter decision-making across the business.

What you will do

• Provide technical leadership across multiple delivery squads, ensuring consistent data engineering standards.
• Coordinate technical solutions and align team efforts across initiatives.
• Design and implement high-performance ELT pipeline patterns using Databricks.
• Translate functional designs into scalable and reusable data engineering solutions.
• Champion engineering best practices, including code quality, testing, and documentation.
• Mentor and develop engineers across the team, creating learning and development plans.

Requirements

• Strong experience with Databricks, Apache Spark, and cloud-based data architectures (preferably Azure).
• Excellent programming capability in Python and SQL.
• Experience orchestrating pipelines with Azure Data Factory and Databricks Workflows.
• Solid understanding of Lakehouse architectures and modern data warehousing principles.
• Experience with data modelling, ELT/ETL pipelines, and testing frameworks.
• Proven experience leading engineering teams or coordinating multiple squads.

Nice to have

• Degree in IT, Computer Science, or related discipline.

Culture & Benefits

• Opportunity to work with cutting-edge technologies.
• Mentor engineers and influence the direction of a high-impact data platform.