Lead Data Engineer

London, UKOn-siteLeaddata £650 – £880/yr Today

We are seeking an experienced Lead Data Engineer with deep expertise in Python, Databricks and Spark to lead the delivery of large-scale data engineering initiatives.

This is a unique opportunity for a technical leader who enjoys combining hands-on engineering with project delivery, team leadership and stakeholder engagement in a complex enterprise environment.

Client Details

Our client is a leading international financial institution with a long-established presence across major global markets. Serving a diverse client base that includes corporates, financial institutions and investors, the organisation delivers a broad range of banking, financing and capital markets services.

With significant investment in digital transformation and data-led innovation, the organisation is modernising its technology estate and expanding its enterprise data capabilities. Data plays a critical role in supporting business operations, regulatory obligations, analytics and strategic decision-making, making this an exciting opportunity to contribute to large-scale, business-critical data initiatives within a complex global environment.

Description

Lead the end-to-end delivery of data engineering projects, ensuring successful outcomes against business objectives, timelines and quality standards
Design, develop and optimise scalable data pipelines using Python, Databricks and PySpark
Build and enhance ETL/ELT workflows, reusable frameworks and modern data engineering solutions
Provide technical leadership and architectural guidance across data platform initiatives
Design and govern Lakehouse architectures leveraging Databricks and Delta Lake
Manage sprint planning, backlog prioritisation and delivery tracking within Agile teams
Identify and mitigate delivery risks, dependencies and technical challenges
Troubleshoot complex data, platform and performance-related issues
Drive engineering best practices across coding standards, testing, performance optimisation and code reviews
Act as a key interface between business stakeholders and engineering teams, translating requirements into scalable technical solutions
Collaborate with architects, product owners and senior stakeholders to shape delivery roadmaps and technical direction
Lead, mentor and develop a team of data engineers, fostering a culture of ownership and engineering excellence
Ensure data quality, governance, reliability and security across data platforms
Implement monitoring, logging and alerting capabilities to support operational excellence
Champion modern engineering practices, including CI/CD, DevOps and automation
Support production environments and drive continuous improvement across data solutions

Profile

You will bring:

10+ years' experience in data or software engineering
Proven experience as a Lead Data Engineer, Technical Lead, Engineering Lead, or similar hands-on leadership role
Strong hands-on expertise in Python
Extensive experience with Databricks, including workflows, notebooks and Delta Lake
Strong experience with Apache Spark / PySpark
Experience building and optimising enterprise-scale ETL pipelines
Strong understanding of modern data architecture and Lakehouse concepts
Advanced SQL and distributed data systems expertise
Experience with cloud platforms such as AWS, Azure or GCP
Experience working within Agile delivery environments
Excellent stakeholder management and communication skills
Ability to balance technical leadership with hands-on delivery responsibilities
Experience delivering data engineering solutions within Banking, Financial Services, Capital Markets, Insurance, or other highly regulated enterprise environments would be highly advantageous

Desirable experience includes: