Data Engineering Lead

Data Engineering Lead

Department: CSG - Operations Hub (Data / Intelligence / Support)

Employment Type: Permanent

Location: London - Woolgate

Description

We are looking for an experienced Data Engineering Lead to play a pivotal role in shaping and delivering our enterprise data platform. This is an exciting opportunity to lead the development of scalable, secure, and AI-ready modern data platform that enable advanced analytics, reporting, automated and AI-enabled processes, and the creation of data powered digital products.

Working closely with Data Architecture, Analytics, and wider business teams, you will be responsible for designing and maintaining robust data pipelines, integrating multiple business systems, and ensuring data remains trusted, accessible, consistent, and fit for purpose.

This role is ideal for a technically strong data engineering professional who enjoys combining hands-on delivery with leadership and continuous improvement.

What you will do?

As our Data Engineering Lead, you will:

• Design, develop and maintain enterprise ETL/ELT pipelines, integrating data from case management, finance, CRM and third-party systems into a central data platform.
• Lead the development and continuous improvement of our cloud-based data platform within Azure.
• Ensure data pipelines are scalable, reliable, cost-effective and resilient.
• Build and maintain datasets that support business reporting, analytics, dashboards and data-enabled products.
• Establish and champion engineering best practices, including CI/CD, automated testing, version control, monitoring and alerting on pipeline failures.
• Work closely with Data Architects to implement target-state data models and modern data engineering principles.
• Partner with data scientists and analysts to operationalise advanced analytics and predictive models.
• Manage integrations with insurer portals and external data providers, ensuring high-quality, timely and secure data flows.
• Support the organisation's ambition to become increasingly data-driven and AI-enabled.

Who you are

Essential Experience & Skills

• Proven experience in a senior Data Engineering or Lead Data Engineer role.
• Strong hands-on experience with Microsoft Azure data technologies, including:
• Azure Data Factory
• Azure Synapse Analytics
• Microsoft Fabric
• Azure Data Lake
• Advanced SQL development skills.
• Proficiency in Python and modern data pipeline orchestration tools, including their monitoring and automation.
• Track record of building reliable, monitored production-grade data pipelines.
• Experience integrating data from multiple business-critical applications and legacy systems.
• Good understanding of data security, data protection and access control principles within cloud-based data platforms.
• Comfortable working in regulated data environments with sensitive personal and litigation data.
• Excellent stakeholder management and communication skills.

Desirable Experience

• Exposure to predictive analytics, machine learning operationalisation or AI platforms.
• Experience working with Dynamics CRM, case management platforms such as VisualFiles, and finance systems such as Elite 3E.
• Understanding of data modelling and data architecture best practices.
• Experience leading or mentoring data engineering teams.
• Experience within insurance, legal services, claims management or financial services is desirable but not mandatory.

Why Join Us?

This is an exciting opportunity to shape the future of data engineering within a growing, data-driven organisation.

You’ll have the opportunity to:

• Be part of an exciting journey reshaping how data and AI are used in claims management.
• Play a key role in our major transformation initiatives and the creation of our modern cloud data platform.
• Work with cutting-edge Azure and AI technologies.
• Influence strategic data decisions across the business.
• Deliver solutions that directly improve operational performance and customer outcomes.
• Join a collaborative and forward-thinking Data & Analytics function.

Success Measures

Success in this role will be measured by:

• Reliability and performance of data pipelines and platform services.
• Delivery of high-quality, trusted data products.
• Reduction in data-related incidents and support issues.
• Improved data accessibility and adoption across the business.
• Successful implementation of engineering best practices and automation.
• Enablement of analytics, reporting and AI initiatives through scalable data pipelines and architecture.

What’s in it for you?

• High levels of flexibility and a great work life balance -
• A well-rounded remuneration package (which includes private medical insurance, income protection insurance and discounted gym membership, amongst many other benefits)
• Opportunities for growth and progression including professional funding
• In person and remote social events
• Opportunity to get involved in a range of Environmental, Social and Governance (ESG) activities

We are dedicated to building a diverse, inclusive and authentic workplace, which aligns closely to our cultural principles (Determined, Clear, Creative and Supportive). If you are excited about this role and being part of our culture, but your past experience does not align perfectly with every qualification in the job description, we encourage you to apply anyway. You may be just the right candidate for this or other roles!

#J-18808-Ljbffr

Full job description from WhatJobs. Apply for this role →