Senior Data Platform Engineer - Python & AWS

At Permutable, we're building real-time AI systems that turn large volumes of global news, market and proprietary data into market intelligence and systematic signals for financial institutions. We're looking for an experienced Data Platform Engineer with strong Python and AWS skills to build and scale the systems behind our data ingestion, processing, storage and delivery. You'll write production Python, develop reliable data pipelines and own the AWS infrastructure that runs them. As we move towards a more real-time, event-driven architecture, you'll help shape how data flows through our platform and reaches our models and clients. Working directly with our founder and engineering team, you'll have significant responsibility for architecture, implementation and production performance.

Build production data pipelines: Design and develop Python services and pipelines to ingest, validate, transform and deliver large volumes of news, market and proprietary data.

Develop our AWS data platform: Build and scale infrastructure across S3, RDS/PostgreSQL, Lambda, ECS and EKS, choosing the right services for each workload.

Build event-driven systems: Develop services using messaging, queues and asynchronous processing to support faster data updates and reliable distributed workflows.

Improve data quality and reliability: Build validation, deduplication, schema handling, retries and recovery into our pipelines. Make it straightforward to identify missing data, investigate failures and reprocess historical datasets.

Own storage and data access: Improve how we store, query and serve data to support research, production models and client-facing applications.

Support AI and quantitative workloads: Work with AI engineers and researchers to provide reliable data inputs and integrate NLP, LLM and quantitative model outputs into the platform.

Automate infrastructure and deployments: Manage infrastructure through Pulumi and build CI/CD workflows using GitHub Actions.

Monitor and optimise production: Develop monitoring and alerting for pipeline health, data freshness and service performance. Diagnose production issues and improve processing speed, resilience and AWS costs.

Strong production Python experience: You can design, build, test and maintain substantial Python services and data pipelines.

Strong AWS experience: Hands-on experience building and operating production systems on AWS, including services such as S3, RDS/PostgreSQL, Lambda, ECS or EKS.

Experience building data platforms: A substantial track record in data platform engineering, data engineering or backend engineering, with ownership of production data systems.

SQL and database skills: Strong SQL and practical experience with PostgreSQL, data modelling, query performance and processing large datasets.

Event-driven and distributed systems experience: A solid understanding of messaging, queues, asynchronous processing and reliable workflows, including retries and failure recovery.

Infrastructure-as-Code and CI/CD: Experience with Pulumi, Terraform or equivalent, alongside automated testing and deployment pipelines.

Production ownership: Comfortable working with Linux, containers, monitoring and logging, and diagnosing problems across application code, pipelines and infrastructure.

Engineering judgement: Able to make practical architecture decisions and balance performance, reliability, cost and delivery speed.

Startup mindset: Proactive, resourceful and comfortable taking responsibility in a small, fast-moving engineering team.

Relevant technical background: A degree in computer science, software engineering or a related discipline, or equivalent practical experience.

Useful additional experience: Apache Airflow, real-time data processing, Kubernetes, AI/ML infrastructure, LLM workflows, GPU infrastructure or financial-market data.

Real responsibility: Own important parts of our data platform and see your work used in production.

An ambitious, collaborative culture: Work closely with our founder, engineers and researchers, with regular opportunities to contribute ideas and influence technical decisions.

A strong startup culture and office vibe: Join a close-knit team that makes time for regular socials, company offsites, sports events and office treats.

Benefits

• Hybrid flexibility: Spend three or more days each week in our Vauxhall hub, with flexibility to work remotely for the remainder.
• Share in Our Growth: Equity options give you the opportunity to share in Permutable's long-term success as you help build the business.

#J-18808-Ljbffr

Full job description from WhatJobs. Apply for this role →