AI Reliability Engineer, HPC

London Source: WhatJobs

See advert for pay

Overview

In this role you ensure the reliability, uptime, and efficiency of large-scale AI HPC infrastructure. You work closely with ML engineers and platform teams to keep model training and inference resilient, scalable, and observable. You will build automation, run incident response, and drive improvements across GPU-accelerated systems. This is a mission-driven opportunity to shape reliable AI infrastructure at scale and enable researchers to move from lab to production.

Responsibilities

Ensure uptime, resiliency, and fault tolerance of HPC clusters powering MAI model training and inference
Design and maintain monitoring, alerting, and logging systems for real-time visibility of GPU, clusters, storage and networking
Build automation for deployments, incident response, scaling, and failover in CPU+GPU environments
Lead on-call rotations, troubleshoot production issues, conduct blameless postmortems, and drive continuous improvements
Ensure data privacy, compliance, and secure operations across model training and serving environments
Collaborate with ML engineers and platform teams to improve developer experience and accelerate research-to-production workflows

Key requirements

Bachelor’s Degree in Computer Science, or related technical discipline
4+ years technical experience in Site Reliability Engineering, DevOps, or Infrastructure Engineering
Experience with Kubernetes, Docker, and CI/CD pipelines for ML workloads
Experience with public cloud platforms (Azure, AWS, GCP) and infrastructure-as-code
Experience with monitoring and observability tools (Grafana, Datadog, OpenTelemetry)
Programming or scripting in Python, Go, or Bash
Experience with distributed systems, networking, storage, and HPC
Experience operating GPU clusters and workload schedulers for ML/AI workloads
Experience with ML training or inference pipelines
Experience with capacity planning and cost optimization for GPU-based infrastructure