Beacon Intelligence delivers data and insights that help R&D scientists develop better pharmaceuticals, faster than ever before.
Our focus is simple: we curate high-accuracy data enriched and transformed with AI. This is delivered through a range of platforms, increasingly via AI-native products and features that enable customers to extract insights faster than ever before.
This is a rare opportunity to make a meaningful impact on patients’ lives worldwide.
The Role
As a Senior Data/AI Engineer, you will play a central role in building and scaling our data products.
This is not a traditional back-end data engineering role. You will work closely with product, commercial and technical colleagues, transforming datasets into commercial, customer-facing solutions.
You will combine deep technical expertise with a strong understanding of how data can be structured, enriched and operationalised through AI systems.
What You’ll Be Doing
Data Product Development (Core Focus)
- Design and build scalable data products underpinning Beacon’s commercial offerings
- Transform raw and third-party data into structured, enriched, product-ready datasets
- Partner with product and commercial teams to define how data is packaged, accessed and monetised
- Enable delivery via APIs, internal tools and customer-facing platforms
- Apply AI and LLM capabilities to enrich and enhance data (e.g. classification, tagging, summarisation, insight generation)
- Design and build LLM-powered product features
- Integrate RAG into pipelines to improve data quality and unlock new features
- Support development of AI-driven products such as recommendation engines, search and insight tools
- Ensure systems are optimised, well-governed and resilient
- Support deployment and lifecycle management of data and AI systems
- Own and evolve the data platform architecture (Databricks, Azure, Airbyte) to support scalability
- Own performance and cost optimisation across Azure Databricks and supporting Azure services, including compute selection, autoscaling, workload monitoring and resource-efficiency improvements.
- Build and maintain robust data pipelines (batch and streaming) delivering reliable, production-ready datasets
- Ensure high standards in data quality, testing and observability
- Improve efficiency through automation, CI/CD and engineering best practice
Commercial & Stakeholder Impact
- Partner with senior stakeholders to identify high-value data product opportunities
- Translate business needs into practical, scalable data and AI solutions
- Act as a bridge between technical teams and commercial/product stakeholders
- Contribute to prioritisation based on business impact
Leadership & Growth
- Shape the evolution of our data product and AI strategy
- Mentor junior engineers and promote a strong engineering and product mindset
- Help build a high-performing, commercially aware data and AI engineering team
What We’re Looking For
Core Engineering Experience
- Deep, hands‑on experience designing, building and operating production data platforms using Azure Databricks and Azure Cloud environments
- Strong experience with Apache Spark, PySpark, Spark SQL and Delta Lake, including incremental processing, schema evolution, performance optimisation and batch and streaming workloads.
- Strong experience implementing Unity Catalog, including catalog and schema design, access control, data lineage, discovery and governance of production data assets.
- Proven experience designing lakehouse and Medallion architectures and converting raw and third‑party data into governed, reusable data products.
- Advanced Python and SQL, focused on production‑quality, scalable solutions
- Proven experience designing and building data pipelines and models
- Experience working with APIs and data delivery mechanisms (critical for productisation)
- Experience deploying Databricks workloads through automated CI/CD, preferably using Azure DevOps or GitHub Actions, automated testing and Databricks Declarative Automation Bundles.
- Experience integrating Azure Databricks with ADLS Gen2, Azure identity, Key Vault, APIs and external data sources.
- Experience building or supporting data products or customer‑facing data solutions
- Strong understanding of how to structure, expose and scale data for end users or clients
- Ability to think beyond pipelines and focus on commercial value
- Experience working closely with product or commercial teams
- Familiarity with generative AI / LLMs (Azure OpenAI, LangChain etc.) and experience applying them to real‑world use cases
- Experience evaluating, monitoring and optimising AI systems
- Experience working with unstructured data (text, documents, web data)
- Familiarity with embeddings, vector databases or semantic search
- Understanding of AI governance and data ethics
- Experience with Airbyte or modern data ingestion tools
- Experience with Terraform or infrastructure‑as‑code
- Exposure to LLM frameworks (e.g. LangChain, LlamaIndex)
- Experience with experimentation or A/B testing frameworks
- Background in building data‑driven / AI‑enabled products
- Domain experience in life sciences
How You Work
- Focused on shipping products, not just experimentation
- Think in terms of products, not just pipelines
- Commercially aware and motivated by impact
- Combine technical excellence with pragmatism and pace
- Collaborate effectively across disciplines and influence stakeholders
- Curious about AI and emerging technologies, with a focus on reliable, scalable delivery
Why Join Beacon Intelligence
- Build products, not just infrastructure – directly helping scientists treat and cure diseases
- Shape our AI future – define how we embed AI into products and workflows
- High‑impact role – influence product strategy and commercial performance
- Modern, evolving stack – work with cutting‑edge tools across data engineering and AI
- Growth and ownership – join early and grow as we scale our capabilities
- Collaborative environment – stakeholders who actively value data and insight
Location & Working Pattern
- London (hybrid working) – 2 days per week minimum in office
- Flexible working arrangements available
60% hands‑on technical
20% management and mentoring
Pay range and compensation package
Salary range: £80,000 to £90,000 + Bonus
Equal Opportunity Statement
Beacon Intelligence is committed to diversity and inclusivity in the workplace.
#J-18808-Ljbffr
Full job description from WhatJobs. Apply for this role →