Software / Data Engineer - Compliance Assurance & Streaming Data

Location: Chester (Hybrid)

A leading global financial institution is seeking a Senior Data Platform Engineer to help build and enhance a strategic enterprise platform responsible for ingesting, processing and assuring large-scale communication, collaboration and transactional data. 

We're looking for an engineer who can own the entire data lifecycle, from source-system integration and streaming ingestion through to transformation, storage and the delivery of production-grade compliance and assurance controls. You'll work at the intersection of software engineering, data engineering and regulatory technology, building scalable solutions used across a highly regulated environment.

The role offers exposure to modern data-platform technologies, complex distributed systems, enterprise-scale datasets and emerging AI-enabled engineering practices.

What You'll Be Doing End-to-End Platform Engineering

• Design and develop resilient enterprise-scale data pipelines
• Build integrations across vendor APIs, streaming technologies and distributed data platforms
• Engineer high-volume batch and near real-time processing solutions
• Develop reusable ingestion and processing frameworks rather than one-off solutions
• Implement robust data handling patterns including replay, backfill, recoverability and schema evolution
• Define and enhance operational telemetry, monitoring and platform observability

Streaming & Distributed Data Processing

• Build and optimise solutions using:
  • Apache Kafka
  • Apache NiFi
  • Apache Spark / PySpark
  • Hadoop ecosystem technologies
  • Hive and large-scale data storage platforms
• Develop scalable matching, enrichment and reconciliation services
• Create canonical data models spanning multiple source systems
• Ensure data lineage, quality and governance requirements are met

Control & Assurance Engineering

• Translate regulatory and operational requirements into deterministic control logic
• Develop reconciliation, completeness, configuration and assurance controls
• Produce auditable outputs, exception reporting and evidence frameworks
• Build monitoring and alerting capabilities for compliance-critical processes
• Work closely with engineering, architecture, risk and operational stakeholders

Software Engineering & Delivery

• Write clean, maintainable and testable Python and PySpark code
• Contribute to CI/CD pipelines and automated deployment processes
• Participate in code reviews and engineering standards initiatives
• Deliver solutions through Development, UAT and Production environments
• Support troubleshooting and root-cause analysis across distributed systems

Required Experience We're interested in engineers who can demonstrate strong expertise across several of the following areas:

Core Technical Skills

• Python development in enterprise environments
• Apache Spark / PySpark
• Apache Kafka
• Apache NiFi
• Hadoop and Hive
• Advanced SQL
• REST API integration and development
• Data modelling and data lineage
• Distributed systems engineering
• CI/CD and modern software delivery practices
• Git and collaborative development workflows

Additional Desirable Experience

• Microsoft Azure
• Microsoft Graph
• Entra ID
• Service principals and secure identity patterns
• Grafana, Splunk, Tableau or Power BI
• Java or another strongly typed language
• AI-assisted development tools including GitHub Copilot
• Exposure to agentic AI, orchestration frameworks or automation tooling

Ideal Background Candidates are likely to come from:

• Financial services
• Banking and capital markets
• RegTech
• Large-scale enterprise data engineering environments
• Telecommunications or communications platforms
• Highly regulated industries

Experience working with voice, communications or collaboration technologies would be advantageous, as would any exposure to assurance, reconciliation or control-focused platforms.

What Success Looks Like

• New data sources are onboarded through reusable engineering patterns
• Pipelines are resilient, observable and fully recoverable
• Control logic is transparent, auditable and trusted
• Data quality and lineage are clearly understood
• Solutions are delivered through a disciplined SDLC into production environments
• The platform scales to support new technologies, controls and business requirements