Tech Lead Data Engineering - Real-Time & Streaming - Finance H/F ClubSec

Contexte de la mission

As part of the development of real-time data products within an international and highly industrialized environment, our client is looking for an experienced Tech Lead Data Engineering to design, build and operate reliable, scalable and secure data solutions.

The mission covers the full data lifecycle, from data ingestion and stream processing to the delivery of APIs and services consumed by internal and external stakeholders. The environment requires strong standards in terms of reliability, security, observability, performance and maintainability.

The ideal candidate will be a highly technical and hands-on leader, able to translate complex business requirements into robust technical designs while supporting engineering teams through architecture discussions, code reviews and knowledge sharing.

Objectifs et livrables

The consultant will be responsible for:

• Designing and implementing real-time data products based on Apache Kafka, Apache Flink and event-driven architectures.

• Building resilient ingestion, processing, transformation and data delivery pipelines for both streaming and batch workloads.

• Developing microservices and well-documented REST APIs using OpenAPI standards.

• Integrating and deploying data services in an AWS environment.

• Defining data models, data quality rules, validation processes and schema governance practices.

• Implementing automated testing and CI/CD pipelines covering build, test, deployment and rollback activities.

• Setting up comprehensive observability across logs, metrics, traces, dashboards and alerts.

• Supporting production operations, including monitoring, incident management, performance optimisation, root-cause analysis and continuous improvement.

• Applying security best practices covering authentication, authorisation, encryption, secrets management and least-privilege access.

• Producing architecture documents, API specifications, operational runbooks and deployment documentation.

• Leading technical discussions and contributing to architecture and code reviews.

• Defining engineering standards and providing mentoring and knowledge transfer to less experienced engineers.

• Delivering production-ready solutions that meet functional, performance, reliability, security and maintainability requirements.

Required skills

• Solid senior-level experience in Data Engineering.

• Strong expertise in Apache Flink and Apache Kafka.

• Proven experience delivering and operating real-time data products in production.

• Good knowledge of event-driven architectures and distributed systems.

• Experience developing REST APIs, OpenAPI specifications and microservices.

• Strong knowledge of AWS services, particularly MSK, Kinesis, Lambda, API Gateway, ECS/EKS, S3 and IAM.

• Experience with data quality, schema management and automated testing.

• Knowledge of CI/CD tools such as GitLab CI, GitHub Actions or equivalent.

• Experience with observability tools and practices, including CloudWatch, Prometheus, Grafana and OpenTelemetry.

• Experience in production support, incident management and performance optimisation.

• Good understanding of cloud and application security principles.

• Ability to produce clear and comprehensive technical documentation.

• Ability to contribute to architecture design, code reviews and technical standards.

• Strong communication skills and the ability to mentor less experienced engineers.

• Experience in financial markets or another regulated industry would be an advantage.

• Fluent in english