Tech Lead / Data Ingestion Developer

Role: Tech Lead / Data Ingestion Developer

Phase: Live / Operational & Modernisation

Role Overview

We are seeking an experienced Tech Lead / Data Ingestion Developer to lead engineering efforts across a critical enterprise Data Ingest Service. The service operates as a trusted, secure, and resilient gateway responsible for ingesting, validating, scanning, and routing approximately half a million files monthly from both public and private sector data providers.

Outputs are delivered across a hybrid ecosystem including on-premises Windows servers, AWS S3, and Google Cloud Storage (GCS). This role balances hands-on technical leadership, operational resilience, and the strategic migration of legacy architectures to a modern cloud-native stack.

Key Responsibilities

• Technical Leadership & Architecture: Lead the technical design, development, and delivery of secure ingest and egress data pipelines while defining standards for reliability, automation, and platform simplification.

• Platform Modernisation: Drive the strategic transition from legacy ingestion pipelines (Apache NiFi 1.x running on AWS EC2 clusters) toward a modernized stack leveraging Apache NiFi 2, AWS Step Functions, and AWS Managed Workspaces for Apache Airflow (MWAA).

• Operational Excellence & Resilience: Establish a balanced operating model that separates live operational support from strategic engineering. Oversee incident resolution, root-cause analysis (RCA), monitoring/observability improvements, and automated platform recovery.

• Security & Compliance: Ensure all incoming files undergo rigorous antivirus/malware scanning (e.g., OPSWAT Metadefender integration) and security checks. Oversee vulnerability remediation, security patching, and SSL/TLS certificate renewals across the platform estate.

• Stakeholder & Onboarding Support: Oversee technical work to onboard new survey and administrative datasets, reduce dataset onboarding times, and build self-service capabilities for internal and external data suppliers.

• Team Leadership & Knowledge Sharing: Mentor engineers, eliminate single-point-of-failure knowledge bottlenecks through structured rotation, and foster cross-functional collaboration across network, security, and cloud platform teams.

Technical Capabilities & Qualifications

• Data Engineering & Orchestration: Deep expertise in Apache NiFi (v1.x and v2.x), AWS Step Functions, and Apache Airflow / AWS MWAA.

• Cloud Architecture (AWS & Multi-Cloud): Hands-on experience with AWS core infrastructure (EC2, S3, IAM, CloudWatch) and multi-cloud data delivery targets (Google Cloud Storage, on-premises systems).

• Security Integration: Demonstrated knowledge of integrating enterprise security controls, automated virus scanning solutions (e.g., OPSWAT or similar), network security protocols, and secure file transfer (SFTP, API, Managed File Transfer).

• Operational Engineering: Strong background in CI/CD pipelines, automated platform patching, infrastructure monitoring, and maintaining operational dashboards (e.g., ServiceNow integration, alert tuning).

• Web Applications & APIs: Experience supporting or integrating lightweight web applications that facilitate user-driven data ingestion.

Core Deliverables & Strategic Objectives

• Reduce Technical Debt: Proactively remediate vulnerabilities, automate platform maintenance, and eliminate recurring failure modes.

• Accelerate Onboarding: Simplify data intake flows and automate processes to shorten the lead time for new dataset ingest requests.

• Reliability Engineering: Transition the operating model from reactive symptom-fixing to preventative, long-term architectural stability.

• Cross-Skilling: Build team resilience by documenting systems, establishing clear engineering standards, and rotating team members between live operational support and strategic projects.