Senior Data Engineer - Real Time Streams

About the role

You will join a team that owns a real‑time data platform, turning operational events into live analytics and warehoused data. The role covers end‑to‑end ownership of streaming jobs, CDC pipelines, Kafka Connect bridges, and downstream sinks across multiple databases.

Key responsibilities

• Design, implement, and operate stateful Apache Flink streaming pipelines (keyed state, windowing, watermarks, timers, side outputs, custom sources/sinks).
• Build and maintain Change Data Capture pipelines using Flink CDC or Debezium, handling snapshots, incremental changes, schema evolution, and idempotency.
• Develop and run Kafka and Kafka Connect pipelines, including topic/partition design, connector configuration, schema management, and dead‑letter routing.
• Own the data flow from Kafka → Flink → PostgreSQL/TimescaleDB/ClickHouse/BigQuery, covering schema design, idempotency, batch tuning, and observability.
• Diagnose and resolve production issues such as checkpoint failures, backpressure, state growth, sink latency, and autoscaler oscillations.
• Harden the platform with delivery guarantees, watermark strategies, DLQ handling, schema migrations, and alerting.
• Conduct code reviews, mentor mid‑level engineers, and contribute to deployment artifacts (Helm charts, ArgoCD apps, GKE config, Grafana dashboards, Prometheus alerts).
• Influence architectural decisions around state backends, schema migrations, and pipeline refactoring.

Required profile

• 5+ years of professional software or data engineering experience with strong Java expertise.
• Production experience with Apache Flink and stateful real‑time streaming pipelines.
• Deep knowledge of Apache Kafka, including Kafka Connect, consumer groups, partitions, delivery guarantees, and schema management.
• Hands‑on CDC experience using Debezium or Flink CDC.
• Experience designing and operating Kafka → Flink → database/warehouse pipelines in production.
• Strong understanding of PostgreSQL and at least one analytical database (ClickHouse or BigQuery).
• Proficiency with Kubernetes and deploying/operating production data workloads.
• Proven ability to troubleshoot and optimise streaming systems (backpressure, checkpoint failures, state growth, latency, sink performance).

Required skills

• Java
• Apache Flink
• Apache Kafka
• Kafka Connect
• Debezium / Flink CDC
• PostgreSQL
• TimescaleDB (optional)
• ClickHouse
• BigQuery
• Kubernetes
• Helm
• ArgoCD
• Google Kubernetes Engine (GKE)
• Grafana
• Prometheus