Senior Data Engineer

About the role

We are seeking a Senior Data Engineer to join our Detection‑Coverage Data Platform team. You will own and evolve high‑volume, production‑grade pipelines that ingest, normalize, and model large‑scale SIEM data from sources such as Splunk, Microsoft Sentinel, and Google SecOps.

Key responsibilities

• Design, build, and maintain durable, back‑pressured, fault‑tolerant ETL pipelines handling thousands of data streams per tenant.
• Develop and evolve graph, relational, and lakehouse data models, selecting optimal storage layers for each access pattern.
• Diagnose and resolve database concurrency and performance issues, including lock contention, deadlocks, and inefficient queries.
• Optimize processing through batching, partitioning, idempotent upserts, and appropriately sized worker fleets.
• Implement end‑to‑end observability with metrics, structured logging, distributed tracing, data‑quality validation, and pipeline SLIs.
• Write production‑grade, async‑first, typed and tested Python code within structured workflow orchestration.
• Containerize and deploy services using Docker, Kubernetes/EKS, and cloud‑native infrastructure.
• Collaborate with data scientists and product engineers to support statistical and ML workflows while maintaining clear data contracts.

Required profile

• Proven experience building and operating data‑intensive backend systems or large‑scale data infrastructure.
• Strong expertise in production‑grade async Python, typing, data validation, testing, and CI pipelines.
• Hands‑on experience with workflow orchestration tools such as Temporal, Airflow, Dagster, or Step Functions.
• Deep knowledge of PostgreSQL, relational schema design, query tuning, and concurrency troubleshooting.
• Experience modeling data with graph databases (Neo4j/Cypher) or lakehouse technologies (Apache Iceberg, Delta Lake, Parquet on S3).
• Solid understanding of distributed data processing, concurrent writes, batching, partitioning, and idempotent processing.
• Familiarity with S3/object storage, columnar formats, and query engines like Athena, Trino, or Spark.
• Strong grasp of observability, data quality, metrics, distributed tracing, and OpenTelemetry.
• Experience with Docker, Kubernetes/EKS, Infrastructure as Code, and cloud‑native deployment practices.

Required skills

• Python (async, type‑hints, testing)
• Workflow orchestration (Temporal, Airflow, Dagster, Step Functions)
• PostgreSQL (schema design, tuning, concurrency)
• Graph databases (Neo4j, Cypher)
• Lakehouse technologies (Apache Iceberg, Delta Lake, Parquet)
• AWS S3 / object storage
• Columnar formats (Parquet)
• Query engines (Athena, Trino, Spark)
• Observability tools (metrics, distributed tracing, OpenTelemetry)
• Docker, Kubernetes/EKS
• Infrastructure as Code (IaC)