Data Engineer

Big Data Engineer - Pipelines

Location: Paris or Toulouse

Salary: €60, ,000

Our mission:

Only 1% of freshwater on earth is usable for humanity's needs and climate change is exacerbating both water scarcity and water hazards – such as floods, droughts, and water pollution.

BWI is an impact startup on a mission to use data and machine learning to provide governments and companies with river flow forecast services in order to help better adapt to the impact of climate change on water management and availability. BWI subscribers benefit from near real-time hydrological forecasts based on dozens of data sources – such as space sensors, weather radars, and other ground sensors.

The role:

After recently securing a major new contract, we're now looking for 3 x Big Data Engineers to join our engineering team in either our Paris or Toulouse offices.

Working as part of either our satellite or water simulation team, you will be responsible for building and operating data ingestion and processing pipelines for terabytes of weather and hydrological data.

You'll be focused on ensuring scalable, reproducible and production-grade delivery of model inputs and outputs.

Responsibilities:

• Implement ingestion (batch/streaming), ELT/ETL steps, and data publishing workflows.
• Handle scientific formats (netCDF, GRIB2) and columnar storage (Parquet); optimize I/O and algorithms.
• Design storage with eventual‑consistency patterns (atomic publishes, manifests, versioned paths) and a metadata catalog.
• Partition and parallelize workloads for distributed compute; compact small files and tune for cost/performance.
• Build and run containerized services and orchestrated workflows; ensure observability, retries, idempotency, and runbooks.
• Collaborate with scientists to define data models and validation rules.

Top paradigms & architectural patterns required:

• ELT-first with ETL where needed; streaming/micro-batch for low-latency sources
• Data lake + metadata catalogue (object storage), data model design for cataloguing
• Partitioned columnar storage & distributed data-parallel processing
• Idempotent, restartable workflows with orchestration
• Versioned datasets, atomic publish patterns, and catalogue as source of truth
• Observability-driven ops and infra-as-code

Essential tools & technologies:

• Python (xarray, netCDF4, pyarrow), PySpark or Dask
• S3-compatible object storage; Parquet format
• PostgreSQL / PostGIS
• Kubernetes + Docker for deployment
• AWS (S3, EKS, EC2) or equivalent cloud; Terraform for IaC

Must-haves:

• 3+ years relevant experience building/operating large data pipelines
• Strong software engineering in Python with testing and CI/CD
• Practical experience with partitioning, and parallelism
• Computer science education
• Understanding of architectural patterns in storing & processing large volumes of data
• Humility & eagerness to accept both: full hands-on coding position or technical leadership depending on circumstances
• Clear communication, teamwork, and analytical problem-solving

Nice-to-haves:

• Hydrology, meteorology, remote-sensing or space-ground-segment experience
• STAC, GeoTIFF, PostGIS, Numba/Cython performance tuning
• Understanding of low-level subject (memory management, HTTP, S3 implementation)
• Willingness to develop managerial skills

Our culture:

• Dynamic, fast paced start-up, working on an important problem for humanity.
• Multicultural Team - With team members from various countries like France, Poland, India, Lebanon and Morocco, you will be exposed to a multicultural work environment.
• Offices in Central Paris and Toulouse
• Diverse Domains - work across software engineering, hydrology, climate and satellite.
• Hybrid working w/ 4 days in office and 1 day remote.