Senior Software Data Engineer

il y a 4 jours

Paris, Île-de-France HG Insights Temps plein

About Us

HG Insights is the global leader in technology intelligence, delivering actionable AI driven insights through advanced data science and scalable big data solutions. Our Big Data Insights Platform processes billions of unstructured documents and powers a vast data lake, enabling enterprises to make strategic, data-driven decisions. Join our team to solve complex data challenges at scale and shape the future of B2B intelligence.

What You'll Do:

  • Design, build, and optimize large-scale distributed data pipelines for processing billions of unstructured documents using Databricks, Apache Spark, and cloud-native big data tools
  • Architect and scale enterprise-grade big-data systems, including data lakes, ETL/ELT workflows, and syndication platforms for customer-facing Insights-as-a-Service (InaaS) products.
  • Collaborate with product teams to develop features across databases, backend services, and frontend UIs that expose actionable intelligence from complex datasets.
  • Implement cutting-edge solutions for data ingestion, transformation, and analytics using Hadoop/Spark ecosystems, Elasticsearch, and cloud services (AWS EC2, S3, EMR).
  • Drive system reliability through automation, CI/CD pipelines (Docker, Kubernetes, Terraform), and infrastructure-as-code practices.

What You'll Be Responsible For

  • Leading the development of our Big Data Insights Platform, ensuring scalability, performance, and cost-efficiency across distributed systems.
  • Mentoring engineers, conducting code reviews, and establishing best practices for Spark optimization, data modeling, and cluster resource management.
  • Building & Troubleshooting complex data pipeline issues, including performance tuning of Spark jobs, query optimization, and data quality enforcement.
  • Collaborating in agile workflows (daily stand-ups, sprint planning) to deliver features rapidly while maintaining system stability.
  • Ensuring security and compliance across data workflows, including access controls, encryption, and governance policies.

What You'll Need

  • BS/MS/Ph.D. in Computer Science or related field, with 5+ years of experience building production-grade big data systems.
  • Expertise in Scala/Java for Spark development, including optimization of batch/streaming jobs and debugging distributed workflows.
  • Proven track record with:
  • Databricks, Hadoop/Spark ecosystems, and SQL/NoSQL databases (MySQL, Elasticsearch).
  • Cloud platforms (AWS EC2, S3, EMR) and infrastructure-as-code tools (Terraform, Kubernetes).
  • RESTful APIs, microservices architectures, and CI/CD automation.
  • Leadership experience as a technical lead, including mentoring engineers and driving architectural decisions.
  • Strong understanding of agile practices, distributed computing principles, and data lake architectures.
  • Airflow orchestration (DAGs, operators, sensors) and integration with Spark/Databricks
  • 7+ years of designing, modeling and building big data pipelines in an enterprise work setting.

Nice-to-Haves

  • Experience with machine learning pipelines (Spark MLlib, Databricks ML) for predictive analytics.
  • Knowledge of data governance frameworks and compliance standards (GDPR, CCPA).
  • Contributions to open-source big data projects or published technical blogs/papers.
  • DevOps proficiency in monitoring tools (Prometheus, Grafana) and serverless architectures.

  • Senior Software Engineer

    il y a 7 jours

    Paris, Île-de-France CAST Software Inc. Temps plein

    We're developing innovative solutions that leverage our Software Intelligence products (CAST Highlight and CAST Imaging) to enhance AI's capabilities in software engineering contexts.As a Senior Software Engineer on this initiative, you will:Design and implement integrations between our Software Intelligence products and modern LLM frameworksDevelop APIs and...

  • Senior Data Engineer

    il y a 4 jours

    Paris, Île-de-France Thurn Partners Temps plein

    Senior Data Engineer - PythonCompany InsightA well-established systematic trading firm, operating across multiple locations, is currently seeking a Senior Data Engineer to join the Data Engineering team in Paris. The opportunity involves building a global, scalable Data Lakehouse platform to support multiple trading desks across the organisation. Founded...

  • Senior Data Engineer

    il y a 2 jours

    Paris, Île-de-France Swile Temps plein

    At Swile, we believe that good products can help reduce friction in daily professional life and boost employee satisfaction. Today, we provide innovative solutions in various areas such as Fintech, Travel, HR, and Employee Benefits to more than 5.5 million users in 85,000 companies in France and Brazil. As a Senior Data Engineer, your primary role is to...

  • EDA Software Engineer

    il y a 4 jours

    Paris, Île-de-France IC Resources Temps plein

    An exciting opportunity for an EDA Software Development Engineer has arisen with a global leader in System-on-Chip (SoC) interconnect IP, based in Paris.As an EDA Software Development Engineer, you will play a key role in advancing their Magillem product line, ensuring scalability, modularity, and high performance for complex semiconductor design...

  • Data Engineer

    il y a 4 jours

    Paris, Île-de-France MP DATA Temps plein

    En tant que Data Engineer Senior, vous jouerez un rôle clé dans la construction, l'optimisation et la fiabilisation de nos pipelines de données à grande échelle, au cœur de notre plateforme analytique. Votre expertise sur Databricks et l'environnement Spark sera essentielle pour garantir des traitements performants, sécurisés et scalables.Vos...

  • Senior Data Engineer

    il y a 5 jours

    Paris, Île-de-France Hydrosat Temps plein

    Job DescriptionHydrosat is a space and data analytics company building a new Earth Observation constellation and software to unlock the power of thermal infrared imagery. Our mission is to capture and deliver high spatio-temporal resolution thermal infrared data, currently not available from other sources, to provide insights that benefit the environment,...

  • Senior Software Engineer

    il y a 4 jours

    Paris, Île-de-France Dioxycle Temps plein

    About Dioxycle Dioxycle is pioneering breakthrough carbon utilization technologies that convert industrial emissions into sustainable chemicals with unprecedented energy and cost efficiency. By displacing fossil fuels for the production of key chemicals, Dioxycle has the potential to ultimately reduce over 800 million tons of CO2 per year, representing over...

  • Data Engineer sénior

    il y a 7 jours

    Paris, Île-de-France a-82ac-4f4c-a6b2-5eb6545b4923 Temps plein

    About the role:We are looking for a result-driven Senior Data Engineer to join our core data engineering team . The successful candidate should have in-depth knowledge in data engineering with the ability to share experience with the team. The company has recently started the Data Transformation Program involving broad-scale modernization of Data...

  • Senior Software Engineer

    il y a 2 semaines

    Paris, Île-de-France Back Market Temps plein

    Hi, we're Back Market.We're here to helpmake tech reliable, affordable, and better than new. We're a global marketplace for refurbished devices, helping lower our collective environmental impact by providing trustworthy, affordable tech with 92% less carbon emissions than new.Yep, you read that right. Turns out refurbished tech is way better for the planet...

  • Senior Software Engineer – Python

    il y a 2 jours

    Paris, Île-de-France Twenty One Talents Temps plein

    Présentation de la société Rejoignez une entreprise en pleine croissance, spécialisée dans les solutions logicielles pour le secteur spatial. Fondée par des experts du domaine, cette entreprise dveloppe une plateforme cloud sécurisée permettant aux opérateurs satellitaires de communiquer efficacement avec leurs satellites via un réseau mondial...