Data Engineer, Aws Data

* Bachelors or Master’s in Computer Science or Engineering, or equivalent experience
- Knowledge of relational databases, SQL, and UNIX/Linux
- Knowledge of the Python programming language

Job summary

The AWS Data & ML Research Division is looking for a data engineer to join the Estel team located in Paris, France. Our research teams extract fundamental scientific problems from AWS products and business needs as informed by customers, and develop innovative solutions that will have a long-lasting, transformative impact on our products and customer experiences.

The Estel team specializes in resource optimization of large-scale cloud data analytics. It aims to bring the best cost performance benefits to customers, in an unprecedented manner as enabled by cutting-edge research. Such innovations include principled multi-objective optimization approaches, and adoption of large-scale machine learning to bear on the process of performance and cost modeling, thereby enabling intelligent optimization decisions based on such models. We envision that such techniques will be broadly applied to a range of AWS products, unlock the power of big data and machine learning for intelligent decision making, and ultimately enable big leaps forward for offering best cost performance benefits to customers and long-term sustainable computing.

If you share our vision and are motivated by a career path that defines new problems, explores in uncharted territory, pushes the scientific boundary, innovates and delivers until a real-world impact is generated, then come and join us — we will work together to make our vision true

Key job responsibilities

As a Data Engineer in the Estel team, you will:

- Develop large ETL pipelines to extract query logs, data statistics, system performance metrics, etc. from production clusters to a processing backend on a daily or hourly basis;
- Handle data cleaning and integration of diverse data sources to build a clean, informative profile of each query executed in recent past and do so in a timely manner to prepare for model training;
- Handle storage and indexing of large numbers of query profiles to enable the development of a history-based optimizer;
- Adapt such ETL pipelines and query profile databases for different AWS products;
- Work within a fast moving, startup environment in a large company, interacting with different product teams to rapidly deliver prototypes and/or products that have a broad business impact.

You should be somebody who enjoys working on large data sets, is customer-centric, is passionate about building quality, performant ETL and related data management software, as well as achieving operational excellence. You should also be a fast learner in order to understand how key AWS products function (including the data API, system architecture, etc.). A commitment to team work, and strong communication skills (both within the research team and with different product teams) are essential. You will work with fellow scientists and engineers to solve challenging problems and have an opportunity to publish your work in the scientific community.
- Masters in Computer Science, with a focus on database systems and big data systems
- 3-5 years of relevant work experience in the query processing or data analysis domain
- Experience with database internals such as query processing and optimization, in particular, query plans, indexes, data statistics, runtime query execution
- Knowledge of ETL tasks such as parsing, data cleaning, and data integration
- Ability to design and develop a storage and retrieval system for extracted query logs and system metrics
- Good understanding of parallel computing, system efficiency, and scalability issues
- Experience in characterizing, debugging, and correcting performance issues in large-scale ETL or data management systems
- Proven ability to drive tasks to completion, work independently, and take on full ownership of projects