R&d Engineer

Le descriptif de l’offre ci-dessous est en Anglais_

**Type de contrat**: CDD

**Niveau de diplôme exigé**: Bac + 5 ou équivalent

**Fonction**: Ingénieur scientifique contractuel

**A propos du centre ou de la direction fonctionnelle**:
The Inria center at Université Côte d'Azur includes 42 research teams and 9 support services. The center’s staff (about 500 people) is made up of scientists of different nationalities, engineers, technicians and administrative staff. The teams are mainly located on the university campuses of Sophia Antipolis and Nice as well as Montpellier, in close collaboration with research and higher education laboratories and establishments (Université Côte d'Azur, CNRS, INRAE, INSERM...), but also with the regional economic players.

With a presence in the fields of computational neuroscience and biology, data science and modeling, software engineering and certification, as well as collaborative robotics, the Inria Centre at Université Côte d'Azur is a major player in terms of scientific excellence through its results and collaborations at both European and international levels.

**Contexte et atouts du poste**:
**Team**

The STARS research team combines advanced theory with cutting-edge practice focusing on cognitive vision systems.

Scientific context

Feature extraction is a challenging computer vision problem which targets extracting relevant information from raw data in order to reduce dimensionality and capture meaningful patterns. When this needs to be done in a dataset and task invariant way, it is referred to as general feature extraction. This is a crucial step in machine learning pipelines and popular methods like VideoSwin and VIdeoMAE work well for the task of action recognition and video understanding. However, these works and also the datasets that they are tested on, like Something-Something and Kinetics, fail to capture information about interactions in daily life.

Towards this research direction, several methods have been proposed to model these complex fine grained interactions using datasets like UDIVA, MPII Group Interactions and Epic-Kitchen. Those datasets encompassing real-world challenges share the following characteristics: Firstly, there is rich multimodal information available where each modality provides important information relevant to the labels. Secondly, there is a lot of irrelevant information that has to be ignored as deep learning models easily identify patterns that are coincidental (local mínima). For example, the colour of the T-shirt could be used to assign a certain personality score to someone if by coincidence the majority of the extrovert people are wearing warm colours. Lastly, the videos in these
datasets are generally very long.

So, the main question is:
How to extract general features from multimodal data with a lot of noise in the form of irrelevant information?

Typical situations that we would like to monitor are daily interactions, responses and reactions and analyse cause and effect in behaviour (it could be humanhuman interaction or human-object interaction).

The system we want to develop will be beneficial for all tasks requiring focus on interactions. Specifically, healthcare for psychological disorders - general feature extraction will allow deep learning models to assist in various subtasks involved in the diagnosis process.

**Mission confiée**:
In this work, we would like to go beyond existing computer vision deep learning models and introduce ways to extend them to utilise information from new modalities. Also, to identify ways to focus on relevant information for interactions in the input. The system should also take into account the long temporal duration of videos in the datasets in this domain. These have to be done in a flexible way, so that there is mínimal change to the original model and hence the original model’s trained weights are useful too.

Existing methods have mostly focused on modelling the variation of visual cues pertinent to the classes provided for video classification tasks. Though they perform these tasks well, changes in the recording setting or addition of noise in the form of irrelevant background information makes it hard for these models to perform well. So, for obtaining a general feature extractor, the models have to be modified to accommodate for these shortcomings.

**Principales activités**:
The Inria STARS team is seeking an engineer with a strong background in computer vision, deep learning, and machine learning.

In this work, we focus on two things: First, in action scenarios, utilizing all available information to obtain relevant features for multiple downstream tasks while ignoring irrelevant background information. Second,
efficient transfer learning for a new recording paradigm. This can include new modalities, changes in recording settings, and different downstream tasks. The first objective can be tackled by forcing attention in transformers to attend to relevant parts of the inp