Internship Research On Joint Encoding of

Le descriptif de l’offre ci-dessous est en Anglais_

**Type de contrat **:Convention de stage

**Niveau de diplôme exigé **:Bac + 5 ou équivalent

**Fonction **:Stagiaire de la recherche

**Niveau d'expérience souhaité **:Jeune diplômé

**A propos du centre ou de la direction fonctionnelle**:
The Inria centre at Université Côte d'Azur includes 37 research teams and 8 support services. The centre's staff (about 500 people) is made up of scientists of different nationalities, engineers, technicians and administrative staff. The teams are mainly located on the university campuses of Sophia Antipolis and Nice as well as Montpellier, in close collaboration with research and higher education laboratories and establishments (Université Côte d'Azur, CNRS, INRAE, INSERM...), but also with the regiona economic players.

With a presence in the fields of computational neuroscience and biology, data science and modeling, software engineering and certification, as well as collaborative robotics, the Inria Centre at Université Côte d'Azur is a major player in terms of scientific excellence through its results and collaborations at both European and international levels.

**Contexte et atouts du poste**:
The work will be embedded in a project in collaboration between Université de Paris Cité (team LIPADE, Paris) and Inria (team EVERGREEN, Montpellier).

By using location on the Earth's surface as the common link between different modalities, a geo-spatial foundation model would be able to incorporate a variety of data sources, including remote sensing imagery, textual descriptions of places, and features in maps. Leveraging the large amounts of available unlabeled geo-spatial data from these different sources, the GEO-ReSeT (Generalized Earth Observation with Remote Sensing and Text) ANR project has the objective to learn a better representation of any geo-spatial location and convey a semantic representation of the information.

By leveraging several data modalities, this foundation model could provide a more comprehensive and accurate understanding of the Earth's surface, enabling more informed decisions and actions. This will be particularly valuable for new potential users in sectors such as journalism, social sciences or environmental monitoring, who may not have the resources or expertise to collect their own training datasets and develop their own methods, thus moving beyond open Earth observation data and democratizing the access to Earth observation information.

**Mission confiée**:
The work to be conducted during the proposed M2 internship will contribute to the ambition of the GEO-ReSeT ANR project by studying a model that is robust to different multi-spectral modalities. Different sensors measure different spectral bands, at different spatial resolutions, which can capture different information about the target. For instance, Sentinel-2 (multi-spectral satellite from the Copernicus program of the European Union) measures 13 spectral bands at resolutions ranging from 10 to 60m. On the other hand, Landsat 9 measures 11 bands at resolutions ranging from 15 to 100m. In addition, hyperspectral sensors which measures hundreds of different spectral bands can be used.

Currently, several approaches exist to jointly work on data obtained from different multi-spectral instruments. One of the most classical one is to train different feature extractors for each modality and to fuse the obtained latent representation. Another approach is to fuse the data at the input level. Finally, it is also possible to make a prediction from each modalities and do a fusion at the prediction level. These approaches tend to perform well. However, they require to train one model for each modality, which generally requires an important amount of supervision and is computationally heavy. A different approach is to translate different modalities to the input space of one of them. This approach has the advantage of reducing the number of different models to learn. However, it will also remove the particularities (in our case in both spatial and spectral resolution) of the other modalities. Recent remote sensing based foundation models can be interpreted as from this last category, even though no explicit conversion is performed.

**Principales activités**:
In this work, our objective is to design and train a model that is able to take as input any multi-spectral acquisition while keeping the physical measurements (i.e. spectral bands and spatial resolution). The work to be performed in this internship will lead to the following three contributions:

- Contribution C: the proposed architecture will be compared to the baselines on a downstream task to demonstrate the relevance of the proposed approach.

In this project, we will evaluate the approach on a setting restricted to Landsat 8/9 and Sentinel-2. We will exploit the Harmonized Landsat and Sentinel-2 product for comparison with a method taking as input a unified represe