Post-doctoral Research Visit F/m a Multi-modal Language Model for Earth Observation

Le descriptif de l’offre ci-dessous est en Anglais_

**Type de contrat**: CDD

**Niveau de diplôme exigé**: Thèse ou équivalent

**Fonction**: Post-Doctorant

**A propos du centre ou de la direction fonctionnelle**:
The Inria centre at Université Côte d'Azur includes 42 research teams and 9 support services. The centre's staff (about 500 people) is made up of scientists of different nationalities, engineers, technicians and administrative staff. The teams are mainly located on the university campuses of Sophia Antipolis and Nice as well as Montpellier, in close collaboration with research and higher education laboratories and establishments (Université Côte d'Azur, CNRS, INRAE, INSERM...), but also with the regiona economic players.

With a presence in the fields of computational neuroscience and biology, data science and modeling, software engineering and certification, as well as collaborative robotics, the Inria Centre at Université Côte d'Azur is a major player in terms of scientific excellence through its results and collaborations at both European and international levels.

**Contexte et atouts du poste**:
This post-doctoral offer is funded by the GEO-ReSeT ANR project, representing a collaboration between Inria (team EVERGREEN, Montpellier) and Université de Paris Cité (team LIPADE, Paris).

Leveraging the large amounts of available geo-spatial data from different sources, the GEO-ReSeT (Generalized Earth Observation with Remote Sensing and Text) project has the objective to learn a rich representation of any geo-spatial location and convey a semantic representation of the information, by improving on existing models and providing a better experience to the end users. By using location on the Earth's surface as the common link between different modalities, a geo-spatial foundation model would be able to incorporate a variety of data sources, including remote sensing imagery, textual descriptions of places, and other generic features.

By leveraging several data modalities, this foundation model could provide a comprehensive and accurate understanding of the Earth's surface, enabling informed decisions and actions. This will be particularly valuable for new potential users in sectors such as journalism, social sciences or environmental monitoring, who may not have the resources or expertise to collect their own training datasets and develop their own methods, thus moving beyond open Earth observation data and democratizing the access to Earth observation information.

**Mission confiée**:
The work to be conducted during the proposed post-doc project will contribute to the ambition of the GEO-ReSeT ANR project by linking textual descriptions of places (e.g., collected from heterogeneous online sources, such as news articles or search engine results), to their approximate geo-location, a task known as geoparsing.

This text-location link will then be used in combination with other geospatial data modalities, with a focus on remote sensing data from sensors such as Sentinel-1 and -2, in order to train multi-modal models that are aware about the way in which people describe locations.

This will be done by first combining information stemming from different databases containing geographic named entities, such as Open Street Map, Wikipedia and gazetteers, such that geographic points or polygons can be linked to each named entity.

In a second step, a Natural Language Processing (NLP) pipeline will be developed to obtain the most likely geographic named entities that are referred to in any piece of text that describes a place.

With respect to existing Named Entity Recognition (NER) methodologies, in order to avoid restricting us to cases where entities' names appear exactly as in the databases or gazetteers, we will leverage pre-trained Large Language Models (LLM) to resolve ambiguities and gather evidence towards the most likely entities that are being described in the text. Such an approach will be trained and validated by using the cases that do match the names in the gazetteer.

We will then move on, in collaboration with the rest of the GEO-ReSeT consortium, to train a multi-modal large language model (MMLLM) that will serve as a foundation model for Earth observation tasks.

This model will finally be evaluated on several agro-environmental tasks.

**Principales activités**:

- Description of the state-of-the-art in unstructured text geoparsing, with a focus on approaches leveraging LLMs.
- Collection of a database of geographic named entities linked to their geographic footprint (e.g. point or polygon). Collection of a database of unstructured online text that is likely to contain a reference to a geographic location.
- Development of an NLP pipeline to link each piece of geographic text to its likely geographic footprint.
- Participate in the design and training of a multi-modal large language model (MMLLM) using remote sensing and geoparsed text.
- Evaluation of the final model on two of the follow