Research engineer Perspective-Grounded Spatial Reasoning in Multimodal Large Language Models

Il y a 1 jour

Paris, Nouvelle-Aquitaine, France Inria Paris Temps plein

Assignment Multimodal large language models (MLLMs) remain notably weak at spatial reasoning from video, particularly at perspective-taking where the model is asked to answer questions from a viewpoint other than the camera's own (e.g., identifying what lies to an observer's left from an allocentric frame). Existing approaches largely rely on either naive data augmentation or opaque latent reasoning tokens whose targets are not grounded in real geometry, both of which have known failure modes. This project aims to build a diagnostic benchmark that isolates genuine perspective-taking failures from confounds (e.g., left-right mirroring, temporal-order artifacts), and to develop a spatial reasoning representation that is explicitly grounded in 3D geometric structure via alignment with a pretrained 3D foundation model, rather than relying on self-consistency alone.

Main activities

  • Analyse and implement related work.
  • Design novel innovative solutions.
  • Write progress reports and papers.
  • Present work at conferences.

Skills

Technical skills and level required : programming skills are required. Languages : English and possibly French. Relational skills : Good communication skills.

#J-18808-Ljbffr