Multimodal Llms

Multimodality, in particular the incorporation of speech representations into LLMs, enables systems to comprehend and respond to spoken language with unparalleled accuracy and depth. Different methods for fusion have been explored recently, ranging from shallow fusion through discrete speech units modeling [1,2] to more complex
approaches for integrating speech representations into the LLM decoding procedure [3].

Beyond conventional text-based interfaces, multimodal LLMs enriched with speech capabilities have the potential to transform various domains, from virtual assistants to accessibility aids and interactive storytelling platforms.

In this internship, the selected student will delve into approaches for integrating speech into LLMs for speech-related tasks.

**Supervisors**: Marcely Zanon Boito and Ioan Calapodescu

Required skills
- Last year MSc or PhD student in speech processing and/or NLP-related domains
- Solid deep learning background
- Experience with pytorch toolkit

References

Application instructions

Please note that applicants must be registered students at a university or other academic institution and that this establishment will need to sign an 'Internship Convention' with NAVER LABS Europe before the student is accepted.

About NAVER LABS

NAVER is the #1 Internet portal in Korea with activities that span a wide range of businesses including search, commerce, content, financial and cloud platforms.

NAVER LABS, co-located in Korea and France, is the organization dedicated to preparing NAVER’s future. NAVER LABS Europe is located in a spectacular setting in Grenoble, in the heart of the French Alps. Scientists at NAVER LABS Europe are empowered to pursue long-term research problems that, if successful, can have significant impact and transform NAVER. We take our ideas as far as research can to create the best technology of its kind. Active participation in the academic community and collaborations with world-class public research groups are, among others, important tools to achieve these goals. Teamwork, focus and persistence are important values for us.

NAVER LABS Europe is an equal opportunity employer.