Title
Perception, reasoning and planning with multimodal models
Abstract
Large multimodal models are expected to perceive, reason about, and plan in complex visual environments, making them candidates for embedding in agentic workflows. Yet these models remain surprisingly brittle. In this talk, I first show that state-of-the-art models miss visual anomalies that humans spot instantly and fail at spatial and commonsense reasoning that human observers handle effortlessly. I argue that the core bottleneck is not language understanding but visual grounding: the ability to anchor reasoning in what is actually present in an image or video. I then present a line of work that addresses this by guiding multimodal models to attend to specific visual elements during reasoning, and show that this principle improves performance on both static visual reasoning and dynamic world modeling, where models must understand and predict how visual scenes evolve over time.
Bio
Syrielle Montariol is a research scientist at the French National Centre for Scientific Research (CNRS) and Sorbonne University (Paris). She obtained her PhD at University Paris-Saclay in 2021 and held postdoctoral positions at INRIA Paris and EPFL (Switzerland), with research visits at NII (Japan), JSI (Slovenia) and UC Berkeley (USA). Her research focuses on developing multilingual and multimodal language models.
Speaker is presenting in person in Oslo.
Practical
The Tuesdays seminar series is devoted to various topics relevant for the Integreat centre´s research focus. Presenters from the Integreat community and beyond have 40 minutes to present, followed by a group discussion.
Seminars are open for attendance for everybody.
For those unable to attend in person: https://uio.zoom.us/j/65632027408
Point of contacts:
- Organiser and moderator: Andrey Kutuzov, University of Oslo
- Integreat Administrative Leader Maria Dikova, University of Oslo