Read full announcement and apply!
Description
While current Vision-Language Models (VLMs) have demonstrated impressive capabilities, their reliance on large, unstructured datasets often leads to a superficial understanding of complex visual scenes. This project introduces a paradigm shift by proposing a structured, graph-based approach to vision-language fusion. The central innovation is the creation of panoptic scene graphs, a rich representation that explicitly models the relationships between objects and concepts in an image. These graphs will be derived from a novel application of superpixels and fused with the advanced reasoning capabilities of Large Language Models (LLMs). This bidirectional flow of information enables the system to move beyond superficial pattern matching and engage in genuine compositional reasoning.
The project aims to produce ground-breaking results on tasks requiring deep contextual understanding, develop a more accurate and explainable machine learning methodology, and unlock a new era where AI reasons over images with the depth of a knowledge graph.
Project duration: three years.
Project requirements
This page describes requirements specific to Project 8. Applicants must also satisfy the general qualification requirements, PhD programme admission requirements, and application requirements described in the main call for PhD fellowships.
Required qualifications
Applicants should satisfy the following qualifications to be eligible for consideration for this project:
- Master’s degree in mathematics, statistics, computer science, data science, machine learning, or another related field.
- Documented background in mathematics, statistics, machine learning, and scientific programming (e.g., Python/PyTorch/Tensorflow).
- Expertise in creating or adapting modern, token-based transformer architectures.
- Expertise in creating or adapting Large Language Models (LLMs) and interfacing them with other models.
Preferred qualifications
The following qualifications are not required but will be considered an advantage in the evaluation process:
- Familiarity with Panoptic Scene Graph Generation (PSG) methodologies.
- Familiarity with Large Language Model fine tuning and alignment.
- Knowledge of graph-based Retrieval-Augmented Generation (Graph RAG) workflows.
- Understanding of probabilistic modeling and information-based regularizers.
Contact for scientific inquiries
For questions regarding the scientific content, research objectives, or project-specific requirements