Transfer learning has led to ground-breaking progress in natural language processing (NLP), computer vision and many application domains, with deep models pre-trained on massive amounts of unlabelled data. The representation of data learned by unsupervised AI models provides a powerful basis for downstream machine learning tasks. Moreover, pre-training has become the default practice in deep learning to substantially save compute costs and data. Pre-trained encoders also called foundation models compress raw data into compact latent embeddings that preserve rich semantic information. Understanding distribution and structure of these embeddings is an important problem. Meanwhile, sharing these embeddings without adequate protection exposes them to unauthorized use in a wide range of malicious activities. Transfer learning is therefore a central topic in AI.
Integreat will explore and analyse the inner representation of deep models, in particular language models. Data-driven approaches leading to these meaningful latent representations can be enhanced by combining knowledge-driven approaches on top of them. While the self-supervised learning of current language models, with next token prediction, targets superficial and local properties of text (e.g., word-in-context prediction), Integreat will study linguistically informed self-supervision, e.g., by estimation of latent variables based on prior knowledge, like lexical relationships between words. The representation of multimodal data is also of key importance with the question of how to interface language models with images, audio or video. Self-supervised learning is not enough to create efficient connections between modalities and Integreat aims at going beyond this paradigm.
Integreat will enable transfer learning across a broader set of domains and modalities by explicitly utilizing prior knowledge of the domains and of the data generation processes, and by developing statistical and information-theoretic principles to ensure knowledge coherence. We will study new aspects of reproducing kernel Hilbert spaces, to match distributions and representations across tasks, and to gain in efficiency, interpretability and sustainability. Further, we will facilitate joint extraction of knowledge (soft and hard) across the domains by investigating domain alignment, which is crucial to combining multi-modal data, and provide knowledge-conditioned alignment solutions. Throughout, it will be important to quantify uncertainty of transferability to guide transfer learning across sources.
Research theme leaders:
- Professor Michael Kampffmeyer, UiT The Arctic University of Oslo
- Associate Professor Ali Ramezani-Kebrya, University of Oslo
- Associate Professor Benjamin Ricaud, UiT The Arctic University of Oslo