Recent breakthroughs in Natural Language Processing (NLP), powered by pre-trained large language models (LLMs), have transformed the landscape of AI. These models have demonstrated impressive capabilities across a wide range of applications by leveraging vast amounts of data and sophisticated neural architectures, showing great promise in enabling natural language as an interface between humans and machines. However, despite their successes, current LLMs exhibit critical limitations: they often lack robustness, produce biased or factually incorrect outputs, struggle with reasoning, and operate as "black boxes" with limited explainability. Furthermore, their development raises concerns about environmental sustainability, due to massive computational demands.
The current dominating LLM paradigm of trying to learn everything through next-token prediction alone, makes for models that are brittle, difficult to control, and extremely resource demanding in terms of both data and compute. This motivates the need for research on incorporating knowledge into various stages of the LLM development cycle, from data preparation to pre-training, post-training, and inference, and, last but not least, evaluation. As with all machine-learning, LLM development is driven by a feedback loop of training and evaluation. However, given the generality of LLMs, the task for evaluating them is monumental; there are many domains of knowledge and many possible modes of problem-solving to cover, not the least with respect to ethical dimensions, and the output space, in terms of both predictions and ground truth, is often open-ended.
In the settings above, relevant knowledge can come in various forms, like linguistic knowledge, world knowledge, domain-specific knowledge, etc., and can be represented in various ways, e.g. human annotations of text, knowledge graphs, databases, logical rules, and more. For example, developing more linguistically informed training objectives can enable self-supervised learning that is more data efficient than the current paradigm. Developing mechanisms that allow models to access and reason with external knowledge sources during prediction, can enhance their ability to explain decisions and adapt to new scenarios. There are also many open research questions pertaining to interpretability and explainability for LLMs.
Ongoing activities
In the following, we briefly summarize some of the core ongoing activities within RT7 so far. Most prominent among these are the two associated PhD projects. This includes a project on assessing ethical and moral judgments by LLMs, by PhD fellow Hannah Clausen (supervised by Erik Velldal, Anna Smajdor, and Andrey Kutuzov), and a project on extraction of knowledge and bias from LLMs by PhD fellow Emil Poiesz (supervised by Ana Ozaki and Lilja Øvrelid, jointly with Samia Touileb from UiB). Both of these projects also have important connections to RT9 on Ethics. The supervisory teams also span different research groups and departments, and even institutions, in alignment with the cross-disciplinary focus of the projects.
Several activities are also tied to the ongoing Integreat reading group on preference optimization and alignment of LLMs. This topic has important research questions pertaining to both modeling and data creation that benefit from competence in across several areas – statistics, NLP, ethics, and more – making it a very good fit for collaboration within Integreat. The regular meetings of the reading group have attracted a lot of interest, from both core members of Integreat and more loosely affiliated researchers from associated research groups. The discussions in the reading group have resulted in concrete research plans that we will now take action to implement. One of these directions involves co-organizing a shared task on assessing LLMs' abilities to reproduce human preferences and justify their choices.
Another important Integreat collaboration has been centered on work by an Integreat-affiliated PhD Fellow Sondre Wold from the Language Technology Group (LTG) at IFI, UiO. Supervised by Integreat-members Lilja Øvrelid and Erik Velldal, his work focuses on the generalization abilities of transformer models and LLMs for tasks that require compositional reasoning, for which he has been working on combining information from Knowledge Graphs and LLMs.
Research theme leaders:
- Associate Professor Andrey Kutuzov, University of Oslo
- Professor Erik Velldal, University of Oslo