About the project
A key step to optimise and control the behaviour of Large Language Models (LLMs) is to align them with curated human feedback. In this fine tuning phase, the LLM is retrained on the basis of a set of data of the type (prompt/input, answer1, answer2, preferred answer), where humans are asked to prefer one of the two possible answers to the given input. Humans can be substituted by algorithms. Reinforcement Learning with Human Feedback (RLHF) was introduced to improve the alignment of LLMs with human preferences (Ouyang et al., 2022). More recently DPO (Rafailov et al., 2023) has been suggested, which avoids the reward model and Reinforcement Learning-based optimization. Learning from human preferences is a form of preference learning from incomplete ranking data. In this context, the mostly used method in LLM fine tuning is the Bradley Terry model. In this project we will explore alternative approaches, including Bayesian ones, with the purpose of preparing the data for fine tuning n a more precise way, as they represent the human (or algorithmical) preferences better.
Project team
Professor Erik Velldal, University of Oslo, Department of Informatics
- Professor Lilja Øvrelid, University of Oslo, Department of Informatics
- Postdoctoral Fellow Anne-Marie George, University of Oslo, Department of Informatics
Associate Professor Valeria Vitelli, University of Oslo, Institute of Basic Medical Sciences
Postdoctoral Fellow Vladislav Mikhailov, University of Oslo, Department of Informatics
Associate Professor Andrey Kutuzov, University of Oslo, Department of Informatics
Associate Professor Ida Scheel, University of Oslo, Department of Mathematics
Professor Arnoldo Frigessi, University of Oslo
PhD Fellow David Samuel, University of Oslo, Department of Informatics