About the project
Answering questions over KGs is a long-standing research topic not only in NLP but also in related fields such as information extraction and semantic technologies. One line of research has been to combine LMs and KGs through some interaction interface where the two modalities can be used jointly, for example through the use of GNNs. By doing this, one can use the knowledge stored in the graph as an external knowledge source for the LM when answering questions. A major limitation of this research direction is that the knowledge contained in most public KGs is typically already included to some extent in the data used for pre-training popular LMs. Consequently, most existing work following this direction cannot demonstrate much beyond absolute performance gain; what type of inferences these models are actually able to make when given access to information stored as a graph, if anything, remains an open research question. This project will use an automated rule-based dataset generation procedure to develop a Question Answering dataset and a corresponding graph structure that is based on information that is guaranteed to be unknown to the pretrained LM and that is annotated with the specific type of reasoning required in order to answer the question given the graph. These types are based on previous work in compositional generalisation and will test the ability of a system to reason over information that is not possible to answer without reasoning over syntactic elements in the graph.
Project team
- PhD Fellow Sondre Wold, University of Oslo, Department of Informatics
- Associate Professor Egor V. Kostylev, University of Oslo, Department of Informatics
- Professor Erik Velldal, University of Oslo, Department of Informatics
- Professor Lilja Øvrelid, University of Oslo, Department of Informatics