Most real-life data are structured; for example, they are often relational and stored in multiple interlinked database tables, hierarchical as in XML, or in the form of knowledge graphs in formats such as RDF and Property Graphs. Logical languages can be used to systematise and formalise complex domain knowledge, for example, in the form of ontologies, and logical inference can be used to enrich explicit structured data with implicit information that logically follows from the explicit data and formalised domain knowledge. Most common Machine Learning methods today can neither ingest nor produce structured data directly, and they are unable to take into account existing domain knowledge formalised in a logical language.
Currently, ML usually deals with structured data by employing ad-hoc embedding techniques and essentially re-learning the structure from training examples. Moreover, recent advances show that ML can learn the impact of logical knowledge indirectly or directly, for example, via embedding methods that incorporate the constraints expressed by the logical knowledge in the loss function. However, indirect approaches lead to difficulties in learning good quality models, unsustainable training (models need to be retrained when the logical knowledge changes), and poor interpretability. Among the most promising approaches that allow MLto cope with structure and logical knowledge are ontology embeddings, graph neural networks (GNNs) of various forms, and representation learning in general. Combining these approaches with other existing methods (Inductive Logic Programming, Markov Logic Networks, ontology-based embeddings, etc), Integreat develops new methods to intertwine ML with logical inference on deterministic and probabilistic knowledge in logical form.
Current work in Integreat has four focus areas:
- Develop and study Graph Learning architectures (Egor Kostylev)
- Derive logical knowledge from an ML model (Ana Ozaki)
- Generalize sequence models to structure models (Basil Ell)
- Develop similarity measures for graph structured data (Elisabeth Wetzer)
Graph Learning Architectures (Egor Kostylev)
One current line of Integreat’s work for integrating structured data into ML aims to consolidate the theoretical foundations of Graph Learning, in particular GNNs, and to investigate the practical applicability of such models. For example, we study the properties of GNN variants, which include establishing their expressive and approximating power, developing ways to incorporate various types of knowledge, and analysing their applicability to practical problems, such as knowledge graph completion, query answering over incomplete graphs, and explainability of neural models. There are many GNN architectures for which seemingly small variations lead to dramatic changes in their properties. One of the key steps here is to design novel translation algorithms from GNN architectures to logical axioms and back. This work will allow us to bridge GNNs and classic logic-based Knowledge Representation formalisms in a principled way and give us an opportunity to leverage strong sides of both statistical and symbolic AI paradigms and to mitigate their weaknesses.
Deriving Logical Knowledge from ML Models (Ana Ozaki)
Large ML models can be overly complex to study without relying on ways of abstracting them. To better understand the information such models have learned, we will investigate active learning approaches for querying ML models, that is, approaches that probe ML models with a series of inputs and combine information from the outputs in a desired structured format. This includes the development of strategies for probing ML models, notably language models, so as to extract rules and decision trees that can help to detect trends and biases, in particular, to detect the presence of potentially harmful social biases. The improvement of question answering capabilities of language models in a variety of domains also facilitates the application of active learning approaches for knowledge acquisition. Building up on these improved capabilities, we are interested in extracting ontologies representing domain knowledge present in trained language models.
Generalizing Sequence Models to Structure Models (Basil Ell)
In another line of work we focus on generalizing sequence models, i.e., models that deal with sequences of tokens, such as language models, to structure models, i.e., models that can natively deal with graph-structured data. Where language models enable to estimate the probability of a sequence and to continue a started sequence, structure models will estimate the probability of a graph and continue/extend a graph. Structure models will be built on graph theory, representation learning, statistics, and logic. We envision that structure models can carry out both inductive and deductive reasoning, might more effectively include formally represented background knowledge, might be easier to update and more explainable than purely neural alternatives. We apply these models in the domain of chemistry and beyond.
Similarity Measures for Graph Structured Data (Elisabeth Wetzer)
Another focus in this research theme is learning similarity measures between labelled graphs of different sizes by incorporating additional knowledge about the relationship of features or labels that reside on their edges and nodes. One approach is to use representation learning techniques to learn representations of graph structures and to use non-Euclidean geometries to embed relational taxonomies of labels in which continuous distance measures between labels can be used to sample augmentations that integrate graph label information. Similarly, useful knowledge embeddings may be found for edge feature relationships that can be incorporated in learning meaningful similarity measures for graph structured data.
Research theme leaders:
- Researcher Basil Ell, University of Oslo
- Professor Martin Giese, University of Oslo
- Associate Professor Egor Kostylev, University of Oslo
- Associate Professor Ana Ozaki, University of Oslo
- Associate Professor Elisabeth Wetzer, UiT The Arctic University of Norway