This RT aims to advance methods for explainability and fairness in complex AI systems. Explainability in AI helps us understand how AI makes decisions, ensuring it is transparent and trustworthy. Further, it is crucial to assess the fairness of such decisions where humans are involved and guarantee non-discriminative, equal treatment of all.
Deep learning is a powerful way of approximating functions. However, its solutions are often “black boxes”, which prevents their use in life- and safety-critical applications in health, transport, law and welfare. “White and grey” statistical models, such as generalised additive and vine copula models, are explainable, allow non-linear effects and complex dependence, but sacrifice complex interactions for interpretability.
Integreat will propose new methods to couple statistical models with deep learning, in order to exploit advantages from both. We will blend statistical sampling approaches with backpropagation-based feature explainability mechanisms, and study regularisations which imposes interpretability. Integreat will work on attributing the significance to each factor in explaining individual predictions taken by machine learning (ML) algorithms.
Methods we will develop originate from classical statistical testing, in-silico controlled knock-outs and game theory, like Shapley values, which we have started to extend to dependent features. We believe that the currently most promising approach is using non-parametric vine copulae. A further important element of trustworthiness of ML models is robustness to model assumptions and inference, as well as to adversarial attacks. We will leverage knowledge about specific artefacts (e.g., adversarial backdoor attacks, Clever Hans effects), to develop robust systems that are self-explanatory. We will develop new ways to understand deep learning systems, utilizing information theoretic concepts and knowledge-infused learning. The general limits of explainability in “black boxes” and the role of uncertainty quantification to provide explainability are further themes of research for us.
Fairness considerations are not to be neglected when making decisions for humans. If not taken into account, biases may persist or be amplified leading to undesirable or unfair decisions in our societies. We will propose approaches that detect and control the bias, starting from counterfactual fairness, that explicitly intervenes on protected features to quantify a potential bias. Furthermore, to determine which counterfactuals to test, we will detect factors that influence large numbers of variables in the causal graph representing the domain knowledge. We also investigate bias dynamics and different policies, e.g. affirmative actions, and action recommendations, e.g., recourse, which may change the population over time. Here, we study their long-term effects on fairness.
Another research direction explores fairness guarantees for decisions made under uncertainty of agent’s preferences over decision outcomes.
Research theme leaders:
- Dr. Philos Kjersti Aas, Norwegian Computing Center
- Associate Professor Anne-Marie George, University of Oslo
- Associate Professor Kristoffer Wickstrøm, UiT The Arctic University of Norway
Publications
2025
- Brandsæter, Andreas & Glad, Ingrid Kristine(2025). XAI in hindsight: Shapley values for explaining prediction accuracy. Expert Systems With Applications. ISSN 0957-4174.273. doi: 10.1016/j.eswa.2025.126845