Important: Note that this guest talk is on Tuesday 11.15-12.00
Title
Enabling AI systems to know when they don’t know: conformal prediction for uncertainty quantification and risk control
Abstract
Uncertainty quantification (UQ) is becoming increasingly important as modern AI systems, including large language models, are deployed in settings where inaccurate or overconfident predictions can have significant consequences. In safety-critical applications, it is not sufficient for a model to achieve high predictive accuracy; it must also communicate when its predictions can be trusted and provide uncertainty estimates that can support reliable decision-making. Conformal prediction (CP) offers an appealing framework for this purpose by transforming the outputs of essentially any predictive model into prediction sets with finite-sample statistical coverage guarantees under relatively mild assumptions.
In this talk, I will first introduce the basic principles of conformal prediction, with particular emphasis on the role of the nonconformity score and the distinction between the statistical validity of a conformal procedure and the informativeness of the resulting uncertainty. I will then discuss several directions pursued by our group to make conformal uncertainty more meaningful and useful in practice. I will examine how the uncertainty encoded by conformal prediction can itself be quantified, enabling more direct comparison with alternative UQ approaches, including Bayesian and evidential methods, and helping us better understand when and where conformal prediction may provide particular advantages.
I will then focus on the design of nonconformity scores as a central component governing the efficiency, adaptivity, and practical usefulness of conformal prediction. I will describe how evidential deep learning can provide richer information for constructing such scores, and how application-specific structure can be incorporated when uncertainty is not equally important everywhere. Finally, I will discuss extending these ideas to large language models, where uncertainty is semantic rather than simply probabilistic. I will present our recent work on combining semantic uncertainty measures with conformal calibration and risk-control mechanisms to identify unreliable responses, support principled abstention, and ultimately detect and control the risk of hallucination.
Bio
Reza Samavi is an Associate Professor in the Department of Electrical and Computer Engineering at Toronto Metropolitan University, a Faculty Affiliate with the Vector Institute for Artificial Intelligence, and an adjunct faculty member at McMaster University.
He is the Director of the Trustworthy Artificial Intelligence Lab (tailab.org), where his research focuses on trustworthy machine learning, particularly uncertainty quantification and conformal prediction, empirical and certified robustness, and the security and reliability of modern AI systems and large language models. His work has received recognition including the IBM Privacy Technology Research Award and the Privacy by Design Research Award from the Information and Privacy Commissioner of Ontario, Canada.
Speaker will present in person.
Practical
Integreat guest talks are open for attendance for everybody.
For those unable to attend in person: https://uio.zoom.us/j/65632027408
Point of contact
- Organiser and moderator: Ali Ramezani-Kebyra, University of Oslo
- Integreat Administrative Leader Maria Dikova, University of Oslo