Title
Using formal verification for text-to-SQL
Abstract
Large Language Models (LLMs) have made significant progress in assisting users to query databases in natural language. While LLM-based techniques provide state-of-the-art results on many standard benchmarks, there are a number of challenging problems that remain unresolved, which we discuss in this talk. First, we provide an overview of the problem of Text-to-SQL and its iterative version, where the agent can ask a user clarification questions. Second, we consider the main challenges that the agent faces in solving the problem, including complex relationships, ambiguous user queries, and weak evaluation procedures. Third, we consider formal verification–driven solutions for improving and evaluating Text-to-SQL. For example, a verification-driven performance evaluation of ten Text-to-SQL methods on the high-profile BIRD dataset suggests that existing methods can often overlook differences between the generated query and the ground-truth, leading to incorrect evaluation results.
Bio
Nina Narodytska is a staff researcher at VMware Research by Broadcom. Prior to VMware, she was a researcher at Samsung Research America. She completed postdoctoral studies in the Carnegie Mellon University School of Computer Science and the University of Toronto.
She received her PhD from the University of New South Wales. She was named one of "AI's 10 to Watch" researchers in the field of AI in 2013. She has presented invited talks and tutorials at FMCAD'18, CP'19, AAAI'20, IJCAI'20, LMML'22, CP'22, ESSAI'23, KR'24, CapeKR'25, and FMCAD'25.
Speaker is presenting in person.
Practical
The Tuesdays seminars series are devoted to various topics relevant for the Integreat's research focus. Presenters from the Integreat community and beyond have 40 minutes to present, followed by a group discussion.
Seminars are open for attendance for everybody.
For those unable to attend in person: https://uio.zoom.us/j/62535744013
Point of contacts:
- Organiser and moderator: Ana Ozaki, University of Oslo
- Integreat Administrative Leader Maria Dikova, University of Oslo