Title
Feature learning in two-layer neural networks under structured data
Abstract
We study the effect of gradient-based optimization on feature learning in two-layer neural networks. We consider a setting where the number of samples is of the same order as the input dimension and show that, when the input data is isotropic, gradient descent always improves upon the initial random features model in terms of prediction risk, for a certain class of targets. Further leveraging the practical observation that data often contains additional structure, i.e., the input covariance has non-trivial alignment with the target, we prove that the class of learnable targets can be significantly extended, demonstrating a clear separation between kernel methods and two-layer neural networks in this regime. We additionally consider sparse settings and show that pruning methods can lead to optimal sample complexity.
Bio
Assistand Professor Murat A. Erdogu, University of Toronto (external link to the institutional page), Department of Computer Science, Department of Statistical Sciences
Vector Institute
Practical
The Tuesdays seminars series are devoted to various topics relevant for the Integreat´s research focus. Presenters from the Integreat community and beyond have 40 minutes to present, followed by a group discussion.
Seminars are open for attendance for everybody. The speaker presents online.
For those unable to attend in person: https://uio.zoom.us/j/65831656210
Point of contacts:
- Organiser and moderator: Associate Professor Ali Ramezani-Kebrya, University of Oslo
- Integreat Administrative Leader Maria Dikova, University of Oslo