Title
The unseen frontier: pushing the limits of LLM sparsity with surrogate-free ADMM
Abstract
Neural network pruning is a promising technique to mitigate the excessive computational and memory requirements of large language models (LLMs). Despite its promise, however, it remains challenging to restore the original performance of unpruned models, and in fact, progress in this area has diminished, as conventional methods are seemingly unable to surpass moderate sparsity levels (50-60%) without severely degrading model performance. I will present a principled and effective method called ELSA and show that this work breaks through the current impasse, achieving extreme sparsity levels of up to 90% while retaining high model fidelity. This is done by identifying several limitations in current practice, all of which can be traced back to their reliance on a surrogate objective formulation. ELSA tackles this issue directly and effectively via standard and well-established constrained optimization techniques based on ADMM. Our extensive experiments across a wide range of models and scales show that ELSA achieves substantial improvements over existing methods; e.g., it achieves 7.8x less perplexity than the best existing method on LLaMA-2-7B at 90% sparsity. Furthermore, we present ELSA-L, a quantized variant that scales to extremely large models (27B), and establish its theoretical convergence guarantees. These results highlight meaningful progress in advancing the frontier of LLM sparsity, while promising that significant opportunities for further advancement may remain in directions that have so far attracted limited exploration.
Bio
Namhoon Lee is an Assistant Professor in the Department of Computer Science and Engineering at POSTECH where he leads the Computational Optimization Lab since 2024. His current research focuses on developing solutions for large-scale machine learning problems including compressing and training foundation models. Previously, he was a visiting researcher at Google, IST Austria, EPFL, ANU, and received his PhD from the University of Oxford in 2020.
Practical
This seminar is organised by the RT 5 Transfer learning at Integreat.
The Tuesday seminar series explores a wide range of topics relevant to our research themes. Each week, invited researchers present for 40 minutes, followed by a group discussion.
Seminars are open for attendance for everybody.
For those unable to attend in person: https://uio.zoom.us/j/62535744013
Point of contacts:
- Organiser and moderator: Associate Professor Ali Ramezani-Kebrya, University of Oslo
- Integreat Administrative Leader Maria Dikova, University of Oslo