About the project
We have significantly accelerated ML in multi-node systems, i.e., handled scalability issues by developing highly communication-efficient and large-scale training in supervised learning, and training deep generative adversarial networks without sacrificing accuracy [1–3].
To make large-scale and data-parallel ML more efficient, we need to handle computation (in chip) and communication costs. In this project, our goal is to develop efficient in-chip elementary operations in lower precisions jointly with communication-efficient model training and inference to make AI training and inference energy-efficient.
- Fartash Faghri, Iman Tabrizian, Ilia Markov, Dan Alistarh, Daniel M. Roy, and Ali Ramezani-Kebrya. Adaptive gradient quantization for data-parallel SGD. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
- Ali Ramezani-Kebrya, Fartash Faghri, Ilya Markov, Vitalii Aksenov, Dan Alistarh, and Daniel M. Roy. NUQSGD: Provably communication-efficient data-parallel SGD via nonuniform quantization. Journal of Machine Learning Research (JMLR), 22(114):1–43, 2021.
- Ali Ramezani-Kebrya, Kimon Antonakopoulos, Igor Krawczuk, Justin Deschenaux, and Volkan Cevher. Distributed extra-gradient with optimal complexity and communication guarantees. In International Conference on Learning Representations (ICLR), 2023.
Project team
-
Associate Professor Ali Ramezani-Kebrya, University of Oslo, Department of Informatics