Title
Register always matters: analysis of LLM pretraining data through the lens of language variation
Abstract
Pretraining data curation is a cornerstone in Large Language Model (LLM) development, leading to growing research on quality filtering of large web corpora. From statistical quality flags to LLM-based labelling systems, datasets are divided into categories, frequently reducing to a binary: those passing the filters are deemed as valuable examples, others are discarded as useless or detrimental. However, a more detailed understanding of the contribution of different kinds of texts to model performance is still largely lacking. We present the first study utilising registers or genres - a widely used standard in corpus linguistics to model linguistic variation - to curate pretraining datasets and investigate the effect of register on the performance of LLMs. We train small generative models with register classified data and evaluate them using standard benchmarks, and show that the register of pretraining data substantially affects model performance. We uncover surprising relationships between the pretraining material and the resulting models: using the News register results in
subpar performance, and on the contrary, including the Opinion class, covering texts such as reviews and opinion blogs, is highly beneficial. While a model trained on the entire unfiltered dataset outperforms those trained on datasets limited to a single register, combining well-performing registers like How-to-Instructions, Informational Description, and Opinion leads to major improvements. Furthermore, analysis of individual benchmark results reveals key differences in the strengths and drawbacks of specific register classes as pretraining data. These findings show that register is an important explainer of model variation and can facilitate more deliberate future data selection practices.
Bio
Sampo Pyysalo is a co-lead of the TurkuNLP group and has specialized in machine learning approaches to NLP since his MSc work. He has pursued this topic as a researcher in leading groups at the University of Tokyo (Japan), the National Centre for Text Mining (UK), and the University of Cambridge (UK). Since returning to the TurkuNLP group in 2019, he has established the leading Finnish academic research team creating deep learning-based language models, including FinBERT, FinGPT, Poro, and the Viking model family. He currently leads the work package on creating LLMs in the OpenEuroLLM project, which creates fully open foundation models for all European languages.
Practical
This seminar is organised by the RT 7 Natural language at Integreat.
The Tuesday seminar series explores a wide range of topics relevant to our research themes. Each week, invited researchers present for 40 minutes, followed by a group discussion.
Seminars are open for attendance for everybody.
For those unable to attend in person: https://uio.zoom.us/j/62535744013
Point of contacts:
- Organiser and moderator: Associate Professor Andrei Kutuzov, University of Oslo
- Integreat Administrative Leader Maria Dikova, University of Oslo