Original title: Porozumění mezijazykovým vlastnostem ve velkých vícejazyčných jazykových modelech
Translated title: Understanding cross-lingual abilities in large multilingual language models
Authors: Del Valle Girón, José Jacobo ; Libovický, Jindřich (advisor) ; Limisiewicz, Tomasz (referee)
Document type: Master’s theses
Year: 2023
Language: eng
Abstract: Cross-lingual abilities have been evident in large multilingual language models over the past few years. However, understanding why and under what circumstances they work is not entirely clear. In this work, we work towards a better understanding of these aspects in a specific subset of multilingual models, namely modular multilingual models with cross-lingual transfer learning abilities. We try to quantify claims in Pfeiffer et al. [2022] regarding their proposed model, X-MOD, as it was tested in a very specific setting which may not align with common low-resource settings. Specifically, we evaluate how the following factors may affect downstream performance: the amount of available pre- training data; hyperparameters such as number of training steps, checkpoint selection criteria, available overlapping lexicon. With the help of our findings, we also aim to provide guidelines on how to best use X-MOD, especially from a low-resource perspective. 1
Keywords: transfer learning|cross-lingual learning|low-resource|language models; transfer learning|cross-lingual learning|low-resource|language models

Institution: Charles University Faculties (theses) (web)
Document availability information: Available in the Charles University Digital Repository.
Original record: http://hdl.handle.net/20.500.11956/184175

Permalink: http://www.nusl.cz/ntk/nusl-534514


The record appears in these collections:
Universities and colleges > Public universities > Charles University > Charles University Faculties (theses)
Academic theses (ETDs) > Master’s theses