XLM-V is multilingual language model with a one million token vocabulary trained on 2.5TB of data from Common Crawl (same as XLM-R). It was introduced in the XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models paper by Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer and Madian Khabsa. **Disclaimer**: The team releasing XLM-V did not write a model card for this model so this model card has been written by the Hugging Face team. This repository documents all necessary integeration steps. > Large multilingual language models typically rely on a single vocabulary shared across 100+ languages. > As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged. > This vocabulary bottleneck limits the representational capabilities of multilingual models like XLM-R. > In this paper, we introduce a new approach for scaling to very large multilingual vocabularies by > de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity > to achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically > more semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V, > a multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we > tested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), and > named entity recognition (WikiAnn) to low-resource tasks (Americas NLI, MasakhaNER).
X-MOD is a multilingual masked language model trained on filtered CommonCrawl data containing 81 languages. It was introduced in the paper Lifting the Curse of Multilinguality by Pre-training Modular Transformers (Pfeiffer et al., NAACL 2022) and first released in this repository. Because it has been pre-trained with language-specific modular components (_language adapters_), X-MOD differs from previous multilingual models like XLM-R. For fine-tuning, the language adapters in each transformer layer are frozen. Because this model uses language adapters, you need to specify the language of your input so that the correct adapter can be activated:
