XLM-RoBERTa-XL model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Larger-Scale Transformers for Multilingual Masked Language Modeling by Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, Alexis Conneau and first released in this repository. Disclaimer: The team releasing XLM-RoBERTa-XL did not write a model card for this model so this model card has been written by the Hugging Face team. XLM-RoBERTa-XL is a extra large multilingual version of RoBERTa. It is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages.
XLM-V is multilingual language model with a one million token vocabulary trained on 2.5TB of data from Common Crawl (same as XLM-R). It was introduced in the XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models paper by Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer and Madian Khabsa. **Disclaimer**: The team releasing XLM-V did not write a model card for this model so this model card has been written by the Hugging Face team. This repository documents all necessary integeration steps. > Large multilingual language models typically rely on a single vocabulary shared across 100+ languages. > As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged. > This vocabulary bottleneck limits the representational capabilities of multilingual models like XLM-R. > In this paper, we introduce a new approach for scaling to very large multilingual vocabularies by > de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity > to achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically > more semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V, > a multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we > tested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), and > named entity recognition (WikiAnn) to low-resource tasks (Americas NLI, MasakhaNER).
X-MOD is a multilingual masked language model trained on filtered CommonCrawl data containing 81 languages. It was introduced in the paper Lifting the Curse of Multilinguality by Pre-training Modular Transformers (Pfeiffer et al., NAACL 2022) and first released in this repository. Because it has been pre-trained with language-specific modular components (_language adapters_), X-MOD differs from previous multilingual models like XLM-R. For fine-tuning, the language adapters in each transformer layer are frozen. Because this model uses language adapters, you need to specify the language of your input so that the correct adapter can be activated:
Adjust provider, category or input/output filters. New models are added continuously.
COST EXPLORER
Great models. Clear costs.
Cost analysis
Adjust monthly token usage to compare estimated API costs.
1 M = 1 million tokens · USD Compares base text-token rates within the same channel
Uses collected, comparable USD-per-million-token rates. Snapshots exclude cache discounts, tiers, extra reasoning output and taxes. Check current source prices.
MODEL HANDBOOK
Understand before you choose
Selection guide & FAQ
How should API prices be compared?+
Separate input, output and cache rates, then estimate the same workload. Image, audio and video rates may use items, duration or resolution rather than tokens.
Do open weights mean free commercial use?+
Not necessarily. Check each version’s license, commercial restrictions and policies. Self-hosting also incurs compute and maintenance costs.
Is the top-ranked model the best for me?+
Rankings depend on tasks, versions and methods. Compare accuracy, latency, reliability and costs on your own workload.
