AI NEWS.ccrubert-tiny
This is a very small distilled version of the bert-base-multilingual-cased model for Russian and English (45 MB, 12M parameters). There is also an **updated version of this model**, rubert-tiny2, with a larger vocabulary and better quality on practically all Russian NLU tasks. This model is useful if you want to fine-tune it for a relatively simple Russian task (e.g. NER or sentiment classification), and you care more about speed and size than about accuracy. It is approximately x10 smaller and faster than a base-sized BERT. Its `[CLS]` embeddings can be used as a sentence representation aligned between Russian and English. It was trained on the Yandex Translate corpus, OPUS-100 and Tatoeba, using MLM loss (distilled from bert-base-multilingual-cased), translation ranking loss, and `[CLS]` embeddings distilled from LaBSE, rubert-base-cased-sentence, Laser and USE.
번역 준비 중입니다. 원문을 표시합니다.
This is a very small distilled version of the bert-base-multilingual-cased model for Russian and English (45 MB, 12M parameters). There is also an **updated version of this model**, rubert-tiny2, with a larger vocabulary and better quality on practically all Russian NLU tasks. This model is useful if you want to fine-tune it for a relatively simple Russian task (e.g. NER or sentiment classification), and you care more about speed and size than about accuracy. It is approximately x10 smaller and faster than a base-sized BERT. Its `[CLS]` embeddings can be used as a sentence representation aligned between Russian and English. It was trained on the Yandex Translate corpus, OPUS-100 and Tatoeba, using MLM loss (distilled from bert-base-multilingual-cased), translation ranking loss, and `[CLS]` embeddings distilled from LaBSE, rubert-base-cased-sentence, Laser and USE.