AI NEWS.ccSONAR_200_text_encoder
This is a port of the multilingual SONAR text encoder (https://huggingface.co/facebook/SONAR) to the `transformers` format from `fairseq2`. Its embeddings are expected be equal to those the official implementation (https://github.com/facebookresearch/SONAR), but the latter stays the source of truth. The encoder supports the same 202 languages as NLLB-200 (see also the source model card and FLORES-200 lang code mapping).
This is a port of the multilingual SONAR text encoder (https://huggingface.co/facebook/SONAR) to the `transformers` format from `fairseq2`. Its embeddings are expected be equal to those the official implementation (https://github.com/facebookresearch/SONAR), but the latter stays the source of truth. The encoder supports the same 202 languages as NLLB-200 (see also the source model card and FLORES-200 lang code mapping).