Crime and Punishment training data was taken from `gs://trax-ml/reformer/crime-and-punishment-2554.txt` and contains roughly 0.5M tokens. The ReformerLM model was trained in flax using colab notebook proposed by authors: https://colab.research.google.com/github/google/trax/blob/master/trax/models/reformer/text_generation.ipynb and the weights were converted to Hugging Face's PyTorch ReformerLM model `ReformerModelWithLMHead`. The model is a language model that operates on small sub-word units. Text can be generated as follows:

翻訳待ち・原文を表示

テキスト生成

入力テキスト→ 出力テキスト

料金未確認マルチモーダル・チャネル料金は個別に確認

詳細を見る ↗

比較に追加出典・文書 ↗

reformer-enwik8

Google

*enwik8* is a dataset based on Wikipedia and is often used to measure the model's ability to *compress* data, *e.g.* in the scope of the *Hutter prize*: https://en.wikipedia.org/wiki/Hutter_Prize. `reformer-enwik8` was pretrained on the first 90M chars of *enwik8* whereas the text was chunked into batches of size 65536 chars (=2^16). The model's weights were taken from https://console.cloud.google.com/storage/browser/trax-ml/reformer/enwik8 and converted to Hugging Face's PyTorch ReformerLM model `ReformerModelWithLMHead`. The model is a language model that operates on characters. Therefore, this model does not need a tokenizer. The following function can instead be used for **encoding** and **decoding**:

翻訳待ち・原文を表示

テキスト生成

入力テキスト→ 出力テキスト

料金未確認マルチモーダル・チャネル料金は個別に確認

詳細を見る ↗

比較に追加出典・文書 ↗

rembert

Google

Pretrained RemBERT model on 110 languages using a masked language modeling (MLM) objective. It was introduced in the paper Rethinking embedding coupling in pre-trained language models. A direct export of the model checkpoint was first made available in this repository. This version of the checkpoint is lightweight since it is meant to be finetuned for classification and excludes the output embedding weights. RemBERT's main difference with mBERT is that the input and output embeddings are not tied. Instead, RemBERT uses small input embeddings and larger output embeddings. This makes the model more efficient since the output embeddings are discarded during fine-tuning. It is also more accurate, especially when reinvesting the input embeddings' parameters into the core model, as is done on RemBERT. You should fine-tune this model for your downstream task. It is meant to be a general-purpose model, similar to mBERT. In our paper, we have successfully applied this model to tasks such as classification, question answering, NER, POS-tagging. For tasks such as text generation you should look at models like GPT2.

翻訳待ち・原文を表示

分類準備中

入力未確認→ 出力未確認

料金未確認マルチモーダル・チャネル料金は個別に確認

詳細を見る ↗

比較に追加出典・文書 ↗