distilcamembert-base

モデル紹介

We present a distillation version of the well named CamemBERT, a RoBERTa French model version, alias DistilCamemBERT. The aim of distillation is to drastically reduce the complexity of the model while preserving the performances. The proof of concept is shown in the DistilBERT paper and the code used for the training is inspired by the code of DistilBERT. The training for the distilled model (student model) is designed to be the closest as possible to the original model (teacher model). To perform this the loss function is composed of 3 parts: * DistilLoss: a distillation loss which measures the silimarity between the probabilities at the outputs of the student and teacher models with a cross-entropy loss on the MLM task ; * CosineLoss: a cosine embedding loss. This loss function is applied on the last hidden layers of student and teacher models to guarantee a collinearity between them ; * MLMLoss: and finaly a Masked Language Modeling (MLM) task loss to perform the student model with the original task of the teacher model. The final loss function is a combination of these three losses functions. We use the following ponderation:

主な特徴

特点摘要待依据来源补齐，不凭模型名称推断能力。

制約と注意点

具体的な制約を判断する出典が不足しています。実際の性能、遅延、データ方針、ライセンス、提供期間を確認してください。

出典別料金と費用分析

料金はチャネル・地域・設定によって異なります。各チャネルの料金は個別に表示し、提供元の統一公式料金とは見なしません。

公開評価と実行設定

暂无已采集的可核验公开评分。模型未上榜不代表能力较低。

よくある質問

出典と追跡記録

许可证：mit。模型介绍与第三方评测均以原来源的具体版本和方法为准。