**ARBERTv2** is the updated version of ARBERT model described in our **ACL 2021 paper** **"ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic"**. **ARBERTv2** is presented in our paper "ORCA: A Challenging Benchmark for Arabic Language Understanding" **ARBERTv2** is trained on MSA data **243 GB** of text and **27.8B tokens**. If you use our models (ARBERTv2) for your scientific publication, or if you find the resources in this repository useful, please cite our paper as follows (to be updated): We gratefully acknowledge support from the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada, Canadian Foundation for Innovation, ComputeCanada and UBC ARC-Sockeye. We also thank the Google TensorFlow Research Cloud (TFRC) program for providing us with free TPU access.

번역 대기 중 · 원문 표시

빈칸 및 마스크

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗

cheetah-base

UBC-NLP

基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。

텍스트 생성

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗

MARBERT

UBC-NLP

**MARBERT** is one of three models described in our **ACL 2021 paper** **"ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic"**. MARBERT is a large-scale pre-trained masked language model focused on both Dialectal Arabic (DA) and MSA. Arabic has multiple varieties. To train MARBERT, we randomly sample 1B Arabic tweets from a large in-house dataset of about 6B tweets. We only include tweets with at least 3 Arabic words, based on character string matching, regardless whether the tweet has non-Arabic string or not. That is, we do not remove non-Arabic so long as the tweet meets the 3 Arabic word criterion. The dataset makes up **128GB of text** (**15.6B tokens**). We use the same network architecture as ARBERT (BERT-base), but without the next sentence prediction (NSP) objective since tweets are short. See our repo for modifying BERT code to remove NSP. For more information about MARBERT, please visit our own GitHub repo. If you use our models (ARBERT, MARBERT, or MARBERTv2) for your scientific publication, or if you find the resources in this repository useful, please cite our paper as follows (to be updated): We gratefully acknowledge support from the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada, Canadian Foundation for Innovation, ComputeCanada and UBC ARC-Sockeye. We also thank the Google TensorFlow Research Cloud (TFRC) program for providing us with free TPU access.

번역 대기 중 · 원문 표시

빈칸 및 마스크

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗

MARBERTv2

UBC-NLP

**MARBERTv2** is one of three models described in our **ACL 2021 paper** **"ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic"**. We find that results with ARBERT and MARBERT on QA are not competitive, a clear discrepancy from what we have observed thus far on other tasksWe hypothesize this is because the two models are pre-trained with a sequence length of only 128, which does not allow them to sufficiently capture both a question and its likely answer within the same sequence window during the pre-training. To rectify this, we further pre-train the stronger model, MARBERT, on the same MSA data as ARBERT in addition to AraNews dataset but with a bigger sequence length of 512 tokens for 40 epochs. We call this further pre-trained model **MARBERTv2**, noting it has **29B tokens**. MARBERTv2 acquires best performance on all but one test set, where XLM-RLarge marginally outperforms us (only in F1).

번역 대기 중 · 원문 표시

빈칸 및 마스크

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗