AI NEWS.ccjetmoe-8b
1. JetMoE-8B is **trained with less than $ 0.1 million** 1 **cost but outperforms LLaMA2-7B from Meta AI**, who has multi-billion-dollar training resources. LLM training can be **much cheaper than people previously thought**. 2. JetMoE-8B is **fully open-sourced and academia-friendly** because: - It **only uses public datasets** for training, and the code is open-sourced. No proprietary resource is needed. - It **can be finetuned with very limited compute budget** (e.g., consumer-grade GPU) that most labs can afford. 3. JetMoE-8B **only has 2.2B active parameters** during inference, which drastically lowers the computational cost. Compared to a model with similar inference computation, like Gemma-2B, JetMoE-8B achieves constantly better performance.
翻訳準備中のため原文を表示しています
1. JetMoE-8B is **trained with less than $ 0.1 million** 1 **cost but outperforms LLaMA2-7B from Meta AI**, who has multi-billion-dollar training resources. LLM training can be **much cheaper than people previously thought**. 2. JetMoE-8B is **fully open-sourced and academia-friendly** because: - It **only uses public datasets** for training, and the code is open-sourced. No proprietary resource is needed. - It **can be finetuned with very limited compute budget** (e.g., consumer-grade GPU) that most labs can afford. 3. JetMoE-8B **only has 2.2B active parameters** during inference, which drastically lowers the computational cost. Compared to a model with similar inference computation, like Gemma-2B, JetMoE-8B achieves constantly better performance.