We expand on our Olmo model series by introducing Olmo Hybrid, a new 7B hybrid RNN model in the Olmo family. Olmo Hybrid dramatically outperforms Olmo 3 in final performance, consistently showing roughly 2x data efficiency on core evals over the course of our pretraining run. We also show gains in performance on long-context benchmarks, as well as improved inference efficiency (throughput and memory) on long-context lengths by a factor of 75%. The training of our hybrid model makes use of Olmo 3 7B, except that we change the learning rate schedule to be a standard cosine schedule rather than the piecewise schedule used by Olmo 3. Additionally, we use the improved data mix of Olmo 3 32B instead of the Olmo 3 7B mix. The table below highlights the architecture differences in our hybrid model. Our overall layer matches the transformer architecture of Olmo 3 7B, except that 75% of layers use gated DeltaNet heads instead of attention heads. The layers alternate so that 3 contain DeltaNet sublayers followed by 1 with a multihead attention sublayer. In particular, each head uses gated DeltaNet heads, extended with negative eigenvaluesWe reduced the number of heads from 32 to 30 while keeping the head dimension fixed at 128 (effectively reducing dmodel from 4096 to 3840). Lastly, head dimension is doubled, which is the default behavior for DeltaNet.

번역 대기 중 · 원문 표시

텍스트 생성

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗

Olmo-Hybrid-Instruct-DPO-7B

allenai

基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。

텍스트 생성

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗

Olmo-Hybrid-Instruct-SFT-7B

allenai

基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。

텍스트 생성

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗

Olmo-Hybrid-Think-SFT-7B

allenai

基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。

기본 정보 등록 · 상세 정보 보완 예정

텍스트 생성

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗

OLMoE-1B-7B-0125

allenai

> OLMoE-1B-7B is a Mixture-of-Experts LLM with 1B active and 7B total parameters released in January 2025 (0125) that is 100% open-source. It is an improved version of OLMoE-09-24, see the paper appendix for details. Important branches: - `step1200000-tokens5033B`: Pretraining checkpoint used for annealing. There are a few more checkpoints after this one but we did not use them. - `main`: Checkpoint annealed from `step1200000-tokens5033B` for an additional 100B tokens (23,842 steps). We use this checkpoint for our adaptation (https://huggingface.co/allenai/OLMoE-1B-7B-0125-SFT & https://huggingface.co/allenai/OLMoE-1B-7B-0125-Instruct). - `fp32`: FP32 version of `main`. The model weights were stored in FP32 during training but we did not observe any performance drop from casting them to BF16 after training so we upload all weights in BF16. If you want the original FP32 checkpoint for `main` you can use this one. You will find that it yields slightly different results but should perform around the same on benchmarks.

번역 대기 중 · 원문 표시

텍스트 생성

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗

OLMoE-1B-7B-0125-DPO

allenai

基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。

기본 정보 등록 · 상세 정보 보완 예정

텍스트 생성

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗

OLMoE-1B-7B-0125-Instruct

allenai

OLMoE-1B-7B-0125-Instruct January 2025 is post-trained variant of the OLMoE-1B-7B January 2025 model, which has undergone supervised finetuning on an OLMo-specific variant of the Tülu 3 dataset and further DPO training on this dataset, and finally RLVR training using this data. Tülu 3 is designed for state-of-the-art performance on a diversity of tasks in addition to chat, such as MATH, GSM8K, and IFEval. Check out the OLMoE paper or Tülu 3 paper for more details! OLMo is a series of **O**pen **L**anguage **Mo**dels designed to enable the science of language models. These models are trained on the Dolma dataset. We are releasing all code, checkpoints, logs (coming soon), and associated training details. The core models released in this batch include the following: - **Model type:** A model trained on a mix of publicly available, synthetic and human-created datasets. - **Language(s) (NLP):** Primarily English - **License:** Apache 2.0 - **Finetuned from model:** allenai/OLMoE-1B-7B-0125-DPO

번역 대기 중 · 원문 표시

텍스트 생성

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗

OLMoE-1B-7B-0125-SFT

allenai

基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。

텍스트 생성

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗

OLMoE-1B-7B-0924

allenai

> OLMoE-1B-7B is a Mixture-of-Experts LLM with 1B active and 7B total parameters released in September 2024 (0924). It yields state-of-the-art performance among models with a similar cost (1B) and is competitive with much larger models like Llama2-13B. OLMoE is 100% open-source. This information and more can also be found on the **OLMoE GitHub repository**. - **Paper**: https://arxiv.org/abs/2409.02060 - **Pretraining** Checkpoints, Code, Data and Logs. - **SFT (Supervised Fine-Tuning)** Checkpoints, Code, Data and Logs. - **DPO/KTO (Direct Preference Optimization/Kahneman-Tversky Optimization)**, Checkpoints, Preference Data, DPO code, KTO code and Logs. Install `transformers` **from source** until a release after this PR & `torch` and run:

번역 대기 중 · 원문 표시

텍스트 생성

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗

OLMoE-1B-7B-0924-Instruct

allenai

> OLMoE-1B-7B-Instruct is a Mixture-of-Experts LLM with 1B active and 7B total parameters released in September 2024 (0924) that has been adapted via SFT and DPO from OLMoE-1B-7B. It yields state-of-the-art performance among models with a similar cost (1B) and is competitive with much larger models like Llama2-13B-Chat. OLMoE is 100% open-source. This information and more can also be found on the **OLMoE GitHub repository**. - **Paper**: https://arxiv.org/abs/2409.02060 - **Pretraining** Checkpoints, Code, Data and Logs. - **SFT (Supervised Fine-Tuning)** Checkpoints, Code, Data and Logs. - **DPO/KTO (Direct Preference Optimization/Kahneman-Tversky Optimization)**, Checkpoints, Preference Data, DPO code, KTO code and Logs. Branches: - `main`: Preference tuned via DPO model of https://hf.co/allenai/OLMoE-1B-7B-0924-SFT (`main` branch) - `load-balancing`: Ablation with load balancing loss during DPO starting from the `load-balancing` branch of https://hf.co/allenai/OLMoE-1B-7B-0924-SFT - `non-annealed`: Ablation starting from the `non-annealed` branch of https://hf.co/allenai/OLMoE-1B-7B-0924-SFT which is an SFT of the pretraining checkpoint prior to annealing (branch `step1200000-tokens5033B` of https://hf.co/allenai/OLMoE-1B-7B-0924) - `kto`: Ablation using KTO instead of DPO. This branch is the checkpoint after 5,000 steps with the RMS optimizer. The other `kto*` branches correspond to the other checkpoints mentioned in the paper.

번역 대기 중 · 원문 표시

텍스트 생성

입력 텍스트→ 출력 텍스트

요금 미확인멀티모달 및 채널 요금 별도 확인

자세히 보기 ↗

비교 추가출처 / 문서 ↗