(a) On MMLU-Pro (4k context length), Kimi Linear achieves 51.0 performance with similar speed as full attention. On RULER (128k context length), it shows Pareto-optimal performance (84.3) and 3.98x speedup. (b) Kimi Linear achieves 6.3x faster TPOT compared to MLA, offering significant speedups at long sequence lengths (1M tokens). Kimi Linear is a hybrid linear attention architecture that outperforms traditional full attention methods across various contexts, including short, long, and reinforcement learning (RL) scaling regimes. At its core is Kimi Delta Attention (KDA)—a refined version of Gated DeltaNet that introduces a more efficient gating mechanism to optimize the use of finite-state RNN memory. Kimi Linear achieves superior performance and hardware efficiency, especially for long-context tasks. It reduces the need for large KV caches by up to 75% and boosts decoding throughput by up to $6\times$ for contexts as long as 1M tokens.
Recently, the Muon optimizer has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: - **Weight Decay**: Critical for scaling to larger models - **Consistent RMS Updates**: Enforcing a consistent root mean square on model updates These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon is $\sim2\times$ more sample efficient than Adam with compute optimal training.
Recently, the Muon optimizer has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: - **Weight Decay**: Critical for scaling to larger models - **Consistent RMS Updates**: Enforcing a consistent root mean square on model updates These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon is $\sim2\times$ more sample efficient than Adam with compute optimal training.
Version 0.5 (git tag `v0.5`). See the changelog. Weight set ID: `2011c0cedde962d19fc43e74c1d409279e398d40d27a6fdb4bcb7b49ea2ea6fb` (see Identifying the weights). Also mirrored on ModelScope: kindlingai/glm-5.3-exlr8-k2-k3-k4. > **These weights cannot be used without E4M3 rounding.** A loader must round every value the trellis > codebook produces to FP8 E4M3 (round to nearest even, saturating) before applying the side vectors. The > encoder searched only over E4M3-rounded values, so the rounding is part of the format: a stock EXL3 decoder, > or any decoder that skips the rounding, reads measurably worse weights from the same bytes.
> **Tip**: Use `--no-think` or `/no_think` mode for faster responses and quality. The thinking tokens consume output budget at ~3.4 tok/s, so disabling thinking gives you more usable output** **This model does not refuse.** It provides complete, detailed, working answers on every tested category. Fine-tuned on the FableForge Mix A dataset (47,824 examples) of agent traces, shell commands, code generation, and multi-step reasoning tasks. 98.3% of the 2.8M formatted examples remain untapped for future training.
A 1.5B model scoring **96%** — punching far above its weight. 30-prompt evaluation (domain knowledge, uncensored domain + general, reasoning, tool use), 0–5 scoring, run on an NVIDIA A40 via Ollama.
A 1.5B model scoring **93%** — punching far above its weight. 30-prompt evaluation (domain knowledge, uncensored domain + general, reasoning, tool use), 0–5 scoring, run on an NVIDIA A40 via Ollama.
A 1.5B model scoring **93%** — punching far above its weight. 30-prompt evaluation (domain knowledge, uncensored domain + general, reasoning, tool use), 0–5 scoring, run on an NVIDIA A40 via Ollama.
A fine-tuned XLS-R 300M CTC model for Urdu automatic speech recognition. It transcribes **16 kHz mono** audio and includes an optional 5-gram KenLM decoder. **Best reported result:** 39.89% WER / 16.70% CER with KenLM decoding on the Urdu Common Voice 8.0 test set. See the Kaggle evaluation notebook for a reproducible example. > **Note:** After installing the packages in a notebook environment, restart the kernel before running the inference code.
Four ROCmFP4/ROCmFPX tiers of `IFM/K2-Horizon-MoVA-36B-A4B`, quantised from the official BF16 GGUF and measured on a Ryzen AI Max+ 395 (Radeon 8060S, gfx1151, 128 GB unified, ROCm 7.2.4). This is the **dual-expert** `k2_horizon` MoVA architecture — 36B total parameters, **~4B active** — with two independent expert systems: a 64-way MoVA value-expert bank (top-4) alongside a 100-way MoE feed-forward (top-8), 48 layers, hidden 2560, and it is a **reasoning** model. **Every number on this card was measured on these exact files.** Nothing is estimated. As an A4B MoE, it decodes far faster than a dense 32B on the same hardware. **Start with `FAST`** — fastest on both axes here, all four answer correctly. Take `AGENT` only if you specifically want 8-bit weights.