This repository contains W4A16 quantized versions of Google's Gemma 3 instruction-tuned models, making them more accessible for deployment on consumer hardware while maintaining good performance. - **abhishekchohan/gemma-3-27b-it-quantized-W4A16** - **abhishekchohan/gemma-3-12b-it-quantized-W4A16** - **abhishekchohan/gemma-3-4b-it-quantized-W4A16** These models use W4A16 quantization via LLM Compressor: - Weights quantized to 4-bit precision - Activations use 16-bit precision - Significantly reduced memory requirements
Input Text · Image→ Output Text
Price not verifiedCheck multimodal and channel pricing separately
View details ↗This repository contains W4A16 quantized versions of Google's Gemma 3 instruction-tuned models, making them more accessible for deployment on consumer hardware while maintaining good performance. These models use W4A16 quantization via LLM Compressor: - Weights quantized to 4-bit precision - Activations use 16-bit precision - Significantly reduced memory requirements
Input Text · Image→ Output Text
Price not verifiedCheck multimodal and channel pricing separately
View details ↗基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。
Basic profile indexed · More details to follow
Category pending
Input Not verified→ Output Not verified
Price not verifiedCheck multimodal and channel pricing separately
View details ↗基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。
Basic profile indexed · More details to follow
Input Text→ Output Video
Price not verifiedCheck multimodal and channel pricing separately
View details ↗基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。
Input Text · Image→ Output Text
Price not verifiedCheck multimodal and channel pricing separately
View details ↗The scales were measured on the quantized model itself, not on the BF16 base, because activation-aware smoothing and int4 rounding change the K/V activations the served model actually writes to cache. Calibration replayed the exact same sequences that were used to calibrate the AWQ model: **112 packed sequences × 262,144 tokens (~29.4M tokens)** from the same Nemotron Post-Training v3 SFT blend (instruction-following chat, math, science, agentic code, multilingual), same shuffle seed, rendered with thinking enabled, so the measured KV distribution matches the deployment regime the weights were tuned for, at the exact context length the model serves at. While the model processed those sequences, we tracked the running maximum absolute value of the K states (after QK-norm) and V states each of the 16 global attention layers writes to the cache. Each scale is `amax / 448` (the fp8-e4m3 representable maximum), symmetric, per tensor, so no cached value can clip. The result is 32 per-layer scalars (`k_scale` / `v_scale`). Only the 16 global attention layers accumulate KV; the 48 DeltaNet linear-attention layers carry recurrent state and have no KV scales.
Input Text · Image→ Output Text
Price not verifiedCheck multimodal and channel pricing separately
View details ↗A 4-bit weight-only **GPTQ** quantization of **Qwen/Qwen3.8-27B** with an **fp8-e4m3 KV cache calibrated alongside the weights**. INT4 weights shrink the footprint ≈2.8× and fp8 KV halves the cache, so the model serves its native **262,144-token** context on a **single 48 GB GPU**. fp8 KV also makes YaRN extension to **524,288 tokens** fit the same card, and up to 1M tokens on 96 GB (retrieval beyond 262k unvalidated). Identical harness and hardware for base and quantized: lm-evaluation-harness with the vLLM 0.27.1 backend on a single NVIDIA RTX PRO 6000 (Blackwell, 96 GB), greedy decoding, seed 1234. The base runs with bf16 KV and the quantized model with fp8-e4m3 KV, each at its recommended serving configuration, so every Δ measures the full int4 + fp8-KV stack rather than weights alone. Generative tasks run with the model's native chat template, thinking enabled, `max_gen_toks=8192`; loglikelihood tasks run untemplated. `minerva_math500` reports `math_verify`, since strict `exact_match` measures answer format, not content, on thinking-model outputs. The BF16 base scores identically (9/9), showing no retrieval degradation through the int4 + fp8-KV stack at the context length it was calibrated at.
Input Text · Image→ Output Text
Price not verifiedCheck multimodal and channel pricing separately
View details ↗