docs: record Qwen3.8-27B running in 3.33GB on an RTX 3090
Measured end to end: split in 2.5 min, init 164s at 0.87GB (vision resident), greedy decode peak 3.33GB, output "Paris". Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -31,7 +31,7 @@
|
||||
* [Bloome — build & run AI agent teams in the cloud, zero setup](https://bloome.im/app?ref=G6BYnov0&utm_medium=github&utm_source=lyogavin-airllm-ivor-202606)
|
||||
|
||||
## Updates
|
||||
[2026/08] **Qwen3.8-27B** support: Qwen's new dense VL (Gated DeltaNet + Gated Attention, native vision). Same one-liner as every other model. Needs `transformers` 5.8+.
|
||||
[2026/08] **Qwen3.8-27B** support: Qwen's new dense VL (Gated DeltaNet + Gated Attention, native vision) runs in **3.33GB** of VRAM, measured end to end on one RTX 3090. Needs `transformers` 5.8+.
|
||||
|
||||
[2026/07] **Kimi K3 (2.8T)** support: the largest open-source model runs on a single card in **3.72GB** of VRAM, measured end to end on one RTX 6000 Ada. Per-expert streaming loads only the experts a token actually routes to. K3 brings three requirements of its own: `pip install compressed-tensors flash-attn` (its model code mandates flash attention regardless of what you request), a CUDA 12 build of torch, since no prebuilt flash-attn wheel exists for CUDA 13 yet, and `transformers` 4.56.x, as its remote code does not load on 5.x.
|
||||
|
||||
@@ -105,7 +105,7 @@ MAX_LENGTH = 128
|
||||
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")
|
||||
|
||||
# go bigger with the exact same one line:
|
||||
#model = AutoModel.from_pretrained("Qwen/Qwen3.8-27B") # 27B dense VL
|
||||
#model = AutoModel.from_pretrained("Qwen/Qwen3.8-27B") # 27B dense VL, 3.33GB
|
||||
#model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B") # 235B, runs in ~3GB
|
||||
#model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3") # 671B, runs in ~12GB
|
||||
|
||||
@@ -285,6 +285,7 @@ The trick: AirLLM only ever keeps **one layer on the GPU at a time**, so the VRA
|
||||
|---|---|---|
|
||||
| Qwen3 / Mistral / Phi (≈8B) | 8B | **~1–2 GB** |
|
||||
| Qwen3-30B / Mixtral (MoE) | 30–47B | **~1–3 GB** |
|
||||
| Qwen3.8-27B (dense VL) | 27B | **3.33 GB** |
|
||||
| Qwen3-235B (MoE) | 235B | **~3 GB** |
|
||||
| Llama 3.x 70B (full precision) | 70B | **~4 GB** |
|
||||
| Llama 3.1 405B | 405B | **~8 GB** |
|
||||
|
||||
Reference in New Issue
Block a user