docs: record Qwen3.8-27B running in 3.33GB on an RTX 3090

Measured end to end: split in 2.5 min, init 164s at 0.87GB (vision
resident), greedy decode peak 3.33GB, output "Paris".

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Yu Li
2026-08-18 14:27:41 -05:00
parent 9d3c1a5e0f
commit c3312e2198
+3 -2
View File
@@ -31,7 +31,7 @@
* [Bloome — build & run AI agent teams in the cloud, zero setup](https://bloome.im/app?ref=G6BYnov0&utm_medium=github&utm_source=lyogavin-airllm-ivor-202606)
## Updates
[2026/08] **Qwen3.8-27B** support: Qwen's new dense VL (Gated DeltaNet + Gated Attention, native vision). Same one-liner as every other model. Needs `transformers` 5.8+.
[2026/08] **Qwen3.8-27B** support: Qwen's new dense VL (Gated DeltaNet + Gated Attention, native vision) runs in **3.33GB** of VRAM, measured end to end on one RTX 3090. Needs `transformers` 5.8+.
[2026/07] **Kimi K3 (2.8T)** support: the largest open-source model runs on a single card in **3.72GB** of VRAM, measured end to end on one RTX 6000 Ada. Per-expert streaming loads only the experts a token actually routes to. K3 brings three requirements of its own: `pip install compressed-tensors flash-attn` (its model code mandates flash attention regardless of what you request), a CUDA 12 build of torch, since no prebuilt flash-attn wheel exists for CUDA 13 yet, and `transformers` 4.56.x, as its remote code does not load on 5.x.
@@ -105,7 +105,7 @@ MAX_LENGTH = 128
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")
# go bigger with the exact same one line:
#model = AutoModel.from_pretrained("Qwen/Qwen3.8-27B") # 27B dense VL
#model = AutoModel.from_pretrained("Qwen/Qwen3.8-27B") # 27B dense VL, 3.33GB
#model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B") # 235B, runs in ~3GB
#model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3") # 671B, runs in ~12GB
@@ -285,6 +285,7 @@ The trick: AirLLM only ever keeps **one layer on the GPU at a time**, so the VRA
|---|---|---|
| Qwen3 / Mistral / Phi (≈8B) | 8B | **~12 GB** |
| Qwen3-30B / Mixtral (MoE) | 3047B | **~13 GB** |
| Qwen3.8-27B (dense VL) | 27B | **3.33 GB** |
| Qwen3-235B (MoE) | 235B | **~3 GB** |
| Llama 3.x 70B (full precision) | 70B | **~4 GB** |
| Llama 3.1 405B | 405B | **~8 GB** |