Harry Cliff
e1f32459ee
* fix: MoE offload speed estimate accounts for DDR bandwidth bottleneck Previously, MoE offload speed was estimated using GPU VRAM bandwidth divided by active parameters with a flat 0.8 factor. This produced unrealistically high estimates (e.g. 80 tok/s for Qwen3-Next-80B on 16GB VRAM). In practice, active experts change per token and must be fetched from system RAM across the PCIe bus. The bottleneck is DDR bandwidth, not GPU bandwidth. Measured: 15.4 tok/s actual vs 80 tok/s estimated. New model: per-token time = expert_fetch_time + gpu_compute_time - expert_fetch = active_expert_gb / ddr_bandwidth (default 50 GB/s) - gpu_compute = active_expert_gb / gpu_bandwidth Result: 15.2 tok/s estimated vs 15.4 tok/s measured on RX 6900 XT. GPU mode for MoE models also corrected: uses full model size for bandwidth calculation since runtimes don't do expert-aware placement. DDR bandwidth is configurable via LLMFIT_DDR_BANDWIDTH env var. All 300 existing tests pass. Co-authored-by: AlexsJones <1235925@users.noreply.github.com> (Addresses review nits: extracted ddr_bandwidth_gbps() helper, removed unreachable!() in favor of config.run_mode_factors) 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com> * chore: enforce LF line endings via .gitattributes * fix: MoE GPU-mode TPS uses disk size + expert cache penalty Two changes to the MoE GPU-mode speed estimate: 1. Use model.estimate_disk_gb() instead of params_b * bpp — MoE routing tables and shared embeddings make the actual quantized file larger than total_params * bytes_per_param. 2. Apply 0.65 MoE overhead factor for cache thrashing from 128+ experts causing irregular memory access patterns. Result: Qwen3-30B-A3B Q2_K on RX 6900 XT now estimates 16.2 tok/s (measured: 16.3 tok/s) — was 337 tok/s before. All 300 tests pass. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com> * fix: scale MoE overhead by expert count + apply RunModeFactors to GPU path The flat 0.65 moe_overhead penalized all MoE models equally — Mixtral 8x7B (8 experts) got the same 35% penalty as Qwen3-30B-A3B (128 experts). Now scales by num_experts: 0.95 for ≤8, stepping down to 0.65 for 128+. Apply RunModeFactors only to the GPU MoE path (default factor 1.0 = no behavior change). The MoeOffload paths already model DDR bottleneck explicitly via physics-based formula, so applying mode_factor there would double-count the offload penalty (0.8 default would silently cut estimates by 20% and invalidate the calibration data). Also fixes misleading comment: estimate_disk_gb uses total params, not "GGUF block quantization overhead" as previously claimed. All 300 tests pass. Addresses review findings from adversarial code review of issue #449. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com> * docs: add DDR bandwidth env var override example to doc comment Per review feedback from three-foxes-in-a-trenchcoat. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com> * refactor: rename model_gb to active_gb + add efficiency coupling comment Per AlexsJones review feedback: - Rename model_gb -> active_gb to clarify it's active params, not full model - Replace hardcoded 0.55 with named fallback_efficiency + COUPLING comment 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com> * style: fix rustfmt formatting 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com> --------- Co-authored-by: RhoMan¢er <github.pentagon482@passmail.net> Co-authored-by: Letta Code <noreply@letta.com>
2 lines
19 B
Plaintext
2 lines
19 B
Plaintext
* text=auto eol=lf
|