github-featured/alexsjones--llmfit
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity

Merge upstream main into ONNX catalog branch

Browse Source
This commit is contained in:
chinhkrb113
2026-07-06 09:55:51 +07:00
parent f2e7dbafce 74cd130619
commit ce89ca581d
62 changed files with 71049 additions and 187755 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/ISSUE_TEMPLATE/bug-report.yml
+129
View File
@@ -0,0 +1,129 @@
name: Bug Report
description: Report a bug in llmfit (incorrect fit analysis, hardware detection issues, TUI rendering problems, etc.)
title: "[Bug]:(short issue description)"
labels: ["bug", "needs-triage"]
body:
- type: markdown
attributes:
value: |
Thanks for reporting a bug! Please fill in the details below so we can reproduce and fix it.
- type: textarea
id: description
attributes:
label: Bug description
description: A clear and concise description of what the bug is.
placeholder: "e.g., llmfit reports 'Perfect' fit for a 70B model on 8GB RAM"
validations:
required: true
- type: textarea
id: expected
attributes:
label: Expected behavior
description: What did you expect to happen?
validations:
required: true
- type: textarea
id: actual
attributes:
label: Actual behavior
description: What actually happened instead?
validations:
required: true
- type: textarea
id: steps
attributes:
label: Steps to reproduce
description: How can we reproduce this issue?
placeholder: |
1. Run `llmfit --cli`
2. Look at model X
3. See incorrect fit level
validations:
required: true
- type: textarea
id: doctor-output
attributes:
label: Diagnostic report (`llmfit doctor`)
description: >
Paste the full output of `llmfit doctor` (v0.9.36+). It captures your
detected specs plus the raw nvidia-smi/rocm-smi/sysfs output detection
relies on — for hardware bugs this is usually all we need to reproduce
and write a regression test. On older versions paste `llmfit system`
instead.
render: text
placeholder: Output of `llmfit doctor` (or `llmfit system` on older versions)
validations:
required: true
- type: input
id: version
attributes:
label: llmfit version
description: Output of `llmfit --version`
placeholder: "llmfit 0.x.x"
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- Linux
- macOS (Intel)
- macOS (Apple Silicon)
- Windows
- WSL
validations:
required: true
- type: dropdown
id: component
attributes:
label: Affected component
description: Which part of llmfit is affected?
options:
- Hardware detection (RAM/CPU/GPU)
- Model fit analysis (scoring, fit levels)
- TUI (rendering, navigation, keybindings)
- CLI output (subcommands, table display)
- Model database (missing/incorrect model data)
- Download manager
- Speed estimation (tok/s)
- Provider detection (Ollama, llama.cpp, MLX, Docker)
- MCP server
- API server
- Other
validations:
required: true
- type: dropdown
id: mode
attributes:
label: Mode used
options:
- TUI (default)
- CLI (--cli flag)
- Subcommand (system, fit, search, list, info)
- API server (serve)
- MCP server
validations:
required: true
- type: textarea
id: gpu-info
attributes:
label: GPU details (if relevant)
description: Paste output of `nvidia-smi`, `rocm-smi`, or `system_profiler SPDisplaysDataType` if the bug is GPU-related.
render: text
- type: textarea
id: additional
attributes:
label: Additional context
description: Any other context, screenshots, or terminal output that might help.
.github/ISSUE_TEMPLATE/config.yml
+5
View File
@@ -0,0 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: Discussions
url: https://github.com/AlexsJones/llmfit/discussions
about: Ask questions, share ideas, or discuss llmfit usage
.github/ISSUE_TEMPLATE/documentation.yml
+54
View File
@@ -0,0 +1,54 @@
name: Documentation Improvement
description: Report unclear, missing, or incorrect documentation
title: "[Docs]: "
labels: ["documentation"]
body:
- type: markdown
attributes:
value: |
Help us improve the llmfit documentation. Report anything that's unclear, outdated, or missing.
- type: dropdown
id: doc-type
attributes:
label: Documentation area
description: Which documentation needs improvement?
options:
- README.md (installation, usage, key bindings)
- AGENTS.md (architecture, conventions)
- CONTRIBUTING.md (contribution guide)
- MODELS.md (model database info)
- API.md (API documentation)
- CHANGELOG.md
- Code comments / inline docs
- CLI help text (--help output)
- Other
validations:
required: true
- type: textarea
id: current
attributes:
label: Current state
description: What is currently documented (or missing)? Link to the specific section if possible.
placeholder: "e.g., The README doesn't explain how hardware simulation works with multi-GPU setups"
validations:
required: true
- type: textarea
id: improvement
attributes:
label: Suggested improvement
description: What should the documentation say instead? What's missing?
validations:
required: true
- type: dropdown
id: willingness
attributes:
label: Would you be willing to submit a PR for this?
options:
- "Yes"
- "No, just reporting"
validations:
required: true
.github/ISSUE_TEMPLATE/feature-request.yml
+74
View File
@@ -0,0 +1,74 @@
name: Feature Request
description: Suggest a new feature or enhancement for llmfit
title: "[Feature]: "
labels: ["enhancement"]
body:
- type: markdown
attributes:
value: |
We'd love to hear your ideas for improving llmfit. Please describe your feature request below.
- type: textarea
id: problem
attributes:
label: Problem or motivation
description: What problem does this feature solve? Is there a workflow that's currently difficult or missing?
placeholder: "e.g., I can't easily compare models side-by-side for my specific use case"
validations:
required: true
- type: textarea
id: solution
attributes:
label: Proposed solution
description: Describe the feature you'd like to see. Be as specific as possible.
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
description: Are there alternative approaches or workarounds you've considered?
- type: dropdown
id: area
attributes:
label: Feature area
description: Which part of llmfit would this feature affect?
options:
- TUI (new keybinding, view, or interaction)
- CLI (new subcommand or flag)
- Hardware detection (new GPU vendor, new hardware type)
- Model database (new models, new metadata fields)
- Fit analysis (scoring algorithm, new fit dimensions)
- Speed estimation (tok/s accuracy, new factors)
- Provider integration (Ollama, llama.cpp, MLX, Docker, LM Studio)
- Download manager
- Plan mode (hardware planning)
- Community leaderboard
- API / MCP server
- Desktop app (Tauri)
- Web interface
- Python bindings
- Cross-platform support
- Other
validations:
required: true
- type: dropdown
id: willingness
attributes:
label: Would you be willing to contribute this?
options:
- "Yes, I'd like to submit a PR"
- "I could help with guidance"
- "No, just suggesting"
validations:
required: true
- type: textarea
id: additional
attributes:
label: Additional context
description: Any mockups, screenshots, links, or references that help explain the feature.
.github/workflows/ci.yml
+8 -8
View File
@@ -32,7 +32,7 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Filter changed paths
id: filter
@@ -57,7 +57,7 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
@@ -120,7 +120,7 @@ jobs:
python-version: '3.12'
- name: Install uv
uses: astral-sh/setup-uv@v8.1.0
uses: astral-sh/setup-uv@v8.2.0
# Tests integration of Rust code with hatch_build.py and key Python tests when
# the full Python test suite is not required. Redundant if Python tests below run.
@@ -142,7 +142,7 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
@@ -160,7 +160,7 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
@@ -215,7 +215,7 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
@@ -268,7 +268,7 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Set up Python
uses: actions/setup-python@v6
@@ -276,7 +276,7 @@ jobs:
python-version: '3.12'
- name: Install uv
uses: astral-sh/setup-uv@v8.1.0
uses: astral-sh/setup-uv@v8.2.0
- name: Ruff lint
run: uv run --only-group=dev ruff check .
.github/workflows/docker.yml
+2 -2
View File
@@ -21,7 +21,7 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Set up QEMU
uses: docker/setup-qemu-action@v4
@@ -48,7 +48,7 @@ jobs:
type=raw,value=latest,enable={{is_default_branch}}
- name: Build and push Docker image
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: .
platforms: linux/amd64,linux/arm64
.github/workflows/release-desktop.yml
+2 -2
View File
@@ -25,7 +25,7 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
@@ -79,7 +79,7 @@ jobs:
path: artifacts
- name: Create GitHub Release
uses: softprops/action-gh-release@b4309332981a82ec1c5618f44dd2e27cc8bfbfda # v3.0.0
uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b # v3.0.1
with:
generate_release_notes: true
files: artifacts/**/*.tar.gz
.github/workflows/release-please.yml
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
config-file: release-please-config.json
manifest-file: .release-please-manifest.json
- uses: actions/checkout@v6
- uses: actions/checkout@v7.0.0
if: steps.release.outputs.pr
with:
ref: ${{ fromJson(steps.release.outputs.pr).headBranchName }}
.github/workflows/release.yml
+12 -6
View File
@@ -51,6 +51,11 @@ jobs:
os: ubuntu-latest
use-cross: true
# Linux RISCV64 (glibc)
- target: riscv64gc-unknown-linux-gnu
os: ubuntu-latest
use-cross: true
# macOS Intel (cross-compiled from ARM64 runner)
- target: x86_64-apple-darwin
os: macos-latest
@@ -75,7 +80,7 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ inputs.tag || github.ref }}
@@ -241,7 +246,7 @@ jobs:
path: artifacts
- name: Create GitHub Release
uses: softprops/action-gh-release@b4309332981a82ec1c5618f44dd2e27cc8bfbfda # v3.0.0
uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b # v3.0.1
with:
tag_name: ${{ inputs.tag || github.ref_name }}
generate_release_notes: true
@@ -263,12 +268,12 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ inputs.tag || github.ref }}
- name: Set up uv
uses: astral-sh/setup-uv@v8.1.0
uses: astral-sh/setup-uv@v8.2.0
- name: Build Python wheels
shell: bash
@@ -280,6 +285,7 @@ jobs:
TARGETS=(
"manylinux_2_17_x86_64:x86_64-unknown-linux-gnu:llmfit:tar.gz"
"manylinux_2_17_aarch64:aarch64-unknown-linux-gnu:llmfit:tar.gz"
"manylinux_2_39_riscv64:riscv64gc-unknown-linux-gnu:llmfit:tar.gz"
"musllinux_1_2_x86_64:x86_64-unknown-linux-musl:llmfit:tar.gz"
"musllinux_1_2_aarch64:aarch64-unknown-linux-musl:llmfit:tar.gz"
"macosx_10_12_x86_64:x86_64-apple-darwin:llmfit:tar.gz"
@@ -323,7 +329,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ inputs.tag || github.ref }}
@@ -354,7 +360,7 @@ jobs:
steps:
- name: Checkout tap
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
repository: AlexsJones/homebrew-llmfit
token: ${{ secrets.HOMEBREW_TAP_TOKEN }}
.github/workflows/weekly-model-update.yml
+85
View File
@@ -0,0 +1,85 @@
name: Weekly Model Update
'on':
schedule:
- cron: '0 2 * * 1'
workflow_dispatch:
concurrency:
group: weekly-model-update
cancel-in-progress: false
permissions:
contents: write
pull-requests: write
jobs:
update-models:
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.12'
# HF_TOKEN raises HuggingFace API rate limits and unlocks gated-model
# metadata (the scraper warns-and-skips gated repos without it).
- name: Refresh model database
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
run: python3 scripts/scrape_hf_models.py -n 5000
# Refresh the community leaderboard cache alongside the catalog. The
# measured tok/s rows also feed the estimate-calibration test below,
# so this doubles as a weekly accuracy audit of estimate_tps.
- name: Refresh benchmark cache
env:
LOCALMAXXING_API_KEY: ${{ secrets.LOCALMAXXING_API_KEY }}
run: python3 scripts/scrape_benchmarks.py --limit 200
- name: Validate generated JSON
run: |
python3 -m json.tool llmfit-core/data/hf_models.json > /dev/null
python3 -m json.tool llmfit-core/data/benchmark_cache.json > /dev/null
- name: Set up Rust
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
- name: Validate against JSON Schema
run: cargo test -p llmfit-core hf_models_match_schema
# Weekly accuracy audit: replay the refreshed measurements through
# estimate_tps; fails on systematic estimator bias (see fit.rs).
- name: Calibrate estimates against measured benchmarks
run: cargo test -p llmfit-core test_estimate_tps_calibration -- --nocapture
- name: Check for changes
id: changes
run: |
if git diff --quiet; then
echo "changed=false" >> "$GITHUB_OUTPUT"
else
echo "changed=true" >> "$GITHUB_OUTPUT"
fi
- name: Create or update pull request
if: steps.changes.outputs.changed == 'true'
uses: peter-evans/create-pull-request@22a9089034f40e5a961c8808d113e2c98fb63676 # v7
with:
token: ${{ github.token }}
commit-message: "chore(models): Weekly model and benchmark data refresh"
title: "[Automated] Weekly model and benchmark data refresh"
body: |
Automated weekly refresh of model metadata (`scripts/scrape_hf_models.py`) and the localmaxxing benchmark cache (`scripts/scrape_benchmarks.py`).
**Note:** CI checks may not automatically trigger with the default `github.token`. If checks don't run, please close and reopen the PR or manually trigger them.
branch: automated/weekly-model-update
delete-branch: true
labels: |
automated
data-update
.release-please-manifest.json
+1 -1
View File
@@ -1,3 +1,3 @@
{
".": "0.9.29"
".": "0.9.38"
}
AGENTS.md
+3 -3
View File
@@ -28,7 +28,7 @@ hardware.rs SystemSpecs::detect() reads RAM/CPU via sysinfo crate.
On unified memory (Apple Silicon), VRAM = system RAM.
No async. No unsafe.
models.rs LlmModel struct. ModelDatabase loads from data/hf_models.json
models.rs LlmModel struct. ModelDatabase loads from llmfit-core/data/hf_models.json
embedded via include_str!() at compile time. No runtime file I/O.
fit.rs FitLevel enum (Perfect, Good, Marginal, TooTight).
@@ -64,7 +64,7 @@ tui_events.rs Keyboard event handling with crossterm. Two modes: Normal
## Model database
- Source: `data/hf_models.json` (33 models).
- Source: `llmfit-core/data/hf_models.json` (33 models).
- Generated by `scripts/scrape_hf_models.py` (Python, stdlib only, no pip deps).
- Embedded at compile time via `include_str!("../data/hf_models.json")`.
- Schema per entry: name, provider, parameter_count, min_ram_gb, recommended_ram_gb, min_vram_gb, quantization, context_length, use_case.
@@ -98,7 +98,7 @@ The scraper has hardcoded fallback entries for gated models that require authent
1. Add the model's HuggingFace repo ID to `TARGET_MODELS` in `scripts/scrape_hf_models.py`.
2. If the model is gated (requires HF auth), add a fallback entry to the `FALLBACK` dict in the same script.
3. Run `python3 scripts/scrape_hf_models.py`.
4. Verify the output in `data/hf_models.json`.
4. Verify the output in `llmfit-core/data/hf_models.json`.
5. Run `cargo build` to verify compilation.
## Adding a new filter
API.md
+19
View File
@@ -34,6 +34,25 @@ http://127.0.0.1:8787
To expose outside localhost, pass `--host 0.0.0.0`.
### Unix domain socket
For same-host consumers that should not touch the network at all (e.g. a
sidecar in a `hostNetwork` Kubernetes pod, where a TCP bind would land on the
node's loopback), listen on a Unix socket instead:
```sh
llmfit serve --unix-socket /run/llmfit/llmfit.sock
```
The socket is created with mode `0660`; a stale socket file from a previous
instance is replaced automatically. All HTTP endpoints are identical:
```sh
curl --unix-socket /run/llmfit/llmfit.sock http://localhost/api/v1/system
```
`--unix-socket` conflicts with `--host`/`--port` and is unix-platforms only.
If you are building from source and want the dashboard embedded in `llmfit`, build web assets first:
```sh
AUDIO_SUPPORT.md
+129
View File
@@ -0,0 +1,129 @@
# Audio Model Support — Implementation Plan
This document describes the Rust changes needed to fully support
`pipeline_tag: automatic-speech-recognition` models (Whisper variants).
The JSON data additions (`llmfit-core/data/hf_models.json`) in this branch are ready.
The Rust integration changes below are the next step — open for discussion.
## Data changes (this branch)
`llmfit-core/data/hf_models.json` — 4 new entries with:
- `pipeline_tag: "automatic-speech-recognition"`
- `capabilities: ["audio"]`
- New fields (custom, don't break existing Rust deserialization via `#[serde(default)]`):
- `_audio_rtf_gpu: f64` — Real-Time Factor on GPU (0.007 = 7x realtime)
- `_audio_rtf_cpu: f64` — RTF on CPU
- `_audio_vram_gb: f64` — VRAM needed at F16
- `_audio_backends: [str]` — supported servers
## Rust changes needed
### 1. `llmfit-core/src/models.rs`
Add `Capability::Audio` to the `Capability` enum:
```rust
pub enum Capability {
Vision,
ToolUse,
Reasoning,
Embedding,
Audio, // ← new
}
```
Extend `LlmModel` deserialization to accept the new `_audio_*` fields:
```rust
// Inside LlmModel or a companion AudioMeta struct
#[serde(default)]
pub audio_rtf_gpu: Option<f64>,
#[serde(default)]
pub audio_rtf_cpu: Option<f64>,
#[serde(default)]
pub audio_vram_gb: Option<f64>,
#[serde(default)]
pub audio_backends: Vec<String>,
```
Add `UseCase::Audio` variant and detect it from `pipeline_tag`:
```rust
pub enum UseCase {
General, Coding, Reasoning, Chat, Multimodal, Embedding,
Audio, // ← new
}
impl UseCase {
pub fn from_model(model: &LlmModel) -> Self {
// existing checks …
if model.pipeline_tag.as_deref() == Some("automatic-speech-recognition")
|| model.capabilities.contains(&Capability::Audio)
{
UseCase::Audio
} else { /* existing logic */ }
}
}
```
### 2. `llmfit-core/src/fit.rs`
Audio models don't use tok/s — they use RTF (Real-Time Factor).
Add an `AudioFit` struct separate from `ModelFit`:
```rust
pub struct AudioFit {
pub model: LlmModel,
pub rtf_gpu: Option<f64>,
pub rtf_cpu: f64,
pub fits_vram: bool,
pub fits_ram: bool,
pub recommended_backend: String,
}
```
Scoring for audio: `score = accuracy_tier - latency_penalty - vram_penalty`.
Lower RTF = faster = better score.
### 3. `llmfit-core/src/providers.rs`
Add Whisper server provider detection:
```rust
/// mlx-openai-server Whisper endpoint (Apple Silicon path).
pub struct MlxWhisperProvider;
impl ModelProvider for MlxWhisperProvider {
fn check_running(&self) -> Option<ProviderInfo> {
probe_http("http://localhost:18000/v1/audio/transcriptions")
.map(|_| ProviderInfo { name: "mlx-openai-server", port: 18000 })
}
}
/// faster-whisper-server (Docker, NVIDIA/CPU path).
pub struct FasterWhisperProvider;
impl ModelProvider for FasterWhisperProvider {
fn check_running(&self) -> Option<ProviderInfo> {
probe_http("http://localhost:8000/health")
.map(|_| ProviderInfo { name: "faster-whisper-server", port: 8000 })
}
}
```
### 4. `llmfit-tui/src/main.rs` / CLI
Add `llmfit fit --kind audio` / `llmfit recommend --kind audio` to filter
to ASR models only (useful for the TLDR smart installer use case).
```bash
llmfit --json fit --kind audio -n 3
```
## Why this matters
Projects like [TLDR](https://github.com/melnikaite/tldr-free) (Chrome extension
that summarizes pages/videos) use an OpenAI-compatible Whisper backend for
audio transcription. Choosing the right Whisper model for your hardware is
just as confusing as choosing an LLM — RTF on a GTX 1660 Ti vs. Apple M3 Pro
is wildly different. This brings llmfit's hardware-aware recommendations to
the audio domain.
CHANGELOG.md
+148
View File
@@ -1,5 +1,153 @@
# Changelog
## [0.9.38](https://github.com/AlexsJones/llmfit/compare/v0.9.37...v0.9.38) (2026-07-04)
### Bug Fixes
* **docker:** drop COPY of removed root data/ directory ([a4445c7](https://github.com/AlexsJones/llmfit/commit/a4445c70885ee399a5b8e39b4ded938a09230064))
* **docker:** drop COPY of removed root data/ directory ([5bb514d](https://github.com/AlexsJones/llmfit/commit/5bb514d130964311617269b0cffd4663a31522e3))
## [0.9.37](https://github.com/AlexsJones/llmfit/compare/v0.9.36...v0.9.37) (2026-07-03)
### Features
* **calibration:** weekly benchmark refresh + estimate-accuracy audit ([#112](https://github.com/AlexsJones/llmfit/issues/112)/[#119](https://github.com/AlexsJones/llmfit/issues/119)) ([5548723](https://github.com/AlexsJones/llmfit/commit/55487236685fde60a2be09580fe892daeb6a9b8b))
* **calibration:** weekly benchmark refresh + estimate-accuracy audit against measured data ([6c40dcf](https://github.com/AlexsJones/llmfit/commit/6c40dcf3127ada85a47cb4023c41fee28d33ec3e))
* **claim:** JSON output for programmatic consumers (llmfit-dra M0) ([f469c22](https://github.com/AlexsJones/llmfit/commit/f469c22cb48cc8e15b33119cd7cf166474ab50a7))
* **claim:** machine-readable JSON output via the global --json flag ([1636e51](https://github.com/AlexsJones/llmfit/commit/1636e51bb8b452b6c9ab8e47f38e0ac764f25f2f))
### Bug Fixes
* **calibration:** exclude speculative-decoding/MTP runs from the accuracy audit ([1c058aa](https://github.com/AlexsJones/llmfit/commit/1c058aa22bd57456487a15aa618155fd1340ef26))
## [0.9.36](https://github.com/AlexsJones/llmfit/compare/v0.9.35...v0.9.36) (2026-07-03)
### Features
* add 'llmfit doctor' hardware diagnostic report for bug reports ([53663bc](https://github.com/AlexsJones/llmfit/commit/53663bcb9935d79c6f959b357d99d622d6cbcef1))
* add `llmfit doctor` hardware diagnostic report ([bcd7dc0](https://github.com/AlexsJones/llmfit/commit/bcd7dc0c89b48909cfdd0ce20813bd58a2e84aeb))
* add issue templates for bug reports ([bab1fe9](https://github.com/AlexsJones/llmfit/commit/bab1fe9166a037aac1e3d4da10484e0f91812bc1))
* request 'llmfit doctor' output in the bug report template ([b4069d9](https://github.com/AlexsJones/llmfit/commit/b4069d927b9f7d271664d9e24324b3fe5716fa29))
* **scoring:** benchmark-informed task alignment per use case ([ce6d37e](https://github.com/AlexsJones/llmfit/commit/ce6d37e034b3978e687a00589ac7ce10e81ae7e9))
* **scoring:** benchmark-informed task alignment per use case (issue [#150](https://github.com/AlexsJones/llmfit/issues/150)) ([c994615](https://github.com/AlexsJones/llmfit/commit/c994615357f1b2577ba5096d7d176cf928f27a56))
* serve over a Unix domain socket (--unix-socket) ([de62c2b](https://github.com/AlexsJones/llmfit/commit/de62c2bc7e2af432771e0036b35e5fb76a50ed7e))
* serve over a Unix domain socket (--unix-socket) ([03d6ca5](https://github.com/AlexsJones/llmfit/commit/03d6ca5170df4f924b3bf24e94f27035fa26bb6a))
* show usable context per model (native→achievable) in TUI, CLI, and JSON ([e33a3a3](https://github.com/AlexsJones/llmfit/commit/e33a3a31be5a7a7b3953c4f33a02732c1790353c)), closes [#621](https://github.com/AlexsJones/llmfit/issues/621)
* usable-context display — native→achievable (closes [#621](https://github.com/AlexsJones/llmfit/issues/621)) ([1706929](https://github.com/AlexsJones/llmfit/commit/1706929ea367fd137117173f56827640bc04404c))
* user-local custom model overlay (closes [#451](https://github.com/AlexsJones/llmfit/issues/451)) ([decef82](https://github.com/AlexsJones/llmfit/commit/decef827b3785df9a5c785b49ed128f98d5c3aaa))
* user-local custom model overlay (custom_models.json) ([e55c8de](https://github.com/AlexsJones/llmfit/commit/e55c8dec388a1788b6093e0ef4c1091149cfe3d9)), closes [#451](https://github.com/AlexsJones/llmfit/issues/451)
### Bug Fixes
* guard optional attribute lookups in generated claim CEL ([c6f481f](https://github.com/AlexsJones/llmfit/commit/c6f481f27557d3e0fd6fc584fd1b1fa008ff95a4))
* guard optional attribute lookups in generated claim CEL ([a6f3c6a](https://github.com/AlexsJones/llmfit/commit/a6f3c6a6bc11e3a4c141dc7e0c3891263958a712))
* **hardware:** correct Intel GPU detection — iGPUs get the unified RAM pool, dGPUs get Vulkan VRAM ([4834767](https://github.com/AlexsJones/llmfit/commit/4834767462a7ac939ddb0306796e9661913f8310))
* **hardware:** detect generic-named MI50-class accelerators (issue [#638](https://github.com/AlexsJones/llmfit/issues/638)) ([180f484](https://github.com/AlexsJones/llmfit/commit/180f4844630e4ca08e8971482a615564d5be64f8))
* **hardware:** don't drop large-VRAM AMD accelerators with generic ROCm names ([bf16eb8](https://github.com/AlexsJones/llmfit/commit/bf16eb8e87b31e1c0f5c6154e82f7b8967d54085))
* **hardware:** Intel iGPU unified-memory pool + discrete Arc VRAM via Vulkan (issue [#609](https://github.com/AlexsJones/llmfit/issues/609)) ([a120aa3](https://github.com/AlexsJones/llmfit/commit/a120aa38d9410b2d874bbe7c70653b4f2c1920b7))
* **schema:** add 'audio' to capability enum ([a812232](https://github.com/AlexsJones/llmfit/commit/a812232f0eb02dc66544e508fe6e5e587acbd1f0))
## [0.9.35](https://github.com/AlexsJones/llmfit/compare/v0.9.34...v0.9.35) (2026-07-02)
### Features
* add claim subcommand generating Kubernetes DRA ResourceClaims ([08547ef](https://github.com/AlexsJones/llmfit/commit/08547efa277dd3f733ecdf72312a3376fbc2a01b))
* add claim subcommand generating Kubernetes DRA ResourceClaims ([3369a49](https://github.com/AlexsJones/llmfit/commit/3369a492630f357c65e2344de18b5a2e35ed4950))
* expose GPU memory bandwidth in system JSON and add Strix Halo entries ([3bfd334](https://github.com/AlexsJones/llmfit/commit/3bfd33406b071f48fddc7fb6e99fe62e55096511))
### Bug Fixes
* reset list cursor to top when the search query changes ([5e4dc7b](https://github.com/AlexsJones/llmfit/commit/5e4dc7b1e578c5a2366a8f7b94220d68178e2fae))
* reset list cursor to top when the search query changes ([43884f5](https://github.com/AlexsJones/llmfit/commit/43884f500a9b392b03bc2d2bc539ba6430d954fd)), closes [#657](https://github.com/AlexsJones/llmfit/issues/657)
## [0.9.34](https://github.com/AlexsJones/llmfit/compare/v0.9.33...v0.9.34) (2026-06-28)
### Bug Fixes
* pin Docker builder to bookworm to match runtime glibc ([f13a633](https://github.com/AlexsJones/llmfit/commit/f13a6332e62713b22bf96b7a26b551f43caed831))
* pin Docker builder to bookworm to match runtime glibc ([48caf17](https://github.com/AlexsJones/llmfit/commit/48caf178baaf7e7c0f81bfd30f4972b404528cf7)), closes [#655](https://github.com/AlexsJones/llmfit/issues/655)
## [0.9.33](https://github.com/AlexsJones/llmfit/compare/v0.9.32...v0.9.33) (2026-06-23)
### Bug Fixes
* **docker:** bump builder image to rust 1.95 (sysinfo MSRV) ([b63a142](https://github.com/AlexsJones/llmfit/commit/b63a14244d93fb1fdd50b2f49c96abaac872b977))
* **docker:** bump builder image to rust 1.95 (sysinfo MSRV) ([b943f3a](https://github.com/AlexsJones/llmfit/commit/b943f3a84a63e77a740acacd38a09c69774dda8c))
## [0.9.32](https://github.com/AlexsJones/llmfit/compare/v0.9.31...v0.9.32) (2026-06-23)
### Features
* add NCAI VAETKI models to curated model database ([b3c04a8](https://github.com/AlexsJones/llmfit/commit/b3c04a82055875b27a8808131df8f41e544ca759))
* **analysis:** add InstalledIndex and build_model_fits to consolidate provider detection ([33dce8b](https://github.com/AlexsJones/llmfit/commit/33dce8bbb9f8a527c48fec07152fa2c3a0983784))
* **data:** add Whisper/ASR model entries + audio support roadmap ([27bae48](https://github.com/AlexsJones/llmfit/commit/27bae4878f798c7069f936de6fa9ed5b1dcc3312))
* **python:** add riscv64 wheel build ([e12bb85](https://github.com/AlexsJones/llmfit/commit/e12bb85abdd752e632930c3086a4afc935e9a4a6))
* **python:** add riscv64 wheel build ([cb377f4](https://github.com/AlexsJones/llmfit/commit/cb377f45595b745d7bf0963ab1c86b807c210b01))
### Bug Fixes
* bundle WebView2 bootstrapper for Windows desktop app ([c72103b](https://github.com/AlexsJones/llmfit/commit/c72103b784134b8cc66725d4e8e9c36b0d01e64d))
* **fit:** CPU-only models that fit with headroom reach Good, not Marginal ([aad4c37](https://github.com/AlexsJones/llmfit/commit/aad4c373285e328bcbab72577ee49d62cca1515d))
* **fit:** CPU-only models that fit with headroom reach Good, not Marginal ([0ced1b7](https://github.com/AlexsJones/llmfit/commit/0ced1b775cb39f783cabf4411d25e1e1aa188015))
* **hardware:** detect all GPUs across sysfs, ROCm tabular, and fit pool ([0989ca2](https://github.com/AlexsJones/llmfit/commit/0989ca2dc443f782cf34cec709e989c23f234178))
* **hardware:** ROCm tabular multi-GPU parsing + summed fit VRAM pool ([4e27617](https://github.com/AlexsJones/llmfit/commit/4e27617baaeaf1902131f59c71763de360ef6837))
* **scoring:** MoE active-param quality, recency bonus, smooth fit curve ([c499c00](https://github.com/AlexsJones/llmfit/commit/c499c000948cb620564d9c8433df5b8b5d36880d))
* **scoring:** MoE active-param quality, recency bonus, smooth fit curve ([2db3e55](https://github.com/AlexsJones/llmfit/commit/2db3e552ad637437589e692bf91bc9ca69350925))
* **scoring:** treat headroom as a perfect fit (one-sided fit curve) ([235be33](https://github.com/AlexsJones/llmfit/commit/235be33b1d8ada30693f72b48b48d50bd48df5a9))
* **tui:** make provider filter input cursor-aware and ASCII-only ([933644e](https://github.com/AlexsJones/llmfit/commit/933644e4d1a80969cb7bfc1c5386423291a50132))
### Performance Improvements
* **analysis:** parallelize provider detection in detect_all with thread::scope ([d0660bc](https://github.com/AlexsJones/llmfit/commit/d0660bc72e601a7707caabb2f4e042a0701c491e))
## [0.9.31](https://github.com/AlexsJones/llmfit/compare/v0.9.30...v0.9.31) (2026-06-09)
### Features
* **lmstudio:** Add LMSTUDIO_API_KEY support for authenticated instances ([0dc2712](https://github.com/AlexsJones/llmfit/commit/0dc2712df4b4d947e0db371c98bf376d4da16c85))
* **lmstudio:** Add LMSTUDIO_API_KEY support for authenticated instances ([302cb53](https://github.com/AlexsJones/llmfit/commit/302cb535830e7e001c0735596d0c06da96fdc467))
* speed up provider navigation when holding shift (gh-571) ([f0d4783](https://github.com/AlexsJones/llmfit/commit/f0d4783dd99f4c12a0fe15d564d8bc787ef7c755))
* speed up provider navigation when holding shift (gh-571) ([7d74d7d](https://github.com/AlexsJones/llmfit/commit/7d74d7d4a93cb8c229868b00d8bc3385005e3551))
* upgrade MiniMax default model to M3 ([d45a546](https://github.com/AlexsJones/llmfit/commit/d45a546ebd9eca78cb5e09f191321e67807112d2))
* upgrade MiniMax default model to M3 ([7407d10](https://github.com/AlexsJones/llmfit/commit/7407d1035ef7dd7b14ed50a1dc1f3f6e21a08a44))
### Bug Fixes
* add Gemma 4 QAT models and fallback HF search ([e40d3b8](https://github.com/AlexsJones/llmfit/commit/e40d3b8e00b1db1fae9234e34fadde8a8a3945a2))
* add Gemma 4 QAT models and fallback HF search ([9b4582f](https://github.com/AlexsJones/llmfit/commit/9b4582f0e45c02ffb09abaef649cc45e01344f63)), closes [#613](https://github.com/AlexsJones/llmfit/issues/613)
* **lmstudio:** empty string guard, README docs, and unit test for LMSTUDIO_API_KEY ([46f70b7](https://github.com/AlexsJones/llmfit/commit/46f70b790e3bd177e001469d04456ce1e20193c5))
* replace unsafe env mutation in tests with pure logic tests ([33047d8](https://github.com/AlexsJones/llmfit/commit/33047d85fab2913d09905922df9a3daef6b5a314))
* **tui:** handle search input cursor and overflow ([66f4c88](https://github.com/AlexsJones/llmfit/commit/66f4c887b37c617944a8bc14a1d0a20aedac9767))
* **tui:** handle search input cursor and overflow ([8596005](https://github.com/AlexsJones/llmfit/commit/85960054bbb88c961ffa4797006dd3475a8b4699))
* **tui:** handle unicode download directory input ([b34d164](https://github.com/AlexsJones/llmfit/commit/b34d164cb9f5baf60f3b14058599d0d13755e212))
* **tui:** handle unicode download directory input ([da120f8](https://github.com/AlexsJones/llmfit/commit/da120f8d6993dd2e6cfed899478b6796b1992221))
* **tui:** ignore modified key input in search ([6c0b697](https://github.com/AlexsJones/llmfit/commit/6c0b69701e8ec9be28a7a98bd0e94812f64a037c))
* **tui:** ignore modified key input in search ([8de0a1b](https://github.com/AlexsJones/llmfit/commit/8de0a1b5a0627327f82360f9c9ced2ca71cfc4a7))
* use GGUF metadata for param counts in GGUF-only repos ([3f9b22b](https://github.com/AlexsJones/llmfit/commit/3f9b22bb032a3b4444750ddfca12acfdf96e27df))
* use GGUF metadata for parameter counts in GGUF-only repos ([ff28cdd](https://github.com/AlexsJones/llmfit/commit/ff28cdd002f4777ab1857b0c6e669c3c7789a1cd)), closes [#622](https://github.com/AlexsJones/llmfit/issues/622)
* wrap env::set_var/remove_var in unsafe blocks for Rust 2024 edition ([55b7979](https://github.com/AlexsJones/llmfit/commit/55b79793d1aa8bd3ca6e6ab7d4f11c5f20f897f6))
## [0.9.30](https://github.com/AlexsJones/llmfit/compare/v0.9.29...v0.9.30) (2026-06-01)
### Features
* **tui:** fuzzy search in provider filter popup ([#600](https://github.com/AlexsJones/llmfit/issues/600)) ([b055a04](https://github.com/AlexsJones/llmfit/commit/b055a04beb50008c3d868aa7f43598ad6b614d33))
## [0.9.29](https://github.com/AlexsJones/llmfit/compare/v0.9.28...v0.9.29) (2026-05-27)
CONTRIBUTING.md
+2 -1
View File
@@ -90,7 +90,8 @@ Keep PRs focused. One bug fix or feature per PR is easier to review than a combi
3. Run `make update-models` to regenerate the database and rebuild.
4. Verify with `./target/release/llmfit list`.
5. Update [MODELS.md](MODELS.md) if needed.
6. Open a PR.
6. Update [llmfit-core/data/schema.json](llmfit-core/data/schema.json) if the model has new or unique metadata fields.
7. Open a PR.
## Code guidelines
Cargo.lock
Generated
+412 -99
View File
File diff suppressed because it is too large Load Diff
Cargo.toml
+1 -1
View File
@@ -4,5 +4,5 @@ default-members = ["llmfit-core", "llmfit-tui"]
resolver = "3"
[workspace.package]
version = "0.9.29"
version = "0.9.38"
license = "MIT"
Dockerfile
+7 -2
View File
@@ -1,6 +1,12 @@
# Multi-stage build for llmfit
# Stage 1: Build the Rust binary
FROM rust:1.88-slim AS builder
# rustc >= 1.95 required: sysinfo 0.39.x bumped its MSRV to 1.95.
# Pin the Debian release to match the runtime stage (bookworm). The default
# rust:1.95-slim base tracks trixie (glibc 2.39), which links the binary
# against symbols the bookworm runtime (glibc 2.36) does not provide, so the
# binary fails to start with "GLIBC_2.39 not found". Keep both stages on the
# same release so the linked glibc is always available at runtime.
FROM rust:1.95-slim-bookworm AS builder
# Install build dependencies
RUN apt-get update && apt-get install -y \
@@ -18,7 +24,6 @@ COPY Cargo.toml Cargo.lock ./
COPY llmfit-core/ ./llmfit-core/
COPY llmfit-tui/ ./llmfit-tui/
COPY llmfit-desktop/ ./llmfit-desktop/
COPY data/ ./data/
# Build release binary for llmfit-tui
RUN cargo build --release -p llmfit
README.ja.md
+990
View File
@@ -0,0 +1,990 @@
# llmfit
<p align="center">
<img src="assets/icon.svg" alt="llmfit アイコン" width="128" height="128">
</p>
<p align="center">
<a href="README.md">English</a> ·
<a href="README.zh.md">中文</a> ·
<b>日本語</b>
</p>
<p align="center">
<a href="https://github.com/AlexsJones/llmfit/actions/workflows/ci.yml"><img src="https://github.com/AlexsJones/llmfit/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
<a href="https://crates.io/crates/llmfit"><img src="https://img.shields.io/crates/v/llmfit.svg" alt="Crates.io"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="ライセンス"></a>
<a href="https://about.signpath.io"><img src="https://img.shields.io/badge/SignPath-signed-brightgreen?logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIxNiIgaGVpZ2h0PSIxNiIgZmlsbD0id2hpdGUiIHZpZXdCb3g9IjAgMCAxNiAxNiI+PHBhdGggZD0iTTEwLjA2NyA0LjU2N2wtNC43MzQgNC43MzMtMS40LTEuNGExIDEgMCAwIDAtMS40MTQgMS40MTRsMi4xIDIuMWExIDEgMCAwIDAgMS40MTQgMGw1LjQ0LTUuNDRhMSAxIDAgMCAwLTEuNDE0LTEuNDE0eiIvPjwvc3ZnPg==" alt="SignPath で署名済み"></a>
</p>
> **新機能: [コミュニティリーダーボード](#コミュニティリーダーボード-b)** — 実際のユーザーから集まった実環境のパフォーマンスデータを閲覧できます。`b` を押すと、自分のものに限らず、あらゆる GPU の実測 tok/s、TTFT、VRAM を確認できます。`H` で 27 種類以上のハードウェアプリセット（RTX 5090 から Apple M1 まで）から選び、購入や自作の前に実数値を比較しましょう。
**数百のモデルとプロバイダー。自分のハードウェアで動くものを見つけるコマンドはひとつ。**
LLM モデルをシステムの RAM、CPU、GPU に合わせて最適化するターミナルツールです。ハードウェアを検出し、各モデルを品質・速度・適合度・コンテキストの各観点でスコアリングして、あなたのマシンで実際に快適に動くものを教えてくれます。
インタラクティブな TUI（デフォルト）と従来型の CLI モードを備えています。マルチ GPU 構成、MoE アーキテクチャ、動的な量子化選択、速度推定、ローカルランタイムプロバイダー（Ollama、llama.cpp、MLX、Docker Model Runner、LM Studio）に対応しています。
**新機能: [コミュニティリーダーボード](#コミュニティリーダーボード-b)（`b`）** — 同じハードウェアを使う他のユーザーから集まった実環境の tok/s、TTFT、VRAM 使用量を確認できます。[localmaxxing.com](https://localmaxxing.com) を利用し、推定パフォーマンスと実測パフォーマンスの差を埋めます。
その他: [ダウンロードマネージャー](#ダウンロードマネージャー-d)（`D`）、[詳細設定](#詳細設定-a)（`A`）、[ハードウェアシミュレーション](#ハードウェアシミュレーション-s) — `D` を押すとダウンロードの管理、履歴の閲覧、モデルの削除、ダウンロードディレクトリの設定ができます。`A` を押すと TPS 効率、実行モード係数、スコアリングの重みを調整できます。`S` を押すと別のハードウェアをシミュレーションできます。
> **姉妹プロジェクト:**
> - [sympozium](https://github.com/sympozium-ai/sympozium/) — Kubernetes でエージェントを管理。
> - [llmserve](https://github.com/AlexsJones/llmserve) — ローカル LLM モデルをサーブするためのシンプルな TUI。モデルを選び、バックエンドを選び、サーブする。
> - [llama-panel](https://github.com/AlexsJones/llama-panel) — ローカルの llama-server インスタンスを管理するネイティブ macOS アプリ。
![demo](assets/demo.gif)
---
## インストール
### Windows
```sh
scoop install llmfit
```
Scoop がインストールされていない場合は、[Scoop インストールガイド](https://scoop.sh/)に従ってください。
### macOS / Linux
#### Homebrew
ビルド済みバイナリ（推奨。すべての macOS/Linux バージョンで動作）:
```sh
brew install AlexsJones/llmfit/llmfit
```
または homebrew-core の formula から。bottle がない macOS バージョンではソースからビルドされます:
```sh
brew install llmfit
```
#### MacPorts
```sh
port install llmfit
```
#### クイックインストール
```sh
curl -fsSL https://llmfit.axjns.dev/install.sh | sh
```
GitHub から最新リリースのバイナリをダウンロードし、`/usr/local/bin`（sudo がない場合は `~/.local/bin`）にインストールします。
**sudo なしで `~/.local/bin` にインストール:**
```sh
curl -fsSL https://llmfit.axjns.dev/install.sh | sh -s -- --local
```
### uv / pip
llmfit をインストールまたは更新するには:
```sh
uv tool install -U llmfit
```
インストールせずに実行するには:
```sh
uvx llmfit
```
pip や uv などのツールを使って、通常の方法で llmfit を Python パッケージとしてインストールすることもできます。
### Docker / Podman
```sh
docker run ghcr.io/alexsjones/llmfit
```
これは `llmfit recommend` コマンドの JSON を出力します。この JSON は `jq` でさらにクエリできます。
```
podman run ghcr.io/alexsjones/llmfit recommend --use-case coding | jq '.models[].name'
```
### ソースから
```sh
git clone https://github.com/AlexsJones/llmfit.git
cd llmfit
cargo build --release
# バイナリは target/release/llmfit にあります
```
---
## 使い方
### TUI（デフォルト）
```sh
llmfit
```
インタラクティブなターミナル UI を起動します。システムスペック（CPU、RAM、GPU 名、VRAM、バックエンド）が上部に表示されます。モデルは複合スコア順に並んだスクロール可能なテーブルに一覧表示されます。各行には、モデルのスコア、推定 tok/s、あなたのハードウェアに最適な量子化、実行モード、メモリ使用量、ユースケースカテゴリが表示されます。
| キー | アクション |
|----------------------------|-----------------------------------------------------------------------|
| `Up` / `Down` または `j` / `k` | モデルを移動 |
| `/` | 検索モードに入る（名前、プロバイダー、パラメータ、ユースケースの部分一致） |
| `Esc` または `Enter` | 検索モードを終了 |
| `Ctrl-U` | 検索をクリア |
| `f` | 適合フィルターを切り替え: All、Runnable、Perfect、Good、Marginal |
| `a` | 利用可否フィルターを切り替え: All、GGUF Avail、Installed |
| `s` | ソート列を切り替え: Score、Params、Mem%、Ctx、Date、Use Case |
| `v` | Visual モードに入る（複数モデルを選択） |
| `V` | Select モードに入る（列ベースのフィルタリング） |
| `t` | カラーテーマを切り替え（自動保存） |
| `p` | 選択したモデルの Plan モードを開く（ハードウェアプランニング） |
| `P` | プロバイダーフィルターのポップアップを開く（入力で曖昧フィルタリング） |
| `U` | ユースケースフィルターのポップアップを開く |
| `C` | 機能フィルターのポップアップを開く |
| `L` | ライセンスフィルターのポップアップを開く |
| `R` | ランタイム/バックエンドフィルターのポップアップを開く（llama.cpp、MLX、vLLM） |
| `S` | ハードウェアシミュレーションのポップアップを開く（RAM/VRAM/CPU を上書き） |
| `A` | 詳細設定のポップアップを開く（効率や実行モード係数を調整） |
| `b` | コミュニティリーダーボードビューを開く（localmaxxing.com） |
| `I` | 推論ベンチビューを開く（あなたのモデルに対するローカル品質スコアリング） |
| `h` | ヘルプポップアップを開く（すべてのキーバインド） |
| `m` | 選択したモデルを比較対象にマーク |
| `c` | 比較ビューを開く（マーク済み vs 選択中） |
| `x` | 比較マークをクリア |
| `i` | インストール済み優先ソートを切り替え（検出された任意のランタイムプロバイダー） |
| `d` | 選択したモデルをダウンロード（複数利用可能ならプロバイダー選択） |
| `D` | ダウンロードマネージャーを開く（履歴、削除、設定） |
| `r` | ランタイムプロバイダーからインストール済みモデルを再読み込み |
| `Enter` | 選択したモデルの詳細ビューを切り替え |
| `PgUp` / `PgDn` | 10 件単位でスクロール |
| `g` / `G` | 先頭 / 末尾にジャンプ |
| `q` | 終了 |
### Vim 風モード
TUI は左下のステータスバーに表示される Vim 由来のモードを使用します。現在のモードによって、どのキーが有効かが決まります。
#### Normal モード
デフォルトのモードです。移動、検索、フィルタリング、ビューの表示を行います。上の表のすべてのキーがここで適用されます。
#### Visual モード（`v`）
一括比較のために連続したモデルの範囲を選択します。`v` を押して現在の行を起点に固定し、`j`/`k` または矢印キーで移動して選択範囲を広げます。選択された行はハイライトされます。
| キー | アクション |
|---------------------|--------------------------------------------------------|
| `j` / `k` または矢印 | 選択範囲を上下に拡張 |
| `c` | 選択したすべてのモデルを比較（マルチ比較ビューを開く） |
| `m` | 現在のモデルを 2 モデル比較用にマーク |
| `Esc` または `v` | Visual モードを終了 |
マルチ比較ビューは、行が属性（Score、tok/s、Fit、Mem%、Params、Mode、Context、Quant など）、列がモデルとなるテーブルを表示します。最良の値はハイライトされます。画面に収まらないほど多くのモデルを選択した場合は、`h`/`l` または矢印キーで横スクロールできます。
#### Select モード（`V`）
列ベースのアクションです。`V`（shift-v）を押して Select モードに入り、`h`/`l` または矢印キーで列ヘッダー間を移動します。アクティブな列は視覚的にハイライトされます。`Enter` または `Space` を押すと、その列の現在のアクションが実行されます。
| 列 | フィルターアクション |
|-------------------------------|---------------------------------------------------------------------------|
| Inst | 利用可否フィルターを切り替え |
| Model | 検索モードに入る |
| Provider | プロバイダーポップアップを開く |
| Params | パラメータサイズのバケットポップアップを開く（<3B、3-7B、7-14B、14-30B、30-70B、70B+） |
| Score, tok/s, Mem%, Ctx, Date | その列でソート |
| Quant | 量子化ポップアップを開く |
| Mode | 実行モードポップアップを開く（GPU、MoE、CPU+GPU、CPU） |
| Fit | 適合フィルターを切り替え |
| Use Case | ユースケースポップアップを開く |
Select モードでも行のナビゲーションは引き続き機能するため、アクションを適用しながらその効果を確認できます: `j`/`k`、矢印キー、`Ctrl-U`、`Ctrl-D`、`PageUp`、`PageDown`、`Home`、`End`。`Esc` を押すと Normal モードに戻ります。
### TUI Plan モード（`p`）
Plan モードは通常の適合分析を反転させます。「何が自分のハードウェアに収まるか?」ではなく、「このモデル構成にはどんなハードウェアが必要か?」を推定します。
選択した行で `p` を押し、次のように操作します:
| キー | アクション |
|------------------------|-----------------------------------------------------------|
| `Tab` / `j` / `k` | 編集可能なフィールド間を移動（Context、Quant、Target TPS） |
| `Left` / `Right` | 現在のフィールド内でカーソルを移動 |
| 入力 | 現在のフィールドを編集 |
| `Backspace` / `Delete` | 文字を削除 |
| `Ctrl-U` | 現在のフィールドをクリア |
| `Esc` または `q` | Plan モードを終了 |
Plan モードは以下の推定値を表示します:
- 最小および推奨の VRAM/RAM/CPU コア数
- 実行可能な実行パス（GPU、CPU オフロード、CPU のみ）
- より良い適合目標に到達するためのアップグレード差分
### ハードウェアシミュレーション（`S`）
`S` を押すとハードウェアシミュレーションのポップアップが開きます。RAM、VRAM、CPU コア数を上書きして、別のターゲットハードウェアでどのモデルが収まるかを確認できます。すべてのモデルスコア、適合レベル、速度推定は、シミュレーションされたスペックに対して即座に再計算されます。
![Hardware Simulation](assets/simulation.png)
| キー | アクション |
|------------------------|-----------------------------------------|
| `Tab` / `j` / `k` | RAM、VRAM、CPU フィールドを切り替え |
| 数字を入力 | 選択したフィールドを編集 |
| `Enter` | シミュレーションを適用 |
| `Ctrl-R` | 実際に検出されたハードウェアにリセット |
| `Esc` | キャンセルして閉じる |
シミュレーションが有効なときは、システムバーとステータスバーに `SIM` バッジが表示されます。リセットするまで、モデルテーブル全体がシミュレーションされたハードウェアを反映します。
### 詳細設定（`A`）
`A` を押すと詳細設定のポップアップが開きます。このパネルでは、TPS 推定、実行モードのペナルティ、複合スコアリングの背後にあるパラメータを調整できます。これは特定のモデル（例: Qwen3 30B）で tok/s が過大評価されていた[issue #449](https://github.com/AlexsJones/llmfit/issues/449)に対応するものです。
すべての変更は即座に適用され、モデルテーブルが再計算されます。`Esc` で確定して閉じるか、`Ctrl-R` でデフォルトにリセットします。
| フィールド | 説明 | デフォルト |
|--------------------|-------------------------------------------------------------------------|---------|
| **Efficiency** | 帯域幅ベースの TPS のグローバル効率係数。オーバーヘッドを考慮 | `0.55` |
| **GPU factor** | 純粋な GPU 推論の速度倍率 | `1.0` |
| **CPU Offload** | 重みがシステム RAM にあふれた場合の速度倍率 | `0.5` |
| **MoE Offload** | Mixture-of-Experts のエキスパート切り替えの速度倍率 | `0.8` |
| **Tensor Par** | テンソル並列推論の速度倍率 | `0.9` |
| **CPU Only** | CPU のみの実行の速度倍率 | `0.3` |
| **Context cap** | メモリ推定に使用する最大コンテキスト長（デフォルトの場合は空欄のまま） | `auto` |
| キー | アクション |
|------------------------|-----------------------------------------|
| `Tab` / `j` / `k` | フィールドを切り替え |
| 数字 / `.` を入力 | 選択したフィールドを編集 |
| `Left` / `Right` | フィールド内でカーソルを移動 |
| `Backspace` / `Delete` | 文字を削除 |
| `Ctrl-U` | 現在のフィールドをクリア |
| `Enter` | 変更を適用してすべてのスコアを再計算 |
| `Esc` / `q` | 適用せずに閉じる |
### ダウンロードマネージャー（`D`）
`D` を押すとダウンロードマネージャービューが開きます。このフルスクリーンビューはメインのモデルテーブルに置き換わり、3 つのセクションを提供します:
- **Active Download** — 進行中の現在のダウンロードを、プログレスバー、モデル名、ステータスメッセージとともに表示します。
- **Config** — GGUF モデルディレクトリを表示（および編集可能）します。設定したパスはセッションをまたいで永続化されます。
- **History** — 過去のダウンロードのナビゲート可能なリスト（新しい順）を、モデル名、プロバイダー、ステータス、日付とともに表示します。失敗したダウンロードは履歴から削除でき、成功したダウンロードはプロバイダーから削除できます。
`Tab` / `Shift-Tab` でセクション間のフォーカスを切り替えます。
| キー | アクション |
|------------------------|--------------------------------------------------|
| `Tab` / `Shift-Tab` | フォーカスを切り替え: Active → Config → History |
| `j` / `k` または矢印 | 履歴リストをナビゲート（History にフォーカス時） |
| `x` | 選択したモデルを削除（確認を求められる） |
| `y` / `n` | 削除を確認またはキャンセル |
| `e` | ダウンロードディレクトリを編集（Config にフォーカス時） |
| `Enter` | ディレクトリ編集を確定 |
| `Esc` / `D` / `q` | 閉じてモデルテーブルに戻る |
失敗したダウンロード（例: 404 エラー）の場合、`x` は履歴からエントリを削除します。成功したダウンロードの場合は、プロバイダーからモデルを削除します（Ollama と llama.cpp でサポート）。
### コミュニティリーダーボード（`b`）
`b` を押すとコミュニティリーダーボードビューが開きます。llmfit の理論的な速度推定だけに頼るのではなく、このビューでは同じハードウェアを使う他のユーザーから集まった**実環境のパフォーマンスデータ** — 実測の tok/s、最初のトークンまでの時間、ピーク VRAM 使用量 — を表示します。
![Community Leaderboard](assets/benchmark.jpeg)
データはコミュニティのベンチマークデータベースである [localmaxxing.com](https://localmaxxing.com) から取得されます。ビューを開くと、llmfit はあなたのハードウェア（GPU モデル、VRAM ティア、Apple Silicon チップファミリー、OS）を自動検出し、一致する結果をクエリします。
| 列 | 説明 |
|--------------|----------------------------------------------------------|
| **Model** | HuggingFace モデル ID |
| **Engine** | 使用された推論ランタイム（llama.cpp、vLLM、Ollama、MLX...） |
| **Quant** | 量子化フォーマット（Q4_K_M、Q8_0 など） |
| **tok/s** | 実測の出力トークン生成速度 |
| **Total t/s**| 総スループット（プロンプト + 生成） |
| **TTFT** | 最初のトークンまでの時間（レイテンシ） |
| **VRAM** | 推論中のピークメモリ使用量 |
| **Ctx** | ベンチマークで使用されたコンテキスト長 |
| **User** | 投稿者（認証済みユーザーは `*` でマーク） |
| キー | アクション |
|------------------------|-----------------------------------------|
| `j` / `k` または矢印 | 結果をナビゲート |
| `H` | ハードウェアピッカーを開く（任意の GPU を閲覧） |
| `r` | API から再取得 / リフレッシュ |
| `b` / `q` / `Esc` | 閉じてモデルテーブルに戻る |
`H` を押すとハードウェアピッカーが開きます。これは 27 種類の人気 GPU とチップ（RTX 5090 から CPU のみまで、加えて Apple Silicon M1–M4 バリアント、AMD RX/MI シリーズ、NVIDIA データセンターカード）のスクロール可能なリストです。1 つ選ぶと、たとえそれが自分の使っているものでなくても、そのハードウェアのベンチマークを即座に読み込めます。「My Hardware (auto-detect)」を選ぶと自分のシステムに戻ります。
#### API キーの設定
公開ベンチマークは認証なしで利用できます。フルアクセスには [localmaxxing.com](https://localmaxxing.com) の API キーを指定します:
```sh
# 環境変数経由（推奨）
export LOCALMAXXING_API_KEY="bhk_your_key_here"
llmfit
# または CLI フラグ経由
llmfit --api-key "bhk_your_key_here"
```
| 変数 | 説明 |
|---|---|
| `LOCALMAXXING_API_KEY` | localmaxxing.com API のベアラートークン |
### 推論ベンチ（`I`）
`I`（大文字）を押すと推論ベンチビューが開きます。これは**ローカルで実行中のプロバイダー** — Ollama、vLLM、MLX — に対して**ライブの推論ベンチマーク**を実行し、実際の推論リクエストで最初のトークンまでの時間（TTFT）、毎秒トークン数（TPS）、総レイテンシを測定します。
コミュニティリーダーボード（他のユーザーから集めたクラウドソースのデータを表示）とは異なり、推論ベンチはあなたの実際のハードウェアであなたの実際のモデルを測定します。
#### TUI での使い方
| キー | アクション |
|-----|--------|
| `I` | 推論ベンチを開く（プロバイダーを自動検出してベンチマークを実行） |
| `I`（再度） | ベンチビュー内からベンチマークを再実行 |
| `j` / `k` または矢印 | モデル結果をナビゲート |
| `Enter` | 選択したモデルの詳細ビューを開く |
| `r` | ルーティングマトリックスビューに切り替え |
| `q` / `Esc` | ベンチビューを閉じる |
結果は `~/.config/llmfit/bench-cache.json` にキャッシュされ、次回以降は即座に読み込まれます。
#### CLI での使い方
```sh
# プロバイダーを自動検出してベンチマーク
llmfit bench
# 実行中のすべてのプロバイダーで検出されたすべてのモデルをベンチマーク
llmfit bench --all
# Ollama 経由で特定のモデルをベンチマーク
llmfit bench --provider ollama llama3.2
# エンドポイント URL を上書き
llmfit bench --provider ollama --url http://my-server:11434 llama3.2
# vLLM エンドポイントを上書き
llmfit bench --provider vllm --url http://localhost:8000
# JSON として出力（スクリプト用）
llmfit bench --json
# 品質ベンチマークを実行（ルーティング用のロールベーススコアリング）
llmfit bench --quality
# ルーティングマトリックスを出力
llmfit bench --quality --routing
```
#### 環境変数
| 変数 | デフォルト | 説明 |
|---|---|---|
| `OLLAMA_HOST` | `http://localhost:11434` | Ollama API のベース URL |
| `VLLM_PORT` | `8000` | vLLM サーバーのポート（`http://localhost:$VLLM_PORT` として使用） |
### テーマ
`t` を押すと 10 種類の組み込みカラーテーマを切り替えられます。選択は `~/.config/llmfit/theme` に自動保存され、次回起動時に復元されます。
| テーマ | 説明 |
|--------------------------|---------------------------------------------------|
| **Default** | オリジナルの llmfit カラー |
| **Dracula** | 暗い紫の背景にパステルのアクセント |
| **Solarized** | Ethan Schoonover の Solarized Dark パレット |
| **Nord** | 北極風の涼しげな青灰色のトーン |
| **Monokai** | Monokai Pro の暖かいシンタックスカラー |
| **Gruvbox** | 暖かいアースカラーのレトログルーヴパレット |
| **Catppuccin Latte** | 🌻 ライトテーマ — 調和の取れたパステルの反転 |
| **Catppuccin Frappé** | 🪴 低コントラストのダーク — 落ち着いた控えめな美学 |
| **Catppuccin Macchiato** | 🌺 中コントラストのダーク — 穏やかで安らぐトーン |
| **Catppuccin Mocha** | 🌿 最も暗いバリアント — 色彩豊かなアクセントで居心地よく |
### Web ダッシュボード
`llmfit` を非 JSON モードで実行すると、`0.0.0.0:8787` でバックグラウンドの Web ダッシュボードが自動的に起動します。同じネットワーク上の任意のブラウザで開けます:
```
http://<your-machine-ip>:8787
```
ホストやポートは環境変数で上書きできます:
```sh
LLMFIT_DASHBOARD_HOST=0.0.0.0 LLMFIT_DASHBOARD_PORT=9000 llmfit
```
| 変数 | デフォルト | 説明 |
|---|---|---|
| `LLMFIT_DASHBOARD_HOST` | `0.0.0.0` | ダッシュボードサーバーをバインドするインターフェース |
| `LLMFIT_DASHBOARD_PORT` | `8787` | ダッシュボードサーバーをバインドするポート |
自動起動するダッシュボードを無効にするには、`--no-dashboard` を渡します:
```sh
llmfit --no-dashboard
```
### CLI モード
`--cli` または任意のサブコマンドを使うと、従来型のテーブル出力が得られます:
```sh
# 適合度でランク付けされたすべてのモデルのテーブル
llmfit --cli
# 完全に適合するモデルのみ、上位 5 件
llmfit fit --perfect -n 5
# 検出されたシステムスペックを表示
llmfit system
# データベース内のすべてのモデルを一覧表示
llmfit list
# 名前、プロバイダー、サイズで検索
llmfit search "llama 8b"
# 単一モデルの詳細ビュー
llmfit info "Mistral-7B"
# 上位 5 件の推奨（JSON、エージェント/スクリプト消費用）
llmfit recommend --json --limit 5
# ユースケースでフィルタリングした推奨
llmfit recommend --json --use-case coding --limit 3
# 特定のランタイムを強制（Apple Silicon での自動 MLX 選択をバイパス）
llmfit recommend --force-runtime llamacpp
llmfit recommend --force-runtime llamacpp --use-case coding --limit 3
# 特定のモデル構成に必要なハードウェアをプラン
llmfit plan "Qwen/Qwen3-4B-MLX-4bit" --context 8192
llmfit plan "Qwen/Qwen3-4B-MLX-4bit" --context 8192 --quant mlx-4bit
llmfit plan "Qwen/Qwen3-4B-MLX-4bit" --context 8192 --target-tps 25 --json
# ノードレベルの REST API として実行（クラスタースケジューラー / アグリゲーター用）
llmfit serve --host 0.0.0.0 --port 8787
```
### REST API（`llmfit serve`）
`llmfit serve` は、TUI/CLI で使われるのと同じ適合度/スコアリングデータを公開する HTTP API を起動します。これにはノードのフィルタリングや上位モデルの選択も含まれます。
```sh
# 生存確認
curl http://localhost:8787/health
# ノードのハードウェア情報
curl http://localhost:8787/api/v1/system
# フィルター付きの完全な適合リスト
curl "http://localhost:8787/api/v1/models?min_fit=marginal&runtime=llamacpp&sort=score&limit=20"
# 主要なスケジューリングエンドポイント: このノードで実行可能な上位モデル
curl "http://localhost:8787/api/v1/models/top?limit=5&min_fit=good&use_case=coding"
# モデル名/プロバイダーのテキストで検索
curl "http://localhost:8787/api/v1/models/Mistral?runtime=any"
```
`models`/`models/top` でサポートされるクエリパラメータ:
- `limit`（または `n`）: 返される行の最大数
- `perfect`: `true|false`（`true` で完全適合のみを強制）
- `min_fit`: `perfect|good|marginal|too_tight`
- `runtime`: `any|mlx|llamacpp`
- `use_case`: `general|coding|reasoning|chat|multimodal|embedding`
- `provider`: プロバイダーのテキストフィルター（部分文字列）
- `search`: 名前/プロバイダー/サイズ/ユースケースにわたる自由テキストフィルター
- `sort`: `score|tps|params|mem|ctx|date|use_case`
- `include_too_tight`: 実行不可能な行を含める（`/top` ではデフォルト `false`、`/models` では `true`）
- `max_context`: メモリ推定のためのリクエストごとのコンテキスト上限
- `force_runtime`: `mlx|llamacpp|vllm` — 分析中の自動ランタイム選択を上書き
API の動作をローカルで検証:
```sh
# サーバーを自動的に起動し、エンドポイント/スキーマ/フィルターのアサーションを実行
python3 scripts/test_api.py --spawn
# またはすでに実行中のサーバーをテスト
python3 scripts/test_api.py --base-url http://127.0.0.1:8787
```
### ハードウェアの上書き
ハードウェアの自動検出は一部のシステム（例: 壊れた `nvidia-smi`、VM、パススルー構成）で失敗することがあります。また、別のターゲットハードウェアに対してモデルの適合度を評価したい場合もあるでしょう。`--memory`、`--ram`、`--cpu-cores` を使って検出された値を上書きできます:
```sh
# GPU VRAM を上書き
llmfit --memory=32G
# システム RAM を上書き
llmfit --ram=128G
# CPU コア数を上書き
llmfit --cpu-cores=16
# 上書きを組み合わせてターゲットハードウェアをシミュレーション
llmfit --memory=24G --ram=64G --cpu-cores=8 fit
llmfit --memory=24G --ram=64G system --json
# すべてのモードで動作: TUI、CLI、サブコマンド
llmfit --memory=24G --cli
llmfit --memory=24G fit --perfect -n 5
llmfit --ram=64G recommend --json
```
`--memory` と `--ram` で使える接尾辞: `G`/`GB`/`GiB`（ギガバイト）、`M`/`MB`/`MiB`（メガバイト）、`T`/`TB`/`TiB`（テラバイト）。大文字小文字は区別されません。GPU が検出されなかった場合、`--memory` は合成 GPU エントリを作成し、モデルが GPU 推論用にスコアリングされるようにします。統合メモリシステム（Apple Silicon）では、`--ram` は VRAM も更新します。VRAM を独立して上書きするには `--memory` を使ってください。
### 推定用のコンテキスト長の上限
`--max-context` を使うと、メモリ推定に使用するコンテキスト長を上限で制限できます（各モデルが公称する最大コンテキストは変更しません）:
```sh
# 4K コンテキストでメモリ適合度を推定
llmfit --max-context 4096 --cli
# サブコマンドで動作
llmfit --max-context 8192 fit --perfect -n 5
llmfit --max-context 16384 recommend --json --limit 5
```
`--max-context` が設定されていない場合、llmfit は利用可能であれば `OLLAMA_CONTEXT_LENGTH` を使用します。
### JSON 出力
任意のサブコマンドに `--json` を追加すると、機械可読な出力が得られます:
```sh
llmfit --json system # ハードウェアスペックを JSON で
llmfit --json fit -n 10 # 上位 10 件の適合を JSON で
llmfit recommend --json # 上位 5 件の推奨（recommend では JSON がデフォルト）
llmfit plan "Qwen/Qwen2.5-Coder-0.5B-Instruct" --context 8192 --json
```
`plan` の JSON には以下の安定したフィールドが含まれます:
- リクエスト（`context`、`quantization`、`target_tps`）
- 推定された最小/推奨ハードウェア
- パスごとの実行可能性（`gpu`、`cpu_offload`、`cpu_only`）
- アップグレード差分
---
## 仕組み
1. **ハードウェア検出** -- `sysinfo` で合計/利用可能 RAM を読み取り、CPU コアを数え、GPU を探索します:
- **NVIDIA** -- `nvidia-smi` によるマルチ GPU サポート。検出されたすべての GPU の VRAM を集約します。レポートが失敗した場合は GPU モデル名から VRAM を推定してフォールバックします。
- **AMD** -- `rocm-smi` で検出。
- **Intel Arc** -- ディスクリート VRAM は sysfs 経由、統合は `lspci` 経由。
- **Apple Silicon** -- `system_profiler` 経由の統合メモリ。VRAM = システム RAM。
- **Ascend** -- `npu-smi` で検出。
- **バックエンド検出** -- 速度推定のために、アクセラレーションバックエンド（CUDA、Metal、ROCm、SYCL、CPU ARM、CPU x86、Ascend）を自動的に識別します。
2. **モデルデータベース** -- HuggingFace API から取得した数百のモデルを `data/hf_models.json` に保存し、コンパイル時に埋め込みます。メモリ要件は、量子化階層（Q8_0 から Q2_K まで）にわたるパラメータ数から計算されます。VRAM は GPU 推論の主要な制約であり、システム RAM は CPU のみの実行のフォールバックです。
**MoE サポート** -- Mixture-of-Experts アーキテクチャを持つモデル（Mixtral、DeepSeek-V2/V3）は自動的に検出されます。トークンごとにアクティブになるのはエキスパートの一部のみなので、実効 VRAM 要件は総パラメータ数が示すよりもはるかに低くなります。例えば、Mixtral 8x7B は総パラメータ 46.7B ですが、トークンごとにアクティブになるのは約 12.9B のみで、エキスパートオフロードにより VRAM が 23.9 GB から約 6.6 GB に削減されます。
3. **動的量子化** -- 固定の量子化を仮定する代わりに、llmfit はあなたのハードウェアに収まる最高品質の量子化を試します。Q8_0（最高品質）から Q2_K（最も圧縮）までの階層をたどり、利用可能なメモリに収まる最高品質のものを選びます。フルコンテキストで何も収まらない場合は、半分のコンテキストで再試行します。
4. **多次元スコアリング** -- 各モデルは 4 つの次元（それぞれ 0〜100）でスコアリングされます:
| 次元 | 測定するもの |
|-------------|--------------------------------------------------------------------------------|
| **Quality** | パラメータ数、モデルファミリーの評判、量子化ペナルティ、タスク整合性 |
| **Speed** | バックエンド、パラメータ、量子化に基づく推定トークン/秒 |
| **Fit** | メモリ使用効率（最適点: 利用可能メモリの 50〜80%） |
| **Context** | ユースケースに対するコンテキストウィンドウ能力 vs ターゲット |
各次元は重み付き複合スコアに統合されます。重みはユースケースカテゴリ（General、Coding、Reasoning、Chat、Multimodal、Embedding）によって異なります。例えば、Chat は Speed を高く重み付け（0.35）し、Reasoning は Quality を高く重み付け（0.55）します。モデルは複合スコアでランク付けされ、実行不可能なモデル（Too Tight）は常に最下位になります。
5. **速度推定** -- LLM 推論におけるトークン生成はメモリ帯域幅に律速されます。各トークンは VRAM からモデルの全重みを一度読み取る必要があります。GPU モデルが認識されると、llmfit はその実際のメモリ帯域幅を使ってスループットを推定します:
計算式: `(bandwidth_GB_s / model_size_GB) × efficiency_factor`
効率係数（0.55）とモードごとの速度倍率は、詳細設定ポップアップ（TUI の `A`）で調整できます。デフォルト値は、カーネルオーバーヘッド、KV キャッシュの読み取り、メモリコントローラーの影響を考慮しています。このアプローチは、llama.cpp の公開ベンチマーク（[Apple Silicon](https://github.com/ggml-org/llama.cpp/discussions/4167)、[NVIDIA T4](https://github.com/ggml-org/llama.cpp/discussions/4225)）および実環境の測定値に対して検証されています。
帯域幅ルックアップテーブルは、NVIDIA（コンシューマー + データセンター）、AMD（RDNA + CDNA）、Apple Silicon ファミリーにわたる約 80 の GPU をカバーしています。
認識されない GPU の場合、llmfit はバックエンドごとの速度定数にフォールバックします:
| バックエンド | 速度定数 |
|--------------|----------------|
| CUDA | 220 |
| Metal | 160 |
| ROCm | 180 |
| SYCL | 100 |
| CPU (ARM) | 90 |
| CPU (x86) | 70 |
| NPU (Ascend) | 390 |
フォールバック計算式: `K / params_b × quant_speed_multiplier`。モードごとのペナルティは詳細設定ポップアップ（TUI の `A`）で調整できます。
6. **適合度分析** -- 各モデルはメモリ互換性について評価されます:
**実行モード:**
- **GPU** -- モデルが VRAM に収まる。高速な推論。
- **MoE** -- エキスパートオフロード付きの Mixture-of-Experts。アクティブなエキスパートは VRAM に、非アクティブは RAM に。
- **CPU+GPU** -- VRAM が不足し、部分的な GPU オフロードでシステム RAM にあふれる。
- **CPU** -- GPU なし。モデルは完全にシステム RAM に読み込まれる。
**適合レベル:**
- **Perfect** -- GPU で推奨メモリを満たす。GPU アクセラレーションが必要。
- **Good** -- 余裕を持って収まる。MoE オフロードや CPU+GPU で達成可能な最良。
- **Marginal** -- ぎりぎりの適合、または CPU のみ（CPU のみは常にここで頭打ち）。
- **Too Tight** -- どこにも十分な VRAM やシステム RAM がない。
---
## モデルデータベース
モデルリストは、HuggingFace REST API をクエリするスタンドアロンの Python スクリプト `scripts/scrape_hf_models.py`（標準ライブラリのみ、pip 依存なし）によって生成されます。Meta Llama、Mistral、Qwen、Google Gemma、Microsoft Phi、DeepSeek、IBM Granite、Allen Institute OLMo、xAI Grok、Cohere、BigCode、01.ai、Upstage、TII Falcon、HuggingFace、Zhipu GLM、Moonshot Kimi、Baidu ERNIE など、数百のモデルとプロバイダーを含みます。スクレイパーは、モデル設定（`num_local_experts`、`num_experts_per_tok`）と既知のアーキテクチャマッピングを通じて MoE アーキテクチャを自動検出します。
モデルカテゴリは、汎用、コーディング（CodeLlama、StarCoder2、WizardCoder、Qwen2.5-Coder、Qwen3-Coder）、推論（DeepSeek-R1、Orca-2）、マルチモーダル/ビジョン（Llama 3.2 Vision、Llama 4 Scout/Maverick、Qwen2.5-VL）、チャット、エンタープライズ（IBM Granite）、埋め込み（nomic-embed、bge）にわたります。
完全なリストは [MODELS.md](MODELS.md) を参照してください。
モデルデータベースはコンパイル時に埋め込まれるため、**エンドユーザー**は llmfit 自体をアップグレード（`brew upgrade llmfit`、`scoop update llmfit`、または新しいリリースのダウンロード）することで更新を受け取ります。以下のコマンドは、ソースからデータベースを更新する**コントリビューター**向けです:
モデルデータベースを更新するには:
```sh
# 自動更新（推奨）
make update-models
# またはスクリプトを直接実行
./scripts/update_models.sh
# または手動で
python3 scripts/scrape_hf_models.py
cargo build --release
```
スクレイパーは `data/hf_models.json` を書き込み、これは `include_str!` を介してバイナリに焼き込まれます。自動更新スクリプトは既存データをバックアップし、JSON 出力を検証し、バイナリを再ビルドします。
デフォルトでは、スクレイパーは [unsloth](https://huggingface.co/unsloth) や [bartowski](https://huggingface.co/bartowski) などのプロバイダーから既知の GGUF ダウンロードソースでモデルを補強します。結果は `data/gguf_sources_cache.json` にキャッシュされ（7 日間の TTL）、API 呼び出しの繰り返しを避けます。補強をスキップしてスクレイプを高速化するには `--no-gguf-sources` を使ってください。
---
## プロジェクト構成
```
src/
main.rs -- CLI 引数解析、エントリポイント、TUI 起動
hardware.rs -- システム RAM/CPU/GPU 検出（マルチ GPU、バックエンド識別）
models.rs -- モデルデータベース、量子化階層、動的量子化選択
fit.rs -- 多次元スコアリング（Q/S/F/C）、速度推定、MoE オフロード
providers.rs -- ランタイムプロバイダー統合（Ollama、llama.cpp、MLX、Docker Model Runner、LM Studio）、インストール検出、pull/ダウンロード
display.rs -- 従来型 CLI テーブルレンダリング + JSON 出力
tui_app.rs -- TUI アプリケーション状態、フィルター、ナビゲーション
tui_ui.rs -- TUI レンダリング（ratatui）
tui_events.rs -- TUI キーボードイベント処理（crossterm）
data/
hf_models.json -- モデルデータベース（206 モデル）
skills/
llmfit-advisor/ -- ハードウェアを考慮したモデル推奨のための OpenClaw スキル
scripts/
scrape_hf_models.py -- HuggingFace API スクレイパー
update_models.sh -- 自動データベース更新スクリプト
install-openclaw-skill.sh -- OpenClaw スキルをインストール
Makefile -- ビルドとメンテナンスのコマンド
```
---
## crates.io への公開
`Cargo.toml` にはすでに必要なメタデータ（description、license、repository）が含まれています。公開するには:
```sh
# 問題を検出するため、まずドライラン
cargo publish --dry-run
# 本番の公開（crates.io API トークンが必要）
cargo login
cargo publish
```
公開する前に、以下を確認してください:
- `Cargo.toml` のバージョンが正しいこと（リリースごとに上げる）。
- リポジトリのルートに `LICENSE` ファイルが存在すること。なければ作成します:
```sh
# MIT ライセンスの場合:
curl -sL https://opensource.org/license/MIT -o LICENSE
# または独自に書く。Cargo.toml は license = "MIT" を宣言しています。
```
- `data/hf_models.json` がコミットされていること。これはコンパイル時に埋め込まれ、公開される crate に含まれている必要があります。
更新を公開するには:
```sh
# バージョンを上げる
# Cargo.toml を編集: version = "0.2.0"
cargo publish
```
---
## 依存関係
| Crate | 目的 |
|------------------------|--------------------------------------------------|
| `clap` | derive マクロによる CLI 引数解析 |
| `sysinfo` | クロスプラットフォームの RAM と CPU の検出 |
| `serde` / `serde_json` | モデルデータベースの JSON デシリアライズ |
| `tabled` | CLI テーブルフォーマット |
| `colored` | CLI のカラー出力 |
| `ureq` | ランタイム/プロバイダー API 統合のための HTTP クライアント |
| `ratatui` | ターミナル UI フレームワーク |
| `crossterm` | ratatui のためのターミナル入出力バックエンド |
---
## ランタイムプロバイダー統合
llmfit は複数のローカルランタイムプロバイダーをサポートします:
- **Ollama**（デーモン/API ベースの pull）
- **llama.cpp**（Hugging Face からの直接 GGUF ダウンロード + ローカルキャッシュ検出）
- **MLX**（Apple Silicon / mlx-community のモデルキャッシュ + オプションのサーバー） — MLX のダウンロードは元のモデル公開者ではなく、HuggingFace の `mlx-community/*` リポジトリにマッピングされます
- **Docker Model Runner**（Docker Desktop 組み込みのモデルサービング）
- **LM Studio**（モデル管理 + ダウンロード用の REST API を備えたローカルモデルサーバー）
あるモデルに対して互換性のあるプロバイダーが複数利用可能な場合、TUI で `d` を押すとプロバイダー選択モーダルが開きます。
### Ollama 統合
llmfit は [Ollama](https://ollama.com) と統合し、すでにインストールされているモデルを検出したり、TUI から直接新しいモデルをダウンロードしたりできます。
### 要件
- **Ollama がインストールされ実行中であること**（`ollama serve` または Ollama デスクトップアプリ）
- llmfit は `http://localhost:11434`（Ollama のデフォルト API ポート）に接続します
- 設定は不要 — Ollama が実行中なら、llmfit は自動的に検出します
### リモートの Ollama インスタンス
別のマシンやポートで実行中の Ollama に接続するには、`OLLAMA_HOST` 環境変数を設定します:
```sh
# 特定の IP とポートの Ollama に接続
OLLAMA_HOST="http://192.168.1.100:11434" llmfit
# ホスト名経由で接続
OLLAMA_HOST="http://ollama-server:666" llmfit
# すべての TUI および CLI コマンドで動作
OLLAMA_HOST="http://192.168.1.100:11434" llmfit --cli
OLLAMA_HOST="http://192.168.1.100:11434" llmfit fit --perfect -n 5
```
これは以下の場合に便利です:
- あるマシンで llmfit を実行し、Ollama を別のマシンからサーブする（例: GPU サーバー + ノート PC クライアント）
- カスタムポートの Docker コンテナで実行中の Ollama に接続する
- リバースプロキシやロードバランサーの背後にある Ollama を使う
### 仕組み
起動時、llmfit は `GET /api/tags` をクエリしてインストール済みの Ollama モデルを一覧表示します。インストール済みの各モデルには、TUI の **Inst** 列に緑の **✓** が付きます。システムバーには `Ollama: ✓ (N installed)` と表示されます。
モデルで `d` を押すと、llmfit は `POST /api/pull` を Ollama に送ってダウンロードします。行はアニメーション付きのプログレスインジケーターでハイライトされ、ダウンロードの進捗がリアルタイムで表示されます。完了すると、モデルはすぐに Ollama で利用可能になります。
Ollama が実行されていない場合、Ollama 固有の操作はスキップされます。TUI は利用可能であれば llama.cpp など他のプロバイダーを引き続きサポートします。
### llama.cpp 統合
llmfit は [llama.cpp](https://github.com/ggml-org/llama.cpp) を、TUI と CLI の両方でランタイム/ダウンロードプロバイダーとして統合します。
要件:
- `llama-cli` または `llama-server` が `PATH` で利用可能であること（ランタイム検出用）
- GGUF ダウンロードのための Hugging Face へのネットワークアクセス
仕組み:
- llmfit は HF モデルを既知の GGUF リポジトリにマッピングします（ヒューリスティックなフォールバック付き）
- GGUF ファイルをローカルの llama.cpp モデルキャッシュにダウンロードします
- 一致する GGUF ファイルがローカルに存在する場合、モデルをインストール済みとしてマークします
#### 環境変数
| 変数 | デフォルト | 説明 |
|---|---|---|
| `LLAMA_CPP_PATH` | *(なし)* | llama.cpp バイナリ（`llama-cli`、`llama-server`）を含むディレクトリ。`PATH` ルックアップの前にチェックされます。 |
| `LLAMA_SERVER_PORT` | `8080` | ランタイム検出のために実行中の `llama-server` のヘルスエンドポイントを探索する際に使用するポート。 |
llama.cpp が標準外の場所にインストールされている場合は、`PATH` に含めることを要求せずに llmfit が見つけられるよう、`LLAMA_CPP_PATH` を設定してください。
### Docker Model Runner 統合
llmfit は Docker Desktop 組み込みのモデルサービング機能である [Docker Model Runner](https://docs.docker.com/desktop/features/model-runner/) と統合します。
要件:
- Model Runner が有効になった Docker Desktop
- デフォルトエンドポイント: `http://localhost:12434`
仕組み:
- llmfit は `GET /engines` をクエリして Docker Model Runner で利用可能なモデルを一覧表示します
- モデルは Ollama スタイルのタグマッピングを使って HF データベースと照合されます（Docker Model Runner は `ai/<tag>` 命名を使用）
- TUI で `d` を押すと `docker model pull` 経由で pull します
### リモートの Docker Model Runner インスタンス
別のホストやポートの Docker Model Runner に接続するには、`DOCKER_MODEL_RUNNER_HOST` 環境変数を設定します:
```sh
DOCKER_MODEL_RUNNER_HOST="http://192.168.1.100:12434" llmfit
```
### LM Studio 統合
llmfit は、組み込みのモデルダウンロード機能を備えたローカルモデルサーバーである [LM Studio](https://lmstudio.ai) と統合します。
要件:
- LM Studio がローカルサーバーを有効にして実行中であること
- デフォルトエンドポイント: `http://127.0.0.1:1234`
仕組み:
- llmfit は `GET /v1/models` をクエリして LM Studio で利用可能なモデルを一覧表示します
- TUI で `d` を押すと `POST /api/v1/models/download` 経由でダウンロードをトリガーします
- ダウンロードの進捗は `GET /api/v1/models/download-status` をポーリングして追跡されます
- LM Studio は HuggingFace のモデル名を直接受け入れるため、名前のマッピングは不要です
### リモートの LM Studio インスタンス
別のホストやポートの LM Studio に接続するには、`LMSTUDIO_HOST` 環境変数を設定します:
```sh
LMSTUDIO_HOST="http://192.168.1.100:1234" llmfit
```
### モデル名のマッピング
llmfit のデータベースは HuggingFace のモデル名（例: `Qwen/Qwen2.5-Coder-14B-Instruct`）を使いますが、Ollama は独自の命名スキーム（例: `qwen2.5-coder:14b`）を使います。llmfit は両者の間の正確なマッピングテーブルを維持し、インストール検出と pull が正しいモデルに解決されるようにします。各マッピングは厳密で、`qwen2.5-coder:14b` はベースの `qwen2.5:14b` ではなく Coder モデルにマッピングされます。
---
## プラットフォームサポート
- **Linux** -- フルサポート。GPU 検出は `nvidia-smi`（NVIDIA）、`rocm-smi`（AMD）、sysfs/`lspci`（Intel Arc）、`npu-smi`（Ascend）経由。
- **macOS (Apple Silicon)** -- フルサポート。`system_profiler` 経由で統合メモリを検出。VRAM = システム RAM（共有プール）。モデルは Metal GPU アクセラレーション経由で実行。
- **macOS (Intel)** -- RAM と CPU の検出が動作。`nvidia-smi` が利用可能ならディスクリート GPU を検出。
- **Windows** -- RAM と CPU の検出が動作。インストールされていれば `nvidia-smi` 経由で NVIDIA GPU を検出。
- **Android / Termux / PRoot** -- CPU と RAM の検出は通常動作しますが、GPU の自動検出は現在サポートされていません。Adreno などのモバイル GPU は、llmfit が使うデスクトップ/サーバーの探索インターフェースからは通常見えません。
### GPU サポート
| ベンダー | 検出方法 | VRAM レポート |
|------------------------|-------------------------------|--------------------------------|
| NVIDIA | `nvidia-smi` | 正確な専用 VRAM |
| AMD | `rocm-smi` | 検出（VRAM は不明な場合あり） |
| Intel Arc（ディスクリート） | sysfs（`mem_info_vram_total`） | 正確な専用 VRAM |
| Intel Arc（統合） | `lspci` | 共有システムメモリ |
| Apple Silicon | `system_profiler` | 統合メモリ（= システム RAM） |
| Ascend | `npu-smi` | 検出（VRAM は不明な場合あり） |
自動検出が失敗したり誤った値を報告したりする場合は、`--memory`、`--ram`、`--cpu-cores` を使って上書きしてください（上記の[ハードウェアの上書き](#ハードウェアの上書き)を参照）。
### Android / Termux に関する注意
**Termux + PRoot** などの Android 構成では、llmfit は標準的な Linux 検出パス（`nvidia-smi`、`rocm-smi`、DRM/sysfs、`lspci` など）を通じてモバイル GPU を見ることが通常できません。そうした環境では、現在の実装では「GPU が検出されない」のが想定どおりの動作です。
それでも統合メモリのスマートフォンやタブレットで GPU スタイルの推奨が欲しい場合は、手動のメモリ上書きを使ってください:
```sh
llmfit --memory=8G fit -n 20
llmfit recommend --json --memory=8G --limit 10
```
これは推奨/スコアリングのみのための回避策であり、真の Android GPU ランタイム検出を提供するものではありません。
---
## コントリビューション
コントリビューション、特に新しいモデルの追加を歓迎します。
### PR を提出する前に
変更をプッシュする前に `cargo fmt` を実行してください。ほとんどの CI チェックの失敗は、フォーマットされていないコードが原因です:
```sh
cargo fmt
```
### モデルの追加
1. モデルの HuggingFace リポジトリ ID（例: `meta-llama/Llama-3.1-8B`）を `scripts/scrape_hf_models.py` の `TARGET_MODELS` リストに追加します。
2. モデルがゲートされている場合（メタデータへのアクセスに HuggingFace 認証が必要な場合）、同じスクリプトの `FALLBACKS` リストにパラメータ数とコンテキスト長を含むフォールバックエントリを追加します。
3. 自動更新スクリプトを実行します:
```sh
make update-models
# または: ./scripts/update_models.sh
```
4. 更新されたモデルリストを確認します: `./target/release/llmfit list`
5. 次を実行して [MODELS.md](MODELS.md) を更新します: `python3 << 'EOF' < scripts/...`（ジェネレータースクリプトはコミット履歴を参照）
6. プルリクエストを開きます。
現在のリストは [MODELS.md](MODELS.md)、アーキテクチャの詳細は [AGENTS.md](AGENTS.md) を参照してください。
---
## OpenClaw 統合
llmfit は [OpenClaw](https://github.com/openclaw/openclaw) スキルとして提供され、エージェントがハードウェアに適したローカルモデルを推奨し、Ollama/vLLM/LM Studio プロバイダーを自動設定できるようにします。
### スキルのインストール
```sh
# llmfit リポジトリから
./scripts/install-openclaw-skill.sh
# または手動で
cp -r skills/llmfit-advisor ~/.openclaw/skills/
```
インストールしたら、OpenClaw エージェントに次のようなことを尋ねられます:
- 「どんなローカルモデルを実行できる?」
- 「私のハードウェアに合うコーディングモデルを推奨して」
- 「私の GPU に最適なモデルで Ollama をセットアップして」
エージェントは内部で `llmfit recommend --json` を呼び出し、結果を解釈して、最適なモデル選択で `openclaw.json` を設定することを提案します。
### 仕組み
このスキルは OpenClaw エージェントに次のことを教えます:
1. `llmfit --json system` でハードウェアを検出する
2. `llmfit recommend --json` でランク付けされた推奨を取得する
3. HuggingFace のモデル名を Ollama/vLLM/LM Studio のタグにマッピングする
4. `openclaw.json` の `models.providers.ollama.models` を設定する
完全なスキル定義は [skills/llmfit-advisor/SKILL.md](skills/llmfit-advisor/SKILL.md) を参照してください。
---
## 代替ツール
別のアプローチをお探しなら、[llm-checker](https://github.com/Pavelevich/llm-checker) をチェックしてください。これは Ollama 統合を備えた Node.js 製の CLI ツールで、モデルを直接 pull してベンチマークできます。スペックから推定するのではなく、実際に Ollama 経由でハードウェア上でモデルを動かすという、より実践的なアプローチを取ります。すでに Ollama がインストールされていて実環境のパフォーマンスをテストしたい場合に適しています。ただし MoE（Mixture-of-Experts）アーキテクチャはサポートしていない点に注意してください。すべてのモデルが密（dense）として扱われるため、Mixtral や DeepSeek-V3 のようなモデルのメモリ推定値は、より小さいアクティブな部分集合ではなく総パラメータ数を反映します。
---
## コード署名
llmfit の Windows リリースバイナリは [SignPath.io](https://about.signpath.io/) によりデジタル署名（Authenticode）されており、コード署名証明書は [SignPath Foundation](https://signpath.org/) から無償で提供されています。
署名は[リリースパイプライン](.github/workflows/release.yml)で自動的に行われます。署名に提出されるのは GitHub Actions によって本リポジトリからビルドされた成果物のみで、署名リクエストはプロジェクトメンテナー（[@AlexsJones](https://github.com/AlexsJones)）が承認します。
**コード署名ポリシー：**[SignPath Foundation のコード署名ポリシーと利用規約](https://signpath.org/terms)を参照してください。
**プライバシー：**本プログラムは、ユーザーまたは本プログラムをインストール・操作する人が明示的に要求しない限り、他のネットワークシステムへ情報を送信することはありません。llmfit が外部サービスにアクセスするのは、該当機能（モデルのダウンロード、ランタイムプロバイダーへの問い合わせ、コミュニティリーダーボードなど）を明示的に使用した場合のみです。
---
## ライセンス
MIT
README.md
+68 -8
View File
@@ -6,7 +6,8 @@
<p align="center">
<b>English</b> ·
<a href="README.zh.md">中文</a>
<a href="README.zh.md">中文</a> ·
<a href="README.ja.md">日本語</a>
</p>
<p align="center">
@@ -98,6 +99,10 @@ This prints JSON from `llmfit recommend` command. The JSON could be further quer
```
podman run ghcr.io/alexsjones/llmfit recommend --use-case coding | jq '.models[].name'
```
To launch the interactive TUI instead, pass the global `--tui` flag:
```sh
docker run --rm -it ghcr.io/alexsjones/llmfit --tui
```
### From source
```sh
@@ -132,7 +137,7 @@ Launches the interactive terminal UI. Your system specs (CPU, RAM, GPU name, VRA
| `V` | Enter Select mode (column-based filtering) |
| `t` | Cycle color theme (saved automatically) |
| `p` | Open Plan mode for selected model (hardware planning) |
| `P` | Open provider filter popup |
| `P` | Open provider filter popup (type to fuzzy-filter providers) |
| `U` | Open use-case filter popup |
| `C` | Open capability filter popup |
| `L` | Open license filter popup |
@@ -433,6 +438,10 @@ llmfit fit --perfect -n 5
# Show detected system specs
llmfit system
# Hardware diagnostic report for bug reports (raw nvidia-smi/rocm-smi/sysfs
# output + what llmfit detected) — paste into a GitHub issue
llmfit doctor
# List all models in the database
llmfit list
@@ -576,7 +585,7 @@ llmfit plan "Qwen/Qwen2.5-Coder-0.5B-Instruct" --context 8192 --json
- **Ascend** -- Detected via `npu-smi`.
- **Backend detection** -- Automatically identifies the acceleration backend (CUDA, Metal, ROCm, SYCL, CPU ARM, CPU x86, Ascend) for speed estimation.
2. **Model database** -- Hundreds models sourced from the HuggingFace API, stored in `data/hf_models.json` and embedded at compile time. Memory requirements are computed from parameter counts across a quantization hierarchy (Q8_0 through Q2_K). VRAM is the primary constraint for GPU inference; system RAM is the fallback for CPU-only execution.
2. **Model database** -- Hundreds models sourced from the HuggingFace API, stored in `llmfit-core/data/hf_models.json` and embedded at compile time. Memory requirements are computed from parameter counts across a quantization hierarchy (Q8_0 through Q2_K). VRAM is the primary constraint for GPU inference; system RAM is the fallback for CPU-only execution.
**MoE support** -- Models with Mixture-of-Experts architectures (Mixtral, DeepSeek-V2/V3) are detected automatically. Only a subset of experts is active per token, so the effective VRAM requirement is much lower than total parameter count suggests. For example, Mixtral 8x7B has 46.7B total parameters but only activates ~12.9B per token, reducing VRAM from 23.9 GB to ~6.6 GB with expert offloading.
@@ -593,6 +602,8 @@ llmfit plan "Qwen/Qwen2.5-Coder-0.5B-Instruct" --context 8192 --json
Dimensions are combined into a weighted composite score. Weights vary by use-case category (General, Coding, Reasoning, Chat, Multimodal, Embedding). For example, Chat weights Speed higher (0.35) while Reasoning weights Quality higher (0.55). Models are ranked by composite score, with unrunnable models (Too Tight) always at the bottom.
Task alignment within the Quality dimension uses a curated per-family benchmark table ([llmfit-core/data/use_case_benchmarks.json](llmfit-core/data/use_case_benchmarks.json), aggregated from public coding/reasoning/chat leaderboards), so a strong coding model outranks a larger generalist for `--use-case coding` even at fewer parameters. Families without an entry fall back to name-based heuristics; corrections to the table are welcome PRs.
5. **Speed estimation** -- Token generation in LLM inference is memory-bandwidth-bound: each token requires reading the full model weights once from VRAM. When the GPU model is recognized, llmfit uses its actual memory bandwidth to estimate throughput:
Formula: `(bandwidth_GB_s / model_size_GB) × efficiency_factor`
@@ -655,7 +666,7 @@ python3 scripts/scrape_hf_models.py
cargo build --release
```
The scraper writes `data/hf_models.json`, which is baked into the binary via `include_str!`. The automated update script backs up existing data, validates JSON output, and rebuilds the binary.
The scraper writes `llmfit-core/data/hf_models.json`, which is baked into the binary via `include_str!`. The automated update script backs up existing data, validates JSON output, and rebuilds the binary.
By default, the scraper enriches models with known GGUF download sources from providers like [unsloth](https://huggingface.co/unsloth) and [bartowski](https://huggingface.co/bartowski). Results are cached in `data/gguf_sources_cache.json` (7-day TTL) to avoid repeated API calls. Use `--no-gguf-sources` to skip enrichment for a faster scrape.
@@ -674,8 +685,8 @@ src/
tui_app.rs -- TUI application state, filters, navigation
tui_ui.rs -- TUI rendering (ratatui)
tui_events.rs -- TUI keyboard event handling (crossterm)
data/
hf_models.json -- Model database (206 models)
llmfit-core/data/
hf_models.json -- Model database (embedded at compile time)
skills/
llmfit-advisor/ -- OpenClaw skill for hardware-aware model recommendations
scripts/
@@ -711,7 +722,7 @@ curl -sL https://opensource.org/license/MIT -o LICENSE
# Or write your own. The Cargo.toml declares license = "MIT".
```
- `data/hf_models.json` is committed. It is embedded at compile time and must be present in the published crate.
- `llmfit-core/data/hf_models.json` is committed. It is embedded at compile time and must be present in the published crate.
To publish updates:
@@ -860,6 +871,15 @@ To connect to LM Studio on a different host or port, set the `LMSTUDIO_HOST` env
LMSTUDIO_HOST="http://192.168.1.100:1234" llmfit
```
### API authentication
If your LM Studio instance has **Require API Key** enabled (required for MCP server access), set the `LMSTUDIO_API_KEY` environment variable to provide a Bearer token with all requests:
```sh
export LMSTUDIO_API_KEY="your-api-key-here"
llmfit
```
### Model name mapping
llmfit's database uses HuggingFace model names (e.g. `Qwen/Qwen2.5-Coder-14B-Instruct`) while Ollama uses its own naming scheme (e.g. `qwen2.5-coder:14b`). llmfit maintains an accurate mapping table between the two so that install detection and pulls resolve to the correct model. Each mapping is exact — `qwen2.5-coder:14b` maps to the Coder model, not the base `qwen2.5:14b`.
@@ -914,7 +934,35 @@ Please run `cargo fmt` before pushing your changes. Most CI check failures are c
cargo fmt
```
### Adding a model
### Adding your own models locally (no rebuild needed)
You don't need to modify llmfit or wait for a release to see extra models. Create a `custom_models.json` in llmfit's data directory:
- Linux: `~/.local/share/llmfit/custom_models.json`
- macOS: `~/Library/Application Support/llmfit/custom_models.json`
- Windows: `%APPDATA%\llmfit\custom_models.json`
(or point the `LLMFIT_CUSTOM_MODELS` env var at any path). The file is a JSON array using the same entry format as the built-in catalog — see [llmfit-core/data/schema.json](llmfit-core/data/schema.json); only a few fields are required:
```json
[
{
"name": "my-org/My-Model-7B",
"provider": "my-org",
"parameter_count": "7B",
"min_ram_gb": 5.0,
"recommended_ram_gb": 8.0,
"min_vram_gb": 5.0,
"quantization": "Q4_K_M",
"context_length": 32768,
"use_case": "General chat"
}
]
```
Custom entries with the same name as a catalog model **override** it; new names are added. Optional fields (`is_moe`, `num_hidden_layers`, `gguf_sources`, …) improve estimate accuracy when provided. You can also run `llmfit update` to fetch trending models from HuggingFace without a rebuild.
### Adding a model to the built-in catalog
1. Add the model's HuggingFace repo ID (e.g., `meta-llama/Llama-3.1-8B`) to the `TARGET_MODELS` list in `scripts/scrape_hf_models.py`.
2. If the model is gated (requires HuggingFace authentication to access metadata), add a fallback entry to the `FALLBACKS` list in the same script with the parameter count and context length.
@@ -972,6 +1020,18 @@ If you're looking for a different approach, check out [llm-checker](https://gith
---
## Code signing
llmfit's Windows release binaries are digitally signed (Authenticode) via [SignPath.io](https://about.signpath.io/), with a free code signing certificate provided by the [SignPath Foundation](https://signpath.org/).
Signing happens automatically in the [release pipeline](.github/workflows/release.yml): only artifacts built by GitHub Actions from this repository are submitted for signing, and signing requests are approved by the project maintainer ([@AlexsJones](https://github.com/AlexsJones)).
**Code signing policy:** see the [SignPath Foundation code signing policy and terms](https://signpath.org/terms).
**Privacy:** this program will not transfer any information to other networked systems unless specifically requested by the user or the person installing or operating it. llmfit only contacts external services when you explicitly use the corresponding feature (e.g. model downloads, runtime provider queries, or the community leaderboard).
---
## License
MIT
README.zh.md
+236 -29
View File
@@ -1,27 +1,33 @@
# llmfit
<p align="center">
<img src="assets/icon.svg" alt="llmfit 图标" width="128" height="128">
<img src="assets/icon.svg" alt="llmfit icon" width="128" height="128">
</p>
<p align="center">
<a href="README.md">English</a> ·
<b>中文</b>
<b>中文</b> ·
<a href="README.ja.md">日本語</a>
</p>
<p align="center">
<a href="https://github.com/AlexsJones/llmfit/actions/workflows/ci.yml"><img src="https://github.com/AlexsJones/llmfit/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
<a href="https://crates.io/crates/llmfit"><img src="https://img.shields.io/crates/v/llmfit.svg" alt="Crates.io"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="许可证"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="License"></a>
<a href="https://about.signpath.io"><img src="https://img.shields.io/badge/SignPath-signed-brightgreen?logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIxNiIgaGVpZ2h0PSIxNiIgZmlsbD0id2hpdGUiIHZpZXdCb3g9IjAgMCAxNiAxNiI+PHBhdGggZD0iTTEwLjA2NyA0LjU2N2wtNC43MzQgNC43MzMtMS40LTEuNGExIDEgMCAwIDAtMS40MTQgMS40MTRsMi4xIDIuMWExIDEgMCAwIDAgMS40MTQgMGw1LjQ0LTUuNDRhMSAxIDAgMCAwLTEuNDE0LTEuNDE0eiIvPjwvc3ZnPg==" alt="Signed with SignPath"></a>
</p>
> **新功能：[社区排行榜](#社区排行榜-b)** — 浏览真实用户的实际性能数据。按 `b` 键查看任意 GPU（不仅限于你的设备）实测的 tok/s、TTFT 和 VRAM 数据。从 RTX 5090 到 Apple M1 的 27+ 种硬件预设中进行选择，按 `H` 键在购买或组装硬件前比较真实数据。
**数百种模型与提供商，一条命令即可找出你的硬件能运行哪些模型。**
一款终端工具，根据你系统的 RAM、CPU 和 GPU 为 LLM 模型匹配合适的规格。自动检测硬件，从质量、速度、适配度和上下文四个维度为每个模型打分，告诉你哪些模型能在你的机器上流畅运行。
内置交互式 TUI（默认）和经典 CLI 模式。支持多 GPU 配置、MoE（混合专家）架构、动态量化选择、速度估算，以及本地运行时提供商（Ollama、llama.cpp、MLX、Docker Model Runner、LM Studio）。
**新功能：[硬件模拟](#硬件模拟-s)** — 在 TUI 中按 `S` 模拟不同硬件。覆盖 RAM、VRAM 和 CPU 核心数，无需离开应用即可查看哪些模型适合目标硬件。
**新功能：[社区排行榜](#社区排行榜-b) (`b`)** — 查看与你使用相同硬件的其他用户的真实 tok/s、TTFT 和 VRAM 占用情况。由 [localmaxxing.com](https://localmaxxing.com) 提供支持，这弥合了预估性能与实际性能之间的差距。
同时：[下载管理器](#下载管理器-d) (`D`)、[高级配置](#高级配置-a) (`A`) 和 [硬件模拟](#硬件模拟-s) — 按 `D` 管理下载、查看历史记录、删除模型以及配置下载目录。按 `A` 调整 TPS 效率、运行模式系数和评分权重。按 `S` 模拟不同的硬件。
> **姐妹项目：**
> - [sympozium](https://github.com/sympozium-ai/sympozium/) — 在 Kubernetes 中管理 Agent。
@@ -44,11 +50,18 @@ scoop install llmfit
### macOS / Linux
#### Homebrew
预编译的二进制文件（推荐，适用于所有 macOS/Linux 版本）：
```sh
brew install AlexsJones/llmfit/llmfit
```
或者从 homebrew-core 公式安装，对于没有预编译包的 macOS 版本将从源码编译：
```sh
brew install llmfit
```
### MacPorts
#### MacPorts
```sh
port install llmfit
```
@@ -65,11 +78,24 @@ curl -fsSL https://llmfit.axjns.dev/install.sh | sh
curl -fsSL https://llmfit.axjns.dev/install.sh | sh -s -- --local
```
### uv / pip
安装或更新 llmfit：
```sh
uv tool install -U llmfit
```
不安装直接运行：
```sh
uvx llmfit
```
你也可以像普通 Python 包一样，使用 pip 或 uv 等工具安装 llmfit。
### Docker / Podman
```sh
docker run ghcr.io/alexsjones/llmfit
```
此命令会输出 `llmfit recommend` 的 JSON 结果，可以用 `jq` 进一步查询。
此命令会输出 `llmfit recommend` 命令的 JSON 结果，可以用 `jq` 进一步查询。
```
podman run ghcr.io/alexsjones/llmfit recommend --use-case coding | jq '.models[].name'
```
@@ -92,7 +118,7 @@ cargo build --release
llmfit
```
启动交互式终端 UI。系统配置（CPU、RAM、GPU 名称、VRAM、后端）显示在顶部。模型按综合评分排序，以可滚动的表格列出。每行显示模型的评分、预估 tok/s、最佳量化方案、运行模式、内存占用和用途分类。
启动交互式终端 UI。系统配置（CPU、RAM、GPU 名称、VRAM、后端）显示在顶部。模型按综合评分排序，以可滚动的表格列出。每行显示模型的评分、预估 tok/s、适合你硬件的最佳量化方案、运行模式、内存占用和用途分类。
| 按键 | 操作 |
|----------------------------|-------------------------------------------------|
@@ -106,19 +132,23 @@ llmfit
| `v` | 进入 Visual 模式（多选模型） |
| `V` | 进入 Select 模式（按列过滤） |
| `t` | 切换颜色主题（自动保存） |
| `p` | 打开 Plan 模式（硬件规划） |
| `P` | 打开提供商过滤弹窗 |
| `p` | 打开 Plan 模式用于选中的模型（硬件规划） |
| `P` | 打开提供商过滤弹窗（输入以模糊过滤提供商） |
| `U` | 打开用途过滤弹窗 |
| `C` | 打开能力过滤弹窗 |
| `L` | 打开许可证过滤弹窗 |
| `R` | 打开运行时/后端过滤弹窗（llama.cpp、MLX、vLLM） |
| `S` | 打开硬件模拟弹窗（覆盖 RAM/VRAM/CPU） |
| `A` | 打开高级配置弹窗（调整效率、运行模式系数） |
| `b` | 打开社区排行榜视图（localmaxxing.com） |
| `I` | 打开推理基准测试视图（对你的模型进行本地质量评分） |
| `h` | 打开帮助弹窗（所有快捷键） |
| `m` | 标记选中模型用于对比 |
| `c` | 打开对比视图（已标记 vs 选中） |
| `x` | 清除对比标记 |
| `i` | 切换已安装优先排序（任何已检测的运行时提供商） |
| `d` | 下载选中模型（多个提供商可用时弹出选择器） |
| `D` | 打开下载管理器（历史记录、删除、配置） |
| `r` | 从运行时提供商刷新已安装模型 |
| `Enter` | 切换选中模型的详情视图 |
| `PgUp` / `PgDn` | 滚动 10 行 |
@@ -148,25 +178,25 @@ TUI 使用类 Vim 模式，当前模式显示在左下角状态栏。当前模
#### Select 模式 (`V`)
按列过滤。按 `V`（shift-v）进入 Select 模式，然后用 `h`/`l` 或方向键在列标题间移动。当前列会高亮显示。按 `Enter` 或 `Space` 激活该列对应的过滤器：
按列过滤。按 `V`（shift-v）进入 Select 模式，然后用 `h`/`l` 或方向键在列标题间移动。当前列会高亮显示。按 `Enter` 或 `Space` 触发该列当前的过滤操作。
| 列 | 过滤操作 |
|---------------------------|-------------------------------------------------------|
| Inst | 切换可用性过滤 |
| Model | 进入搜索模式 |
| Provider | 打开提供商弹窗 |
| Params | 打开参数量分组弹窗（<3B、3-7B、7-14B、14-30B、30-70B、70B+） |
| Score、tok/s、Mem%、Ctx、Date | 按该列排序 |
| Quant | 打开量化弹窗 |
| Mode | 打开运行模式弹窗（GPU、MoE、CPU+GPU、CPU） |
| Fit | 切换适配度过滤 |
| Use Case | 打开用途弹窗 |
| 列 | 过滤操作 |
|-------------------------------|-------------------------------------------------------|
| Inst | 切换可用性过滤 |
| Model | 进入搜索模式 |
| Provider | 打开提供商弹窗 |
| Params | 打开参数量分组弹窗（<3B、3-7B、7-14B、14-30B、30-70B、70B+） |
| Score, tok/s, Mem%, Ctx, Date | 按该列排序 |
| Quant | 打开量化弹窗 |
| Mode | 打开运行模式弹窗（GPU、MoE、CPU+GPU、CPU） |
| Fit | 切换适配度过滤 |
| Use Case | 打开用途弹窗 |
在 Select 模式下仍可用 `j`/`k`、方向键、`Ctrl-U`、`Ctrl-D`、`PageUp`、`PageDown`、`Home` 和 `End` 浏览行，以便在应用过滤器时查看效果。按 `Esc` 返回 Normal 模式。
### TUI Plan 模式 (`p`)
Plan 模式与常规适配分析相反：不是问"我的硬件能跑什么？"，而是估算"这个模型配置需要什么硬件？"。
Plan 模式与常规适配分析相反：不是问“我的硬件能跑什么？”，而是估算“这个模型配置需要什么硬件？”。
在选中的行上按 `p`，然后：
@@ -186,7 +216,7 @@ Plan 模式显示以下估算：
### 硬件模拟 (`S`)
在 TUI 中按 `S` 打开硬件模拟弹窗。覆盖 RAM、VRAM 和 CPU 核心数，查看哪些模型适合不同的目标硬件。所有模型评分、适配等级和速度估算会根据模拟配置即时重新计算。
按 `S` 打开硬件模拟弹窗。覆盖 RAM、VRAM 和 CPU 核心数，查看哪些模型适合不同的目标硬件。所有模型评分、适配等级和速度估算会根据模拟配置即时重新计算。
![硬件模拟](assets/simulation.png)
@@ -200,6 +230,154 @@ Plan 模式显示以下估算：
模拟激活时，系统栏和状态栏会显示 `SIM` 标识。整个模型表格会反映模拟硬件，直到你重置。
### 高级配置 (`A`)
按 `A` 打开高级配置弹窗。该面板允许你调整 TPS 估算、运行模式惩罚和综合评分背后的参数 — 解决部分模型（如 Qwen3 30B）tok/s 被高估的 [问题 #449](https://github.com/AlexsJones/llmfit/issues/449)。
所有更改会立即应用并重新计算模型表格。按 `Esc` 接受，或按 `Ctrl-R` 重置为默认值。
| 字段 | 描述 | 默认值 |
|-------------------|-------------------------------------------------------------------------|---------|
| **Efficiency** | 基于带宽的 TPS 的全局效率系数。计算开销 | `0.55` |
| **GPU factor** | 纯 GPU 推理的速度乘数 | `1.0` |
| **CPU Offload** | 当权重溢出到系统 RAM 时的速度乘数 | `0.5` |
| **MoE Offload** | 混合专家模型进行专家切换时的速度乘数 | `0.8` |
| **Tensor Par** | 张量并行推理的速度乘数 | `0.9` |
| **CPU Only** | 纯 CPU 执行的速度乘数 | `0.3` |
| **Context cap** | 内存估算使用的最大上下文长度（留空为默认值） | `auto` |
| 按键 | 操作 |
|------------------------|-----------------------------------------|
| `Tab` / `j` / `k` | 在字段间切换 |
| 输入数字 / `.` | 编辑选中的字段 |
| `Left` / `Right` | 在字段内移动光标 |
| `Backspace` / `Delete` | 删除字符 |
| `Ctrl-U` | 清空当前字段 |
| `Enter` | 应用更改并重新计算所有评分 |
| `Esc` / `q` | 关闭且不应用更改 |
### 下载管理器 (`D`)
按 `D` 打开下载管理器视图。该全屏视图会替换主模型表格并提供三个部分：
- **当前下载 (Active Download)** — 显示当前正在进行的下载及进度条、模型名称和状态信息。
- **配置 (Config)** — 显示（并允许编辑）GGUF 模型目录。配置的路径会在会话间持久保留。
- **历史记录 (History)** — 以最新优先排列的历史下载列表，包含模型名称、提供商、状态和日期。可以从历史记录中移除失败的下载，也可以从提供商中删除已成功的下载模型。
使用 `Tab` / `Shift-Tab` 在各个部分间循环切换焦点。
| 按键 | 操作 |
|------------------------|------------------------------------------------|
| `Tab` / `Shift-Tab` | 循环切换焦点：当前下载 → 配置 → 历史记录 |
| `j` / `k` 或方向键 | 浏览历史记录列表（历史记录为焦点时） |
| `x` | 删除选中的模型（会弹出确认提示） |
| `y` / `n` | 确认或取消删除 |
| `e` | 编辑下载目录（配置为焦点时） |
| `Enter` | 确认目录编辑 |
| `Esc` / `D` / `q` | 关闭并返回模型表格 |
对于失败的下载（如 404 错误），`x` 会将其从历史记录中移除。对于成功的下载，`x` 将从提供商中删除模型（支持 Ollama 和 llama.cpp）。
### 社区排行榜 (`b`)
按 `b` 打开社区排行榜视图。此视图不仅仅依赖于 llmfit 的理论速度估算，而是展示使用相同硬件的其他用户的**真实性能数据** — 实际测量的 tok/s、首次 token 响应时间和峰值 VRAM 使用率。
![社区排行榜](assets/benchmark.jpeg)
数据来源于 [localmaxxing.com](https://localmaxxing.com)（一个社区基准测试数据库）。当你打开该视图时，llmfit 会自动检测你的硬件（GPU 型号、VRAM 层级、Apple Silicon 芯片系列、操作系统）并查询匹配结果。
| 列 | 描述 |
|---------------|------------------------------------------------------------|
| **Model** | HuggingFace 模型 ID |
| **Engine** | 使用的推理运行时（llama.cpp、vLLM、Ollama、MLX 等） |
| **Quant** | 量化格式（Q4_K_M、Q8_0 等） |
| **tok/s** | 测量的输出 token 生成速度 |
| **Total t/s** | 总吞吐量（提示词 + 生成） |
| **TTFT** | 首次 token 响应时间（延迟） |
| **VRAM** | 推理期间的峰值内存使用量 |
| **Ctx** | 基准测试中使用的上下文长度 |
| **User** | 提交者（已验证用户标有 `*`） |
| 按键 | 操作 |
|------------------------|-----------------------------------------|
| `j` / `k` 或方向键 | 浏览结果 |
| `H` | 打开硬件选择器（浏览任何 GPU） |
| `r` | 从 API 刷新 / 重新获取 |
| `b` / `q` / `Esc` | 关闭并返回模型表格 |
按 `H` 打开硬件选择器 — 一个包含 27 种流行 GPU 和芯片的滚动列表（涵盖 RTX 5090 到纯 CPU、Apple Silicon M1-M4 变体、AMD RX/MI 系列以及 NVIDIA 数据中心显卡）。选择任何一种即可立即加载该硬件的基准测试，即使它不是你当前正在使用的硬件。选择 "My Hardware (auto-detect)" 将切回至你自己的系统。
#### API 密钥配置
公开的基准测试无需认证即可工作。要获得完全访问权限，请提供你的 [localmaxxing.com](https://localmaxxing.com) API 密钥：
```sh
# 通过环境变量（推荐）
export LOCALMAXXING_API_KEY="bhk_你的密钥"
llmfit
# 或通过 CLI 标志
llmfit --api-key "bhk_你的密钥"
```
| 变量 | 描述 |
|---|---|
| `LOCALMAXXING_API_KEY` | localmaxxing.com API 的 Bearer token |
### 推理基准测试 (`I`)
按 `I`（大写）打开推理基准测试 (Inference Bench) 视图。这会对你**本地运行的提供商**（Ollama、vLLM 和 MLX）运行**实时推理基准测试**，并通过真实的推理请求测量首次 token 响应时间 (TTFT)、每秒 token 数 (TPS) 和总延迟。
与社区排行榜（展示其他用户的众包数据）不同，推理基准测试在你的实际硬件上测试你的实际模型。
#### TUI 使用
| 按键 | 操作 |
|-----|--------|
| `I` | 打开推理基准测试（自动检测提供商并运行基准测试） |
| `I` (再次) | 在测试视图内重新运行基准测试 |
| `j` / `k` 或方向键 | 浏览模型结果 |
| `Enter` | 打开选中模型的详细视图 |
| `r` | 切换到路由矩阵视图 |
| `q` / `Esc` | 关闭基准测试视图 |
结果会缓存到 `~/.config/llmfit/bench-cache.json`，在后续打开时可立即加载。
#### CLI 使用
```sh
# 自动检测提供商并进行基准测试
llmfit bench
# 测试所有运行中提供商的所有已发现模型
llmfit bench --all
# 通过 Ollama 测试特定模型
llmfit bench --provider ollama llama3.2
# 覆盖端点 URL
llmfit bench --provider ollama --url http://my-server:11434 llama3.2
# 覆盖 vLLM 端点
llmfit bench --provider vllm --url http://localhost:8000
# 以 JSON 格式输出（供脚本使用）
llmfit bench --json
# 运行质量基准测试（用于路由的基于角色的评分）
llmfit bench --quality
# 输出路由矩阵
llmfit bench --quality --routing
```
#### 环境变量
| 变量 | 默认值 | 描述 |
|---|---|---|
| `OLLAMA_HOST` | `http://localhost:11434` | Ollama API 基础 URL |
| `VLLM_PORT` | `8000` | vLLM 服务器端口（被用作 `http://localhost:$VLLM_PORT`） |
### 主题
按 `t` 可在 10 种内置颜色主题间切换。选择会自动保存到 `~/.config/llmfit/theme`，下次启动时恢复。
@@ -420,7 +598,7 @@ llmfit plan "Qwen/Qwen2.5-Coder-0.5B-Instruct" --context 8192 --json
公式：`(bandwidth_GB_s / model_size_GB) × efficiency_factor`
效率因子（0.55）考虑了内核开销、KV 缓存读取和内存控制器效应。该方法已通过 llama.cpp 的公开基准测试验证（[Apple Silicon](https://github.com/ggml-org/llama.cpp/discussions/4167)、[NVIDIA T4](https://github.com/ggml-org/llama.cpp/discussions/4225)）及实际测量数据。
效率因子（0.55）和各模式速度乘数可通过高级配置弹窗（TUI 中的 `A`）进行调整。默认值考虑了内核开销、KV 缓存读取和内存控制器效应。该方法已通过 llama.cpp 的公开基准测试验证（[Apple Silicon](https://github.com/ggml-org/llama.cpp/discussions/4167)、[NVIDIA T4](https://github.com/ggml-org/llama.cpp/discussions/4225)）及实际测量数据。
带宽查找表涵盖约 80 种 GPU，覆盖 NVIDIA（消费级 + 数据中心级）、AMD（RDNA + CDNA）和 Apple Silicon 系列。
@@ -436,7 +614,7 @@ llmfit plan "Qwen/Qwen2.5-Coder-0.5B-Instruct" --context 8192 --json
| CPU (x86) | 70 |
| NPU (Ascend) | 390 |
回退公式：`K / params_b × quant_speed_multiplier`，对 CPU 卸载（0.5x）、纯 CPU（0.3x）和 MoE 专家切换（0.8x）施加惩罚。
回退公式：`K / params_b × quant_speed_multiplier`，通过高级配置弹窗（TUI 中的 `A`）可调整各种运行模式的惩罚系数。
6. **适配分析** -- 评估每个模型的内存兼容性：
@@ -631,10 +809,10 @@ llmfit 与 [llama.cpp](https://github.com/ggml-org/llama.cpp) 集成，在 TUI
| 变量 | 默认值 | 说明 |
|---|---|---|
| `LLAMA_CPP_PATH` | *(无)* | 包含 llama.cpp 二进制文件（`llama-cli`、`llama-server`）的目录。优先于 `PATH` 查找��� |
| `LLAMA_CPP_PATH` | *(无)* | 包含 llama.cpp 二进制文件（`llama-cli`、`llama-server`）的目录。优先于 `PATH` 查找。 |
| `LLAMA_SERVER_PORT` | `8080` | 探测运行中的 `llama-server` 健康端点以进行运行时检测时使用的端口。 |
如果 llama.cpp 安装在非标准位置，设置 `LLAMA_CPP_PATH` 以便 llmfit 无需将其加入 `PATH` 即可找���它。
如果 llama.cpp 安装在非标准位置，设置 `LLAMA_CPP_PATH` 以便 llmfit 无需将其加入 `PATH` 即可找到它。
### Docker Model Runner 集成
@@ -683,6 +861,15 @@ llmfit 与 [LM Studio](https://lmstudio.ai) 集成，作为本地模型服务器
LMSTUDIO_HOST="http://192.168.1.100:1234" llmfit
```
### API 认证
如果你的 LM Studio 实例启用了**Require API Key**（访问 MCP 服务器需要），请设置 `LMSTUDIO_API_KEY` 环境变量以在所有请求中提供 Bearer token：
```sh
export LMSTUDIO_API_KEY="你的-api-key"
llmfit
```
### 模型名称映射
llmfit 的数据库使用 HuggingFace 模型名称（例如 `Qwen/Qwen2.5-Coder-14B-Instruct`），而 Ollama 使用自己的命名方案（例如 `qwen2.5-coder:14b`）。llmfit 维护了一个精确的映射表，确保安装检测和拉取操作解析到正确的模型。每个映射都是精确的 -- `qwen2.5-coder:14b` 映射到 Coder 模型，而不是基础的 `qwen2.5:14b`。
@@ -708,7 +895,7 @@ llmfit 的数据库使用 HuggingFace 模型名称（例如 `Qwen/Qwen2.5-Coder-
| Apple Silicon | `system_profiler` | 统一内存（= 系统 RAM） |
| Ascend | `npu-smi` | 已检测（VRAM 可能未知） |
如果自动检测失败或报告的值不正确，使用 `--memory`、`--ram` 或 `--cpu-cores` 覆盖（参见上方[硬件覆盖](#硬���覆盖)）。
如果自动检测失败或报告的值不正确，使用 `--memory`、`--ram` 或 `--cpu-cores` 覆盖（参见上方[硬件覆盖](#硬件覆盖)）。
### Android / Termux 说明
@@ -729,6 +916,14 @@ llmfit recommend --json --memory=8G --limit 10
欢迎贡献，特别是添加新模型。
### 提交 PR 前
在推送更改前，请运行 `cargo fmt`。大多数 CI 检查失败都是由于未格式化的代码引起的：
```sh
cargo fmt
```
### 添加模型
1. 在 `scripts/scrape_hf_models.py` 的 `TARGET_MODELS` 列表中添加模型的 HuggingFace 仓库 ID（例如 `meta-llama/Llama-3.1-8B`）。
@@ -787,10 +982,22 @@ Agent 会在后台调用 `llmfit recommend --json`，解读结果，并提议用
---
## 代码签名
llmfit 的 Windows 发布二进制文件通过 [SignPath.io](https://about.signpath.io/) 进行数字签名（Authenticode），代码签名证书由 [SignPath Foundation](https://signpath.org/) 免费提供。
签名在[发布流水线](.github/workflows/release.yml)中自动完成：只有由 GitHub Actions 从本仓库构建的产物才会提交签名，签名请求由项目维护者（[@AlexsJones](https://github.com/AlexsJones)）审批。
**代码签名政策：**参见 [SignPath Foundation 代码签名政策与条款](https://signpath.org/terms)。
**隐私：**除非用户或安装/运行本程序的人员明确请求，本程序不会向其他联网系统传输任何信息。llmfit 仅在你明确使用相应功能时才会访问外部服务（例如模型下载、运行时提供商查询或社区排行榜）。
---
## 许可证
MIT
---
*本文档由 [@JasonYeYuhe](https://github.com/JasonYeYuhe) 翻译并维护。如果您发现任何翻译问题或需要增加新特性说明，欢迎提交 Issue 或与我联系。*
*本文档由 [@JasonYeYuhe](https://github.com/JasonYeYuhe) 翻译并维护。如果您发现任何翻译问题或需要增加新特性说明，欢迎提交 Issue 或与我联系。*
data/benchmark_cache.json
-7396
View File
File diff suppressed because it is too large Load Diff
data/hf_models.json
-160753
View File
File diff suppressed because it is too large Load Diff
llmfit-core/Cargo.toml
+4 -1
View File
@@ -18,6 +18,9 @@ regex = "1"
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
serde_yml = "0.0"
sysinfo = "0.38"
sysinfo = "0.39"
ureq = { version = "3.2", features = ["json"] }
which = "8.0.2"
[dev-dependencies]
jsonschema = { version = "0.46", default-features = false }
llmfit-core/data/benchmark_cache.json
+25212 -3307
View File
File diff suppressed because it is too large Load Diff
llmfit-core/data/hf_models.json
+38531 -15196
View File
File diff suppressed because it is too large Load Diff
llmfit-core/data/schema.json
+324
View File
@@ -0,0 +1,324 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://llmfit.dev/schemas/hf_models.schema.json",
"title": "llmfit HuggingFace model catalog",
"description": "Schema for llmfit-core/data/hf_models.json",
"type": "array",
"items": {
"$ref": "#/$defs/model"
},
"$defs": {
"nonNegativeNumberOrNull": {
"oneOf": [
{
"type": "number",
"minimum": 0
},
{
"type": "null"
}
]
},
"positiveIntegerOrNull": {
"oneOf": [
{
"type": "integer",
"minimum": 1
},
{
"type": "null"
}
]
},
"capability": {
"type": "string",
"enum": [
"vision",
"tool_use",
"audio",
"tts"
]
},
"modelFormat": {
"type": "string",
"enum": [
"gguf",
"awq",
"gptq",
"autoround",
"mlx",
"safetensors"
]
},
"quantization": {
"type": "string"
},
"parameterCount": {
"type": "string",
"pattern": "^[0-9]+(?:\\.[0-9]+)?[KMBT]$"
},
"dateOrNull": {
"oneOf": [
{
"type": "string",
"format": "date"
},
{
"type": "null"
}
]
},
"ggufSource": {
"type": "object",
"properties": {
"repo": {
"type": "string",
"minLength": 1
},
"provider": {
"type": "string",
"minLength": 1
}
},
"required": [
"repo",
"provider"
],
"additionalProperties": false
},
"attentionLayout": {
"type": "object",
"properties": {
"full": {
"type": "integer",
"minimum": 0
},
"linear": {
"type": "integer",
"minimum": 0
}
},
"required": [
"full",
"linear"
],
"additionalProperties": false
},
"model": {
"type": "object",
"properties": {
"name": {
"type": "string",
"pattern": "^.+/.+$"
},
"provider": {
"type": "string",
"minLength": 1
},
"parameter_count": {
"$ref": "#/$defs/parameterCount"
},
"parameters_raw": {
"type": "integer",
"minimum": 1
},
"min_ram_gb": {
"type": "number",
"minimum": 0
},
"recommended_ram_gb": {
"type": "number",
"minimum": 0
},
"min_vram_gb": {
"$ref": "#/$defs/nonNegativeNumberOrNull"
},
"quantization": {
"$ref": "#/$defs/quantization"
},
"format": {
"$ref": "#/$defs/modelFormat"
},
"context_length": {
"type": "integer",
"minimum": 1
},
"use_case": {
"type": "string",
"minLength": 1
},
"capabilities": {
"type": "array",
"items": {
"$ref": "#/$defs/capability"
},
"uniqueItems": true
},
"languages": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
},
"uniqueItems": true
},
"pipeline_tag": {
"type": "string"
},
"architecture": {
"oneOf": [
{
"type": "string",
"minLength": 1
},
{
"type": "null"
}
]
},
"hf_downloads": {
"type": "integer",
"minimum": 0
},
"hf_likes": {
"type": "integer",
"minimum": 0
},
"release_date": {
"$ref": "#/$defs/dateOrNull"
},
"is_moe": {
"type": "boolean"
},
"num_experts": {
"$ref": "#/$defs/positiveIntegerOrNull"
},
"active_experts": {
"$ref": "#/$defs/positiveIntegerOrNull"
},
"active_parameters": {
"$ref": "#/$defs/positiveIntegerOrNull"
},
"num_hidden_layers": {
"$ref": "#/$defs/positiveIntegerOrNull"
},
"num_attention_heads": {
"$ref": "#/$defs/positiveIntegerOrNull"
},
"num_key_value_heads": {
"$ref": "#/$defs/positiveIntegerOrNull"
},
"head_dim": {
"$ref": "#/$defs/positiveIntegerOrNull"
},
"hidden_size": {
"$ref": "#/$defs/positiveIntegerOrNull"
},
"vocab_size": {
"$ref": "#/$defs/positiveIntegerOrNull"
},
"moe_intermediate_size": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "null"
}
]
},
"shared_expert_intermediate_size": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "null"
}
]
},
"gguf_sources": {
"type": "array",
"items": {
"$ref": "#/$defs/ggufSource"
}
},
"license": {
"oneOf": [
{
"type": "string",
"minLength": 1
},
{
"type": "null"
}
]
},
"attention_layout": {
"oneOf": [
{
"$ref": "#/$defs/attentionLayout"
},
{
"type": "null"
}
]
},
"_audio_backends": {
"type": "array",
"items": {
"type": "string"
}
},
"_audio_rtf_cpu": {
"type": "number",
"minimum": 0
},
"_audio_rtf_gpu": {
"type": "number",
"minimum": 0
},
"_audio_vram_gb": {
"$ref": "#/$defs/nonNegativeNumberOrNull"
},
"_discovered": {
"type": "boolean"
}
},
"required": [
"name",
"provider",
"parameter_count",
"min_ram_gb",
"recommended_ram_gb",
"quantization",
"context_length",
"use_case"
],
"additionalProperties": false,
"allOf": [
{
"if": {
"properties": {
"is_moe": {
"const": true
}
},
"required": [
"is_moe"
]
},
"then": {
"required": [
"num_experts",
"active_experts",
"active_parameters"
]
}
}
]
}
}
}
llmfit-core/data/use_case_benchmarks.json
+25
View File
@@ -0,0 +1,25 @@
{
"_comment": "Curated per-family use-case scores (0-100), aggregated from public leaderboards (LiveCodeBench/HumanEval-class for coding, GPQA/MMLU-Pro-class for reasoning, arena-style preference for chat). Values encode RELATIVE family strength per task, not absolute benchmark numbers. Longest matching substring of the lowercased model name wins. Refresh alongside the weekly model-database update; models without an entry fall back to the name heuristic in fit.rs.",
"families": [
{ "match": ["qwen3.5-coder", "qwen3-coder", "qwen2.5-coder"], "scores": { "coding": 90, "reasoning": 78, "chat": 72 } },
{ "match": ["qwen3.5"], "scores": { "coding": 82, "reasoning": 84, "chat": 82 } },
{ "match": ["qwen3"], "scores": { "coding": 78, "reasoning": 80, "chat": 78 } },
{ "match": ["deepseek-v4"], "scores": { "coding": 90, "reasoning": 92, "chat": 84 } },
{ "match": ["deepseek-v3"], "scores": { "coding": 86, "reasoning": 88, "chat": 82 } },
{ "match": ["deepseek-r1", "deepseek-r2"], "scores": { "coding": 84, "reasoning": 93, "chat": 76 } },
{ "match": ["llama-4", "llama4"], "scores": { "coding": 74, "reasoning": 80, "chat": 84 } },
{ "match": ["llama-3.3", "llama3.3"], "scores": { "coding": 70, "reasoning": 76, "chat": 82 } },
{ "match": ["gemma-4", "gemma4"], "scores": { "coding": 72, "reasoning": 76, "chat": 84 } },
{ "match": ["gemma-3", "gemma3"], "scores": { "coding": 68, "reasoning": 72, "chat": 80 } },
{ "match": ["phi-4", "phi4"], "scores": { "coding": 74, "reasoning": 82, "chat": 72 } },
{ "match": ["codestral"], "scores": { "coding": 84, "reasoning": 68, "chat": 62 } },
{ "match": ["devstral"], "scores": { "coding": 86, "reasoning": 70, "chat": 60 } },
{ "match": ["mixtral"], "scores": { "coding": 66, "reasoning": 72, "chat": 74 } },
{ "match": ["mistral"], "scores": { "coding": 68, "reasoning": 70, "chat": 76 } },
{ "match": ["glm-4", "glm4"], "scores": { "coding": 80, "reasoning": 82, "chat": 78 } },
{ "match": ["kimi"], "scores": { "coding": 78, "reasoning": 84, "chat": 80 } },
{ "match": ["granite"], "scores": { "coding": 70, "reasoning": 72, "chat": 70 } },
{ "match": ["starcoder"], "scores": { "coding": 78, "reasoning": 55, "chat": 50 } },
{ "match": ["olmo"], "scores": { "coding": 62, "reasoning": 66, "chat": 70 } }
]
}
llmfit-core/src/analysis.rs
+160
View File
@@ -0,0 +1,160 @@
use crate::fit::{InferenceRuntime, ModelFit};
use crate::hardware::SystemSpecs;
use crate::models::ModelDatabase;
use crate::providers::{
self, DockerModelRunnerProvider, LlamaCppProvider, LmStudioProvider, MlxProvider,
ModelProvider, OllamaProvider, VllmProvider,
};
use std::collections::HashSet;
/// Aggregated installed-model sets from all supported inference providers.
///
/// A single point of truth used by both the CLI and the TUI to check which
/// models are locally installed. Replaces the scattered `HashSet<String>` fields
/// that used to live on each caller's struct.
#[derive(Debug, Clone)]
pub struct InstalledIndex {
pub ollama: HashSet<String>,
pub ollama_count: usize,
pub mlx: HashSet<String>,
pub llamacpp: HashSet<String>,
pub llamacpp_count: usize,
pub docker_mr: HashSet<String>,
pub docker_mr_count: usize,
pub lmstudio: HashSet<String>,
pub lmstudio_count: usize,
pub vllm: HashSet<String>,
pub vllm_count: usize,
}
impl InstalledIndex {
/// Build an empty index — used as a placeholder while providers load.
pub fn empty() -> Self {
Self {
ollama: HashSet::new(),
ollama_count: 0,
mlx: HashSet::new(),
llamacpp: HashSet::new(),
llamacpp_count: 0,
docker_mr: HashSet::new(),
docker_mr_count: 0,
lmstudio: HashSet::new(),
lmstudio_count: 0,
vllm: HashSet::new(),
vllm_count: 0,
}
}
/// Detect installed models across all providers in parallel.
///
/// Each provider query is issued on its own thread so that a single
/// offline/slow backend (worst case ~1.5 s timeout) doesn't serialize
/// into ~9 s of total blocking time for the CLI path.
pub fn detect_all() -> Self {
std::thread::scope(|s| {
let ollama = s.spawn(|| {
let p = OllamaProvider::new();
p.installed_models_counted()
});
let mlx = s.spawn(|| MlxProvider::new().installed_models());
let llamacpp = s.spawn(|| {
let p = LlamaCppProvider::new();
p.installed_models_counted()
});
let docker_mr = s.spawn(|| {
let p = DockerModelRunnerProvider::new();
p.installed_models_counted()
});
let lmstudio = s.spawn(|| {
let p = LmStudioProvider::new();
p.installed_models_counted()
});
let vllm = s.spawn(|| {
let p = VllmProvider::new();
p.installed_models_counted()
});
let (ollama, ollama_count) = ollama.join().unwrap();
let mlx = mlx.join().unwrap();
let (llamacpp, llamacpp_count) = llamacpp.join().unwrap();
let (docker_mr, docker_mr_count) = docker_mr.join().unwrap();
let (lmstudio, lmstudio_count) = lmstudio.join().unwrap();
let (vllm, vllm_count) = vllm.join().unwrap();
Self {
ollama,
ollama_count,
mlx,
llamacpp,
llamacpp_count,
docker_mr,
docker_mr_count,
lmstudio,
lmstudio_count,
vllm,
vllm_count,
}
})
}
/// Returns `true` when the model is installed in **any** provider.
pub fn is_installed(&self, model_name: &str) -> bool {
providers::is_model_installed(model_name, &self.ollama)
|| providers::is_model_installed_mlx(model_name, &self.mlx)
|| providers::is_model_installed_llamacpp(model_name, &self.llamacpp)
|| providers::is_model_installed_docker_mr(model_name, &self.docker_mr)
|| providers::is_model_installed_lmstudio(model_name, &self.lmstudio)
|| providers::is_model_installed_vllm(model_name, &self.vllm)
}
/// Returns the display names of all providers that have this model
/// installed. Used by the detail panel in the TUI.
pub fn installed_providers(&self, model_name: &str) -> Vec<&'static str> {
let mut out = Vec::new();
if providers::is_model_installed(model_name, &self.ollama) {
out.push("Ollama");
}
if providers::is_model_installed_mlx(model_name, &self.mlx) {
out.push("MLX");
}
if providers::is_model_installed_llamacpp(model_name, &self.llamacpp) {
out.push("llama.cpp");
}
if providers::is_model_installed_docker_mr(model_name, &self.docker_mr) {
out.push("Docker");
}
if providers::is_model_installed_lmstudio(model_name, &self.lmstudio) {
out.push("LM Studio");
}
if providers::is_model_installed_vllm(model_name, &self.vllm) {
out.push("vLLM");
}
out
}
}
/// Build a complete `Vec<ModelFit>` with installed markers populated.
///
/// Filters models that are backend-incompatible, runs fit analysis, marks
/// each fit's `installed` flag from the given index, and returns the results
/// **unsorted** so the caller can apply its own sort criteria.
pub fn build_model_fits(
db: &ModelDatabase,
specs: &SystemSpecs,
installed: &InstalledIndex,
context_limit: Option<u32>,
forced_runtime: Option<InferenceRuntime>,
) -> Vec<ModelFit> {
use crate::fit::backend_compatible;
db.get_all_models()
.iter()
.filter(|m| backend_compatible(m, specs))
.map(|m| {
let mut fit =
ModelFit::analyze_with_forced_runtime(m, specs, context_limit, forced_runtime);
fit.installed = installed.is_installed(&m.name);
fit
})
.collect()
}
llmfit-core/src/bench.rs
+14 -16
View File
@@ -362,13 +362,12 @@ pub fn auto_detect_target(model_hint: Option<&str>) -> Result<BenchTarget, Strin
.build()
.call()
.is_ok()
&& let Ok(model_name) = detect_ollama_model(&ollama_url, model_hint)
{
if let Ok(model_name) = detect_ollama_model(&ollama_url, model_hint) {
return Ok(BenchTarget::Ollama {
url: ollama_url,
model: model_name,
});
}
return Ok(BenchTarget::Ollama {
url: ollama_url,
model: model_name,
});
}
// Check MLX
@@ -380,13 +379,12 @@ pub fn auto_detect_target(model_hint: Option<&str>) -> Result<BenchTarget, Strin
.build()
.call()
.is_ok()
&& let Ok(model_name) = detect_openai_model(&mlx_url, model_hint)
{
if let Ok(model_name) = detect_openai_model(&mlx_url, model_hint) {
return Ok(BenchTarget::Mlx {
url: mlx_url,
model: model_name,
});
}
return Ok(BenchTarget::Mlx {
url: mlx_url,
model: model_name,
});
}
Err("No inference provider found. Start Ollama, vLLM, or MLX first.".to_string())
@@ -518,10 +516,10 @@ fn detect_openai_model(base_url: &str, hint: Option<&str>) -> Result<String, Str
if let Some(hint) = hint {
let hint_lower = hint.to_lowercase();
for m in models {
if let Some(id) = m.get("id").and_then(|i: &serde_json::Value| i.as_str()) {
if id.to_lowercase().contains(&hint_lower) {
return Ok(id.to_string());
}
if let Some(id) = m.get("id").and_then(|i: &serde_json::Value| i.as_str())
&& id.to_lowercase().contains(&hint_lower)
{
return Ok(id.to_string());
}
}
}
llmfit-core/src/benchmarks.rs
+27
View File
@@ -101,6 +101,8 @@ pub struct LeaderboardEntry {
#[serde(default)]
pub engine: Option<LeaderboardEngine>,
#[serde(default)]
pub engine_flags: Option<LeaderboardEngineFlags>,
#[serde(default)]
pub user: Option<LeaderboardUser>,
}
@@ -119,6 +121,19 @@ pub struct LeaderboardModel {
pub is_mo_e: Option<bool>,
}
/// Per-run engine acceleration flags. Speculative decoding and MTP runs
/// measure draft-accelerated throughput, which can exceed the memory
/// bandwidth roofline that plain autoregressive estimates model — consumers
/// comparing against `estimate_tps` must filter these out.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct LeaderboardEngineFlags {
#[serde(default)]
pub spec_decoding: Option<bool>,
#[serde(default)]
pub mtp_enabled: Option<bool>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct LeaderboardEngine {
@@ -340,6 +355,18 @@ pub fn cache_timestamp() -> Option<&'static str> {
embedded_cache().scraped_at.as_deref()
}
/// All preset labels present in the embedded benchmark cache. Used by the
/// estimate-calibration test to replay every cached measurement.
pub fn cached_preset_labels() -> Vec<&'static str> {
let mut labels: Vec<&'static str> = embedded_cache()
.presets
.keys()
.map(|s| s.as_str())
.collect();
labels.sort_unstable();
labels
}
// ── Fetch functions ──────────────────────────────────────────────────
/// Fetch benchmarks matching the user's hardware.
llmfit-core/src/claim.rs
+364
View File
@@ -0,0 +1,364 @@
//! Kubernetes DRA claim generation.
//!
//! Turns a (model, quantization, min-tok/s) target into a ResourceClaim or
//! ResourceClaimTemplate whose CEL selector encodes the fit inequality
//! against attributes published by the llmfit.ai DRA driver (llmfit-dra).
//! The driver publishes physics inputs (memory capacity, bandwidth); this
//! module inlines the model-specific constants from the database so the
//! kube-scheduler can evaluate fit at claim time. Nothing here runs in the
//! serving path — the output is plain YAML for kubectl/GitOps.
use crate::models::LlmModel;
/// Attribute/driver domain used by llmfit-dra.
pub const DRIVER_DOMAIN: &str = "llmfit.ai";
/// KV-cache / runtime headroom multiplier applied when the requested quant
/// differs from the database entry (whose min_vram_gb already includes it).
const WEIGHTS_HEADROOM: f64 = 1.2;
#[derive(Debug, Clone)]
pub struct ClaimTarget {
pub min_tps: f64,
/// Backend efficiency as a percentage (fit.rs default_efficiency = 0.55).
pub efficiency_pct: u32,
pub device_class: String,
/// Emit a ResourceClaimTemplate (for pod templates) instead of a bare
/// ResourceClaim.
pub template: bool,
/// Override the database entry's quantization.
pub quant: Option<String>,
/// Override the generated metadata.name.
pub name: Option<String>,
}
impl Default for ClaimTarget {
fn default() -> Self {
Self {
min_tps: 20.0,
efficiency_pct: 55,
device_class: DRIVER_DOMAIN.to_string(),
template: false,
quant: None,
name: None,
}
}
}
/// The two constants the CEL selector needs.
#[derive(Debug, PartialEq)]
pub struct FitBounds {
/// Device memory floor, binary gibibytes (weights + headroom).
pub memory_gi: u64,
/// Bandwidth floor in GB/s such that bw × efficiency / weights ≥ min_tps.
pub min_bandwidth_gbs: u64,
/// Weights size used for the bound, for provenance comments.
pub weights_gb: f64,
pub quant: String,
}
pub fn fit_bounds(model: &LlmModel, target: &ClaimTarget) -> Result<FitBounds, String> {
if target.min_tps <= 0.0 {
return Err("--min-tps must be > 0".to_string());
}
if target.efficiency_pct == 0 || target.efficiency_pct > 100 {
return Err("--efficiency must be in 1..=100".to_string());
}
let quant = target
.quant
.clone()
.unwrap_or_else(|| model.quantization.clone());
let weights_gb = model.estimate_disk_gb(&quant);
if weights_gb <= 0.0 {
return Err(format!(
"cannot size model '{}' (unknown parameter count)",
model.name
));
}
// Memory floor: the database's min_vram_gb is authoritative for the
// entry's own quant (it already includes KV/runtime headroom); for a
// quant override, fall back to weights × headroom.
let memory_gb = if quant == model.quantization {
model
.min_vram_gb
.unwrap_or(model.min_ram_gb)
.max(weights_gb)
} else {
weights_gb * WEIGHTS_HEADROOM
};
// tok/s ≈ bandwidth × efficiency / weights ⇒ bandwidth ≥ tps × weights / eff
let min_bw = target.min_tps * weights_gb * 100.0 / f64::from(target.efficiency_pct);
Ok(FitBounds {
memory_gi: memory_gb.ceil() as u64,
min_bandwidth_gbs: min_bw.ceil() as u64,
weights_gb,
quant,
})
}
/// DNS-label-safe name derived from the model name, e.g.
/// "Qwen2.5 32B Instruct" → "qwen2-5-32b-instruct-fit".
pub fn claim_name(model: &LlmModel) -> String {
let mut s: String = model
.name
.to_lowercase()
.chars()
.map(|c| if c.is_ascii_alphanumeric() { c } else { '-' })
.collect();
while s.contains("--") {
s = s.replace("--", "-");
}
let s = s.trim_matches('-');
let mut base = s.chars().take(48).collect::<String>();
base = base.trim_matches('-').to_string();
format!("{base}-fit")
}
/// Render the resolved fit bounds as machine-readable JSON, for programmatic
/// consumers (the llmfit-dra ModelClaim controller renders its own
/// ResourceClaimTemplate from these numbers instead of scraping YAML).
/// `resolver_version` is the binary version, recorded so consumers can
/// re-resolve when the model database advances.
pub fn render_json(
model: &LlmModel,
target: &ClaimTarget,
resolver_version: &str,
) -> Result<String, String> {
let b = fit_bounds(model, target)?;
let name = target.name.clone().unwrap_or_else(|| claim_name(model));
let out = serde_json::json!({
"model": model.name,
"claimName": name,
"quant": b.quant,
"weightsGb": (b.weights_gb * 10.0).round() / 10.0,
"memoryGi": b.memory_gi,
"minBandwidthGBs": b.min_bandwidth_gbs,
"minTps": target.min_tps,
"efficiencyPct": target.efficiency_pct,
"deviceClass": target.device_class,
"resolverVersion": resolver_version,
});
serde_json::to_string_pretty(&out).map_err(|e| format!("JSON serialization failed: {e}"))
}
/// Render the claim YAML. Built as a template string (not serde) so the
/// output carries provenance comments explaining where every constant came
/// from — the file is meant to be committed to GitOps repos and read by
/// humans.
pub fn render(model: &LlmModel, target: &ClaimTarget) -> Result<String, String> {
let b = fit_bounds(model, target)?;
let name = target.name.clone().unwrap_or_else(|| claim_name(model));
let eff = target.efficiency_pct;
let ind = if target.template { " " } else { " " };
// Continuation lines must sit at exactly the block scalar's content
// indentation (first line = ind + 16) for clean YAML folding.
let pad = format!("{ind} ");
// Every optional lookup is guarded with CEL map membership: a missing
// attribute must mean "device does not match", not a CEL runtime error.
// Unguarded access errors on any device without the attribute (the cpu0
// fallback, unindexed virtual display adapters on servers with BMC
// framebuffers) and can wrongly disqualify allocations.
let cel = format!(
"'memory' in device.capacity['{d}'] &&\n\
{pad}device.capacity['{d}'].memory.compareTo(quantity('{mem}Gi')) >= 0 &&\n\
{pad}'memoryBandwidthGBs' in device.attributes['{d}'] &&\n\
{pad}device.attributes['{d}'].memoryBandwidthGBs >= {bw} &&\n\
{pad}'healthy' in device.attributes['{d}'] &&\n\
{pad}device.attributes['{d}'].healthy",
d = DRIVER_DOMAIN,
mem = b.memory_gi,
bw = b.min_bandwidth_gbs,
);
let header = format!(
"# Generated by llmfit claim — do not compute these constants by hand.\n\
# model: {name} ({params:.1}B params, {quant} ≈ {weights:.1} GB weights)\n\
# fit: tok/s ≈ bandwidth × {eff}% / {weights:.1} GB ⇒ bandwidth ≥ {bw} GB/s for ≥ {tps} tok/s\n\
# memory: ≥ {mem} Gi (weights + KV/runtime headroom)\n",
name = model.name,
params = model.params_b(),
quant = b.quant,
weights = b.weights_gb,
eff = eff,
bw = b.min_bandwidth_gbs,
tps = target.min_tps,
mem = b.memory_gi,
);
let devices = format!(
"devices:\n\
{i} requests:\n\
{i} - name: model\n\
{i} exactly:\n\
{i} deviceClassName: {class}\n\
{i} selectors:\n\
{i} - cel:\n\
{i} expression: >-\n\
{i} {cel}",
i = ind,
class = target.device_class,
cel = cel,
);
let body = if target.template {
format!(
"apiVersion: resource.k8s.io/v1\n\
kind: ResourceClaimTemplate\n\
metadata:\n\
\x20 name: {name}\n\
spec:\n\
\x20 spec:\n\
\x20 {devices}\n"
)
} else {
format!(
"apiVersion: resource.k8s.io/v1\n\
kind: ResourceClaim\n\
metadata:\n\
\x20 name: {name}\n\
spec:\n\
\x20 {devices}\n"
)
};
Ok(format!("{header}{body}"))
}
#[cfg(test)]
mod tests {
use super::*;
fn model(quant: &str, min_vram: Option<f64>) -> LlmModel {
serde_json::from_value(serde_json::json!({
"name": "Test Model 7B",
"provider": "test",
"parameter_count": "7B",
"parameters_raw": 7_000_000_000u64,
"min_ram_gb": 8.0,
"recommended_ram_gb": 10.0,
"min_vram_gb": min_vram,
"quantization": quant,
"context_length": 8192,
"use_case": "general",
}))
.unwrap()
}
#[test]
fn render_json_golden() {
let json = render_json(
&model("Q4_K_M", Some(6.0)),
&ClaimTarget::default(),
"9.9.9-test",
)
.unwrap();
let v: serde_json::Value = serde_json::from_str(&json).unwrap();
assert_eq!(v["model"], "Test Model 7B");
assert_eq!(v["claimName"], "test-model-7b-fit");
assert_eq!(v["quant"], "Q4_K_M");
assert_eq!(v["memoryGi"], 6);
assert_eq!(v["minBandwidthGBs"], 148);
assert_eq!(v["minTps"], 20.0);
assert_eq!(v["efficiencyPct"], 55);
assert_eq!(v["deviceClass"], "llmfit.ai");
assert_eq!(v["resolverVersion"], "9.9.9-test");
// weightsGb rounded to one decimal
assert!(v["weightsGb"].as_f64().unwrap() > 3.0);
}
#[test]
fn render_json_propagates_resolution_errors() {
let mut t = ClaimTarget::default();
t.min_tps = 0.0;
assert!(render_json(&model("Q4_K_M", None), &t, "x").is_err());
}
#[test]
fn bounds_use_db_memory_for_entry_quant() {
// Q4_K_M: 7B × 0.58 bpp = 4.06 GB weights; db min_vram 6.0 wins.
let b = fit_bounds(&model("Q4_K_M", Some(6.0)), &ClaimTarget::default()).unwrap();
assert_eq!(b.memory_gi, 6);
// bw ≥ 20 × 4.06 / 0.55 = 147.6… → 148
assert_eq!(b.min_bandwidth_gbs, 148);
assert_eq!(b.quant, "Q4_K_M");
}
#[test]
fn bounds_apply_headroom_on_quant_override() {
let t = ClaimTarget {
quant: Some("Q8_0".to_string()),
..ClaimTarget::default()
};
// Q8_0: 7B × 1.05 bpp = 7.35 GB weights; ×1.2 headroom = 8.82 → 9 Gi.
let b = fit_bounds(&model("Q4_K_M", Some(6.0)), &t).unwrap();
assert_eq!(b.memory_gi, 9);
assert_eq!(b.min_bandwidth_gbs, 268); // 20 × 7.35 / 0.55 = 267.3…
}
#[test]
fn bounds_reject_nonsense() {
assert!(
fit_bounds(
&model("Q4_K_M", None),
&ClaimTarget {
min_tps: 0.0,
..ClaimTarget::default()
}
)
.is_err()
);
assert!(
fit_bounds(
&model("Q4_K_M", None),
&ClaimTarget {
efficiency_pct: 0,
..ClaimTarget::default()
}
)
.is_err()
);
}
#[test]
fn name_is_dns_label_safe() {
let mut m = model("Q4_K_M", None);
m.name = "Qwen2.5 32B Instruct".to_string();
assert_eq!(claim_name(&m), "qwen2-5-32b-instruct-fit");
}
#[test]
fn render_claim_yaml_shape() {
let y = render(&model("Q4_K_M", Some(6.0)), &ClaimTarget::default()).unwrap();
assert!(y.contains("kind: ResourceClaim\n"));
assert!(y.contains("name: test-model-7b-fit"));
assert!(y.contains("deviceClassName: llmfit.ai"));
assert!(y.contains("quantity('6Gi')"));
assert!(y.contains("memoryBandwidthGBs >= 148"));
assert!(y.contains(".healthy"));
}
#[test]
fn render_guards_every_optional_lookup() {
// Missing attributes must be a non-match, not a CEL runtime error:
// each capacity/attribute access is preceded by an `in` guard.
let y = render(&model("Q4_K_M", Some(6.0)), &ClaimTarget::default()).unwrap();
for guard in [
"'memory' in device.capacity['llmfit.ai']",
"'memoryBandwidthGBs' in device.attributes['llmfit.ai']",
"'healthy' in device.attributes['llmfit.ai']",
] {
assert!(y.contains(guard), "missing guard: {guard}\n{y}");
}
// Guard must appear before the corresponding access.
let mem_guard = y.find("'memory' in").unwrap();
let mem_access = y.find(".memory.compareTo").unwrap();
assert!(mem_guard < mem_access);
}
#[test]
fn render_template_wraps_spec() {
let t = ClaimTarget {
template: true,
..ClaimTarget::default()
};
let y = render(&model("Q4_K_M", None), &t).unwrap();
assert!(y.contains("kind: ResourceClaimTemplate\n"));
assert!(y.contains("spec:\n spec:\n"));
}
}
llmfit-core/src/doctor.rs
+249
View File
@@ -0,0 +1,249 @@
//! Hardware diagnostic dump for bug reports.
//!
//! `llmfit doctor` captures the raw output of every external tool the GPU
//! detection paths in [`crate::hardware`] shell out to, alongside what llmfit
//! actually detected. Users paste the dump into GitHub issues; each report
//! then doubles as a parser regression fixture (the verbatim tool output can
//! be dropped straight into `hardware.rs` tests).
use crate::hardware::SystemSpecs;
use std::fmt::Write as _;
/// Cap each captured section so a pathological tool can't flood the report.
const MAX_SECTION_BYTES: usize = 16 * 1024;
/// Run `cmd args…` and return its combined stdout/stderr, or a note that the
/// tool is unavailable. Never fails: missing tools are part of the diagnosis.
fn capture(cmd: &str, args: &[&str]) -> String {
match std::process::Command::new(cmd).args(args).output() {
Ok(out) => {
let mut text = String::new();
let stdout = String::from_utf8_lossy(&out.stdout);
let stderr = String::from_utf8_lossy(&out.stderr);
if !stdout.trim().is_empty() {
text.push_str(stdout.trim_end());
}
if !stderr.trim().is_empty() {
if !text.is_empty() {
text.push_str("\n--- stderr ---\n");
}
text.push_str(stderr.trim_end());
}
if text.is_empty() {
text = format!("(no output, exit status: {})", out.status);
}
truncate(text)
}
Err(e) => format!("(not available: {e})"),
}
}
fn truncate(mut text: String) -> String {
if text.len() > MAX_SECTION_BYTES {
// Truncate on a char boundary at or below the cap.
let mut cut = MAX_SECTION_BYTES;
while !text.is_char_boundary(cut) {
cut -= 1;
}
text.truncate(cut);
text.push_str("\n… (truncated)");
}
text
}
fn section(report: &mut String, title: &str, body: &str) {
let _ = writeln!(report, "## {title}\n```\n{body}\n```\n");
}
/// Walk `/sys/class/drm/card*` and report the fields the sysfs detection
/// paths read: vendor, device id, driver, and dedicated VRAM if exposed.
fn sysfs_drm_summary() -> String {
let entries = match std::fs::read_dir("/sys/class/drm") {
Ok(e) => e,
Err(e) => return format!("(not available: {e})"),
};
let mut cards: Vec<String> = Vec::new();
for entry in entries.flatten() {
let path = entry.path();
let Some(name) = path.file_name().and_then(|f| f.to_str()) else {
continue;
};
if !name.starts_with("card") || name.contains('-') {
continue;
}
let device = path.join("device");
let read = |f: &str| {
std::fs::read_to_string(device.join(f))
.map(|s| s.trim().to_string())
.unwrap_or_else(|_| "-".to_string())
};
let driver = std::fs::read_to_string(device.join("uevent"))
.ok()
.and_then(|u| {
u.lines()
.find(|l| l.starts_with("DRIVER="))
.map(|l| l.trim_start_matches("DRIVER=").to_string())
})
.unwrap_or_else(|| "-".to_string());
cards.push(format!(
"{name}: vendor={} device={} driver={driver} mem_info_vram_total={}",
read("vendor"),
read("device"),
read("mem_info_vram_total"),
));
}
if cards.is_empty() {
"(no /sys/class/drm cardN entries)".to_string()
} else {
cards.sort();
cards.join("\n")
}
}
/// Build the full diagnostic report as Markdown.
///
/// `version` is the binary version string (core doesn't know the crate
/// version of the caller).
pub fn collect_diagnostics(version: &str) -> String {
let mut report = String::new();
let _ = writeln!(report, "# llmfit doctor report\n");
let _ = writeln!(
report,
"Paste this whole report into a GitHub issue at \
https://github.com/AlexsJones/llmfit/issues — the raw tool output \
below is what lets detection bugs become regression tests. It \
contains hardware model names and driver info only.\n"
);
let _ = writeln!(
report,
"- llmfit version: {version}\n- OS: {} ({})\n",
std::env::consts::OS,
std::env::consts::ARCH
);
// What llmfit concluded — shown first so mismatches with the raw
// output below are immediately visible.
let specs = SystemSpecs::detect();
section(&mut report, "Detected by llmfit", &format!("{specs:#?}"));
// NVIDIA
section(
&mut report,
"nvidia-smi (extended query)",
&capture(
"nvidia-smi",
&[
"--query-gpu=addressing_mode,memory.total,name",
"--format=csv,noheader,nounits",
],
),
);
section(
&mut report,
"nvidia-smi (standard query)",
&capture(
"nvidia-smi",
&[
"--query-gpu=memory.total,name",
"--format=csv,noheader,nounits",
],
),
);
// AMD ROCm
section(
&mut report,
"rocm-smi --showmeminfo vram",
&capture("rocm-smi", &["--showmeminfo", "vram"]),
);
section(
&mut report,
"rocm-smi --showproductname",
&capture("rocm-smi", &["--showproductname"]),
);
if cfg!(target_os = "linux") {
section(&mut report, "sysfs DRM cards", &sysfs_drm_summary());
section(&mut report, "lspci (display controllers)", &{
let full = capture("lspci", &["-nn"]);
let filtered: Vec<&str> = full
.lines()
.filter(|l| {
let lower = l.to_lowercase();
lower.contains("vga")
|| lower.contains("3d controller")
|| lower.contains("display controller")
|| lower.starts_with("(not available")
})
.collect();
if filtered.is_empty() {
"(no display controllers listed)".to_string()
} else {
filtered.join("\n")
}
});
}
if cfg!(target_os = "macos") {
section(
&mut report,
"system_profiler SPDisplaysDataType",
&capture("system_profiler", &["SPDisplaysDataType"]),
);
}
if cfg!(target_os = "windows") {
section(
&mut report,
"PowerShell Win32_VideoController",
&capture(
"powershell",
&[
"-NoProfile",
"-Command",
"Get-CimInstance Win32_VideoController | Select-Object Name,AdapterRAM | ForEach-Object { $_.Name + '|' + $_.AdapterRAM }",
],
),
);
}
// Vulkan (fallback path on Linux/Windows) and NPUs — cheap to include
// everywhere; reported as unavailable where the tool is missing.
section(
&mut report,
"vulkaninfo --summary",
&capture("vulkaninfo", &["--summary"]),
);
section(&mut report, "npu-smi info", &capture("npu-smi", &["info"]));
report
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_capture_missing_tool_is_note_not_panic() {
let out = capture("definitely-not-a-real-tool-xyz", &[]);
assert!(out.starts_with("(not available:"), "{out}");
}
#[test]
fn test_truncate_caps_section_and_marks_it() {
let big = "x".repeat(MAX_SECTION_BYTES + 100);
let out = truncate(big);
assert!(out.len() <= MAX_SECTION_BYTES + 20);
assert!(out.ends_with("(truncated)"));
}
#[test]
fn test_report_contains_key_sections() {
let report = collect_diagnostics("0.0.0-test");
assert!(report.contains("# llmfit doctor report"));
assert!(report.contains("llmfit version: 0.0.0-test"));
assert!(report.contains("## Detected by llmfit"));
assert!(report.contains("## rocm-smi --showmeminfo vram"));
assert!(report.contains("## nvidia-smi (extended query)"));
}
}
llmfit-core/src/fit.rs
+644 -67
View File
@@ -111,6 +111,7 @@ pub enum InferenceRuntime {
LlamaCpp, // llama.cpp / Ollama
Mlx, // Apple MLX framework
Vllm, // vLLM (for AWQ/GPTQ/AutoRound pre-quantized models)
Unsupported,
}
impl InferenceRuntime {
@@ -119,6 +120,7 @@ impl InferenceRuntime {
InferenceRuntime::LlamaCpp => "llama.cpp",
InferenceRuntime::Mlx => "MLX",
InferenceRuntime::Vllm => "vLLM",
InferenceRuntime::Unsupported => "unsupported",
}
}
}
@@ -217,6 +219,11 @@ pub struct ModelFit {
pub installed: bool, // model found in a local runtime provider
pub fits_with_turboquant: bool, // TooTight at fp16 KV but fits with TurboQuant KV
pub effective_context_length: u32, // context length used for memory estimation
/// Context (tokens) that actually fits in this run mode's memory pool
/// after weights and overhead, capped at the model's native window.
/// A "Perfect" fit with an 8k usable context out of a 262k window is a
/// very different proposition for coding work (issue #621).
pub usable_context: u32,
}
impl ModelFit {
@@ -292,6 +299,38 @@ impl ModelFit {
));
}
if model.requires_specialized_runtime() {
notes.push(
"Requires a specialized TTS runtime; llama.cpp/MLX/vLLM fit is not supported yet"
.to_string(),
);
return ModelFit {
model: model.clone(),
fit_level: FitLevel::TooTight,
run_mode: RunMode::CpuOnly,
memory_required_gb: default_mem_required,
memory_available_gb: 0.0,
utilization_pct: 0.0,
notes,
moe_offloaded_gb: None,
score: 0.0,
score_components: ScoreComponents {
quality: 0.0,
speed: 0.0,
fit: 0.0,
context: 0.0,
},
estimated_tps: 0.0,
best_quant: model.quantization.clone(),
use_case,
runtime: InferenceRuntime::Unsupported,
installed: false,
fits_with_turboquant: false,
effective_context_length: estimation_ctx,
usable_context: 0,
};
}
// Determine inference runtime up front so path selection can use
// the correct quantization hierarchy.
// Honour the force_runtime override first if provided; otherwise
@@ -524,6 +563,22 @@ impl ModelFit {
));
}
// Usable context: how many tokens of KV cache the pool can actually
// hold once weights and runtime overhead are resident. The KV formula
// is linear in ctx, so derive a per-token cost from a fixed reference
// window. Suggested by @MrMarble in issue #621.
let usable_context = {
const REF_CTX: u32 = 4096;
let fixed_mem = model.estimate_memory_gb(&best_quant_str, 0);
let leftover = (mem_available - fixed_mem).max(0.0);
let per_token_gb = model.kv_cache_gb(REF_CTX, KvQuant::Fp16) / f64::from(REF_CTX);
if per_token_gb > 0.0 {
((leftover / per_token_gb) as u32).min(model.context_length)
} else {
model.context_length
}
};
// Check if a TooTight model would fit with TurboQuant KV compression.
// Only compute on CUDA systems — TurboQuant requires vLLM + CUDA.
let fits_with_turboquant =
@@ -554,9 +609,28 @@ impl ModelFit {
installed: false, // set later by App after provider detection
fits_with_turboquant,
effective_context_length: estimation_ctx,
usable_context,
}
}
/// Context column text: `"262k→14k"` when the memory pool constrains
/// context below the model's native window, plain `"262k"` otherwise.
/// See [`fmt_ctx_tokens`] for the token formatting.
pub fn context_display(&self) -> String {
let native = fmt_ctx_tokens(self.model.context_length);
if self.usable_context < self.model.context_length {
format!("{native}\u{2192}{}", fmt_ctx_tokens(self.usable_context))
} else {
native
}
}
/// True when the usable context is too small for real work (below 4k),
/// so UIs can highlight the constraint.
pub fn context_severely_limited(&self) -> bool {
self.usable_context < 4096 && self.usable_context < self.model.context_length
}
pub fn fit_emoji(&self) -> &str {
match self.fit_level {
FitLevel::Perfect => "🟢",
@@ -593,7 +667,8 @@ impl ModelFit {
/// Pure memory headroom scoring.
/// - GPU (including Apple Silicon unified memory): can reach Perfect.
/// - CpuOffload: caps at Good.
/// - CpuOnly: caps at Marginal -- CPU-only inference is always a compromise.
/// - CpuOnly: caps at Good -- no GPU acceleration so never Perfect, but a model
/// that fits with comfortable headroom is genuinely runnable, not Marginal.
fn score_fit(
mem_required: f64,
mem_available: f64,
@@ -632,8 +707,15 @@ fn score_fit(
}
}
RunMode::CpuOnly => {
// CPU-only is always a compromise -- cap at Marginal
FitLevel::Marginal
// CPU-only never reaches Perfect (that requires a GPU), but a model
// that fits with comfortable headroom is genuinely runnable -- cap
// at Good rather than hiding every CPU-only model behind Marginal.
// (Matches the FitLevel::Good contract: "GPU tight, or CPU comfortable".)
if mem_available >= mem_required * 1.2 {
FitLevel::Good
} else {
FitLevel::Marginal
}
}
}
}
@@ -793,7 +875,9 @@ fn best_quant_for_runtime_budget(
}
pub fn backend_compatible(model: &LlmModel, system: &SystemSpecs) -> bool {
if model.is_mlx_model() {
if model.requires_specialized_runtime() {
false
} else if model.is_mlx_model() {
system.backend == GpuBackend::Metal && system.unified_memory
} else if model.is_prequantized() {
if !matches!(system.backend, GpuBackend::Cuda | GpuBackend::Rocm) {
@@ -878,7 +962,13 @@ pub fn rank_models_by_fit_opts_col(
.utilization_pct
.partial_cmp(&a.utilization_pct)
.unwrap_or(std::cmp::Ordering::Equal),
SortColumn::Ctx => b.model.context_length.cmp(&a.model.context_length),
// Sort by the context that actually fits on this machine, not the
// advertised window — that's the number that constrains real work
// (issue #621). Native window breaks ties.
SortColumn::Ctx => b
.usable_context
.cmp(&a.usable_context)
.then(b.model.context_length.cmp(&a.model.context_length)),
SortColumn::ReleaseDate => {
let a_date = a.model.release_date.as_deref().unwrap_or("");
let b_date = b.model.release_date.as_deref().unwrap_or("");
@@ -1232,6 +1322,7 @@ fn estimate_tps(
// Used when the GPU is not recognized (custom/unnamed GPUs,
// synthetic entries from --memory override, etc.).
let k: f64 = match (system.backend, runtime) {
(_, InferenceRuntime::Unsupported) => 0.0,
(GpuBackend::Metal, InferenceRuntime::Mlx) => 250.0,
(GpuBackend::Metal, InferenceRuntime::LlamaCpp) => 160.0,
(GpuBackend::Metal, InferenceRuntime::Vllm) => 160.0,
@@ -1329,18 +1420,27 @@ fn compute_scores(
fn quality_score(model: &LlmModel, quant: &str, use_case: UseCase) -> f64 {
let params = model.params_b();
// Base quality by parameter count
let base = if params < 1.0 {
// For the base quality tier, MoE models are scored on their *active*
// parameters per token rather than the total across all experts. A model
// like Qwen3-Coder-Next (80B total / 3B active) infers at a quality closer
// to a small dense model, so using the 80B total would inflate its tier.
let quality_params = model
.active_parameters
.map(|a| a as f64 / 1_000_000_000.0)
.unwrap_or(params);
// Base quality by (active) parameter count
let base = if quality_params < 1.0 {
30.0
} else if params < 3.0 {
} else if quality_params < 3.0 {
45.0
} else if params < 7.0 {
} else if quality_params < 7.0 {
60.0
} else if params < 10.0 {
} else if quality_params < 10.0 {
75.0
} else if params < 20.0 {
} else if quality_params < 20.0 {
82.0
} else if params < 40.0 {
} else if quality_params < 40.0 {
89.0
} else {
95.0
@@ -1370,39 +1470,118 @@ fn quality_score(model: &LlmModel, quant: &str, use_case: UseCase) -> f64 {
// Generation bonus: newer model generations get a quality bump
let gen_bonus = models::generation_quality_bonus(model.architecture.as_deref(), &model.name);
// Recency bonus: same-size models improve over time, so a freshly released
// model edges out an identically-sized older one. Uses the catalog
// `release_date` (YYYY-MM-DD); models without a date get no bonus.
let recency_bonus = model
.release_date
.as_deref()
.and_then(|d| months_since(d, current_year_month()))
.map(|months| {
if months < 3 {
3.0
} else if months < 9 {
1.5
} else {
0.0
}
})
.unwrap_or(0.0);
// Quantization penalty
let q_penalty = models::quant_quality_penalty(quant);
// Task alignment bump
let task_bump = match use_case {
UseCase::Coding => {
if name_lower.contains("code")
|| name_lower.contains("starcoder")
|| name_lower.contains("wizard")
// Task alignment bump. Curated benchmark aggregates (per-family table in
// data/use_case_benchmarks.json) take precedence over name heuristics:
// the family's measured task strength, centered on a 72-point baseline,
// maps to a bounded adjustment so it composes with the existing quality
// machinery instead of replacing it (issue #150). Families without an
// entry keep the original heuristics.
let bench_task_key = match use_case {
UseCase::Coding => Some("coding"),
UseCase::Reasoning => Some("reasoning"),
UseCase::Chat => Some("chat"),
_ => None,
};
let bench_score = bench_task_key.and_then(|k| crate::task_bench::score(&name_lower, k));
let task_bump = match bench_score {
Some(bench) => ((bench - 72.0) * 0.4).clamp(-8.0, 9.0),
None => match use_case {
UseCase::Coding => {
if name_lower.contains("code")
|| name_lower.contains("starcoder")
|| name_lower.contains("wizard")
{
6.0
} else {
0.0
}
}
UseCase::Reasoning => {
if params >= 13.0 {
5.0
} else {
0.0
}
}
UseCase::Multimodal
if (name_lower.contains("vision")
|| model.use_case.to_lowercase().contains("vision")) =>
{
6.0
} else {
0.0
}
}
UseCase::Reasoning => {
if params >= 13.0 {
5.0
} else {
0.0
}
}
UseCase::Multimodal => {
if name_lower.contains("vision") || model.use_case.to_lowercase().contains("vision") {
6.0
} else {
0.0
}
}
_ => 0.0,
_ => 0.0,
},
};
(base + family_bump + gen_bonus + q_penalty + task_bump).clamp(0.0, 100.0)
(base + family_bump + gen_bonus + recency_bonus + q_penalty + task_bump).clamp(0.0, 100.0)
}
/// Token count as a compact column string: `"32k"` for ≥1000, raw otherwise.
fn fmt_ctx_tokens(tokens: u32) -> String {
if tokens >= 1000 {
format!("{}k", tokens / 1000)
} else {
tokens.to_string()
}
}
/// Whole months elapsed between a `release_date` (`YYYY-MM-DD`, only the year
/// and month are read) and `now` as a `(year, month)` pair. Returns `None` if
/// the date can't be parsed; negative spans (future dates) clamp to 0.
fn months_since(release_date: &str, now: (i32, u32)) -> Option<u32> {
let mut parts = release_date.split('-');
let year: i32 = parts.next()?.trim().parse().ok()?;
let month: i32 = parts.next()?.trim().parse().ok()?;
let (now_year, now_month) = now;
let diff = (now_year - year) * 12 + (now_month as i32 - month);
Some(diff.max(0) as u32)
}
/// Current `(year, month)` in UTC, derived from the system clock. Falls back to
/// the Unix epoch if the clock is before 1970 (which only removes the bonus).
fn current_year_month() -> (i32, u32) {
let secs = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
civil_from_days((secs / 86_400) as i64)
}
/// Convert days since 1970-01-01 to `(year, month)` in the proleptic Gregorian
/// calendar (Howard Hinnant's `civil_from_days`). Exact — no 365-day/30-day
/// approximations, so the recency bonus doesn't drift across leap years.
fn civil_from_days(z: i64) -> (i32, u32) {
let z = z + 719_468;
let era = if z >= 0 { z } else { z - 146_096 } / 146_097;
let doe = z - era * 146_097; // [0, 146096]
let yoe = (doe - doe / 1460 + doe / 36524 - doe / 146096) / 365; // [0, 399]
let y = yoe + era * 400;
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); // [0, 365]
let mp = (5 * doy + 2) / 153; // [0, 11]
let month = if mp < 10 { mp + 3 } else { mp - 9 }; // [1, 12]
let year = if month <= 2 { y + 1 } else { y };
(year as i32, month as u32)
}
/// Speed score: normalize estimated TPS against target for the use case.
@@ -1421,19 +1600,15 @@ fn fit_score(required: f64, available: f64) -> f64 {
return 0.0;
}
let ratio = required / available;
// Sweet spot: 50-80% utilization scores highest
if ratio <= 0.5 {
// Under-utilizing: still good but not optimal
60.0 + (ratio / 0.5) * 40.0
} else if ratio <= 0.8 {
100.0
} else if ratio <= 0.9 {
// Getting tight
70.0
} else {
// Very tight
50.0
}
// Headroom is good: anything that fits with room to spare is a perfect fit,
// so the score holds a flat 100 up to a comfortable utilization, then eases
// down as memory gets tight via a one-sided Gaussian falloff. This removes
// the old step function's 100 -> 70 cliff at 80% (79% scored 100, 81%
// scored 70) without penalizing models that leave headroom.
const COMFORT: f64 = 0.70;
const SIGMA: f64 = 0.20;
let z = ((ratio - COMFORT) / SIGMA).max(0.0);
(100.0 * (-0.5 * z * z).exp()).clamp(0.0, 100.0)
}
/// Context score: context window capability vs target for the use case.
@@ -1494,6 +1669,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: models::ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -1569,12 +1745,28 @@ mod tests {
}
#[test]
fn test_score_fit_cpu_caps_at_marginal() {
// CPU-only never reaches Perfect
fn test_score_fit_cpu_comfortable_is_good() {
// CPU-only with comfortable headroom (4 GB of 32 GB) is runnable -> Good.
// It never reaches Perfect (recommended 8 GB is met, but Perfect needs a GPU).
let fit = score_fit(4.0, 32.0, 8.0, RunMode::CpuOnly);
assert_eq!(fit, FitLevel::Good);
}
#[test]
fn test_score_fit_cpu_tight_is_marginal() {
// CPU-only that only just fits (under the 1.2x headroom bar) stays Marginal.
let fit = score_fit(8.0, 8.5, 16.0, RunMode::CpuOnly);
assert_eq!(fit, FitLevel::Marginal);
}
#[test]
fn test_score_fit_cpu_never_perfect() {
// Even with enormous headroom, CPU-only caps at Good (no GPU -> not Perfect).
let fit = score_fit(1.0, 64.0, 2.0, RunMode::CpuOnly);
assert_ne!(fit, FitLevel::Perfect);
assert_eq!(fit, FitLevel::Good);
}
#[test]
fn test_score_fit_cpu_offload_caps_at_good() {
// CpuOffload with plenty of headroom caps at Good
@@ -1619,8 +1811,10 @@ mod tests {
// Should use CPU path
assert_eq!(fit.run_mode, RunMode::CpuOnly);
// CPU-only caps at Marginal
assert_eq!(fit.fit_level, FitLevel::Marginal);
// CPU-only with comfortable headroom (7B in 16 GB) is runnable -> Good,
// and never Perfect (that requires a GPU).
assert_eq!(fit.fit_level, FitLevel::Good);
assert_ne!(fit.fit_level, FitLevel::Perfect);
}
#[test]
@@ -1663,6 +1857,26 @@ mod tests {
assert_eq!(fit.fit_level, FitLevel::TooTight);
}
#[test]
fn test_tts_requires_unsupported_runtime() {
let mut model = test_model("82M", 1.0, Some(0.5));
model.quantization = "F16".to_string();
model.format = models::ModelFormat::Safetensors;
model.capabilities = vec![models::Capability::Audio, models::Capability::Tts];
let system = test_system(16.0, true, Some(8.0));
let fit = ModelFit::analyze(&model, &system);
assert_eq!(fit.runtime, InferenceRuntime::Unsupported);
assert_eq!(fit.fit_level, FitLevel::TooTight);
assert_eq!(fit.score, 0.0);
assert!(
fit.notes
.iter()
.any(|n| n.contains("specialized TTS runtime"))
);
}
#[test]
fn test_moe_offload_tries_lower_quantization() {
let model = LlmModel {
@@ -1683,6 +1897,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: models::ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -1727,6 +1942,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: models::ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -1830,28 +2046,43 @@ mod tests {
#[test]
fn test_fit_score_sweet_spot() {
// Sweet spot: 50-80% utilization
let score = fit_score(6.0, 10.0);
assert!(score >= 95.0); // Should be near perfect
// Comfortable utilization (room to spare) is a perfect fit.
assert!((fit_score(6.5, 10.0) - 100.0).abs() < 0.01); // 65%
assert!((fit_score(6.0, 10.0) - 100.0).abs() < 0.01); // 60%
let score2 = fit_score(8.0, 10.0);
assert_eq!(score2, 100.0);
// Past the comfort point the smooth curve eases down instead of holding
// a flat 100 right up to the old 80% cliff.
let score2 = fit_score(8.0, 10.0); // 80%
assert!(score2 > 70.0 && score2 < 100.0);
}
#[test]
fn test_fit_score_under_utilized() {
// Under-utilizing: still good but not optimal
let score = fit_score(2.0, 10.0);
assert!(score >= 60.0);
assert!(score < 100.0);
// Plenty of headroom is a good thing, not waste -- it stays a perfect
// fit rather than being penalized.
assert!((fit_score(2.0, 10.0) - 100.0).abs() < 0.01); // 20%
assert!((fit_score(5.0, 10.0) - 100.0).abs() < 0.01); // 50%
}
#[test]
fn test_fit_score_tight() {
// Very tight fit
let score = fit_score(9.5, 10.0);
assert!(score >= 50.0);
assert!(score < 80.0);
// Very tight fit: still positive but well off the peak, and no longer
// pinned to the old flat 50 floor.
let score = fit_score(9.5, 10.0); // 95% utilization
assert!(score > 0.0 && score < 50.0);
// Smoothly monotonic as it tightens past the peak.
assert!(score < fit_score(8.5, 10.0));
}
#[test]
fn test_fit_score_smooth_no_cliffs() {
// Across the old 80% step boundary, neighbouring ratios stay close
// together instead of jumping 100 -> 70.
// Old step: 79% -> 100, 81% -> 70 (a 30-point cliff). The smooth curve
// keeps neighbours within a few points of each other.
let below = fit_score(7.9, 10.0);
let above = fit_score(8.1, 10.0);
assert!((below - above).abs() < 8.0);
}
#[test]
@@ -1977,6 +2208,77 @@ mod tests {
assert!((score - 70.0).abs() < 0.01, "Got {}", score);
}
#[test]
fn test_quality_score_moe_uses_active_params() {
// 80B total / 3B active MoE: the base tier should follow the 3B active
// count (45 tier), not the 80B total (95 tier).
let mut moe = test_model("80B", 48.0, Some(48.0));
moe.active_parameters = Some(3_000_000_000);
let moe_score = quality_score(&moe, "Q4_K_M", UseCase::General);
// A plain 80B dense model (no active_parameters) keeps the top tier.
let dense = test_model("80B", 48.0, Some(48.0));
let dense_score = quality_score(&dense, "Q4_K_M", UseCase::General);
assert!(
dense_score > moe_score + 30.0,
"MoE (active 3B) {} should be far below dense 80B {}",
moe_score,
dense_score
);
// And it should land near a real 3B dense model's tier.
let small = test_model("3B", 2.0, Some(2.0));
let small_score = quality_score(&small, "Q4_K_M", UseCase::General);
assert!(
(moe_score - small_score).abs() < 0.01,
"MoE active-3B {} should match dense 3B {}",
moe_score,
small_score
);
}
#[test]
fn test_quality_score_recency_bonus() {
// Two otherwise-identical models; the newer one scores higher purely on
// its release date. months_since/current_year_month back the bonus, so
// we exercise the pure helper directly for determinism below.
let mut fresh = test_model("7B", 4.0, Some(4.0));
fresh.release_date = Some("2099-01-01".to_string()); // far future -> 0 months
let mut old = test_model("7B", 4.0, Some(4.0));
old.release_date = Some("2000-01-01".to_string()); // ancient -> no bonus
let fresh_score = quality_score(&fresh, "Q4_K_M", UseCase::General);
let old_score = quality_score(&old, "Q4_K_M", UseCase::General);
assert!(
fresh_score > old_score,
"fresh {} should beat old {}",
fresh_score,
old_score
);
// Fresh gets the full +3 on top of the no-bonus baseline of 70.
assert!((fresh_score - 73.0).abs() < 0.01, "Got {}", fresh_score);
assert!((old_score - 70.0).abs() < 0.01, "Got {}", old_score);
}
#[test]
fn test_months_since_is_deterministic() {
// Pure date math — no dependency on the system clock.
assert_eq!(months_since("2026-06-01", (2026, 6)), Some(0));
assert_eq!(months_since("2026-04-01", (2026, 6)), Some(2)); // < 3 -> +3
assert_eq!(months_since("2025-12-01", (2026, 6)), Some(6)); // < 9 -> +1.5
assert_eq!(months_since("2024-06-01", (2026, 6)), Some(24)); // old -> 0
assert_eq!(months_since("2099-01-01", (2026, 6)), Some(0)); // future clamps
assert_eq!(months_since("not-a-date", (2026, 6)), None);
}
#[test]
fn test_civil_from_days_known_dates() {
assert_eq!(civil_from_days(0), (1970, 1)); // epoch
assert_eq!(civil_from_days(59), (1970, 3)); // 1970-03-01
assert_eq!(civil_from_days(20_454), (2026, 1)); // 2026-01-01
}
#[test]
fn test_weighted_score_composition() {
let components = ScoreComponents {
@@ -2067,6 +2369,268 @@ mod tests {
assert!(capped.notes.iter().any(|n| n.contains("Context capped at")));
}
// ── Estimate calibration against measured community benchmarks ──────
/// Build simulated SystemSpecs for a leaderboard hardware preset label
/// like "RTX 3090 (24 GB)" or "Apple M4 Max (128 GB)". Returns None for
/// presets the calibration can't model faithfully (e.g. "CPU Only",
/// where the CPU model — and thus memory bandwidth — is unknown).
fn specs_for_preset_label(label: &str) -> Option<SystemSpecs> {
let (name, rest) = label.split_once(" (")?;
let vram_gb: f64 = rest
.trim_end_matches(')')
.trim_end_matches(" GB")
.trim()
.parse()
.ok()?;
if name == "CPU Only" {
return None;
}
let unified = name.starts_with("Apple");
let backend = if unified {
GpuBackend::Metal
} else if name.starts_with("RX ") || name.contains("Radeon") {
GpuBackend::Rocm
} else {
GpuBackend::Cuda
};
// The estimator is bandwidth-driven: without a bandwidth entry for
// this GPU the replay would exercise the generic fallback and tell
// us nothing about the preset.
crate::hardware::gpu_memory_bandwidth_gbps(name)?;
let total_ram_gb = if unified {
vram_gb
} else {
(2.0 * vram_gb).max(32.0)
};
Some(SystemSpecs {
total_ram_gb,
available_ram_gb: total_ram_gb * 0.85,
total_cpu_cores: 16,
cpu_name: "calibration".to_string(),
has_gpu: true,
gpu_vram_gb: Some(vram_gb),
total_gpu_vram_gb: Some(vram_gb),
gpu_name: Some(name.to_string()),
gpu_count: 1,
unified_memory: unified,
backend,
gpus: vec![crate::hardware::GpuInfo {
name: name.to_string(),
vram_gb: Some(vram_gb),
backend,
count: 1,
unified_memory: unified,
}],
cluster_mode: false,
cluster_node_count: 0,
})
}
/// Replay every usable measurement in the embedded localmaxxing cache
/// through estimate_tps and check the estimator's overall accuracy.
///
/// This is the estimate↔reality feedback loop (#112/#119): the cache is
/// refreshed weekly, so a drift in either the estimator or the real
/// world shows up here. The bounds are deliberately generous — the test
/// exists to catch egregious regressions (e.g. a 3× systematic bias like
/// #449), not to enforce per-row precision.
#[test]
fn test_estimate_tps_calibration_against_leaderboard() {
let db = crate::models::ModelDatabase::embedded();
let models = db.get_all_models();
let config = CalcConfig::default();
// (preset label, est/measured ratio)
let mut ratios: Vec<(String, f64)> = Vec::new();
let mut skipped_unknown_model = 0usize;
for label in crate::benchmarks::cached_preset_labels() {
let Some(specs) = specs_for_preset_label(label) else {
continue;
};
let Some(resp) = crate::benchmarks::cached_leaderboard_for_preset(label) else {
continue;
};
for row in &resp.rows {
let Some(measured) = row.tok_s_out.filter(|t| *t > 0.5) else {
continue;
};
// Single-request generation throughput only: batched serving
// measures a different quantity than estimate_tps models.
if row.batch_size.unwrap_or(1) > 1 {
continue;
}
// Draft-accelerated runs (speculative decoding / MTP) exceed
// the memory-bandwidth roofline plain autoregressive
// estimates model — e.g. 577 tok/s for a 9B on an 800 GB/s
// card. Comparing against them reads as a 3-4× estimator
// "bias" that isn't one.
if row.engine_flags.as_ref().is_some_and(|f| {
f.spec_decoding.unwrap_or(false) || f.mtp_enabled.unwrap_or(false)
}) {
continue;
}
let hf_id = row.hf_id();
if hf_id.is_empty() {
continue;
}
let slug = crate::models::canonical_slug(hf_id);
let Some(model) = models
.iter()
.find(|m| crate::models::canonical_slug(&m.name) == slug)
else {
skipped_unknown_model += 1;
continue;
};
let quant = {
let q = row.quantization();
if q.is_empty() {
model.quantization.clone()
} else {
q.to_string()
}
};
let engine = row.engine_name().to_lowercase();
let runtime = if engine.contains("mlx") {
InferenceRuntime::Mlx
} else if engine.contains("vllm") {
InferenceRuntime::Vllm
} else {
InferenceRuntime::LlamaCpp
};
// Pure-GPU rows only: offload splits depend on unknown
// per-run layer placement, so estimates aren't comparable.
let ctx = row
.context_length
.unwrap_or(4096)
.min(DEFAULT_ESTIMATION_CTX);
let mem = model.estimate_memory_gb(&quant, ctx);
let fits_gpu =
specs.unified_memory || specs.gpu_vram_gb.map(|v| mem <= v).unwrap_or(false);
if !fits_gpu {
continue;
}
let est = estimate_tps(model, &quant, &specs, RunMode::Gpu, runtime, &config);
if est <= 0.0 {
continue;
}
ratios.push((label.to_string(), est / measured));
}
}
assert!(
ratios.len() >= 15,
"calibration needs a workable sample; got {} rows \
({skipped_unknown_model} skipped as not in catalog) — did the \
cache or catalog shrink drastically?",
ratios.len()
);
let mut sorted: Vec<f64> = ratios.iter().map(|(_, r)| *r).collect();
sorted.sort_by(|a, b| a.partial_cmp(b).unwrap());
let pct = |p: f64| sorted[((sorted.len() - 1) as f64 * p) as usize];
let (p10, median, p90) = (pct(0.10), pct(0.50), pct(0.90));
// Per-preset medians for the report.
let mut by_preset: std::collections::BTreeMap<String, Vec<f64>> = Default::default();
for (label, r) in &ratios {
by_preset.entry(label.clone()).or_default().push(*r);
}
println!(
"calibration: {} rows across {} presets ({} rows skipped: model not in catalog)",
ratios.len(),
by_preset.len(),
skipped_unknown_model
);
println!(" est/measured overall: p10={p10:.2} median={median:.2} p90={p90:.2}");
for (label, mut rs) in by_preset {
rs.sort_by(|a, b| a.partial_cmp(b).unwrap());
println!(
" {label}: n={} median={:.2}",
rs.len(),
rs[(rs.len() - 1) / 2]
);
}
// Guardrails: a median outside this band means a systematic bias
// approaching the #449 bug — investigate before loosening. Baseline
// when set (2026-07, 152 rows): median 0.87, per-preset 0.67–1.19.
assert!(
(0.5..=2.0).contains(&median),
"estimate_tps median est/measured ratio {median:.2} is outside \
[0.5, 2.0] — systematic estimator bias against {} measured runs",
ratios.len()
);
}
// ── Usable context (issue #621) ─────────────────────────────────────
#[test]
fn test_usable_context_constrained_by_tight_pool() {
// 7B model on a 10 GB card: weights leave a few GB for KV cache, so
// the usable context must land strictly below a 200k native window.
let mut model = test_model("7B", 4.0, Some(4.0));
model.context_length = 200_000;
let system = test_system(32.0, true, Some(10.0));
let fit = ModelFit::analyze(&model, &system);
assert!(
fit.usable_context < model.context_length,
"usable {} should be below native {}",
fit.usable_context,
model.context_length
);
assert!(fit.usable_context > 0);
assert!(
fit.context_display().contains('\u{2192}'),
"{}",
fit.context_display()
);
}
#[test]
fn test_usable_context_uncapped_when_pool_is_ample() {
// Small window + huge pool: the full native window fits.
let mut model = test_model("7B", 4.0, Some(4.0));
model.context_length = 8192;
let system = test_system(128.0, true, Some(80.0));
let fit = ModelFit::analyze(&model, &system);
assert_eq!(fit.usable_context, 8192);
assert_eq!(fit.context_display(), "8k");
assert!(!fit.context_severely_limited());
}
#[test]
fn test_ctx_sort_uses_usable_context() {
// Big-window model that can't use it vs small-window model that can:
// on a tight system the honest ranking puts the achievable context
// first when sorting by Ctx.
let mut big_window = test_model("13B", 8.0, Some(8.0));
big_window.context_length = 262_144;
big_window.name = "big-window".into();
let mut small_window = test_model("1B", 1.0, Some(1.0));
small_window.context_length = 32_768;
small_window.name = "small-window".into();
let system = test_system(16.0, true, Some(10.0));
let fits = rank_models_by_fit_opts_col(
vec![
ModelFit::analyze(&big_window, &system),
ModelFit::analyze(&small_window, &system),
],
false,
SortColumn::Ctx,
);
assert!(
fits[0].usable_context >= fits[1].usable_context,
"sorted by usable: {} then {}",
fits[0].usable_context,
fits[1].usable_context
);
}
#[test]
fn test_estimate_tps_run_mode_penalties() {
let model = test_model("7B", 4.0, Some(4.0));
@@ -2394,6 +2958,16 @@ mod tests {
assert!(backend_compatible(&gguf_model, &metal_sys));
}
#[test]
fn test_tts_backend_incompatible_until_runtime_supported() {
let mut model = test_model("82M", 1.0, Some(0.5));
model.format = models::ModelFormat::Safetensors;
model.capabilities = vec![models::Capability::Audio, models::Capability::Tts];
let cuda_sys = test_system(64.0, true, Some(24.0));
assert!(!backend_compatible(&model, &cuda_sys));
}
#[test]
fn test_awq_incompatible_on_volta_v100() {
// V100 is Volta (cc 7.0) — AWQ requires cc >= 7.5
@@ -2493,6 +3067,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: models::ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2778,6 +3353,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: models::ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -3061,6 +3637,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: models::ModelFormat::default(),
num_attention_heads: Some(num_attention_heads),
num_key_value_heads: Some(num_key_value_heads),
llmfit-core/src/hardware.rs
+624 -125
View File
@@ -85,15 +85,26 @@ impl SystemSpecs {
let gpus = Self::detect_all_gpus(total_ram_gb, &cpu_name);
// Primary GPU = the one with the most VRAM (best for inference).
// For fit scoring, we use the primary GPU's VRAM pool.
// Per-card display values come from the primary; the fit-scoring pool
// and GPU count are aggregated across every detected GPU so that
// multi-GPU systems (including mixed models, e.g. RX 7600 + R9700)
// contribute their full combined VRAM, not just the primary's.
let primary = gpus.first();
let has_gpu = !gpus.is_empty();
let gpu_vram_gb = primary.and_then(|g| g.vram_gb);
// Total VRAM = per-card VRAM * count (for multi-GPU tensor splitting)
let total_gpu_vram_gb = primary.and_then(|g| g.vram_gb.map(|vram| vram * g.count as f64));
let gpu_name = primary.map(|g| g.name.clone());
let gpu_count = primary.map(|g| g.count).unwrap_or(0);
let unified_memory = primary.map(|g| g.unified_memory).unwrap_or(false);
// Total VRAM = sum of per-card VRAM * count across all GPUs (for
// multi-GPU tensor splitting). Unified-memory GPUs report the shared
// system pool as their VRAM; with a single such GPU this is correct.
let total_gpu_vram_gb = {
let sum: f64 = gpus
.iter()
.filter_map(|g| g.vram_gb.map(|vram| vram * g.count as f64))
.sum();
if sum > 0.0 { Some(sum) } else { None }
};
let gpu_count: u32 = gpus.iter().map(|g| g.count).sum();
let cpu_backend =
if cfg!(target_arch = "aarch64") || cpu_name.to_lowercase().contains("apple") {
@@ -140,9 +151,7 @@ impl SystemSpecs {
// AMD GPUs via rocm-smi or sysfs
let amd_rocm = Self::detect_amd_gpu_rocm_info();
if amd_rocm.is_empty() {
if let Some(amd) = Self::detect_amd_gpu_sysfs_info() {
gpus.push(amd);
}
gpus.extend(Self::detect_amd_gpu_sysfs_info());
} else {
gpus.extend(amd_rocm);
}
@@ -207,17 +216,12 @@ impl SystemSpecs {
}
}
// Intel Arc via sysfs
if let Some(vram) = Self::detect_intel_gpu() {
// Intel GPUs (integrated or discrete Arc) via lspci/sysfs
let intel_gpus = Self::detect_intel_gpus(total_ram_gb);
if !intel_gpus.is_empty() {
let already_found = gpus.iter().any(|g| g.name.to_lowercase().contains("intel"));
if !already_found {
gpus.push(GpuInfo {
name: "Intel Arc".to_string(),
vram_gb: Some(vram),
backend: GpuBackend::Sycl,
count: 1,
unified_memory: false,
});
gpus.extend(intel_gpus);
}
}
@@ -259,10 +263,22 @@ impl SystemSpecs {
}
}
let dominated = gpus
.iter()
.any(|existing| Self::is_same_gpu_name(&existing.name, &vulkan_gpu.name));
if !dominated {
gpus.push(vulkan_gpu);
.iter_mut()
.find(|existing| Self::is_same_gpu_name(&existing.name, &vulkan_gpu.name));
match dominated {
Some(existing) => {
// The earlier detection path may know the device but not
// its VRAM (e.g. discrete Intel Arc, where i915/xe expose
// no sysfs VRAM file) — the Vulkan device heap is real
// data, so adopt it rather than dropping it (issue #609).
if !existing.unified_memory
&& existing.vram_gb.unwrap_or(0.0) == 0.0
&& vulkan_gpu.vram_gb.unwrap_or(0.0) > 0.0
{
existing.vram_gb = vulkan_gpu.vram_gb;
}
}
None => gpus.push(vulkan_gpu),
}
}
@@ -570,45 +586,155 @@ impl SystemSpecs {
Self::parse_rocm_smi_output(&vram_text, product_text.as_deref())
}
/// Parse per-GPU VRAM totals (bytes) from `rocm-smi --showmeminfo vram`.
///
/// Handles both output formats:
/// * Block (ROCm 5.x / 6.x default): one line per field, e.g.
/// `GPU[0] : VRAM Total Memory (B): 8589934592`. The total line is
/// matched by containing "total" and not "used".
/// * Tabular (newer rocm-smi): a header row followed by one row per
/// device, e.g.
/// `Device Node VRAM Total Memory (B) VRAM Total Used Memory (B)`
/// `0 2 34342961152 16893`.
/// Device rows begin with the integer device index; the Total column
/// precedes the Used column, so the first VRAM-sized number on the row
/// is the total.
fn parse_rocm_vram_bytes(vram_text: &str) -> Vec<u64> {
let mut out: Vec<u64> = Vec::new();
// Block format.
for line in vram_text.lines() {
let lower = line.to_lowercase();
if lower.contains("total")
&& !lower.contains("used")
&& let Some(val) = line
.split_whitespace()
.filter_map(|w| w.parse::<u64>().ok())
.next_back()
&& val > 0
{
out.push(val);
}
}
if !out.is_empty() {
return out;
}
// Tabular format fallback. A device row starts with the integer device
// index; pick the first number large enough to be a VRAM total (>= 64
// MB), which skips the device/node index columns and lands on the
// Total column before the Used column.
const MIN_VRAM_BYTES: u64 = 64 * 1024 * 1024; // 64 MB
for line in vram_text.lines() {
let mut tokens = line.split_whitespace();
match tokens.next() {
Some(first) if first.parse::<u32>().is_ok() => {}
_ => continue, // not a device data row
}
if let Some(total) = line
.split_whitespace()
.filter_map(|w| w.parse::<u64>().ok())
.find(|&v| v >= MIN_VRAM_BYTES)
{
out.push(total);
}
}
out
}
/// Parse per-GPU product names from `rocm-smi --showproductname`.
///
/// Handles both the block format (`GPU[0] : Card Series: AMD Radeon RX
/// 7600`) and the tabular format where `Card Series` is a column header
/// and each device row carries its model in that column. Returns one
/// name per device, in device order.
fn parse_rocm_product_names(text: &str) -> Vec<String> {
// Block format: name is after the last colon on a "Card Series" line,
// e.g. "GPU[0] : Card Series: AMD Radeon RX 7600". The colon guard
// avoids matching a tabular "Card Series" column header (no colon).
let mut block: Vec<String> = Vec::new();
let mut gfx_versions: Vec<String> = Vec::new();
for line in text.lines() {
let lower = line.to_lowercase();
if lower.contains("card series")
&& line.contains(':')
&& let Some(name) = line.rsplit(':').next().map(|n| n.trim().to_string())
&& !name.is_empty()
{
block.push(name);
} else if lower.contains("gfx version")
&& line.contains(':')
&& let Some(gfx) = line.rsplit(':').next().map(|g| g.trim().to_string())
&& !gfx.is_empty()
{
gfx_versions.push(gfx);
}
}
if !block.is_empty() {
// Disambiguate generic series names with the GFX version when
// available: some accelerators (e.g. Instinct MI50/MI60) report
// `Card Series: AMD Radeon Graphics`, which would otherwise be
// indistinguishable from — and grouped with — an APU iGPU that
// reports the same generic name (issue #638).
if gfx_versions.len() == block.len() {
for (name, gfx) in block.iter_mut().zip(&gfx_versions) {
if Self::is_integrated_gpu_name(name) {
*name = format!("{name} ({gfx})");
}
}
}
return block;
}
// Tabular format: slice the "Card Series" column out of each device
// row using the header column offsets. The column runs from the start
// of "Card Series" to the start of the next known column header.
let mut out: Vec<String> = Vec::new();
let Some(header) = text
.lines()
.find(|l| l.to_lowercase().contains("card series"))
else {
return out;
};
let header_lower = header.to_lowercase();
let Some(start) = header_lower.find("card series") else {
return out;
};
let end = ["card model", "card vendor", "card sku", "card partition"]
.iter()
.filter_map(|h| header_lower.find(h))
.filter(|&i| i > start)
.min()
.unwrap_or(header.len());
for line in text.lines() {
// Device rows start with the integer device index.
match line.split_whitespace().next() {
Some(first) if first.parse::<u32>().is_ok() => {}
_ => continue,
}
if line.len() <= start {
out.push("AMD GPU".to_string());
continue;
}
let slice_end = end.min(line.len());
let name = line[start..slice_end].trim().to_string();
out.push(if name.is_empty() {
"AMD GPU".to_string()
} else {
name
});
}
out
}
/// Parse rocm-smi `--showmeminfo vram` and `--showproductname` output
/// into one `GpuInfo` per distinct GPU model. Identical models are
/// grouped with a `count` field, like `parse_nvidia_smi_list`.
fn parse_rocm_smi_output(vram_text: &str, product_text: Option<&str>) -> Vec<GpuInfo> {
// Parse per-GPU VRAM total.
// Typical format: "GPU[0] : VRAM Total Memory (B): 8589934592"
let mut per_gpu_vram_bytes: Vec<u64> = Vec::new();
for line in vram_text.lines() {
let lower = line.to_lowercase();
if lower.contains("total") && !lower.contains("used") {
if let Some(val) = line
.split_whitespace()
.filter_map(|w| w.parse::<u64>().ok())
.next_back()
&& val > 0
{
per_gpu_vram_bytes.push(val);
}
}
}
// Parse per-GPU names from --showproductname.
// Format: "GPU[0] : Card Series: AMD Radeon RX 7600"
let per_gpu_names: Vec<String> = product_text
.map(|text| {
text.lines()
.filter_map(|line| {
let lower = line.to_lowercase();
if lower.contains("card series") {
line.rsplit(':')
.next()
.map(|n| n.trim().to_string())
.filter(|n| !n.is_empty())
} else {
None
}
})
.collect()
})
let per_gpu_vram_bytes = Self::parse_rocm_vram_bytes(vram_text);
let per_gpu_names = product_text
.map(Self::parse_rocm_product_names)
.unwrap_or_default();
// Filter out integrated GPUs (iGPUs) that have very little VRAM.
@@ -664,19 +790,30 @@ impl SystemSpecs {
.collect()
}
/// Detect AMD GPU via sysfs on Linux (works without ROCm installed).
/// AMD vendor ID is 0x1002.
fn detect_amd_gpu_sysfs_info() -> Option<GpuInfo> {
/// Detect AMD GPUs via sysfs on Linux (works without ROCm installed).
/// AMD vendor ID is 0x1002. Enumerates every `cardN` entry in
/// `/sys/class/drm`, groups identical models with a `count` (like the
/// ROCm and NVIDIA paths), and returns one `GpuInfo` per distinct model
/// so multi-GPU setups are reported in full.
fn detect_amd_gpu_sysfs_info() -> Vec<GpuInfo> {
if !cfg!(target_os = "linux") {
return None;
return Vec::new();
}
let mut slot_hints: Vec<String> = Vec::new();
let entries = std::fs::read_dir("/sys/class/drm").ok()?;
let entries = match std::fs::read_dir("/sys/class/drm") {
Ok(e) => e,
Err(_) => return Vec::new(),
};
// Collect per-card (name, vram) pairs.
let mut cards: Vec<(String, Option<f64>)> = Vec::new();
for entry in entries.flatten() {
let card_path = entry.path();
let fname = card_path.file_name()?.to_str()?.to_string();
let fname = match card_path.file_name().and_then(|f| f.to_str()) {
Some(f) => f.to_string(),
None => continue,
};
// Only look at cardN entries, not cardN-DP-1 etc.
if !fname.starts_with("card") || fname.contains('-') {
continue;
@@ -684,12 +821,9 @@ impl SystemSpecs {
let device_path = card_path.join("device");
let vendor_path = device_path.join("vendor");
if let Ok(vendor) = std::fs::read_to_string(&vendor_path) {
if vendor.trim() != "0x1002" {
continue;
}
} else {
continue;
match std::fs::read_to_string(&vendor_path) {
Ok(vendor) if vendor.trim() == "0x1002" => {}
_ => continue,
}
// Found an AMD GPU. Try to read VRAM.
@@ -702,6 +836,8 @@ impl SystemSpecs {
vram_gb = Some(vram_bytes as f64 / (1024.0 * 1024.0 * 1024.0));
}
// Resolve this card's PCI slot so lspci yields a per-card name.
let mut slot_hints: Vec<String> = Vec::new();
if let Ok(uevent) = std::fs::read_to_string(device_path.join("uevent")) {
for line in uevent.lines() {
if let Some(slot) = line.strip_prefix("PCI_SLOT_NAME=") {
@@ -722,16 +858,42 @@ impl SystemSpecs {
}
}
// AMD GPU without ROCm — Vulkan is the most likely inference backend
return Some(GpuInfo {
name,
vram_gb,
backend: GpuBackend::Vulkan,
count: 1,
unified_memory: false,
cards.push((name, vram_gb));
}
// Group identical models, tracking count and max per-card VRAM.
let mut grouped: BTreeMap<String, (u32, Option<f64>)> = BTreeMap::new();
for (name, vram_gb) in cards {
let entry = grouped.entry(name).or_insert((0, None));
entry.0 += 1;
match (entry.1, vram_gb) {
(Some(existing), Some(new)) if new > existing => entry.1 = Some(new),
(None, Some(_)) => entry.1 = vram_gb,
_ => {}
}
}
// Filter out integrated GPUs when discrete GPUs are present.
let has_discrete = grouped.iter().any(|(name, (_, vram))| {
!Self::is_integrated_gpu_name(name) && vram.unwrap_or(0.0) > 2.0
});
if has_discrete {
grouped.retain(|name, (_, vram)| {
!Self::is_integrated_gpu_name(name) && vram.unwrap_or(0.0) > 2.0
});
}
None
grouped
.into_iter()
.map(|(name, (count, vram_gb))| GpuInfo {
name,
// AMD GPU without ROCm — Vulkan is the most likely backend
vram_gb,
backend: GpuBackend::Vulkan,
count,
unified_memory: false,
})
.collect()
}
/// Extract AMD GPU name from lspci output.
@@ -977,7 +1139,7 @@ impl SystemSpecs {
fn prefer_discrete_gpus(gpus: Vec<GpuInfo>) -> Vec<GpuInfo> {
let discrete: Vec<GpuInfo> = gpus
.iter()
.filter(|g| !Self::is_integrated_gpu_name(&g.name))
.filter(|g| !Self::is_integrated_gpu(&g.name, g.vram_gb))
.cloned()
.collect();
@@ -989,6 +1151,24 @@ impl SystemSpecs {
}
}
/// VRAM-aware integrated-GPU check.
///
/// Intel iGPU product names (UHD/HD/Iris) are conclusive, but the AMD
/// "Radeon Graphics" pattern is ambiguous: datacenter accelerators like
/// the Instinct MI50/MI60 report the generic `Card Series: AMD Radeon
/// Graphics` through rocm-smi on some firmware. No true iGPU has this
/// much *dedicated* VRAM, so a large-VRAM AMD-generic device is treated
/// as discrete rather than dropped (issue #638).
fn is_integrated_gpu(name: &str, vram_gb: Option<f64>) -> bool {
const AMD_GENERIC_DISCRETE_VRAM_GB: f64 = 8.0;
if !Self::is_integrated_gpu_name(name) {
return false;
}
let lower = name.to_lowercase();
let amd_generic = lower.contains("radeon") && !lower.contains("(integrated)");
!(amd_generic && vram_gb.unwrap_or(0.0) >= AMD_GENERIC_DISCRETE_VRAM_GB)
}
/// Heuristic: returns true when the GPU name matches known integrated GPU
/// patterns on Windows (Intel UHD/HD/Iris, AMD Radeon Graphics without a
/// discrete model number like RX).
@@ -1055,65 +1235,113 @@ impl SystemSpecs {
}
}
/// Detect Intel Arc / Intel integrated GPU via sysfs or lspci.
/// Intel Arc GPUs (A370M, A770, etc.) have dedicated VRAM exposed via
/// the DRM subsystem at /sys/class/drm/card*/device/. Even integrated
/// Intel GPUs that share system RAM are useful for inference via SYCL/oneAPI.
fn detect_intel_gpu() -> Option<f64> {
// Try sysfs first: works for Intel discrete (Arc) GPUs on Linux.
// Walk /sys/class/drm/card*/device/ looking for Intel vendor ID (0x8086).
/// Detect Intel GPUs (integrated or discrete Arc) via lspci, with a sysfs
/// vendor-ID fallback when lspci is unavailable.
///
/// Intel's i915/xe drivers do **not** expose `mem_info_vram_total` — that
/// sysfs file is amdgpu-specific — so dedicated VRAM for discrete Arc
/// cards cannot be read here; it is left as `None` for the Vulkan
/// fallback to fill in from the device's memory heap (issue #609).
/// Integrated GPUs (always at PCI address 00:02.0 on Intel platforms)
/// share system RAM and are reported as unified-memory devices with the
/// full RAM pool, matching the AMD APU and Apple Silicon conventions.
fn detect_intel_gpus(total_ram_gb: f64) -> Vec<GpuInfo> {
if let Some(text) = Self::lspci_output() {
let gpus = Self::parse_intel_gpus_from_lspci(&text, total_ram_gb);
if !gpus.is_empty() {
return gpus;
}
}
// Fallback: lspci unavailable — sysfs vendor ID at least tells us an
// Intel GPU exists, but not whether it's integrated or discrete.
if let Ok(entries) = std::fs::read_dir("/sys/class/drm") {
for entry in entries.flatten() {
let card_path = entry.path();
let device_path = card_path.join("device");
// Check vendor ID matches Intel (0x8086)
let vendor_path = device_path.join("vendor");
if let Ok(vendor) = std::fs::read_to_string(&vendor_path)
&& vendor.trim() != "0x8086"
{
let fname = match card_path.file_name().and_then(|f| f.to_str()) {
Some(f) => f,
None => continue,
};
if !fname.starts_with("card") || fname.contains('-') {
continue;
}
// Look for total VRAM via DRM memory info
// Intel discrete GPUs expose this under drm/card*/device/mem_info_vram_total
let vram_path = card_path.join("device/mem_info_vram_total");
if let Ok(vram_str) = std::fs::read_to_string(&vram_path)
&& let Ok(vram_bytes) = vram_str.trim().parse::<u64>()
&& vram_bytes > 0
if let Ok(vendor) = std::fs::read_to_string(card_path.join("device/vendor"))
&& vendor.trim() == "0x8086"
{
let vram_gb = vram_bytes as f64 / (1024.0 * 1024.0 * 1024.0);
return Some(vram_gb);
}
// For integrated Intel GPUs, check if it's an Arc-class device
// by looking for "Arc" in the device name via lspci
if let Some(text) = Self::lspci_output() {
for line in text.lines() {
let lower = line.to_lowercase();
if lower.contains("intel") && lower.contains("arc") {
// Intel Arc integrated (e.g. Arc Graphics in Meteor Lake)
// These share system RAM; report None for VRAM and
// let the caller know a GPU exists.
return Some(0.0);
}
}
return vec![GpuInfo {
name: "Intel Graphics".to_string(),
vram_gb: None,
backend: GpuBackend::Sycl,
count: 1,
unified_memory: false,
}];
}
}
}
// Fallback: check lspci directly for Intel Arc devices
// (covers cases where sysfs isn't available or card dirs don't exist)
if let Some(text) = Self::lspci_output() {
for line in text.lines() {
let lower = line.to_lowercase();
if lower.contains("intel") && lower.contains("arc") {
return Some(0.0);
}
Vec::new()
}
/// Classify Intel display controllers from `lspci -nnD` output.
/// Separated from [`Self::detect_intel_gpus`] so real lspci captures can
/// be used as regression fixtures.
fn parse_intel_gpus_from_lspci(text: &str, total_ram_gb: f64) -> Vec<GpuInfo> {
let mut gpus = Vec::new();
for line in text.lines() {
let lower = line.to_lowercase();
let is_display = lower.contains("vga compatible")
|| lower.contains("3d controller")
|| lower.contains("display controller");
if !is_display || !line.contains("[8086:") {
continue;
}
let name = Self::intel_name_from_lspci_line(line);
// Intel iGPUs live at PCI 00:02.0 on the root complex; discrete
// cards enumerate behind a bridge on a nonzero bus.
let addr = line.split_whitespace().next().unwrap_or("");
let integrated = addr.ends_with(":00:02.0") || addr == "00:02.0";
if integrated {
gpus.push(GpuInfo {
name: format!("{name} (integrated)"),
vram_gb: Some(total_ram_gb),
backend: GpuBackend::Sycl,
count: 1,
unified_memory: true,
});
} else {
gpus.push(GpuInfo {
name,
vram_gb: None, // filled by the Vulkan fallback when available
backend: GpuBackend::Sycl,
count: 1,
unified_memory: false,
});
}
}
gpus
}
None
/// Extract a readable GPU name from an Intel lspci line, e.g.
/// `"... Intel Corporation Core Ultra 200V Series Processors Arc Graphics
/// 130V/140V GPU [8086:64a0] (rev 04)"` → `"Intel Arc Graphics 130V/140V"`.
fn intel_name_from_lspci_line(line: &str) -> String {
let after = line
.split_once("Intel Corporation")
.map(|(_, r)| r)
.unwrap_or(line);
let cleaned = after.split(" [8086:").next().unwrap_or(after).trim();
let mut name = if let Some(idx) = cleaned.find("Arc") {
// Codename lines bracket the marketing name: "DG2 [Arc A770]".
format!("Intel {}", cleaned[idx..].trim_end_matches(']'))
} else if cleaned.is_empty() {
"Intel Graphics".to_string()
} else {
format!("Intel {cleaned}")
};
if let Some(stripped) = name.strip_suffix(" GPU") {
name = stripped.to_string();
}
name
}
/// Detect Apple Silicon GPU via system_profiler.
@@ -1238,6 +1466,26 @@ impl SystemSpecs {
}
}
// Intel: lspci reports platform names ("Intel Arc Graphics 130V/140V
// (integrated)") while Mesa/Vulkan reports codenames ("Intel(R)
// Arc(tm) Graphics (LNL)"). Same-model-number matches (A770 vs
// "Arc A770") are the same device; an integrated entry also matches
// a Vulkan Intel device with no model number of its own, since a
// platform has at most one Intel iGPU.
let is_intel = |s: &str| s.contains("intel");
if is_intel(&e_lower) && is_intel(&c_lower) {
let e_nums = Self::extract_gpu_model_numbers(&e_lower);
let c_nums = Self::extract_gpu_model_numbers(&c_lower);
if !e_nums.is_empty() && e_nums.iter().any(|n| c_nums.contains(n)) {
return true;
}
if (e_lower.contains("(integrated)") && c_nums.is_empty())
|| (c_lower.contains("(integrated)") && e_nums.is_empty())
{
return true;
}
}
false
}
@@ -1496,7 +1744,7 @@ impl SystemSpecs {
#[cfg(target_os = "linux")]
{
let text = std::fs::read_to_string("/proc/cpuinfo").ok()?;
return Self::parse_cpu_name_from_cpuinfo(&text);
Self::parse_cpu_name_from_cpuinfo(&text)
}
#[cfg(not(target_os = "linux"))]
@@ -1540,7 +1788,7 @@ impl SystemSpecs {
return None;
}
return Some(model.to_string());
Some(model.to_string())
}
#[cfg(not(target_os = "linux"))]
@@ -1986,6 +2234,19 @@ pub fn gpu_memory_bandwidth_gbps(name: &str) -> Option<f64> {
return Some(448.0);
}
// ── AMD unified-memory APUs (Strix Halo) ───────────────────────
// Ryzen AI MAX / MAX+ (Radeon 8050S/8060S): 256-bit LPDDR5X-8000.
// Names vary by detection path: lspci ("Strix Halo [Radeon ...]"),
// marketing ("Radeon 8060S"), or the cpu-derived fallback
// ("AMD Ryzen AI MAX+ 395 w/ Radeon 8060S (integrated)").
if lower.contains("8060s")
|| lower.contains("8050s")
|| lower.contains("strix halo")
|| lower.contains("ryzen ai max")
{
return Some(256.0);
}
// ── AMD Discrete (RDNA) ────────────────────────────────────────
// RX 9000 series (RDNA 4)
if lower.contains("9070 xt") {
@@ -3445,4 +3706,242 @@ GPU[0] : VRAM Total Used Memory (B): 200000";
assert_eq!(gpus[0].name, "AMD GPU");
assert!(gpus[0].vram_gb.unwrap() > 31.0);
}
// Newer rocm-smi emits a tabular layout instead of one line per field.
// Models the dual Instinct MI50 setup from issue #638 (both cards share
// the same product name and 32 GB VRAM), which the block-only parser
// collapsed to a single card.
#[test]
fn test_parse_rocm_smi_tabular_identical_gpus() {
let vram_text = "\
====================== ROCm System Management Interface ======================
================================ Memory Usage ================================
Device Node VRAM Total Memory (B) VRAM Total Used Memory (B)
0 2 34342961152 16893952
1 1 34342961152 33678336
=============================================================================";
let product_text = "\
====================== ROCm System Management Interface ======================
================================ Product Info ================================
Device Card Series Card Model Card Vendor
0 Instinct MI60 / MI50 0x66a1 Advanced Micro Devices, Inc. [AMD/ATI]
1 Instinct MI60 / MI50 0x66a1 Advanced Micro Devices, Inc. [AMD/ATI]
=============================================================================";
let gpus = SystemSpecs::parse_rocm_smi_output(vram_text, Some(product_text));
assert_eq!(gpus.len(), 1, "identical tabular GPUs should be grouped");
assert_eq!(gpus[0].count, 2, "both MI50s should be detected");
assert!(gpus[0].name.contains("MI50"), "name: {}", gpus[0].name);
assert!(gpus[0].vram_gb.unwrap() > 31.0 && gpus[0].vram_gb.unwrap() < 33.0);
}
#[test]
fn test_parse_rocm_smi_tabular_two_different_gpus() {
let vram_text = "\
Device Node VRAM Total Memory (B) VRAM Total Used Memory (B)
0 2 8573157376 60448768
1 1 34208743424 33732509696";
let product_text = "\
Device Card Series Card Model
0 AMD Radeon RX 7600 0x7480
1 AMD Radeon AI PRO R9700 0x7551";
let gpus = SystemSpecs::parse_rocm_smi_output(vram_text, Some(product_text));
assert_eq!(gpus.len(), 2, "should detect two distinct tabular GPUs");
let rx7600 = gpus.iter().find(|g| g.name.contains("RX 7600")).unwrap();
let r9700 = gpus.iter().find(|g| g.name.contains("R9700")).unwrap();
assert_eq!(rx7600.count, 1);
assert_eq!(r9700.count, 1);
assert!(rx7600.vram_gb.unwrap() > 7.0 && rx7600.vram_gb.unwrap() < 9.0);
assert!(r9700.vram_gb.unwrap() > 31.0 && r9700.vram_gb.unwrap() < 33.0);
}
// Tabular VRAM without parseable product names still yields the right
// count (names fall back to "AMD GPU" but cards are not lost).
#[test]
fn test_parse_rocm_smi_tabular_vram_no_names() {
let vram_text = "\
Device Node VRAM Total Memory (B) VRAM Total Used Memory (B)
0 2 34342961152 16893952
1 1 34342961152 33678336";
let gpus = SystemSpecs::parse_rocm_smi_output(vram_text, None);
assert_eq!(gpus.len(), 1);
assert_eq!(gpus[0].count, 2);
assert_eq!(gpus[0].name, "AMD GPU");
}
// Regression for issue #638 (keyz182): verbatim rocm-smi block output
// from a mixed system — a 32 GB MI50 that reports the generic
// `Card Series: AMD Radeon Graphics`, a 16 GB MI50 with the proper
// Instinct name, and a 512 MB Cezanne iGPU. The generic-named 32 GB
// card must survive both the iGPU VRAM filter and prefer_discrete_gpus,
// and must not be grouped with the iGPU that shares its generic name.
#[test]
fn test_parse_rocm_smi_mixed_mi50s_generic_name_and_igpu() {
let vram_text = "\
============================ ROCm System Management Interface ============================
================================== Memory Usage (Bytes) ==================================
GPU[0]\t\t: VRAM Total Memory (B): 34342961152
GPU[0]\t\t: VRAM Total Used Memory (B): 25227759616
GPU[1]\t\t: VRAM Total Memory (B): 17163091968
GPU[1]\t\t: VRAM Total Used Memory (B): 7695077376
GPU[2]\t\t: VRAM Total Memory (B): 536870912
GPU[2]\t\t: VRAM Total Used Memory (B): 18165760
==========================================================================================
================================== End of ROCm SMI Log ===================================";
let product_text = "\
============================ ROCm System Management Interface ============================
====================================== Product Info ======================================
GPU[0]\t\t: Card Series: \t\tAMD Radeon Graphics
GPU[0]\t\t: Card Model: \t\t0x66a0
GPU[0]\t\t: Card Vendor: \t\tAdvanced Micro Devices, Inc. [AMD/ATI]
GPU[0]\t\t: Card SKU: \t\tD1640200
GPU[0]\t\t: Subsystem ID: \t0x081e
GPU[0]\t\t: Device Rev: \t\t0x00
GPU[0]\t\t: Node ID: \t\t1
GPU[0]\t\t: GUID: \t\t45854
GPU[0]\t\t: GFX Version: \t\tgfx906
GPU[1]\t\t: Card Series: \t\tAMD Instinct MI60 / MI50
GPU[1]\t\t: Card Model: \t\t0x66a1
GPU[1]\t\t: Card Vendor: \t\tAdvanced Micro Devices, Inc. [AMD/ATI]
GPU[1]\t\t: Card SKU: \t\tD1631400
GPU[1]\t\t: Subsystem ID: \t0x0834
GPU[1]\t\t: Device Rev: \t\t0x02
GPU[1]\t\t: Node ID: \t\t2
GPU[1]\t\t: GUID: \t\t28640
GPU[1]\t\t: GFX Version: \t\tgfx906
GPU[2]\t\t: Card Series: \t\tAMD Radeon Graphics
GPU[2]\t\t: Card Model: \t\t0x1638
GPU[2]\t\t: Card Vendor: \t\tAdvanced Micro Devices, Inc. [AMD/ATI]
GPU[2]\t\t: Card SKU: \t\tCEZANNE
GPU[2]\t\t: Subsystem ID: \t0x1636
GPU[2]\t\t: Device Rev: \t\t0xc8
GPU[2]\t\t: Node ID: \t\t3
GPU[2]\t\t: GUID: \t\t48746
GPU[2]\t\t: GFX Version: \t\tgfx90c
==========================================================================================
================================== End of ROCm SMI Log ===================================";
let gpus = SystemSpecs::parse_rocm_smi_output(vram_text, Some(product_text));
assert_eq!(
gpus.len(),
2,
"both MI50s must be detected, iGPU excluded: {gpus:?}"
);
let big = gpus
.iter()
.find(|g| g.vram_gb.unwrap_or(0.0) > 30.0)
.expect("32 GB MI50 missing");
let small = gpus
.iter()
.find(|g| {
let v = g.vram_gb.unwrap_or(0.0);
v > 15.0 && v < 17.0
})
.expect("16 GB MI50 missing");
// Generic name disambiguated with the GFX version.
assert_eq!(big.name, "AMD Radeon Graphics (gfx906)");
assert!(small.name.contains("MI60 / MI50"));
// The generic-named 32 GB accelerator must survive the global
// discrete-preference filter alongside the properly named card.
let filtered = SystemSpecs::prefer_discrete_gpus(gpus);
assert_eq!(
filtered.len(),
2,
"prefer_discrete_gpus must not drop a 32 GB accelerator: {filtered:?}"
);
}
// Real `lspci -nnD` line from a Lunar Lake laptop (Core Ultra 7 258V,
// Arc 140V iGPU): must classify as integrated/unified with the RAM pool,
// not a 0-VRAM discrete device (issue #609 family).
#[test]
fn test_parse_intel_igpu_from_lspci_lunar_lake() {
let text = "0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Core Ultra 200V Series Processors Arc Graphics 130V/140V GPU [8086:64a0] (rev 04)";
let gpus = SystemSpecs::parse_intel_gpus_from_lspci(text, 32.0);
assert_eq!(gpus.len(), 1, "{gpus:?}");
assert_eq!(gpus[0].name, "Intel Arc Graphics 130V/140V (integrated)");
assert!(gpus[0].unified_memory);
assert_eq!(gpus[0].vram_gb, Some(32.0));
}
// Discrete Arc cards enumerate behind a bridge (nonzero bus). i915/xe
// expose no sysfs VRAM file, so VRAM stays None for the Vulkan fallback
// to fill in — but the card must be detected and named (issue #609).
#[test]
fn test_parse_intel_dgpu_from_lspci() {
let a770 = "0000:03:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A770] [8086:56a0] (rev 08)";
let gpus = SystemSpecs::parse_intel_gpus_from_lspci(a770, 32.0);
assert_eq!(gpus.len(), 1, "{gpus:?}");
assert_eq!(gpus[0].name, "Intel Arc A770");
assert!(!gpus[0].unified_memory);
assert_eq!(gpus[0].vram_gb, None);
let b70 = "0000:03:00.0 VGA compatible controller [0300]: Intel Corporation Battlemage G21 [Arc Pro B70] [8086:e211]";
let gpus = SystemSpecs::parse_intel_gpus_from_lspci(b70, 32.0);
assert_eq!(gpus.len(), 1, "{gpus:?}");
assert_eq!(gpus[0].name, "Intel Arc Pro B70");
assert!(!gpus[0].unified_memory);
}
#[test]
fn test_parse_intel_igpu_and_dgpu_together() {
let text = "\
0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Raptor Lake-S UHD Graphics [8086:a780] (rev 04)
0000:03:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A770] [8086:56a0] (rev 08)";
let gpus = SystemSpecs::parse_intel_gpus_from_lspci(text, 64.0);
assert_eq!(gpus.len(), 2, "{gpus:?}");
assert!(gpus[0].unified_memory && gpus[0].name.contains("(integrated)"));
assert_eq!(gpus[1].name, "Intel Arc A770");
assert!(!gpus[1].unified_memory);
}
// Mesa/Vulkan reports Intel devices by codename ("(LNL)") — must dedupe
// against the lspci-derived integrated entry, but a discrete Arc with a
// model number must NOT be swallowed by the iGPU entry.
#[test]
fn test_is_same_gpu_name_intel_igpu_vs_vulkan_codename() {
assert!(SystemSpecs::is_same_gpu_name(
"Intel Arc Graphics 130V/140V (integrated)",
"Intel(R) Arc(tm) Graphics (LNL)"
));
assert!(!SystemSpecs::is_same_gpu_name(
"Intel Arc Graphics 130V/140V (integrated)",
"Intel(R) Arc(tm) A770 Graphics"
));
assert!(SystemSpecs::is_same_gpu_name(
"Intel Arc A770",
"Intel(R) Arc(tm) A770 Graphics"
));
}
#[test]
fn test_prefer_discrete_gpus_drops_small_generic_radeon_keeps_large() {
use super::GpuBackend;
let mk = |name: &str, vram: f64| super::GpuInfo {
name: name.to_string(),
vram_gb: Some(vram),
backend: GpuBackend::Rocm,
count: 1,
unified_memory: false,
};
let gpus = vec![
mk("AMD Radeon Graphics", 32.0), // mislabeled MI50-class accelerator
mk("AMD Radeon(TM) Graphics", 0.5), // true APU iGPU
mk("AMD Instinct MI60 / MI50", 16.0),
];
let result = SystemSpecs::prefer_discrete_gpus(gpus);
assert_eq!(result.len(), 2, "{result:?}");
assert!(result.iter().any(|g| g.vram_gb == Some(32.0)));
assert!(result.iter().any(|g| g.name.contains("Instinct")));
}
}
llmfit-core/src/lib.rs
+5
View File
@@ -1,13 +1,18 @@
pub mod analysis;
pub mod bench;
pub mod benchmarks;
pub mod claim;
pub mod doctor;
pub mod fit;
pub mod hardware;
pub mod models;
pub mod plan;
pub mod providers;
pub mod quality;
pub mod task_bench;
pub mod update;
pub use analysis::{InstalledIndex, build_model_fits};
pub use fit::{FitLevel, InferenceRuntime, ModelFit, RunMode, ScoreComponents, SortColumn};
pub use hardware::{GpuBackend, SystemSpecs};
pub use models::{Capability, LlmModel, ModelDatabase, ModelFormat, UseCase};
llmfit-core/src/models.rs
+369 -65
View File
@@ -134,8 +134,7 @@ pub fn parse_generation(architecture: Option<&str>, name: &str) -> Option<f64> {
return Some(1.0);
}
// Qwen: qwen2, qwen3, qwen3_moe, qwen3_5, qwen3_5_moe, qwen3_next
if arch_lower.starts_with("qwen") {
let suffix = &arch_lower["qwen".len()..];
if let Some(suffix) = arch_lower.strip_prefix("qwen") {
if suffix.starts_with("3_5") || suffix.starts_with("3.5") {
return Some(3.5);
}
@@ -154,16 +153,14 @@ pub fn parse_generation(architecture: Option<&str>, name: &str) -> Option<f64> {
return Some(1.0);
}
// Llama: llama, llama4
if arch_lower.starts_with("llama") {
let suffix = &arch_lower["llama".len()..];
if let Some(suffix) = arch_lower.strip_prefix("llama") {
if suffix.starts_with('4') {
return Some(4.0);
}
// Architecture is just "llama" — fall through to name-based parsing
}
// Gemma: gemma, gemma2, gemma3, gemma4
if arch_lower.starts_with("gemma") {
let suffix = &arch_lower["gemma".len()..];
if let Some(suffix) = arch_lower.strip_prefix("gemma") {
if suffix.starts_with('4') {
return Some(4.0);
}
@@ -176,8 +173,7 @@ pub fn parse_generation(architecture: Option<&str>, name: &str) -> Option<f64> {
return Some(1.0);
}
// Phi: phi, phi3, phimoe
if arch_lower.starts_with("phi") {
let suffix = &arch_lower["phi".len()..];
if let Some(suffix) = arch_lower.strip_prefix("phi") {
if suffix.starts_with('4') {
return Some(4.0);
}
@@ -194,24 +190,21 @@ pub fn parse_generation(architecture: Option<&str>, name: &str) -> Option<f64> {
return Some(1.0);
}
// Cohere: cohere, cohere2
if arch_lower.starts_with("cohere") {
let suffix = &arch_lower["cohere".len()..];
if let Some(suffix) = arch_lower.strip_prefix("cohere") {
if suffix.starts_with('2') {
return Some(2.0);
}
return Some(1.0);
}
// Falcon: falcon, falcon3
if arch_lower.starts_with("falcon") {
let suffix = &arch_lower["falcon".len()..];
if let Some(suffix) = arch_lower.strip_prefix("falcon") {
if suffix.starts_with('3') {
return Some(3.0);
}
return Some(1.0);
}
// Granite: granite, granite4
if arch_lower.starts_with("granite") {
let suffix = &arch_lower["granite".len()..];
if let Some(suffix) = arch_lower.strip_prefix("granite") {
if suffix.starts_with('4') {
return Some(4.0);
}
@@ -319,6 +312,8 @@ pub fn generation_quality_bonus(architecture: Option<&str>, name: &str) -> f64 {
pub enum Capability {
Vision,
ToolUse,
Audio,
Tts,
}
impl Capability {
@@ -326,11 +321,18 @@ impl Capability {
match self {
Capability::Vision => "Vision",
Capability::ToolUse => "Tool Use",
Capability::Audio => "Audio",
Capability::Tts => "Text-to-Speech",
}
}
pub fn all() -> &'static [Capability] {
&[Capability::Vision, Capability::ToolUse]
&[
Capability::Vision,
Capability::ToolUse,
Capability::Audio,
Capability::Tts,
]
}
/// Infer capabilities from model metadata when not explicitly set in JSON.
@@ -369,6 +371,26 @@ impl Capability {
caps.push(Capability::ToolUse);
}
// Audio (speech-to-text) detection — Whisper / distil-whisper family.
// The scraper does not set capabilities=["audio"] for new ASR models,
// so infer it from the architecture / name / use_case the way Vision and
// ToolUse are inferred above.
let architecture = model.architecture.as_deref().unwrap_or("").to_lowercase();
if !caps.contains(&Capability::Audio)
&& (architecture.contains("whisper")
|| name.contains("whisper")
|| use_case.contains("text-to-speech")
|| use_case.contains("transcription")
|| use_case.contains("speech")
|| use_case.contains("audio"))
{
caps.push(Capability::Audio);
}
if !caps.contains(&Capability::Tts) && use_case.contains("text-to-speech") {
caps.push(Capability::Tts);
}
caps
}
}
@@ -475,6 +497,9 @@ pub struct LlmModel {
/// Model capabilities (vision, tool use, etc.)
#[serde(default)]
pub capabilities: Vec<Capability>,
/// Explicitly declared supported languages from HuggingFace metadata.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub languages: Vec<String>,
/// Model weight format (gguf, awq, gptq, autoround, mlx, safetensors, onnx)
#[serde(default)]
pub format: ModelFormat,
@@ -635,10 +660,20 @@ impl std::fmt::Display for KvQuant {
/// Returns true if a model's license matches any in the comma-separated filter string.
/// Models without a license never match.
pub fn matches_license_filter(license: &Option<String>, filter: &str) -> bool {
let allowed: Vec<String> = filter.split(',').map(|s| s.trim().to_lowercase()).collect();
let allowed: Vec<String> = filter
.split(',')
.map(|s| s.trim().to_lowercase())
.filter(|s| !s.is_empty())
.collect();
license
.as_ref()
.map(|l| allowed.contains(&l.to_lowercase()))
.map(|licenses| {
licenses
.split(',')
.map(|s| s.trim().to_lowercase())
.any(|license| allowed.contains(&license))
})
.unwrap_or(false)
}
@@ -666,6 +701,12 @@ impl LlmModel {
self.format.is_prequantized()
}
/// Returns true for catalog entries that need a task-specific runtime not
/// yet modeled by llmfit's llama.cpp/MLX/vLLM fit paths.
pub fn requires_specialized_runtime(&self) -> bool {
self.capabilities.contains(&Capability::Tts)
}
/// Returns true if the model's attention/KV heads are evenly divisible
/// by `tp_size`, meaning it can be split across that many devices.
/// TP=1 always returns true.
@@ -969,6 +1010,8 @@ struct HfModelEntry {
#[serde(default)]
capabilities: Vec<Capability>,
#[serde(default)]
languages: Vec<String>,
#[serde(default)]
format: ModelFormat,
#[serde(default)]
hf_downloads: u64,
@@ -1065,7 +1108,7 @@ pub(crate) fn canonical_slug(name: &str) -> String {
/// - Numeric fields (params, RAM, context): higher wins.
/// - MoE info: if either entry is MoE the result is MoE.
/// - `release_date`: later wins.
/// - `capabilities`, `gguf_sources`: union (no duplicates).
/// - `capabilities`, `languages`, `gguf_sources`: union (no duplicates).
/// - `hf_downloads`, `hf_likes`: maximum.
/// - Architecture fields (`num_attention_heads`, etc.): first non-`None` wins.
fn dedupe_hf_entries(entries: Vec<HfModelEntry>) -> Vec<HfModelEntry> {
@@ -1111,6 +1154,12 @@ fn dedupe_hf_entries(entries: Vec<HfModelEntry>) -> Vec<HfModelEntry> {
existing.capabilities.push(*cap);
}
}
// Merge languages (union, no duplicates).
for lang in &entry.languages {
if !existing.languages.contains(lang) {
existing.languages.push(lang.clone());
}
}
// Merge gguf_sources (union by repo).
for src in &entry.gguf_sources {
if !existing.gguf_sources.iter().any(|s| s.repo == src.repo) {
@@ -1149,6 +1198,51 @@ fn dedupe_hf_entries(entries: Vec<HfModelEntry>) -> Vec<HfModelEntry> {
map.into_values().collect()
}
/// Map a JSON catalog entry to an [`LlmModel`], inferring capabilities and
/// attention layout where the entry doesn't carry them explicitly.
fn entry_to_model(e: HfModelEntry) -> LlmModel {
let mut model = LlmModel {
name: e.name,
provider: e.provider,
parameter_count: e.parameter_count,
parameters_raw: e.parameters_raw,
min_ram_gb: e.min_ram_gb,
recommended_ram_gb: e.recommended_ram_gb,
min_vram_gb: e.min_vram_gb,
quantization: e.quantization,
context_length: e.context_length,
use_case: e.use_case,
is_moe: e.is_moe,
num_experts: e.num_experts,
active_experts: e.active_experts,
active_parameters: e.active_parameters,
release_date: e.release_date,
gguf_sources: e.gguf_sources,
capabilities: e.capabilities,
languages: e.languages,
format: e.format,
num_attention_heads: e.num_attention_heads,
num_key_value_heads: e.num_key_value_heads,
num_hidden_layers: e.num_hidden_layers,
head_dim: e.head_dim,
attention_layout: None,
hidden_size: e.hidden_size,
moe_intermediate_size: e.moe_intermediate_size,
vocab_size: e.vocab_size,
shared_expert_intermediate_size: e.shared_expert_intermediate_size,
license: e.license,
architecture: e.architecture,
};
model.capabilities = Capability::infer(&model);
// Auto-populate attention_layout from name heuristic for known
// hybrid families. Explicit metadata still wins (model.attention_layout
// stays None until the scraper is taught to read it from config.json).
if model.attention_layout.is_none() {
model.attention_layout = infer_attention_layout_from_name(&model.name);
}
model
}
fn parse_parameter_count_raw(parameter_count: &str) -> Option<u64> {
let trimmed = parameter_count.trim().to_uppercase();
let (number, multiplier) = if let Some(number) = trimmed.strip_suffix('B') {
@@ -1272,6 +1366,7 @@ impl OnnxModelEntry {
release_date: None,
gguf_sources: vec![],
capabilities: self.capabilities,
languages: vec![],
format: ModelFormat::Onnx,
num_attention_heads: None,
num_key_value_heads: None,
@@ -1298,47 +1393,7 @@ fn load_embedded() -> Vec<LlmModel> {
// for the same model slug with conflicting metadata.
let mut models: Vec<LlmModel> = dedupe_hf_entries(entries)
.into_iter()
.map(|e| {
let mut model = LlmModel {
name: e.name,
provider: e.provider,
parameter_count: e.parameter_count,
parameters_raw: e.parameters_raw,
min_ram_gb: e.min_ram_gb,
recommended_ram_gb: e.recommended_ram_gb,
min_vram_gb: e.min_vram_gb,
quantization: e.quantization,
context_length: e.context_length,
use_case: e.use_case,
is_moe: e.is_moe,
num_experts: e.num_experts,
active_experts: e.active_experts,
active_parameters: e.active_parameters,
release_date: e.release_date,
gguf_sources: e.gguf_sources,
capabilities: e.capabilities,
format: e.format,
num_attention_heads: e.num_attention_heads,
num_key_value_heads: e.num_key_value_heads,
num_hidden_layers: e.num_hidden_layers,
head_dim: e.head_dim,
attention_layout: None,
hidden_size: e.hidden_size,
moe_intermediate_size: e.moe_intermediate_size,
vocab_size: e.vocab_size,
shared_expert_intermediate_size: e.shared_expert_intermediate_size,
license: e.license,
architecture: e.architecture,
};
model.capabilities = Capability::infer(&model);
// Auto-populate attention_layout from name heuristic for known
// hybrid families. Explicit metadata still wins (model.attention_layout
// stays None until the scraper is taught to read it from config.json).
if model.attention_layout.is_none() {
model.attention_layout = infer_attention_layout_from_name(&model.name);
}
model
})
.map(entry_to_model)
.collect();
let onnx_entries: Vec<OnnxModelEntry> =
@@ -1356,6 +1411,34 @@ fn load_embedded() -> Vec<LlmModel> {
models
}
/// Full path to the user's custom model overlay file, alongside the update
/// cache (e.g. `~/.local/share/llmfit/custom_models.json` on Linux).
/// The `LLMFIT_CUSTOM_MODELS` env var overrides the location.
pub fn custom_models_file() -> Option<std::path::PathBuf> {
if let Ok(path) = std::env::var("LLMFIT_CUSTOM_MODELS") {
return Some(std::path::PathBuf::from(path));
}
Some(crate::update::cache_dir()?.join("custom_models.json"))
}
/// Load user-defined models from a JSON file using the same entry schema as
/// the embedded catalog (`hf_models.json`). Returns an error string for a
/// present-but-invalid file so callers can warn instead of silently dropping
/// hand-written entries; a missing file is `Ok(vec![])`.
fn load_custom_models_from(path: &std::path::Path) -> Result<Vec<LlmModel>, String> {
if !path.exists() {
return Ok(vec![]);
}
let content = std::fs::read_to_string(path)
.map_err(|e| format!("cannot read {}: {e}", path.display()))?;
let entries: Vec<HfModelEntry> = serde_json::from_str(&content)
.map_err(|e| format!("invalid JSON in {}: {e}", path.display()))?;
Ok(dedupe_hf_entries(entries)
.into_iter()
.map(entry_to_model)
.collect())
}
impl ModelDatabase {
/// Load only the compile-time embedded model list (no cache).
/// Used internally by the updater to determine which models are already known.
@@ -1365,23 +1448,41 @@ impl ModelDatabase {
}
}
/// Load the embedded model list **and** merge any locally cached models.
/// Load the embedded model list **and** merge user custom models and any
/// locally cached models.
///
/// Cached models are appended after the embedded ones; if an ID already
/// exists in the embedded list it is skipped to avoid duplication.
/// Silently ignores a missing or corrupt cache file.
/// Precedence: custom models (see [`custom_models_file`]) replace embedded
/// entries with the same canonical slug; cached models (from
/// `llmfit update`) are appended only for slugs not already present.
/// A missing cache/custom file is ignored; a *corrupt* custom file prints
/// a warning to stderr so hand-written entries don't vanish silently.
pub fn new() -> Self {
let mut models = load_embedded();
// Overlay user-defined models: same slug replaces the embedded entry,
// new slugs are appended.
if let Some(path) = custom_models_file() {
match load_custom_models_from(&path) {
Ok(custom) if !custom.is_empty() => {
let custom_keys: std::collections::HashSet<String> =
custom.iter().map(|m| canonical_slug(&m.name)).collect();
models.retain(|m| !custom_keys.contains(&canonical_slug(&m.name)));
models.extend(custom);
}
Ok(_) => {}
Err(e) => eprintln!("Warning: skipping custom models: {e}"),
}
}
// Merge cached models (from `llmfit update`) without duplicating.
// canonical_slug normalizes org/ prefix, case, and separators so that
// e.g. `meta-llama/Llama-3.1-8B` and `meta-llama/llama-3.1-8b` are
// treated as the same model.
let embedded_keys: std::collections::HashSet<String> =
let existing_keys: std::collections::HashSet<String> =
models.iter().map(|m| canonical_slug(&m.name)).collect();
for cached in crate::update::load_cache() {
if !embedded_keys.contains(&canonical_slug(&cached.name)) {
if !existing_keys.contains(&canonical_slug(&cached.name)) {
models.push(cached);
}
}
@@ -1610,6 +1711,92 @@ fn infer_heads_from_name(name: &str, params_b: f64) -> (u32, u32) {
mod tests {
use super::*;
// ────────────────────────────────────────────────────────────────────
// Custom model overlay tests
// ────────────────────────────────────────────────────────────────────
const CUSTOM_ENTRY_JSON: &str = r#"[{
"name": "acme/CustomNet-7B",
"provider": "acme",
"parameter_count": "7B",
"min_ram_gb": 5.0,
"recommended_ram_gb": 8.0,
"min_vram_gb": 5.0,
"quantization": "Q4_K_M",
"context_length": 32768,
"use_case": "Testing"
}]"#;
fn write_temp_json(name: &str, content: &str) -> std::path::PathBuf {
let path = std::env::temp_dir().join(format!("llmfit-test-{}-{name}", std::process::id()));
std::fs::write(&path, content).unwrap();
path
}
#[test]
fn test_load_custom_models_missing_file_is_empty() {
let path = std::path::Path::new("/nonexistent/llmfit-custom-models.json");
assert_eq!(load_custom_models_from(path).unwrap().len(), 0);
}
#[test]
fn test_load_custom_models_parses_minimal_entry() {
let path = write_temp_json("minimal.json", CUSTOM_ENTRY_JSON);
let models = load_custom_models_from(&path).unwrap();
std::fs::remove_file(&path).ok();
assert_eq!(models.len(), 1);
let m = &models[0];
assert_eq!(m.name, "acme/CustomNet-7B");
assert_eq!(m.context_length, 32768);
assert_eq!(m.quantization, "Q4_K_M");
}
#[test]
fn test_load_custom_models_invalid_json_is_error_not_empty() {
let path = write_temp_json("broken.json", "[{\"name\": ");
let result = load_custom_models_from(&path);
std::fs::remove_file(&path).ok();
let err = result.unwrap_err();
assert!(err.contains("invalid JSON"), "unexpected error: {err}");
}
#[test]
fn test_custom_overlay_replaces_embedded_entry_by_slug() {
// Simulate the overlay step in ModelDatabase::new() against the real
// embedded catalog: a custom entry whose slug matches an embedded
// model must replace it rather than duplicate it.
let mut models = load_embedded();
let original_len = models.len();
let victim = models[0].name.clone();
let json = CUSTOM_ENTRY_JSON.replace("acme/CustomNet-7B", &victim);
let path = write_temp_json("override.json", &json);
let custom = load_custom_models_from(&path).unwrap();
std::fs::remove_file(&path).ok();
let custom_keys: std::collections::HashSet<String> =
custom.iter().map(|m| canonical_slug(&m.name)).collect();
models.retain(|m| !custom_keys.contains(&canonical_slug(&m.name)));
models.extend(custom);
assert_eq!(models.len(), original_len, "override must not duplicate");
let replaced = models.iter().find(|m| m.name == victim).unwrap();
assert_eq!(replaced.use_case, "Testing");
}
#[test]
fn test_matches_license_filter_handles_comma_separated_model_licenses() {
let license = Some("apache-2.0,mit".to_string());
assert!(matches_license_filter(&license, "apache-2.0"));
assert!(matches_license_filter(&license, "mit"));
assert!(matches_license_filter(&license, "bsd-3-clause,mit"));
assert!(!matches_license_filter(&license, "cc-by-nc-4.0"));
assert!(!matches_license_filter(&None, "mit"));
}
// ────────────────────────────────────────────────────────────────────
// Quantization function tests
// ────────────────────────────────────────────────────────────────────
@@ -1686,6 +1873,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -1768,6 +1956,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -1804,6 +1993,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -1840,6 +2030,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -1876,6 +2067,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -1920,6 +2112,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -1970,6 +2163,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2004,6 +2198,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2046,6 +2241,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2080,6 +2276,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2124,6 +2321,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2160,6 +2358,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2196,6 +2395,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2249,6 +2449,7 @@ mod tests {
provider: "test".to_string(),
}],
capabilities: vec![Capability::Vision],
languages: vec!["en".to_string()],
format: ModelFormat::Safetensors,
hf_downloads: 10_000,
hf_likes: 500,
@@ -2285,6 +2486,7 @@ mod tests {
provider: "unsloth".to_string(),
}],
capabilities: vec![Capability::ToolUse],
languages: vec!["de".to_string()],
format: ModelFormat::Gguf,
hf_downloads: 100,
hf_likes: 10,
@@ -2333,6 +2535,9 @@ mod tests {
assert!(m.capabilities.contains(&Capability::Vision));
assert!(m.capabilities.contains(&Capability::ToolUse));
// Languages: union of explicit metadata
assert_eq!(m.languages, vec!["en", "de"]);
// GGUF sources: both repos present
assert_eq!(m.gguf_sources.len(), 2);
assert!(m.gguf_sources.iter().any(|s| s.repo == "test/model-a-gguf"));
@@ -2416,6 +2621,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2455,6 +2661,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2493,6 +2700,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2530,6 +2738,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![Capability::Vision],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2604,6 +2813,47 @@ mod tests {
}"#;
let entry: HfModelEntry = serde_json::from_str(json).unwrap();
assert!(entry.gguf_sources.is_empty());
assert!(entry.languages.is_empty());
}
#[test]
fn test_capability_infer_tts_adds_audio_and_tts() {
let model = LlmModel {
name: "hexgrad/Kokoro-82M".to_string(),
provider: "hexgrad".to_string(),
parameter_count: "82M".to_string(),
parameters_raw: Some(82_000_000),
min_ram_gb: 1.0,
recommended_ram_gb: 2.0,
min_vram_gb: Some(0.5),
quantization: "Q4_K_M".to_string(),
context_length: 4096,
use_case: "Text-to-speech".to_string(),
is_moe: false,
num_experts: None,
active_experts: None,
active_parameters: None,
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
num_hidden_layers: None,
head_dim: None,
attention_layout: None,
hidden_size: None,
moe_intermediate_size: None,
vocab_size: None,
shared_expert_intermediate_size: None,
architecture: None,
license: None,
};
let caps = Capability::infer(&model);
assert!(caps.contains(&Capability::Audio));
assert!(caps.contains(&Capability::Tts));
}
#[test]
@@ -2704,6 +2954,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: attn_heads,
num_key_value_heads: kv_heads,
@@ -2812,6 +3063,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: ModelFormat::default(),
num_attention_heads: Some(32),
num_key_value_heads: Some(8),
@@ -3078,4 +3330,56 @@ mod tests {
total_known_family
);
}
#[test]
fn test_embedded_database_includes_whisper_audio_models() {
// Regression guard for PR #603: the audio/ASR entries must live in the
// *embedded* data file (llmfit-core/data/hf_models.json). Editing only
// the repo-root copy is a silent no-op because the binary embeds the
// core copy via include_str! — this test would catch that (whisper
// count: 0) at `cargo test` time.
let db = ModelDatabase::embedded();
let whisper: Vec<_> = db
.get_all_models()
.iter()
.filter(|m| m.name.to_lowercase().contains("whisper"))
.collect();
assert!(
!whisper.is_empty(),
"embedded database has no Whisper models — audio entries are \
missing from llmfit-core/data/hf_models.json"
);
// Each Whisper entry must carry the Audio capability (set explicitly in
// JSON and re-derived by Capability::infer).
for m in &whisper {
assert!(
m.capabilities.contains(&Capability::Audio),
"Whisper model {:?} is missing Capability::Audio",
m.name
);
}
}
#[test]
fn test_embedded_database_includes_tts_models() {
let db = ModelDatabase::embedded();
let tts: Vec<_> = db
.get_all_models()
.iter()
.filter(|m| m.capabilities.contains(&Capability::Tts))
.collect();
assert!(
!tts.is_empty(),
"embedded database has no TTS models with Capability::Tts"
);
for m in &tts {
assert!(
m.capabilities.contains(&Capability::Audio),
"TTS model {:?} is missing broad Capability::Audio",
m.name
);
}
}
}
llmfit-core/src/plan.rs
+1
View File
@@ -855,6 +855,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: crate::models::ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
llmfit-core/src/providers.rs
+83 -47
View File
@@ -1343,6 +1343,13 @@ pub fn llamacpp_models_dir() -> PathBuf {
}
}
/// Check whether a binary is available on the system PATH.
/// Cross-platform: uses the `which` crate rather than shelling out to a
/// Unix-only `which` command, so it works on Windows too.
pub fn command_exists(name: &str) -> bool {
which::which(name).is_ok()
}
/// Find a binary by checking `LLAMA_CPP_PATH` env var, common install
/// locations, and finally the system PATH via `which`.
fn find_binary(name: &str) -> Option<String> {
@@ -1685,6 +1692,7 @@ impl ModelProvider for DockerModelRunnerProvider {
/// `POST /api/v1/models/download` and listed via `GET /v1/models`.
pub struct LmStudioProvider {
base_url: String,
api_key: Option<String>,
}
fn normalize_lmstudio_host(raw: &str) -> Option<String> {
@@ -1720,7 +1728,10 @@ impl Default for LmStudioProvider {
normalized
})
.unwrap_or_else(|| "http://127.0.0.1:1234".to_string());
Self { base_url }
let api_key = std::env::var("LMSTUDIO_API_KEY")
.ok()
.filter(|k| !k.is_empty());
Self { base_url, api_key }
}
}
@@ -1744,12 +1755,16 @@ impl LmStudioProvider {
/// Returns `(available, installed_models, count)`.
pub fn detect_with_installed(&self) -> (bool, HashSet<String>, usize) {
let mut set = HashSet::new();
let Ok(resp) = ureq::get(&self.models_url())
.config()
.timeout_global(Some(std::time::Duration::from_millis(800)))
.build()
.call()
else {
let Ok(resp) = ({
let mut req = ureq::get(&self.models_url())
.config()
.timeout_global(Some(std::time::Duration::from_millis(800)))
.build();
if let Some(ref key) = self.api_key {
req = req.header("Authorization", &format!("Bearer {}", key));
}
req.call()
}) else {
return (false, set, 0);
};
@@ -1818,12 +1833,14 @@ impl ModelProvider for LmStudioProvider {
}
fn is_available(&self) -> bool {
ureq::get(&self.models_url())
let mut req = ureq::get(&self.models_url())
.config()
.timeout_global(Some(std::time::Duration::from_secs(2)))
.build()
.call()
.is_ok()
.build();
if let Some(ref key) = self.api_key {
req = req.header("Authorization", &format!("Bearer {}", key));
}
req.call().is_ok()
}
fn installed_models(&self) -> HashSet<String> {
@@ -1834,6 +1851,7 @@ impl ModelProvider for LmStudioProvider {
fn start_pull(&self, model_tag: &str) -> Result<PullHandle, String> {
let download_url = self.download_url();
let models_url = self.models_url();
let api_key = self.api_key.clone();
let tag = match lmstudio_pull_tag(model_tag) {
Some(t) => t,
None => {
@@ -1858,11 +1876,14 @@ impl ModelProvider for LmStudioProvider {
// close the stream while the download proceeds in the background.
// In the latter case we poll the installed models list to detect
// eventual completion.
let resp = ureq::post(&download_url)
let mut req = ureq::post(&download_url)
.config()
.timeout_global(Some(std::time::Duration::from_secs(3600)))
.build()
.send_json(&body);
.build();
if let Some(ref key) = api_key {
req = req.header("Authorization", &format!("Bearer {}", key));
}
let resp = req.send_json(&body);
match resp {
Ok(resp) => {
@@ -1969,12 +1990,14 @@ impl ModelProvider for LmStudioProvider {
for poll_num in 0..max_polls {
std::thread::sleep(poll_interval);
let Ok(resp) = ureq::get(&models_url)
let mut req = ureq::get(&models_url)
.config()
.timeout_global(Some(std::time::Duration::from_secs(5)))
.build()
.call()
else {
.build();
if let Some(ref key) = api_key {
req = req.header("Authorization", &format!("Bearer {}", key));
}
let Ok(resp) = req.call() else {
continue;
};
@@ -2109,10 +2132,10 @@ fn lmstudio_find_gguf_url(hf_name: &str) -> Option<String> {
let budget_gb = system_ram_gb * 0.85;
// Try known mappings first
if let Some(repo) = lookup_gguf_repo(hf_name) {
if let Some(url) = try_gguf_repo(repo, budget_gb) {
return Some(url);
}
if let Some(repo) = lookup_gguf_repo(hf_name)
&& let Some(url) = try_gguf_repo(repo, budget_gb)
{
return Some(url);
}
// Try heuristic candidates (bartowski/, ggml-org/, TheBloke/)
@@ -2123,10 +2146,10 @@ fn lmstudio_find_gguf_url(hf_name: &str) -> Option<String> {
}
// Try the base repo itself (some repos host GGUF directly)
if hf_name.contains('/') {
if let Some(url) = try_gguf_repo(hf_name, budget_gb) {
return Some(url);
}
if hf_name.contains('/')
&& let Some(url) = try_gguf_repo(hf_name, budget_gb)
{
return Some(url);
}
None
@@ -3436,6 +3459,26 @@ mod tests {
assert_eq!(candidates.len(), 1);
}
#[test]
fn test_lmstudio_api_key_filtering() {
// Test the api_key filtering logic without mutating the process
// environment. LmStudioProvider::default() applies
// `.filter(|k| !k.is_empty())` to the env var value.
fn filter_key(val: Option<&str>) -> Option<String> {
val.map(String::from).filter(|k| !k.is_empty())
}
// Missing env var → None
assert!(filter_key(None).is_none());
// Real value → Some
assert_eq!(
filter_key(Some("my-secret-key")),
Some("my-secret-key".to_string())
);
// Empty string → None (must not produce Some(""))
assert!(filter_key(Some("")).is_none());
}
#[test]
fn test_is_model_installed_mlx_with_owner_prefixed_repo_id() {
let mut installed = HashSet::new();
@@ -4425,28 +4468,21 @@ mod tests {
}
#[test]
fn test_docker_desktop_running_via_env_var() {
// Test 1: Non-empty value should detect Docker Desktop
unsafe {
std::env::set_var("DOCKER_MODEL_RUNNER_HOST", "localhost:12434");
fn test_docker_model_runner_host_filtering() {
// Test the DOCKER_MODEL_RUNNER_HOST filtering logic without mutating the
// process environment. is_docker_desktop_running() applies
// `!v.trim().is_empty()` to the env var value.
fn host_is_set(val: Option<&str>) -> bool {
val.map(|v| !v.trim().is_empty()).unwrap_or(false)
}
assert!(is_docker_desktop_running());
// Test 2: Empty string should NOT count as running
unsafe {
std::env::set_var("DOCKER_MODEL_RUNNER_HOST", "");
}
assert!(!is_docker_desktop_running());
// Test 3: Whitespace-only should NOT count as running
unsafe {
std::env::set_var("DOCKER_MODEL_RUNNER_HOST", " ");
}
assert!(!is_docker_desktop_running());
// Cleanup
unsafe {
std::env::remove_var("DOCKER_MODEL_RUNNER_HOST");
}
// Non-empty value should count as set
assert!(host_is_set(Some("localhost:12434")));
// Empty string should NOT count
assert!(!host_is_set(Some("")));
// Whitespace-only should NOT count
assert!(!host_is_set(Some(" ")));
// Missing env var should NOT count
assert!(!host_is_set(None));
}
}
llmfit-core/src/quality.rs
+4 -4
View File
@@ -354,10 +354,10 @@ where
for (role_name, role_def) in &config.roles {
// Skip roles not in the filter (if a filter is set)
if let Some(filter) = role_filter {
if !filter.iter().any(|f| f == role_name) {
continue;
}
if let Some(filter) = role_filter
&& !filter.iter().any(|f| f == role_name)
{
continue;
}
let mut role_results = Vec::new();
llmfit-core/src/task_bench.rs
+90
View File
@@ -0,0 +1,90 @@
//! Curated per-family use-case benchmark scores (issue #150).
//!
//! Maps model families to relative task-strength scores (0-100) aggregated
//! from public leaderboards, so a strong coding model can outrank a larger
//! generalist when the user asks for a coding recommendation. The table is
//! embedded from `data/use_case_benchmarks.json` and refreshed alongside the
//! weekly model-database update; [`score`] returns `None` for models without
//! an entry, which fall back to the name heuristics in `fit.rs`.
use std::collections::HashMap;
use std::sync::OnceLock;
const TASK_BENCH_JSON: &str = include_str!("../data/use_case_benchmarks.json");
#[derive(serde::Deserialize)]
struct FamilyEntry {
#[serde(rename = "match")]
patterns: Vec<String>,
scores: HashMap<String, f64>,
}
#[derive(serde::Deserialize)]
struct BenchFile {
families: Vec<FamilyEntry>,
}
fn table() -> &'static [FamilyEntry] {
static TABLE: OnceLock<Vec<FamilyEntry>> = OnceLock::new();
TABLE.get_or_init(|| {
serde_json::from_str::<BenchFile>(TASK_BENCH_JSON)
.expect("embedded use_case_benchmarks.json is invalid")
.families
})
}
/// Benchmark score for a model on a task (`"coding"`, `"reasoning"`,
/// `"chat"`), or `None` if no family entry matches.
///
/// `name_lower` must already be lowercased. When several patterns match
/// (e.g. `qwen3` and `qwen3-coder`), the longest — most specific — wins.
pub fn score(name_lower: &str, task: &str) -> Option<f64> {
let mut best: Option<(usize, f64)> = None;
for entry in table() {
for pattern in &entry.patterns {
if name_lower.contains(pattern.as_str())
&& let Some(s) = entry.scores.get(task)
&& best.is_none_or(|(len, _)| pattern.len() > len)
{
best = Some((pattern.len(), *s));
}
}
}
best.map(|(_, s)| s)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_embedded_table_parses() {
assert!(!table().is_empty());
}
#[test]
fn test_longest_pattern_wins() {
// "qwen3.5-coder-…" matches both "qwen3.5" and "qwen3.5-coder";
// the coder entry must win for the coding task.
let coder = score("qwen/qwen3.5-coder-32b-instruct", "coding").unwrap();
let base = score("qwen/qwen3.5-32b-instruct", "coding").unwrap();
assert!(coder > base, "coder {coder} <= base {base}");
}
#[test]
fn test_unknown_family_is_none() {
assert_eq!(score("acme/customnet-7b", "coding"), None);
assert_eq!(score("qwen/qwen3.5-32b", "nonexistent-task"), None);
}
#[test]
fn test_coding_specialist_beats_generalist_at_coding_only() {
let starcoder_code = score("bigcode/starcoder2-15b", "coding").unwrap();
let llama_code = score("meta-llama/llama-3.3-70b", "coding").unwrap();
assert!(starcoder_code > llama_code);
let starcoder_chat = score("bigcode/starcoder2-15b", "chat").unwrap();
let llama_chat = score("meta-llama/llama-3.3-70b", "chat").unwrap();
assert!(llama_chat > starcoder_chat);
}
}
llmfit-core/src/update.rs
+195 -30
View File
@@ -12,13 +12,29 @@ use serde::Deserialize;
use std::collections::HashSet;
use std::path::PathBuf;
use crate::models::{LlmModel, ModelFormat};
use crate::models::{Capability, LlmModel, ModelFormat};
const HF_API: &str = "https://huggingface.co/api/models";
/// Bump this when the `LlmModel` schema changes in a breaking way.
/// A cache written by an older version will be discarded and re-fetched.
const CACHE_VERSION: u32 = 3;
const CACHE_VERSION: u32 = 4;
const ACCEPTED_PIPELINES: &[&str] = &[
"text-generation",
"image-text-to-text",
"any-to-any",
"text-to-speech",
];
const PRIMARY_UPDATE_PIPELINE: &str = "text-generation";
fn pipeline_query_limit(limit: usize, pipeline: &str) -> usize {
if pipeline == PRIMARY_UPDATE_PIPELINE {
limit
} else {
(limit / ACCEPTED_PIPELINES.len()).max(1)
}
}
// ── Cache helpers ─────────────────────────────────────────────────────────────
@@ -103,6 +119,9 @@ struct HfApiModel {
safetensors: Option<SafetensorsInfo>,
#[serde(default)]
license: Option<String>,
/// GGUF metadata (parameter count, context length) for GGUF-only repos.
#[serde(default)]
gguf: Option<GgufInfo>,
}
#[derive(Deserialize, Debug, Default)]
@@ -111,6 +130,14 @@ struct SafetensorsInfo {
total: Option<u64>,
}
#[derive(Deserialize, Debug, Default)]
struct GgufInfo {
#[serde(default)]
total: Option<u64>,
#[serde(default)]
context_length: Option<u32>,
}
// ── Parameter extraction ──────────────────────────────────────────────────────
/// Parse "7B" → 7_000_000_000u64, "1.5B" → 1_500_000_000u64, "500M" → 500_000_000u64.
@@ -224,7 +251,9 @@ fn infer_use_case(model_id: &str, tags: &[String]) -> String {
model_id.to_lowercase(),
tags.join(" ").to_lowercase()
);
if lower.contains("embed") || lower.contains("bge") || lower.contains("-e5-") {
if lower.contains("text-to-speech") {
"Text-to-speech".to_string()
} else if lower.contains("embed") || lower.contains("bge") || lower.contains("-e5-") {
"Embedding".to_string()
} else if lower.contains("code") || lower.contains("starcoder") || lower.contains("coder") {
"Code generation".to_string()
@@ -247,6 +276,59 @@ fn infer_use_case(model_id: &str, tags: &[String]) -> String {
}
}
fn infer_capabilities(pipeline_tag: Option<&str>, use_case: &str) -> Vec<Capability> {
if pipeline_tag == Some("text-to-speech") || use_case.eq_ignore_ascii_case("text-to-speech") {
vec![Capability::Audio, Capability::Tts]
} else {
vec![]
}
}
fn normalize_language(value: &str) -> Option<String> {
let mut lang = value.trim().to_lowercase().replace('_', "-");
let mut prefixed = false;
for prefix in ["language:", "languages:", "lang:"] {
if let Some(stripped) = lang.strip_prefix(prefix) {
lang = stripped.to_string();
prefixed = true;
break;
}
}
let mut parts = lang.split('-');
let Some(primary) = parts.next() else {
return None;
};
if !primary.chars().all(|c| c.is_ascii_alphabetic()) {
return None;
}
if primary.len() == 3 && !prefixed {
return None;
}
if primary.len() != 2 && primary.len() != 3 {
return None;
}
if parts.all(|part| {
(2..=8).contains(&part.len()) && part.chars().all(|c| c.is_ascii_alphanumeric())
}) {
Some(lang)
} else {
None
}
}
fn infer_languages(tags: &[String]) -> Vec<String> {
let mut languages = Vec::new();
for tag in tags {
if let Some(lang) = normalize_language(tag)
&& !languages.contains(&lang)
{
languages.push(lang);
}
}
languages
}
// ── Context-length inference ──────────────────────────────────────────────────
fn infer_context_length(model_id: &str, params_raw: Option<u64>) -> u32 {
@@ -381,10 +463,15 @@ fn resolve_head_dim(cfg: &HfConfig) -> Option<u32> {
// ── HF API fetching ───────────────────────────────────────────────────────────
fn hf_get_list(sort: &str, limit: usize, token: Option<&str>) -> Result<Vec<HfApiModel>, String> {
fn hf_get_list_for_pipeline(
pipeline: &str,
sort: &str,
limit: usize,
token: Option<&str>,
) -> Result<Vec<HfApiModel>, String> {
let url = format!(
"{}?pipeline_tag=text-generation&sort={}&limit={}",
HF_API, sort, limit
"{}?pipeline_tag={}&sort={}&limit={}&expand[]=gguf",
HF_API, pipeline, sort, limit
);
let resp = if let Some(t) = token {
ureq::get(&url)
@@ -428,18 +515,31 @@ fn hf_get_list(sort: &str, limit: usize, token: Option<&str>) -> Result<Vec<HfAp
/// (`num_hidden_layers`, `num_attention_heads`, `num_key_value_heads`,
/// `head_dim`). The fetch is best-effort and silently degrades to `None`.
fn map_to_llm_model(hf: HfApiModel, token: Option<&str>) -> Option<LlmModel> {
let is_tg = hf.pipeline_tag.as_deref() == Some("text-generation")
|| hf.tags.iter().any(|t| t == "text-generation");
let is_tg = hf
.pipeline_tag
.as_deref()
.is_some_and(|p| ACCEPTED_PIPELINES.contains(&p))
|| hf
.tags
.iter()
.any(|t| ACCEPTED_PIPELINES.contains(&t.as_str()));
if !is_tg {
return None;
}
// Use safetensors for an exact parameter count when available, but always
// run name-based parsing for MoE architecture hints — safetensors only
// reports total parameters and would cause MoE models (e.g. Mixtral) to
// lose their MoE classification and receive inaccurate VRAM estimates.
// Use safetensors or GGUF metadata for an exact parameter count when
// available, but always run name-based parsing for MoE architecture
// hints — safetensors/GGUF only report total parameters and would cause
// MoE models (e.g. Mixtral, DeepSeek) to lose their MoE classification
// and receive inaccurate VRAM estimates.
let exact_total = hf
.safetensors
.as_ref()
.and_then(|s| s.total)
.or_else(|| hf.gguf.as_ref().and_then(|g| g.total));
let (param_str, params_raw, is_moe, num_experts, active_experts, active_params) =
if let Some(total) = hf.safetensors.as_ref().and_then(|s| s.total) {
if let Some(total) = exact_total {
let (_, _, is_moe, num_experts, active_experts, active_params) =
extract_model_params(&hf.id);
let b = total / 1_000_000_000;
@@ -467,7 +567,15 @@ fn map_to_llm_model(hf: HfApiModel, token: Option<&str>) -> Option<LlmModel> {
let raw = params_raw.unwrap_or(7_000_000_000);
let use_case = infer_use_case(&hf.id, &hf.tags);
let context_length = infer_context_length(&hf.id, params_raw);
let capabilities = infer_capabilities(hf.pipeline_tag.as_deref(), &use_case);
let languages = infer_languages(&hf.tags);
let is_tts = capabilities.contains(&Capability::Tts);
// Prefer GGUF-reported context length (authoritative), fall back to heuristic.
let context_length = hf
.gguf
.as_ref()
.and_then(|g| g.context_length)
.unwrap_or_else(|| infer_context_length(&hf.id, params_raw));
let (min_ram, rec_ram, min_vram) = estimate_ram(raw, is_moe, active_params);
let provider = hf
@@ -550,10 +658,9 @@ fn map_to_llm_model(hf: HfApiModel, token: Option<&str>) -> Option<LlmModel> {
min_ram_gb: min_ram,
recommended_ram_gb: rec_ram,
min_vram_gb: min_vram,
// Q4_K_M is used as a conservative approximation for all fetched models.
// Actual available quantizations depend on the GGUF files published for
// each model. RAM/VRAM estimates downstream reflect this assumption.
quantization: "Q4_K_M".to_string(),
// Q4_K_M is used as a conservative approximation for LLMs. TTS models
// are not GGUF/llama.cpp-compatible here, so keep their HF format.
quantization: if is_tts { "F16" } else { "Q4_K_M" }.to_string(),
context_length,
use_case,
is_moe,
@@ -562,8 +669,13 @@ fn map_to_llm_model(hf: HfApiModel, token: Option<&str>) -> Option<LlmModel> {
active_parameters: active_params,
release_date,
gguf_sources: vec![],
capabilities: vec![],
format: ModelFormat::default(),
capabilities,
languages,
format: if is_tts {
ModelFormat::Safetensors
} else {
ModelFormat::default()
},
num_attention_heads,
num_key_value_heads,
num_hidden_layers,
@@ -638,12 +750,22 @@ pub fn update_model_cache(
"Fetching {} trending models from HuggingFace...",
opts.trending_limit
));
match hf_get_list("trendingScore", opts.trending_limit, token) {
Ok(list) => {
progress(&format!(" Received {} trending models", list.len()));
all_hf.extend(list);
for pipeline in ACCEPTED_PIPELINES {
let limit = pipeline_query_limit(opts.trending_limit, pipeline);
match hf_get_list_for_pipeline(pipeline, "trendingScore", limit, token) {
Ok(list) => {
progress(&format!(
" Received {} trending {} models",
list.len(),
pipeline
));
all_hf.extend(list);
}
Err(e) => progress(&format!(
" Warning: trending {} fetch failed — {e}",
pipeline
)),
}
Err(e) => progress(&format!(" Warning: trending fetch failed — {e}")),
}
}
@@ -652,12 +774,22 @@ pub fn update_model_cache(
"Fetching {} top-downloaded models...",
opts.downloads_limit
));
match hf_get_list("downloads", opts.downloads_limit, token) {
Ok(list) => {
progress(&format!(" Received {} download-ranked models", list.len()));
all_hf.extend(list);
for pipeline in ACCEPTED_PIPELINES {
let limit = pipeline_query_limit(opts.downloads_limit, pipeline);
match hf_get_list_for_pipeline(pipeline, "downloads", limit, token) {
Ok(list) => {
progress(&format!(
" Received {} download-ranked {} models",
list.len(),
pipeline
));
all_hf.extend(list);
}
Err(e) => progress(&format!(
" Warning: downloads {} fetch failed — {e}",
pipeline
)),
}
Err(e) => progress(&format!(" Warning: downloads fetch failed — {e}")),
}
}
@@ -761,6 +893,39 @@ mod tests {
assert!(uc.to_lowercase().contains("embed"), "got: {}", uc);
}
#[test]
fn test_infer_use_case_tts() {
let uc = infer_use_case("hexgrad/Kokoro-82M", &["text-to-speech".to_string()]);
assert_eq!(uc, "Text-to-speech");
}
#[test]
fn test_infer_capabilities_tts() {
let caps = infer_capabilities(Some("text-to-speech"), "Text-to-speech");
assert!(caps.contains(&Capability::Audio));
assert!(caps.contains(&Capability::Tts));
}
#[test]
fn test_pipeline_query_limit_preserves_primary_budget() {
assert_eq!(pipeline_query_limit(100, "text-generation"), 100);
assert_eq!(pipeline_query_limit(100, "text-to-speech"), 25);
assert_eq!(pipeline_query_limit(2, "text-to-speech"), 1);
}
#[test]
fn test_infer_languages_from_explicit_tags_only() {
let tags = vec![
"text-to-speech".to_string(),
"language:en".to_string(),
"language:tir".to_string(),
"fr".to_string(),
"tts".to_string(),
"not-a-language".to_string(),
];
assert_eq!(infer_languages(&tags), vec!["en", "tir", "fr"]);
}
#[test]
fn test_infer_context_length_keywords() {
assert_eq!(infer_context_length("model-128k", None), 131_072);
llmfit-core/tests/schema_validation.rs
+44
View File
@@ -0,0 +1,44 @@
use jsonschema::Validator;
use serde_json::Value;
use std::path::Path;
fn load_schema() -> Value {
let path = Path::new(env!("CARGO_MANIFEST_DIR")).join("data/schema.json");
let text = std::fs::read_to_string(&path)
.unwrap_or_else(|e| panic!("cannot read {}: {e}", path.display()));
serde_json::from_str(&text).unwrap_or_else(|e| panic!("cannot parse schema JSON: {e}"))
}
fn validate_file(schema: &Validator, rel_path: &str) {
let path = Path::new(env!("CARGO_MANIFEST_DIR")).join(rel_path);
let text = std::fs::read_to_string(&path)
.unwrap_or_else(|e| panic!("cannot read {}: {e}", path.display()));
let data: Value = serde_json::from_str(&text)
.unwrap_or_else(|e| panic!("cannot parse {}: {e}", path.display()));
let errors: Vec<String> = schema
.iter_errors(&data)
.take(30)
.map(|e| format!(" [{}] {}", e.instance_path(), e))
.collect();
assert!(
errors.is_empty(),
"{rel_path}: {} schema violation(s):\n{}",
errors.len(),
errors.join("\n")
);
let count = data.as_array().map(|a| a.len()).unwrap_or(0);
println!("✓ {rel_path} ({count} models)");
}
#[test]
fn hf_models_match_schema() {
let schema_value = load_schema();
let schema = jsonschema::validator_for(&schema_value)
.expect("schema itself is invalid — check llmfit-core/data/schema.json");
validate_file(&schema, "data/hf_models.json");
}
llmfit-desktop/src/main.rs
+1
View File
@@ -125,6 +125,7 @@ fn get_model_fits() -> Result<Vec<ModelFitInfo>, String> {
InferenceRuntime::LlamaCpp => "llama.cpp".to_string(),
InferenceRuntime::Mlx => "MLX".to_string(),
InferenceRuntime::Vllm => "vLLM".to_string(),
InferenceRuntime::Unsupported => "unsupported".to_string(),
},
installed: f.installed,
notes: f.notes.clone(),
llmfit-desktop/tauri.conf.json
+5
View File
@@ -5,6 +5,11 @@
"build": {
"frontendDist": "./ui"
},
"bundle": {
"windows": {
"webviewInstallMode": "downloadBootstrapper"
}
},
"app": {
"withGlobalTauri": true,
"windows": [
llmfit-python/hatch_build.py
+1
View File
@@ -41,6 +41,7 @@ from packaging.tags import sys_tags
TARGET_CONFIGS: dict[str, tuple[str, str]] = {
"manylinux_2_17_x86_64": ("x86_64-unknown-linux-gnu", "llmfit"),
"manylinux_2_17_aarch64": ("aarch64-unknown-linux-gnu", "llmfit"),
"manylinux_2_39_riscv64": ("riscv64gc-unknown-linux-gnu", "llmfit"),
"musllinux_1_2_x86_64": ("x86_64-unknown-linux-musl", "llmfit"),
"musllinux_1_2_aarch64": ("aarch64-unknown-linux-musl", "llmfit"),
"macosx_10_12_x86_64": ("x86_64-apple-darwin", "llmfit"),
llmfit-tui/Cargo.toml
+5 -3
View File
@@ -26,17 +26,19 @@ clap = { version = "4.6", features = ["derive", "env"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
schemars = "1.0"
tabled = "0.20"
tabled = "0.21"
colored = "3.1"
csv = "1.4"
ratatui = "0.30"
crossterm = "0.29"
unicode-segmentation = "1.13"
unicode-width = "0.2"
arboard = "3.4"
dirs = "6.0"
axum = "0.8"
tokio = { version = "1.52", features = ["rt-multi-thread", "signal", "net", "io-std"] }
rmcp = { version = "1.6", features = ["server", "macros", "transport-io"] }
async-nats = { version = "0.48", optional = true }
rmcp = { version = "1.7", features = ["server", "macros", "transport-io"] }
async-nats = { version = "0.49", optional = true }
[dev-dependencies]
http-body-util = "0.1"
llmfit-tui/src/display.rs
+5 -1
View File
@@ -130,7 +130,7 @@ pub fn display_model_fits(fits: &[ModelFit]) {
runtime: fit.runtime_text().to_string(),
mode: fit.run_mode_text().to_string(),
mem_use: format!("{:.1}%", fit.utilization_pct),
context: format!("{}k", fit.model.context_length / 1000),
context: fit.context_display(),
release_date: fit
.model
.release_date
@@ -682,6 +682,7 @@ fn system_json(specs: &SystemSpecs) -> serde_json::Value {
"backend": g.backend.label(),
"count": g.count,
"unified_memory": g.unified_memory,
"memory_bandwidth_gbps": llmfit_core::hardware::gpu_memory_bandwidth_gbps(&g.name),
})
})
.collect();
@@ -709,6 +710,7 @@ fn fit_to_json(fit: &ModelFit) -> serde_json::Value {
"params_b": round2(fit.model.params_b()),
"context_length": fit.model.context_length,
"effective_context_length": fit.effective_context_length,
"usable_context": fit.usable_context,
"use_case": fit.model.use_case,
"category": fit.use_case.label(),
"release_date": fit.model.release_date,
@@ -955,6 +957,7 @@ mod tests {
provider: "test".to_string(),
}],
capabilities: vec![],
languages: vec![],
format: ModelFormat::Gguf,
num_attention_heads: None,
num_key_value_heads: None,
@@ -989,6 +992,7 @@ mod tests {
installed: false,
fits_with_turboquant: false,
effective_context_length: 8_192,
usable_context: 8_192,
}
}
llmfit-tui/src/main.rs
+209 -78
View File
@@ -168,6 +168,12 @@ struct Cli {
#[arg(long, value_name = "TOKENS", value_parser = clap::value_parser!(u32).range(1..))]
max_context: Option<u32>,
/// Force the interactive TUI, ignoring any subcommand or output flags.
/// Useful in Docker where a baked-in CMD would otherwise run a subcommand:
/// docker run --rm -it ghcr.io/alexsjones/llmfit --tui
#[arg(long, global = true)]
tui: bool,
/// Do not auto-start the background dashboard server
#[arg(long, global = true)]
no_dashboard: bool,
@@ -204,6 +210,76 @@ AGENT USAGE:
gpu_backend, unified_memory, os } }")]
System,
/// Print a hardware diagnostic report for bug reports
#[command(long_about = "\
Print a hardware diagnostic report for GitHub bug reports.
Captures the raw output of every external tool GPU detection shells out to
(nvidia-smi, rocm-smi, sysfs, lspci, system_profiler, WMI, vulkaninfo,
npu-smi) alongside what llmfit actually detected, so detection bugs can be
reproduced — and turned into regression tests — from the report alone.
PRECONDITIONS:
None. Missing tools are reported as unavailable, which is itself useful.
SIDE EFFECTS:
None — read-only. Output contains hardware model names and driver info
only; no hostnames, usernames, or serial numbers.
EXIT CODES:
0 Success
AGENT USAGE:
llmfit doctor > llmfit-doctor.md
Output is Markdown; attach or paste it into a GitHub issue.")]
Doctor,
/// Generate a Kubernetes DRA ResourceClaim encoding the model's fit
#[command(long_about = "\
Generate a Kubernetes DRA ResourceClaim (or ResourceClaimTemplate) whose CEL
selector encodes the model's fit inequality against attributes published by
the llmfit.ai DRA driver (llmfit-dra). Constants (weights size, memory floor,
bandwidth floor) are resolved from the model database and inlined; the YAML
is printed on stdout with provenance comments.
PRECONDITIONS:
None locally. Applying the output requires a cluster running llmfit-dra
(Kubernetes >= 1.34) with its shipped DeviceClasses.
SIDE EFFECTS:
None — prints YAML; pipe to kubectl to apply.
EXIT CODES:
0 Success
1 Unknown/ambiguous model, or invalid bounds
AGENT USAGE:
llmfit claim qwen2.5-32b --min-tps 20 | kubectl apply -f -
llmfit claim mistral-7b --template > claim-template.yaml")]
Claim {
/// Model name (exact or unambiguous partial match)
model: String,
/// Minimum acceptable decode speed, tokens/second
#[arg(long, default_value_t = 20.0)]
min_tps: f64,
/// Override the database entry's quantization (e.g. Q4_K_M, Q8_0)
#[arg(long)]
quant: Option<String>,
/// Backend efficiency percentage used in the fit inequality
#[arg(long, default_value_t = 55)]
efficiency: u32,
/// DeviceClass the claim requests against
#[arg(long, default_value = "llmfit.ai")]
device_class: String,
/// Emit a ResourceClaimTemplate (for pod templates) instead of a ResourceClaim
#[arg(long)]
template: bool,
/// metadata.name for the generated object (default: derived from the model name)
#[arg(long)]
name: Option<String>,
},
/// List all available LLM models
#[command(long_about = "\
List all available LLM models.
@@ -479,7 +555,7 @@ AGENT USAGE:
#[arg(long, value_name = "RUNTIME")]
force_runtime: Option<String>,
/// Filter by capability: vision, tool_use (comma-separated for multiple)
/// Filter by capability: vision, tool_use, audio, tts (comma-separated)
#[arg(long, value_name = "CAPS")]
capability: Option<String>,
@@ -682,6 +758,14 @@ AGENT USAGE:
#[arg(long, default_value = "8787")]
port: u16,
/// Listen on a Unix domain socket instead of TCP (unix platforms
/// only). Any stale socket file is replaced; the socket is created
/// with mode 0660. Intended for same-host/same-pod consumers (e.g.
/// the llmfit-dra DRA driver sidecar) where a TCP port on the host
/// network is undesirable.
#[arg(long, value_name = "PATH", conflicts_with_all = ["host", "port"])]
unix_socket: Option<std::path::PathBuf>,
/// Run as MCP server on stdio instead of HTTP
#[arg(long)]
mcp: bool,
@@ -957,11 +1041,6 @@ fn run_fit(
overrides: &HardwareOverrides,
context_limit: Option<u32>,
) {
use llmfit_core::providers::{
self as provs, DockerModelRunnerProvider, LlamaCppProvider, LmStudioProvider, MlxProvider,
ModelProvider, OllamaProvider,
};
let specs = detect_specs(overrides);
let db = ModelDatabase::new();
@@ -969,13 +1048,7 @@ fn run_fit(
specs.display();
}
// Query installed models across local providers so that `fit.installed`
// is populated in both text and JSON output — same behaviour as `recommend`.
let ollama_installed = OllamaProvider::new().installed_models();
let mlx_installed = MlxProvider::new().installed_models();
let llamacpp_installed = LlamaCppProvider::new().installed_models();
let docker_mr_installed = DockerModelRunnerProvider::new().installed_models();
let lmstudio_installed = LmStudioProvider::new().installed_models();
let installed = llmfit_core::analysis::InstalledIndex::detect_all();
let hidden: usize = db
.get_all_models()
@@ -983,20 +1056,8 @@ fn run_fit(
.filter(|m| !backend_compatible(m, &specs))
.count();
let mut fits: Vec<ModelFit> = db
.get_all_models()
.iter()
.filter(|m| backend_compatible(m, &specs))
.map(|m| {
let mut fit = ModelFit::analyze_with_context_limit(m, &specs, context_limit);
fit.installed = provs::is_model_installed(&m.name, &ollama_installed)
|| provs::is_model_installed_mlx(&m.name, &mlx_installed)
|| provs::is_model_installed_llamacpp(&m.name, &llamacpp_installed)
|| provs::is_model_installed_docker_mr(&m.name, &docker_mr_installed)
|| provs::is_model_installed_lmstudio(&m.name, &lmstudio_installed);
fit
})
.collect();
let mut fits =
llmfit_core::analysis::build_model_fits(&db, &specs, &installed, context_limit, None);
if perfect {
fits.retain(|f| f.fit_level == llmfit_core::fit::FitLevel::Perfect);
@@ -1307,35 +1368,10 @@ fn run_recommend(
}
});
// Query installed models across local providers so that `fit.installed`
// is populated for CLI output (same behavior as the TUI). This also causes
// backends like Docker Model Runner to receive a probe request when
// DOCKER_MODEL_RUNNER_HOST is set.
use llmfit_core::providers::{
self as provs, DockerModelRunnerProvider, LlamaCppProvider, LmStudioProvider, MlxProvider,
ModelProvider, OllamaProvider,
};
let ollama_installed = OllamaProvider::new().installed_models();
let mlx_installed = MlxProvider::new().installed_models();
let llamacpp_installed = LlamaCppProvider::new().installed_models();
let docker_mr_installed = DockerModelRunnerProvider::new().installed_models();
let lmstudio_installed = LmStudioProvider::new().installed_models();
let installed = llmfit_core::analysis::InstalledIndex::detect_all();
let mut fits: Vec<ModelFit> = db
.get_all_models()
.iter()
.filter(|m| backend_compatible(m, &specs))
.map(|m| {
let mut fit =
ModelFit::analyze_with_forced_runtime(m, &specs, context_limit, forced_rt);
fit.installed = provs::is_model_installed(&m.name, &ollama_installed)
|| provs::is_model_installed_mlx(&m.name, &mlx_installed)
|| provs::is_model_installed_llamacpp(&m.name, &llamacpp_installed)
|| provs::is_model_installed_docker_mr(&m.name, &docker_mr_installed)
|| provs::is_model_installed_lmstudio(&m.name, &lmstudio_installed);
fit
})
.collect();
let mut fits =
llmfit_core::analysis::build_model_fits(&db, &specs, &installed, context_limit, forced_rt);
// Filter by minimum fit level
let min_level = match min_fit.to_lowercase().as_str() {
@@ -1390,11 +1426,14 @@ fn run_recommend(
// Filter by capability if specified
if let Some(ref caps_str) = capability {
let requested: Vec<&str> = caps_str.split(',').map(|s| s.trim()).collect();
fits.retain(|f| {
requested
.iter()
.all(|req| match req.to_lowercase().as_str() {
let requested: Vec<String> = caps_str
.split(',')
.map(|s| s.trim().to_lowercase())
.filter(|s| !s.is_empty())
.collect();
if !requested.is_empty() {
fits.retain(|f| {
requested.iter().all(|req| match req.as_str() {
"vision" => f
.model
.capabilities
@@ -1403,9 +1442,18 @@ fn run_recommend(
.model
.capabilities
.contains(&llmfit_core::models::Capability::ToolUse),
_ => true,
"audio" => f
.model
.capabilities
.contains(&llmfit_core::models::Capability::Audio),
"tts" | "text-to-speech" | "text_to_speech" => f
.model
.capabilities
.contains(&llmfit_core::models::Capability::Tts),
_ => false,
})
});
});
}
}
// Filter by license if specified
@@ -2434,18 +2482,9 @@ fn display_routing_matrix_full(
let total_tests: usize = results.iter().map(|r| r.roles.len()).sum();
println!();
println!(
"{}",
"╔══════════════════════════════════════════════════════╗"
);
println!(
"{}",
"║ MODEL ROUTING MATRIX ║"
);
println!(
"{}",
"╚══════════════════════════════════════════════════════╝"
);
println!("╔══════════════════════════════════════════════════════╗");
println!("║ MODEL ROUTING MATRIX ║");
println!("╚══════════════════════════════════════════════════════╝");
println!();
println!(
"Provider: {} • Models: {} • Roles: {} • Tests: {}",
@@ -2544,8 +2583,8 @@ fn main() {
cpu_cores: cli.cpu_cores,
};
let auto_dashboard = !cli.no_dashboard
&& !cli.json
&& !matches!(cli.command.as_ref(), Some(Commands::Serve { .. }));
&& (cli.tui
|| (!cli.json && !matches!(cli.command.as_ref(), Some(Commands::Serve { .. }))));
let _dashboard_guard = if auto_dashboard {
ensure_dashboard_available(&overrides, context_limit)
@@ -2553,6 +2592,17 @@ fn main() {
None
};
// --tui forces the interactive TUI regardless of any subcommand or
// output flags, so a Docker image with a baked-in CMD can still launch
// the TUI: docker run --rm -it ghcr.io/alexsjones/llmfit --tui
if cli.tui {
if let Err(e) = run_tui(&overrides, context_limit, cli.api_key) {
eprintln!("Error running TUI: {}", e);
std::process::exit(1);
}
return;
}
// If a subcommand is given, use classic CLI mode
if let Some(command) = cli.command {
match command {
@@ -2565,6 +2615,53 @@ fn main() {
}
}
Commands::Doctor => {
print!(
"{}",
llmfit_core::doctor::collect_diagnostics(env!("CARGO_PKG_VERSION"))
);
}
Commands::Claim {
model,
min_tps,
quant,
efficiency,
device_class,
template,
name,
} => {
let db = ModelDatabase::new();
let target = llmfit_core::claim::ClaimTarget {
min_tps,
efficiency_pct: efficiency,
device_class,
template,
quant,
name,
};
let rendered = resolve_model_selector(db.get_all_models(), &model).and_then(|m| {
if cli.json {
llmfit_core::claim::render_json(m, &target, env!("CARGO_PKG_VERSION"))
} else {
llmfit_core::claim::render(m, &target)
}
});
match rendered {
Ok(out) => {
if cli.json {
println!("{}", out)
} else {
print!("{}", out)
}
}
Err(err) => {
eprintln!("Error: {}", err);
std::process::exit(1);
}
}
}
Commands::List { sort } => {
let db = ModelDatabase::new();
if cli.json {
@@ -2599,7 +2696,30 @@ fn main() {
Commands::Search { query } => {
let db = ModelDatabase::new();
let results = db.find_model(&query);
display::display_search_results(&results, &query);
if results.is_empty() {
// Fallback: search HuggingFace directly for GGUF models
use llmfit_core::providers::LlamaCppProvider;
println!(
"\nNo local models found matching '{}'. Searching HuggingFace...\n",
query
);
let hf_results = LlamaCppProvider::search_hf_gguf(&query);
if hf_results.is_empty() {
println!("No models found matching '{}'.", query);
} else {
println!("{:<50} Type", "Repository");
println!("{}", "-".repeat(65));
for (id, desc) in hf_results.iter().take(20) {
println!("{:<50} {}", id, desc);
}
println!("\nTo download: llmfit download <repository>");
println!(
"Tip: run 'llmfit update' to add trending models to the local index."
);
}
} else {
display::display_search_results(&results, &query);
}
}
Commands::Info { model } => {
@@ -2728,6 +2848,7 @@ fn main() {
Commands::Serve {
host,
port,
unix_socket,
mcp,
send_events,
nats_url,
@@ -2753,7 +2874,13 @@ fn main() {
eprintln!("NATS events enabled, publishing to {}", nats_url);
}
if let Err(err) = serve_api::run_serve(&host, port, &overrides, context_limit) {
if let Err(err) = serve_api::run_serve(
&host,
port,
unix_socket.as_deref(),
&overrides,
context_limit,
) {
eprintln!("Error: {}", err);
std::process::exit(1);
}
@@ -2848,6 +2975,7 @@ mod tests {
release_date: Some("2025-01-01".to_string()),
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: llmfit_core::models::ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2882,6 +3010,7 @@ mod tests {
installed: false,
fits_with_turboquant: false,
effective_context_length: 8192,
usable_context: 8192,
}
}
@@ -2934,6 +3063,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: llmfit_core::models::ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
@@ -2965,6 +3095,7 @@ mod tests {
release_date: None,
gguf_sources: vec![],
capabilities: vec![],
languages: vec![],
format: llmfit_core::models::ModelFormat::default(),
num_attention_heads: None,
num_key_value_heads: None,
llmfit-tui/src/mcp_server.rs
+8 -8
View File
@@ -308,14 +308,14 @@ impl LlmfitMcpServer {
let mut models = Vec::new();
while let Some(result) = set.join_next().await {
if let Ok((name, available, installed)) = result {
if available {
for model_name in installed {
models.push(InstalledModel {
name: model_name,
runtime: name.to_string(),
});
}
if let Ok((name, available, installed)) = result
&& available
{
for model_name in installed {
models.push(InstalledModel {
name: model_name,
runtime: name.to_string(),
});
}
}
}
llmfit-tui/src/serve_api.rs
+72 -29
View File
@@ -141,14 +141,10 @@ type ApiResult<T> = Result<T, ApiError>;
pub fn run_serve(
host: &str,
port: u16,
unix_socket: Option<&std::path::Path>,
overrides: &super::HardwareOverrides,
context_limit: Option<u32>,
) -> Result<(), String> {
let ip: IpAddr = host
.parse()
.map_err(|_| format!("invalid --host value: '{host}'"))?;
let addr = SocketAddr::new(ip, port);
let specs = super::detect_specs(overrides);
let db = ModelDatabase::new();
let all_models = db.get_all_models().clone();
@@ -170,36 +166,83 @@ pub fn run_serve(
let app = build_router(state);
println!("llmfit dashboard listening on http://{}/", addr);
println!(" API models: http://{}/api/v1/models", addr);
println!(" GET /health");
println!(" GET /api/v1/system");
println!(" GET /api/v1/models?limit=20&min_fit=marginal&sort=score");
println!(" GET /api/v1/models/top?limit=5&use_case=coding&min_fit=good");
println!(" GET /api/v1/models/<name>");
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.map_err(|e| format!("failed to start tokio runtime: {e}"))?;
runtime
.block_on(async move {
let listener = tokio::net::TcpListener::bind(addr)
.await
.map_err(|e| ApiError::internal(format!("bind failed on {addr}: {e}")))?;
match unix_socket {
Some(path) => {
#[cfg(unix)]
{
println!("llmfit dashboard listening on unix://{}", path.display());
println!(" GET /health");
println!(" GET /api/v1/system");
println!(" GET /api/v1/models?limit=20&min_fit=marginal&sort=score");
runtime
.block_on(async move {
if let Some(parent) = path.parent() {
let _ = std::fs::create_dir_all(parent);
}
// A stale file from a previous instance blocks bind
// (the path can outlive the process, e.g. a pod
// volume across container restarts).
let _ = std::fs::remove_file(path);
let listener = tokio::net::UnixListener::bind(path).map_err(|e| {
ApiError::internal(format!("bind failed on {}: {e}", path.display()))
})?;
// Same-host peers only; group-accessible, not world.
use std::os::unix::fs::PermissionsExt;
let _ =
std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o660));
axum::serve(listener, app.into_make_service())
.with_graceful_shutdown(async {
let _ = tokio::signal::ctrl_c().await;
})
.await
.map_err(|e| ApiError::internal(format!("server error: {e}")))
})
.map_err(|e| e.message)
}
#[cfg(not(unix))]
{
let _ = path;
Err("--unix-socket is only supported on unix platforms".to_string())
}
}
None => {
let ip: IpAddr = host
.parse()
.map_err(|_| format!("invalid --host value: '{host}'"))?;
let addr = SocketAddr::new(ip, port);
axum::serve(
listener,
app.into_make_service_with_connect_info::<SocketAddr>(),
)
.with_graceful_shutdown(async {
let _ = tokio::signal::ctrl_c().await;
})
.await
.map_err(|e| ApiError::internal(format!("server error: {e}")))
})
.map_err(|e| e.message)
println!("llmfit dashboard listening on http://{}/", addr);
println!(" API models: http://{}/api/v1/models", addr);
println!(" GET /health");
println!(" GET /api/v1/system");
println!(" GET /api/v1/models?limit=20&min_fit=marginal&sort=score");
println!(" GET /api/v1/models/top?limit=5&use_case=coding&min_fit=good");
println!(" GET /api/v1/models/<name>");
runtime
.block_on(async move {
let listener = tokio::net::TcpListener::bind(addr)
.await
.map_err(|e| ApiError::internal(format!("bind failed on {addr}: {e}")))?;
axum::serve(
listener,
app.into_make_service_with_connect_info::<SocketAddr>(),
)
.with_graceful_shutdown(async {
let _ = tokio::signal::ctrl_c().await;
})
.await
.map_err(|e| ApiError::internal(format!("server error: {e}")))
})
.map_err(|e| e.message)
}
}
}
fn build_router(state: Arc<AppState>) -> Router {
llmfit-tui/src/serve_shared.rs
+1
View File
@@ -94,6 +94,7 @@ pub fn runtime_code(runtime: InferenceRuntime) -> &'static str {
InferenceRuntime::Mlx => "mlx",
InferenceRuntime::LlamaCpp => "llamacpp",
InferenceRuntime::Vllm => "vllm",
InferenceRuntime::Unsupported => "unsupported",
}
}
llmfit-tui/src/tui_app.rs
+539 -168
View File
@@ -4,20 +4,73 @@ use llmfit_core::models::{Capability, ModelDatabase, UseCase};
use llmfit_core::plan::{PlanEstimate, PlanRequest, estimate_model_plan};
use llmfit_core::providers::{
self, DockerModelRunnerProvider, LlamaCppProvider, LmStudioProvider, MlxProvider,
ModelProvider, OllamaProvider, PullEvent, PullHandle, VllmProvider,
ModelProvider, OllamaProvider, PullEvent, PullHandle, VllmProvider, command_exists,
};
use llmfit_core::quality;
use std::collections::{HashMap, HashSet};
use std::sync::mpsc;
use std::thread;
use std::{cmp, thread};
use ratatui::widgets::TableState;
use unicode_segmentation::UnicodeSegmentation;
use crate::download_history::{DownloadHistory, DownloadRecord, DownloadResult};
use crate::filter_config::FilterConfig;
use crate::theme::Theme;
fn floor_char_boundary(value: &str, index: usize) -> usize {
let mut index = index.min(value.len());
while index > 0 && !value.is_char_boundary(index) {
index -= 1;
}
index
}
fn previous_grapheme_boundary(value: &str, index: usize) -> usize {
let index = floor_char_boundary(value, index);
value[..index]
.grapheme_indices(true)
.next_back()
.map(|(idx, _)| idx)
.unwrap_or(0)
}
fn next_grapheme_boundary(value: &str, index: usize) -> usize {
let index = floor_char_boundary(value, index);
value[index..]
.grapheme_indices(true)
.nth(1)
.map(|(idx, _)| index + idx)
.unwrap_or_else(|| value.len())
}
fn insert_ascii_graphic_input(value: &mut String, cursor: &mut usize, c: char) {
if !c.is_ascii_graphic() {
return;
}
*cursor = floor_char_boundary(value, *cursor);
value.insert(*cursor, c);
*cursor += c.len_utf8();
}
fn backspace_grapheme_input(value: &mut String, cursor: &mut usize) {
if *cursor > 0 {
let prev = previous_grapheme_boundary(value, *cursor);
value.drain(prev..*cursor);
*cursor = prev;
}
}
fn delete_grapheme_input(value: &mut String, cursor: usize) {
if cursor < value.len() {
let cursor = floor_char_boundary(value, cursor);
let next = next_grapheme_boundary(value, cursor);
value.drain(cursor..next);
}
}
/// Messages sent from background provider-detection threads back to the main TUI.
pub enum ProviderDetectionMsg {
Ollama {
@@ -471,6 +524,26 @@ fn sort_column_from_label(s: &str) -> SortColumn {
}
}
/// Case-insensitive subsequence ("fuzzy") match: returns true when every
/// character of `query` appears in `candidate` in order (not necessarily
/// contiguous). An empty query matches everything.
fn fuzzy_match(query: &str, candidate: &str) -> bool {
if query.is_empty() {
return true;
}
let mut q = query.chars().flat_map(char::to_lowercase).peekable();
for c in candidate.chars().flat_map(char::to_lowercase) {
if let Some(&qc) = q.peek() {
if c == qc {
q.next();
}
} else {
break;
}
}
q.peek().is_none()
}
pub struct App {
pub should_quit: bool,
pub input_mode: InputMode,
@@ -520,6 +593,8 @@ pub struct App {
// Provider popup
pub provider_cursor: usize,
pub provider_search: String,
pub provider_search_cursor_position: usize,
pub use_case_cursor: usize,
pub capability_cursor: usize,
pub download_provider_cursor: usize,
@@ -529,28 +604,18 @@ pub struct App {
// Provider state
pub ollama_available: bool,
pub ollama_binary_available: bool,
pub ollama_installed: HashSet<String>,
pub ollama_installed_count: usize,
pub installed: llmfit_core::analysis::InstalledIndex,
ollama: OllamaProvider,
pub mlx_available: bool,
pub mlx_installed: HashSet<String>,
mlx: MlxProvider,
pub llamacpp_available: bool,
pub llamacpp_installed: HashSet<String>,
pub llamacpp_installed_count: usize,
pub llamacpp_detection_hint: String,
llamacpp: LlamaCppProvider,
pub docker_mr_available: bool,
pub docker_mr_installed: HashSet<String>,
pub docker_mr_installed_count: usize,
docker_mr: DockerModelRunnerProvider,
pub lmstudio_available: bool,
pub lmstudio_installed: HashSet<String>,
pub lmstudio_installed_count: usize,
lmstudio: LmStudioProvider,
pub vllm_available: bool,
pub vllm_installed: HashSet<String>,
pub vllm_installed_count: usize,
vllm: VllmProvider,
// Download state
@@ -715,23 +780,17 @@ impl App {
let ollama = OllamaProvider::new();
let ollama_available = false;
let ollama_binary_available = false;
let ollama_installed = HashSet::new();
let ollama_installed_count = 0;
let mlx = MlxProvider::new();
let mlx_available = false;
let mlx_installed = HashSet::new();
let docker_mr = DockerModelRunnerProvider::new();
let docker_mr_available = false;
let docker_mr_installed = HashSet::new();
let docker_mr_installed_count = 0;
let lmstudio = LmStudioProvider::new();
let lmstudio_available = false;
let lmstudio_installed = HashSet::new();
let lmstudio_installed_count = 0;
let vllm = VllmProvider::new();
let vllm_available = false;
let vllm_installed = HashSet::new();
let vllm_installed_count = 0;
let mut installed = llmfit_core::analysis::InstalledIndex::empty();
installed.llamacpp = llamacpp_installed;
installed.llamacpp_count = llamacpp_installed_count;
// Spawn background provider detection for network-based providers
let (provider_tx, provider_detection_rx) = mpsc::channel();
@@ -812,12 +871,7 @@ impl App {
.filter(|m| backend_compatible(m, &specs))
.map(|m| {
let mut fit = ModelFit::analyze_with_context_limit(m, &specs, context_limit);
fit.installed = providers::is_model_installed(&m.name, &ollama_installed)
|| providers::is_model_installed_mlx(&m.name, &mlx_installed)
|| providers::is_model_installed_llamacpp(&m.name, &llamacpp_installed)
|| providers::is_model_installed_docker_mr(&m.name, &docker_mr_installed)
|| providers::is_model_installed_lmstudio(&m.name, &lmstudio_installed)
|| providers::is_model_installed_vllm(&m.name, &vllm_installed);
fit.installed = installed.is_installed(&m.name);
fit
})
.collect();
@@ -1018,6 +1072,8 @@ impl App {
plan_estimate: None,
plan_error: None,
provider_cursor: 0,
provider_search: String::new(),
provider_search_cursor_position: 0,
use_case_cursor: 0,
capability_cursor: 0,
download_provider_cursor: 0,
@@ -1025,28 +1081,18 @@ impl App {
download_provider_model: None,
ollama_available,
ollama_binary_available,
ollama_installed,
ollama_installed_count,
installed,
ollama,
mlx_available,
mlx_installed,
mlx,
llamacpp_available,
llamacpp_installed,
llamacpp_installed_count,
llamacpp_detection_hint,
llamacpp,
docker_mr_available,
docker_mr_installed,
docker_mr_installed_count,
docker_mr,
lmstudio_available,
lmstudio_installed,
lmstudio_installed_count,
lmstudio,
vllm_available,
vllm_installed,
vllm_installed_count,
vllm,
pull_active: None,
pull_status: None,
@@ -1665,28 +1711,49 @@ impl App {
pub fn search_input(&mut self, c: char) {
self.search_query.insert(self.cursor_position, c);
self.cursor_position += 1;
self.cursor_position += c.len_utf8();
// Changing the query should snap the list back to the top so all
// matches are visible regardless of the prior cursor position.
self.selected_row = 0;
self.apply_filters();
}
pub fn search_backspace(&mut self) {
if self.cursor_position > 0 {
self.cursor_position -= 1;
self.search_query.remove(self.cursor_position);
let prev = previous_grapheme_boundary(&self.search_query, self.cursor_position);
self.search_query.drain(prev..self.cursor_position);
self.cursor_position = prev;
self.selected_row = 0;
self.apply_filters();
}
}
pub fn search_delete(&mut self) {
if self.cursor_position < self.search_query.len() {
self.search_query.remove(self.cursor_position);
let next = next_grapheme_boundary(&self.search_query, self.cursor_position);
self.search_query.drain(self.cursor_position..next);
self.selected_row = 0;
self.apply_filters();
}
}
pub fn search_cursor_left(&mut self) {
if self.cursor_position > 0 {
self.cursor_position =
previous_grapheme_boundary(&self.search_query, self.cursor_position);
}
}
pub fn search_cursor_right(&mut self) {
if self.cursor_position < self.search_query.len() {
self.cursor_position = next_grapheme_boundary(&self.search_query, self.cursor_position);
}
}
pub fn clear_search(&mut self) {
self.search_query.clear();
self.cursor_position = 0;
self.selected_row = 0;
self.apply_filters();
}
@@ -1724,6 +1791,47 @@ impl App {
self.dm_editing_dir = true;
}
pub fn insert_dm_dir_char(&mut self, c: char) {
self.dm_dir_cursor = floor_char_boundary(&self.dm_dir_input, self.dm_dir_cursor);
self.dm_dir_input.insert(self.dm_dir_cursor, c);
self.dm_dir_cursor += c.len_utf8();
}
pub fn dm_dir_backspace(&mut self) {
self.dm_dir_cursor = floor_char_boundary(&self.dm_dir_input, self.dm_dir_cursor);
if self.dm_dir_cursor > 0 {
let prev = previous_grapheme_boundary(&self.dm_dir_input, self.dm_dir_cursor);
self.dm_dir_input.drain(prev..self.dm_dir_cursor);
self.dm_dir_cursor = prev;
}
}
pub fn dm_dir_delete(&mut self) {
self.dm_dir_cursor = floor_char_boundary(&self.dm_dir_input, self.dm_dir_cursor);
if self.dm_dir_cursor < self.dm_dir_input.len() {
let next = next_grapheme_boundary(&self.dm_dir_input, self.dm_dir_cursor);
self.dm_dir_input.drain(self.dm_dir_cursor..next);
}
}
pub fn dm_dir_cursor_left(&mut self) {
if self.dm_dir_cursor > 0 {
self.dm_dir_cursor = previous_grapheme_boundary(&self.dm_dir_input, self.dm_dir_cursor);
}
}
pub fn dm_dir_cursor_right(&mut self) {
self.dm_dir_cursor = floor_char_boundary(&self.dm_dir_input, self.dm_dir_cursor);
if self.dm_dir_cursor < self.dm_dir_input.len() {
self.dm_dir_cursor = next_grapheme_boundary(&self.dm_dir_input, self.dm_dir_cursor);
}
}
pub fn dm_dir_clear(&mut self) {
self.dm_dir_input.clear();
self.dm_dir_cursor = 0;
}
pub fn apply_download_dir(&mut self) {
let path = std::path::PathBuf::from(&self.dm_dir_input);
if let Err(e) = std::fs::create_dir_all(&path) {
@@ -1976,16 +2084,13 @@ impl App {
// Try each preset and see if the GPU name matches
for preset in llmfit_core::benchmarks::HardwarePreset::all() {
if let Some(hw_name) = preset.hardware_name {
if lower.contains(&hw_name.to_lowercase()) {
if let Some(cached) =
llmfit_core::benchmarks::cached_leaderboard_for_preset(preset.label)
{
if !cached.rows.is_empty() {
return Some(cached);
}
}
}
if let Some(hw_name) = preset.hardware_name
&& lower.contains(&hw_name.to_lowercase())
&& let Some(cached) =
llmfit_core::benchmarks::cached_leaderboard_for_preset(preset.label)
&& !cached.rows.is_empty()
{
return Some(cached);
}
}
None
@@ -2228,11 +2333,90 @@ impl App {
pub fn open_provider_popup(&mut self) {
self.input_mode = InputMode::ProviderPopup;
// Don't reset cursor -- keep it where it was last time
self.provider_search.clear();
self.provider_search_cursor_position = 0;
self.provider_cursor = 0;
}
pub fn close_provider_popup(&mut self) {
self.input_mode = InputMode::Normal;
self.provider_search.clear();
self.provider_search_cursor_position = 0;
}
/// Indices into `self.providers` that match the current fuzzy search query,
/// in display order. With an empty query this is every provider.
pub fn provider_filtered_indices(&self) -> Vec<usize> {
self.providers
.iter()
.enumerate()
.filter(|(_, name)| fuzzy_match(&self.provider_search, name))
.map(|(i, _)| i)
.collect()
}
pub fn provider_search_input(&mut self, c: char) {
insert_ascii_graphic_input(
&mut self.provider_search,
&mut self.provider_search_cursor_position,
c,
);
self.clamp_provider_cursor();
}
pub fn provider_search_backspace(&mut self) {
backspace_grapheme_input(
&mut self.provider_search,
&mut self.provider_search_cursor_position,
);
self.clamp_provider_cursor();
}
pub fn provider_search_delete(&mut self) {
delete_grapheme_input(
&mut self.provider_search,
self.provider_search_cursor_position,
);
self.clamp_provider_cursor();
}
pub fn provider_search_cursor_left(&mut self) {
if self.provider_search_cursor_position > 0 {
self.provider_search_cursor_position = previous_grapheme_boundary(
&self.provider_search,
self.provider_search_cursor_position,
);
}
}
pub fn provider_search_cursor_right(&mut self) {
if self.provider_search_cursor_position < self.provider_search.len() {
self.provider_search_cursor_position =
next_grapheme_boundary(&self.provider_search, self.provider_search_cursor_position);
}
}
pub fn provider_search_cursor_home(&mut self) {
self.provider_search_cursor_position = 0;
}
pub fn provider_search_cursor_end(&mut self) {
self.provider_search_cursor_position = self.provider_search.len();
}
pub fn provider_search_clear(&mut self) {
self.provider_search.clear();
self.provider_search_cursor_position = 0;
self.clamp_provider_cursor();
}
fn clamp_provider_cursor(&mut self) {
let len = self.provider_filtered_indices().len();
if len == 0 {
self.provider_cursor = 0;
} else if self.provider_cursor >= len {
self.provider_cursor = len - 1;
}
}
pub fn open_use_case_popup(&mut self) {
@@ -2244,38 +2428,45 @@ impl App {
self.input_mode = InputMode::Normal;
}
pub fn provider_popup_up(&mut self) {
pub fn provider_popup_up(&mut self, step: usize) {
if self.provider_cursor > 0 {
self.provider_cursor -= 1;
self.provider_cursor = self.provider_cursor.saturating_sub(step);
}
}
pub fn provider_popup_down(&mut self) {
if self.provider_cursor + 1 < self.providers.len() {
self.provider_cursor += 1;
pub fn provider_popup_down(&mut self, step: usize) {
let len = self.provider_filtered_indices().len();
if self.provider_cursor + 1 < len {
self.provider_cursor = cmp::min(len - 1, self.provider_cursor + step);
}
}
pub fn provider_popup_toggle(&mut self) {
if self.provider_cursor < self.selected_providers.len() {
self.selected_providers[self.provider_cursor] =
!self.selected_providers[self.provider_cursor];
let filtered = self.provider_filtered_indices();
if let Some(&idx) = filtered.get(self.provider_cursor) {
self.selected_providers[idx] = !self.selected_providers[idx];
self.apply_filters();
}
}
/// Toggle all currently-visible (matching) providers. If they are all
/// selected, deselect them; otherwise select them all.
pub fn provider_popup_select_all(&mut self) {
let all_selected = self.selected_providers.iter().all(|&s| s);
let filtered = self.provider_filtered_indices();
if filtered.is_empty() {
return;
}
let all_selected = filtered.iter().all(|&i| self.selected_providers[i]);
let new_val = !all_selected;
for s in &mut self.selected_providers {
*s = new_val;
for &i in &filtered {
self.selected_providers[i] = new_val;
}
self.apply_filters();
}
pub fn provider_popup_clear_all(&mut self) {
for s in &mut self.selected_providers {
*s = false;
for &i in &self.provider_filtered_indices() {
self.selected_providers[i] = false;
}
self.apply_filters();
}
@@ -2735,11 +2926,7 @@ impl App {
.map(|m| {
let mut fit =
ModelFit::analyze_with_context_limit(m, &self.specs, self.context_limit);
fit.installed = providers::is_model_installed(&m.name, &self.ollama_installed)
|| providers::is_model_installed_mlx(&m.name, &self.mlx_installed)
|| providers::is_model_installed_llamacpp(&m.name, &self.llamacpp_installed)
|| providers::is_model_installed_docker_mr(&m.name, &self.docker_mr_installed)
|| providers::is_model_installed_lmstudio(&m.name, &self.lmstudio_installed);
fit.installed = self.installed.is_installed(&m.name);
fit
})
.collect();
@@ -3159,11 +3346,7 @@ impl App {
.map(|m| {
let mut fit =
ModelFit::analyze_with_config(m, &self.specs, self.calc_config.clone());
fit.installed = providers::is_model_installed(&m.name, &self.ollama_installed)
|| providers::is_model_installed_mlx(&m.name, &self.mlx_installed)
|| providers::is_model_installed_llamacpp(&m.name, &self.llamacpp_installed)
|| providers::is_model_installed_docker_mr(&m.name, &self.docker_mr_installed)
|| providers::is_model_installed_lmstudio(&m.name, &self.lmstudio_installed);
fit.installed = self.installed.is_installed(&m.name);
fit
})
.collect();
@@ -3572,38 +3755,27 @@ impl App {
/// Re-query all providers for installed models and update all_fits.
pub fn refresh_installed(&mut self) {
let (ollama_set, ollama_count) = self.ollama.installed_models_counted();
self.ollama_installed = ollama_set;
self.ollama_installed_count = ollama_count;
self.mlx_installed = self.mlx.installed_models();
let (llamacpp_set, llamacpp_count) = self.llamacpp.installed_models_counted();
self.llamacpp_installed = llamacpp_set;
self.llamacpp_installed_count = llamacpp_count;
let (docker_mr_set, docker_mr_count) = self.docker_mr.installed_models_counted();
self.docker_mr_installed = docker_mr_set;
self.docker_mr_installed_count = docker_mr_count;
let (lmstudio_set, lmstudio_count) = self.lmstudio.installed_models_counted();
self.lmstudio_installed = lmstudio_set;
self.lmstudio_installed_count = lmstudio_count;
let (vllm_set, vllm_count) = self.vllm.installed_models_counted();
self.vllm_installed = vllm_set;
self.vllm_installed_count = vllm_count;
let (ollama, ollama_count) = self.ollama.installed_models_counted();
let mlx = self.mlx.installed_models();
let (llamacpp, llamacpp_count) = self.llamacpp.installed_models_counted();
let (docker_mr, docker_mr_count) = self.docker_mr.installed_models_counted();
let (lmstudio, lmstudio_count) = self.lmstudio.installed_models_counted();
let (vllm, vllm_count) = self.vllm.installed_models_counted();
self.installed = llmfit_core::analysis::InstalledIndex {
ollama,
ollama_count,
mlx,
llamacpp,
llamacpp_count,
docker_mr,
docker_mr_count,
lmstudio,
lmstudio_count,
vllm,
vllm_count,
};
for fit in &mut self.all_fits {
fit.installed = providers::is_model_installed(&fit.model.name, &self.ollama_installed)
|| providers::is_model_installed_mlx(&fit.model.name, &self.mlx_installed)
|| providers::is_model_installed_llamacpp(
&fit.model.name,
&self.llamacpp_installed,
)
|| providers::is_model_installed_docker_mr(
&fit.model.name,
&self.docker_mr_installed,
)
|| providers::is_model_installed_lmstudio(
&fit.model.name,
&self.lmstudio_installed,
)
|| providers::is_model_installed_vllm(&fit.model.name, &self.vllm_installed);
fit.installed = self.installed.is_installed(&fit.model.name);
}
self.re_sort();
self.enqueue_capability_probes_for_visible(24);
@@ -3708,8 +3880,8 @@ impl App {
} => {
self.ollama_available = available;
self.ollama_binary_available = binary_available;
self.ollama_installed = installed;
self.ollama_installed_count = installed_count;
self.installed.ollama = installed;
self.installed.ollama_count = installed_count;
self.ollama = provider;
}
ProviderDetectionMsg::Mlx {
@@ -3717,7 +3889,7 @@ impl App {
installed,
} => {
self.mlx_available = available;
self.mlx_installed = installed;
self.installed.mlx = installed;
}
ProviderDetectionMsg::DockerMr {
available,
@@ -3725,8 +3897,8 @@ impl App {
installed_count,
} => {
self.docker_mr_available = available;
self.docker_mr_installed = installed;
self.docker_mr_installed_count = installed_count;
self.installed.docker_mr = installed;
self.installed.docker_mr_count = installed_count;
}
ProviderDetectionMsg::LmStudio {
available,
@@ -3734,8 +3906,8 @@ impl App {
installed_count,
} => {
self.lmstudio_available = available;
self.lmstudio_installed = installed;
self.lmstudio_installed_count = installed_count;
self.installed.lmstudio = installed;
self.installed.lmstudio_count = installed_count;
}
ProviderDetectionMsg::Vllm {
available,
@@ -3743,8 +3915,8 @@ impl App {
installed_count,
} => {
self.vllm_available = available;
self.vllm_installed = installed;
self.vllm_installed_count = installed_count;
self.installed.vllm = installed;
self.installed.vllm_count = installed_count;
}
}
}
@@ -3758,25 +3930,7 @@ impl App {
if got_any {
// Re-mark installed status for all models
for fit in &mut self.all_fits {
fit.installed =
providers::is_model_installed(&fit.model.name, &self.ollama_installed)
|| providers::is_model_installed_mlx(&fit.model.name, &self.mlx_installed)
|| providers::is_model_installed_llamacpp(
&fit.model.name,
&self.llamacpp_installed,
)
|| providers::is_model_installed_docker_mr(
&fit.model.name,
&self.docker_mr_installed,
)
|| providers::is_model_installed_lmstudio(
&fit.model.name,
&self.lmstudio_installed,
)
|| providers::is_model_installed_vllm(
&fit.model.name,
&self.vllm_installed,
);
fit.installed = self.installed.is_installed(&fit.model.name);
}
self.re_sort();
}
@@ -3809,10 +3963,8 @@ impl App {
self.live_bench_scroll = 0;
self.bench_view_mode = BenchViewMode::Results;
if !self.bench_running && self.ollama_available {
if !self.load_bench_cache() {
self.start_bench();
}
if !self.bench_running && self.ollama_available && !self.load_bench_cache() {
self.start_bench();
}
}
@@ -3839,7 +3991,8 @@ impl App {
let _ = std::fs::create_dir_all(parent);
}
let mut installed: Vec<String> = self
.ollama_installed
.installed
.ollama
.iter()
.map(|m| m.strip_suffix(":latest").unwrap_or(m).to_string())
.collect();
@@ -3870,7 +4023,8 @@ impl App {
// Check if installed models match
let mut installed: Vec<String> = self
.ollama_installed
.installed
.ollama
.iter()
.map(|m| m.strip_suffix(":latest").unwrap_or(m).to_string())
.collect();
@@ -3886,25 +4040,22 @@ impl App {
return false;
}
if let Some(results) = cache.get("results") {
if let Ok(r) =
if let Some(results) = cache.get("results")
&& let Ok(r) =
serde_json::from_value::<Vec<quality::ModelQualityResult>>(results.clone())
{
self.bench_results = r;
}
{
self.bench_results = r;
}
if let Some(routing) = cache.get("routing") {
if let Ok(r) =
if let Some(routing) = cache.get("routing")
&& let Ok(r) =
serde_json::from_value::<Vec<quality::RoutingRecommendation>>(routing.clone())
{
self.bench_routing = r;
}
{
self.bench_routing = r;
}
if let Some(ru) = cache.get("runner_ups") {
if let Ok(r) = serde_json::from_value::<Vec<quality::RoutingRecommendation>>(ru.clone())
{
self.bench_runner_ups = r;
}
if let Some(ru) = cache.get("runner_ups")
&& let Ok(r) = serde_json::from_value::<Vec<quality::RoutingRecommendation>>(ru.clone())
{
self.bench_runner_ups = r;
}
// Rebuild model status from cached results
@@ -3963,7 +4114,8 @@ impl App {
// Deduplicate: strip ":latest" suffix and remove dupes
let mut seen = std::collections::HashSet::new();
let mut models: Vec<String> = self
.ollama_installed
.installed
.ollama
.iter()
.map(|m| m.strip_suffix(":latest").unwrap_or(m).to_string())
.filter(|m| seen.insert(m.clone()))
@@ -4164,22 +4316,35 @@ impl App {
}
}
fn command_exists(name: &str) -> bool {
std::process::Command::new("which")
.arg(name)
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false)
}
#[cfg(test)]
mod tests {
use super::*;
use llmfit_core::fit::{InferenceRuntime, RunMode, ScoreComponents};
use llmfit_core::hardware::GpuBackend;
use llmfit_core::models::{LlmModel, ModelFormat, UseCase};
fn test_app() -> App {
App::with_specs_and_context(
SystemSpecs {
total_ram_gb: 16.0,
available_ram_gb: 12.0,
total_cpu_cores: 8,
cpu_name: "Test CPU".to_string(),
has_gpu: false,
gpu_vram_gb: None,
total_gpu_vram_gb: None,
gpu_name: None,
gpu_count: 0,
unified_memory: false,
backend: GpuBackend::CpuX86,
gpus: Vec::new(),
cluster_mode: false,
cluster_node_count: 0,
},
None,
)
}
fn test_model(name: &str) -> LlmModel {
LlmModel {
name: name.to_string(),
@@ -4199,6 +4364,7 @@ mod tests {
release_date: None,
gguf_sources: Vec::new(),
capabilities: Vec::new(),
languages: Vec::new(),
format: ModelFormat::Gguf,
num_attention_heads: None,
num_key_value_heads: None,
@@ -4238,6 +4404,7 @@ mod tests {
installed: false,
fits_with_turboquant: false,
effective_context_length: 8192,
usable_context: 8192,
}
}
@@ -4263,4 +4430,208 @@ mod tests {
assert_eq!(App::initial_best_fit_row(&[0, 1], &fits), 0);
assert_eq!(App::initial_best_fit_row(&[], &fits), 0);
}
#[test]
fn fuzzy_match_empty_query_matches_everything() {
assert!(fuzzy_match("", "Ollama"));
assert!(fuzzy_match("", ""));
}
#[test]
fn fuzzy_match_is_case_insensitive_subsequence() {
// Contiguous substring
assert!(fuzzy_match("oll", "Ollama"));
// Non-contiguous subsequence, mixed case
assert!(fuzzy_match("opn", "OpenAI"));
assert!(fuzzy_match("MLX", "mlx"));
// Order matters
assert!(!fuzzy_match("alo", "Ollama"));
// Char not present
assert!(!fuzzy_match("ollamax", "Ollama"));
assert!(!fuzzy_match("z", "Anthropic"));
}
#[test]
fn ascii_graphic_input_inserts_at_cursor_and_rejects_non_ascii() {
let mut value = "Ollma".to_string();
let mut cursor = "Oll".len();
insert_ascii_graphic_input(&mut value, &mut cursor, 'a');
assert_eq!(value, "Ollama");
assert_eq!(cursor, "Olla".len());
insert_ascii_graphic_input(&mut value, &mut cursor, '你');
insert_ascii_graphic_input(&mut value, &mut cursor, ' ');
assert_eq!(value, "Ollama");
assert_eq!(cursor, "Olla".len());
}
#[test]
fn grapheme_input_backspace_and_delete_edit_at_cursor() {
let mut value = "LLaamCpp".to_string();
let mut cursor = "LLa".len();
backspace_grapheme_input(&mut value, &mut cursor);
assert_eq!(value, "LLamCpp");
assert_eq!(cursor, "LL".len());
delete_grapheme_input(&mut value, cursor);
assert_eq!(value, "LLmCpp");
assert_eq!(cursor, "LL".len());
}
#[test]
fn search_grapheme_boundaries_keep_emoji_sequences_together() {
let query = "a👩‍💻b";
let after_a = "a".len();
let after_emoji = after_a + "👩‍💻".len();
assert_eq!(next_grapheme_boundary(query, after_a), after_emoji);
assert_eq!(previous_grapheme_boundary(query, after_emoji), after_a);
}
#[test]
fn search_grapheme_boundaries_handle_multibyte_text() {
let query = "a你好";
let after_a = "a".len();
let after_ni = after_a + "你".len();
assert_eq!(next_grapheme_boundary(query, after_a), after_ni);
assert_eq!(previous_grapheme_boundary(query, after_ni), after_a);
}
#[test]
fn download_dir_input_handles_multibyte_text_without_invalid_boundaries() {
let mut app = test_app();
app.insert_dm_dir_char('模');
app.insert_dm_dir_char('型');
app.insert_dm_dir_char('一');
assert_eq!(app.dm_dir_input, "模型一");
assert_eq!(app.dm_dir_cursor, "模型一".len());
app.dm_dir_cursor_left();
assert_eq!(app.dm_dir_cursor, "模型".len());
app.insert_dm_dir_char('二');
assert_eq!(app.dm_dir_input, "模型二一");
assert_eq!(app.dm_dir_cursor, "模型二".len());
app.dm_dir_backspace();
assert_eq!(app.dm_dir_input, "模型一");
assert_eq!(app.dm_dir_cursor, "模型".len());
}
#[test]
fn download_dir_input_deletes_whole_emoji_graphemes() {
let mut app = test_app();
app.dm_dir_input = "a👩‍💻b".to_string();
app.dm_dir_cursor = "a".len();
app.dm_dir_delete();
assert_eq!(app.dm_dir_input, "ab");
assert_eq!(app.dm_dir_cursor, "a".len());
app.insert_dm_dir_char('🚀');
assert_eq!(app.dm_dir_input, "a🚀b");
assert_eq!(app.dm_dir_cursor, "a🚀".len());
app.dm_dir_backspace();
assert_eq!(app.dm_dir_input, "ab");
assert_eq!(app.dm_dir_cursor, "a".len());
}
#[test]
fn download_dir_input_repairs_non_boundary_cursor_before_editing() {
let mut app = test_app();
app.dm_dir_input = "一a".to_string();
app.dm_dir_cursor = 1;
app.insert_dm_dir_char('二');
assert_eq!(app.dm_dir_input, "二一a");
assert_eq!(app.dm_dir_cursor, "二".len());
}
#[test]
fn download_dir_backspace_repairs_non_boundary_cursor_before_editing() {
let mut app = test_app();
app.dm_dir_input = "一a".to_string();
app.dm_dir_cursor = 2;
app.dm_dir_backspace();
assert_eq!(app.dm_dir_input, "一a");
assert_eq!(app.dm_dir_cursor, 0);
}
// Reset every filter that `with_specs_and_context` may restore from a
// persisted filters.json so search tests are deterministic regardless of
// the developer's saved llmfit state.
fn clear_persisted_filters(app: &mut App) {
app.search_query.clear();
app.cursor_position = 0;
app.fit_filter = FitFilter::All;
app.availability_filter = AvailabilityFilter::All;
app.tp_filter = TpFilter::All;
app.filter_params_min_input.clear();
app.filter_params_max_input.clear();
app.filter_mem_pct_min_input.clear();
app.filter_mem_pct_max_input.clear();
}
#[test]
fn changing_search_query_resets_selection_to_top() {
let mut app = test_app();
clear_persisted_filters(&mut app);
app.all_fits = vec![
test_fit("gemma-2b", FitLevel::Good, 90.0),
test_fit("gemma-7b", FitLevel::Good, 80.0),
test_fit("llama-7b", FitLevel::Good, 70.0),
];
app.providers = vec!["Test".to_string()];
app.selected_providers = vec![true];
app.apply_filters();
assert_eq!(app.filtered_fits.len(), 3);
// Cursor parked deep in the full list, mimicking the user having
// navigated far down before searching.
app.selected_row = app.filtered_fits.len() - 1;
// Typing a query must snap the viewport back to the top so every
// match is visible (issue #657).
app.search_input('g');
assert!(!app.filtered_fits.is_empty());
assert_eq!(app.selected_row, 0);
// Clearing the search also resets to the top.
app.selected_row = app.filtered_fits.len() - 1;
app.clear_search();
assert_eq!(app.filtered_fits.len(), 3);
assert_eq!(app.selected_row, 0);
// Backspacing the query resets to the top too.
app.search_input('g');
app.selected_row = app.filtered_fits.len() - 1;
app.search_backspace();
assert!(!app.filtered_fits.is_empty());
assert_eq!(app.selected_row, 0);
}
#[test]
fn search_query_with_no_matches_keeps_selection_at_zero() {
let mut app = test_app();
clear_persisted_filters(&mut app);
app.all_fits = vec![test_fit("gemma-2b", FitLevel::Good, 90.0)];
app.providers = vec!["Test".to_string()];
app.selected_providers = vec![true];
app.apply_filters();
// A query that matches nothing must not panic on the clamp path and
// leaves the selection at the top.
app.search_input('z');
app.search_input('z');
app.search_input('z');
assert!(app.filtered_fits.is_empty());
assert_eq!(app.selected_row, 0);
}
}
llmfit-tui/src/tui_events.rs
+104 -21
View File
@@ -288,7 +288,10 @@ fn handle_search_mode(app: &mut App, key: KeyEvent) {
app.clear_search();
}
KeyCode::Char(c) => app.search_input(c),
KeyCode::Left => app.search_cursor_left(),
KeyCode::Right => app.search_cursor_right(),
KeyCode::Char(c) if allows_search_text_input(key.modifiers) => app.search_input(c),
// Allow navigation while searching
KeyCode::Up => app.move_up(),
@@ -299,21 +302,63 @@ fn handle_search_mode(app: &mut App, key: KeyEvent) {
}
fn handle_provider_popup_mode(app: &mut App, key: KeyEvent) {
let ctrl = key.modifiers.contains(KeyModifiers::CONTROL);
let shift = key.modifiers.contains(KeyModifiers::SHIFT);
match key.code {
KeyCode::Esc | KeyCode::Char('P') | KeyCode::Char('q') => app.close_provider_popup(),
KeyCode::Esc => app.close_provider_popup(),
KeyCode::Up | KeyCode::Char('k') => app.provider_popup_up(),
KeyCode::Down | KeyCode::Char('j') => app.provider_popup_down(),
KeyCode::Up if shift => app.provider_popup_up(25),
KeyCode::Down if shift => app.provider_popup_down(25),
KeyCode::Up => app.provider_popup_up(1),
KeyCode::Down => app.provider_popup_down(1),
KeyCode::Char(' ') | KeyCode::Enter => app.provider_popup_toggle(),
KeyCode::Left => app.provider_search_cursor_left(),
KeyCode::Right => app.provider_search_cursor_right(),
KeyCode::Home => app.provider_search_cursor_home(),
KeyCode::End => app.provider_search_cursor_end(),
KeyCode::Char('a') => app.provider_popup_select_all(),
KeyCode::Char('c') => app.provider_popup_clear_all(),
// Space toggles too (provider names never contain spaces).
KeyCode::Enter | KeyCode::Char(' ') => app.provider_popup_toggle(),
KeyCode::Backspace => app.provider_search_backspace(),
KeyCode::Delete => app.provider_search_delete(),
// Ctrl shortcuts (typing plain letters filters, so these are modified).
KeyCode::Char('u') if ctrl => app.provider_search_clear(),
KeyCode::Char('a') if ctrl => app.provider_popup_select_all(),
KeyCode::Char('n') if ctrl => app.provider_popup_clear_all(),
// Plain printable ASCII filters the provider list. Reject modified
// character events such as macOS Option/Command-arrow artifacts.
KeyCode::Char(c) if is_plain_provider_filter_char(c, key.modifiers) => {
app.provider_search_input(c)
}
_ => {}
}
}
fn allows_search_text_input(modifiers: KeyModifiers) -> bool {
!modifiers.intersects(
KeyModifiers::CONTROL
| KeyModifiers::ALT
| KeyModifiers::SUPER
| KeyModifiers::HYPER
| KeyModifiers::META,
)
}
fn is_plain_provider_filter_char(c: char, modifiers: KeyModifiers) -> bool {
c.is_ascii_graphic()
&& !modifiers.intersects(
KeyModifiers::CONTROL
| KeyModifiers::ALT
| KeyModifiers::SUPER
| KeyModifiers::HYPER
| KeyModifiers::META,
)
}
fn handle_plan_mode(app: &mut App, key: KeyEvent) {
match key.code {
KeyCode::Esc | KeyCode::Char('q') => app.close_plan_mode(),
@@ -554,28 +599,22 @@ fn handle_download_manager_mode(app: &mut App, key: KeyEvent) {
app.dm_editing_dir = false;
}
KeyCode::Backspace => {
if app.dm_dir_cursor > 0 {
app.dm_dir_cursor -= 1;
app.dm_dir_input.remove(app.dm_dir_cursor);
}
app.dm_dir_backspace();
}
KeyCode::Delete => {
app.dm_dir_delete();
}
KeyCode::Left => {
if app.dm_dir_cursor > 0 {
app.dm_dir_cursor -= 1;
}
app.dm_dir_cursor_left();
}
KeyCode::Right => {
if app.dm_dir_cursor < app.dm_dir_input.len() {
app.dm_dir_cursor += 1;
}
app.dm_dir_cursor_right();
}
KeyCode::Char('u') if key.modifiers.contains(KeyModifiers::CONTROL) => {
app.dm_dir_input.clear();
app.dm_dir_cursor = 0;
app.dm_dir_clear();
}
KeyCode::Char(c) => {
app.dm_dir_input.insert(app.dm_dir_cursor, c);
app.dm_dir_cursor += 1;
app.insert_dm_dir_char(c);
}
_ => {}
}
@@ -688,3 +727,47 @@ fn handle_benchmarks_mode(app: &mut App, key: KeyEvent) {
_ => {}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn search_text_accepts_unmodified_and_shift_modified_input() {
assert!(allows_search_text_input(KeyModifiers::NONE));
assert!(allows_search_text_input(KeyModifiers::SHIFT));
}
#[test]
fn search_text_rejects_modified_navigation_artifacts() {
assert!(!allows_search_text_input(KeyModifiers::ALT));
assert!(!allows_search_text_input(KeyModifiers::SUPER));
assert!(!allows_search_text_input(KeyModifiers::CONTROL));
assert!(!allows_search_text_input(KeyModifiers::META));
assert!(!allows_search_text_input(KeyModifiers::HYPER));
assert!(!allows_search_text_input(
KeyModifiers::ALT | KeyModifiers::SHIFT
));
assert!(!allows_search_text_input(
KeyModifiers::SUPER | KeyModifiers::SHIFT
));
}
#[test]
fn provider_filter_text_accepts_plain_ascii_graphic_chars() {
assert!(is_plain_provider_filter_char('o', KeyModifiers::NONE));
assert!(is_plain_provider_filter_char('O', KeyModifiers::SHIFT));
assert!(is_plain_provider_filter_char('-', KeyModifiers::NONE));
}
#[test]
fn provider_filter_text_rejects_non_ascii_space_and_modified_chars() {
assert!(!is_plain_provider_filter_char('你', KeyModifiers::NONE));
assert!(!is_plain_provider_filter_char(' ', KeyModifiers::NONE));
assert!(!is_plain_provider_filter_char('b', KeyModifiers::ALT));
assert!(!is_plain_provider_filter_char('f', KeyModifiers::ALT));
assert!(!is_plain_provider_filter_char('a', KeyModifiers::SUPER));
assert!(!is_plain_provider_filter_char('e', KeyModifiers::SUPER));
assert!(!is_plain_provider_filter_char('x', KeyModifiers::CONTROL));
}
}
llmfit-tui/src/tui_ui.rs
+431 -240
View File
@@ -18,7 +18,10 @@ use crate::tui_app::{
};
use llmfit_core::fit::{FitLevel, ModelFit, SortColumn};
use llmfit_core::hardware::is_running_in_wsl;
use llmfit_core::providers;
use unicode_segmentation::UnicodeSegmentation;
use unicode_width::UnicodeWidthStr;
const DM_MODELS_DIR_LABEL: &str = " Models dir: ";
pub fn draw(frame: &mut Frame, app: &mut App) {
let tc = app.theme.colors();
@@ -131,7 +134,7 @@ fn draw_system_bar(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeColors) {
};
let ollama_info = if app.ollama_available {
format!("Ollama: ✓ ({} installed)", app.ollama_installed_count)
format!("Ollama: ✓ ({} installed)", app.installed.ollama_count)
} else {
"Ollama: ✗".to_string()
};
@@ -142,15 +145,15 @@ fn draw_system_bar(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeColors) {
};
let mlx_info = if app.mlx_available {
format!("MLX: ✓ ({} installed)", app.mlx_installed.len())
} else if !app.mlx_installed.is_empty() {
format!("MLX: ({} cached)", app.mlx_installed.len())
format!("MLX: ✓ ({} installed)", app.installed.mlx.len())
} else if !app.installed.mlx.is_empty() {
format!("MLX: ({} cached)", app.installed.mlx.len())
} else {
"MLX: ✗".to_string()
};
let mlx_color = if app.mlx_available {
tc.good
} else if !app.mlx_installed.is_empty() {
} else if !app.installed.mlx.is_empty() {
tc.warning
} else {
tc.muted
@@ -158,25 +161,25 @@ fn draw_system_bar(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeColors) {
let llamacpp_info = if app.llamacpp_available {
if app.llamacpp_detection_hint.is_empty() {
format!("llama.cpp: ✓ ({} models)", app.llamacpp_installed_count)
format!("llama.cpp: ✓ ({} models)", app.installed.llamacpp_count)
} else {
format!("llama.cpp: ✓ ({})", app.llamacpp_detection_hint)
}
} else if !app.llamacpp_installed.is_empty() {
format!("llama.cpp: ({} cached)", app.llamacpp_installed_count)
} else if !app.installed.llamacpp.is_empty() {
format!("llama.cpp: ({} cached)", app.installed.llamacpp_count)
} else {
format!("llama.cpp: ✗ ({})", app.llamacpp_detection_hint)
};
let llamacpp_color = if app.llamacpp_available {
tc.good
} else if !app.llamacpp_installed.is_empty() {
} else if !app.installed.llamacpp.is_empty() {
tc.warning
} else {
tc.muted
};
let docker_mr_info = if app.docker_mr_available {
format!("Docker: ✓ ({} models)", app.docker_mr_installed_count)
format!("Docker: ✓ ({} models)", app.installed.docker_mr_count)
} else {
"Docker: ✗".to_string()
};
@@ -187,7 +190,7 @@ fn draw_system_bar(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeColors) {
};
let lmstudio_info = if app.lmstudio_available {
format!("LM Studio: ✓ ({} models)", app.lmstudio_installed_count)
format!("LM Studio: ✓ ({} models)", app.installed.lmstudio_count)
} else {
"LM Studio: ✗".to_string()
};
@@ -198,7 +201,7 @@ fn draw_system_bar(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeColors) {
};
let vllm_info = if app.vllm_available {
format!("vLLM: ✓ ({} models)", app.vllm_installed_count)
format!("vLLM: ✓ ({} models)", app.installed.vllm_count)
} else {
"vLLM: ✗".to_string()
};
@@ -285,6 +288,93 @@ fn draw_system_bar(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeColors) {
frame.render_widget(paragraph, area);
}
fn visible_search_query(query: &str, cursor_position: usize, width: usize) -> (String, u16) {
if width == 0 {
return (String::new(), 0);
}
let cursor_position = cursor_position.min(query.len());
let cursor_position = floor_grapheme_boundary(query, cursor_position);
let text_width = width.saturating_sub(1);
if text_width == 0 {
return (String::new(), 0);
}
if UnicodeWidthStr::width(query) <= text_width {
return (
query.to_string(),
UnicodeWidthStr::width(&query[..cursor_position]).min(width.saturating_sub(1)) as u16,
);
}
let graphemes: Vec<(usize, &str, usize)> = query
.grapheme_indices(true)
.map(|(idx, grapheme)| (idx, grapheme, UnicodeWidthStr::width(grapheme)))
.collect();
let cursor_grapheme = graphemes
.iter()
.take_while(|(idx, _, _)| *idx < cursor_position)
.count();
let mut start = cursor_grapheme;
let mut cells_before_cursor = 0;
while start > 0 {
let previous_width = graphemes[start - 1].2;
if cells_before_cursor + previous_width > text_width {
break;
}
cells_before_cursor += previous_width;
start -= 1;
}
let start_byte = graphemes.get(start).map(|(idx, _, _)| *idx).unwrap_or(0);
let mut end = start;
let mut visible_cells = 0;
while let Some((_, _, grapheme_width)) = graphemes.get(end) {
if visible_cells + grapheme_width > text_width {
break;
}
visible_cells += grapheme_width;
end += 1;
}
let end_byte = graphemes
.get(end)
.map(|(idx, _, _)| *idx)
.unwrap_or_else(|| query.len());
let visible = query[start_byte..end_byte].to_string();
let cursor_offset = UnicodeWidthStr::width(&query[start_byte..cursor_position])
.min(width.saturating_sub(1)) as u16;
(visible, cursor_offset)
}
fn visible_dm_dir_input(input: &str, cursor: usize, inner_width: u16) -> (String, u16) {
let label_width = UnicodeWidthStr::width(DM_MODELS_DIR_LABEL) as u16;
let input_width = inner_width.saturating_sub(label_width) as usize;
visible_search_query(input, cursor, input_width)
}
fn floor_grapheme_boundary(value: &str, index: usize) -> usize {
let mut index = index.min(value.len());
while index > 0 && !value.is_char_boundary(index) {
index -= 1;
}
if index == value.len() {
return index;
}
for (start, grapheme) in value.grapheme_indices(true) {
if start + grapheme.len() > index {
return start;
}
}
index
}
fn draw_search_and_filters(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeColors) {
let chunks = Layout::default()
.direction(Direction::Horizontal)
@@ -325,13 +415,17 @@ fn draw_search_and_filters(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeC
| InputMode::Benchmarks => Style::default().fg(tc.muted),
};
let search_inner_width = chunks[0].width.saturating_sub(2) as usize;
let (visible_query, cursor_offset) =
visible_search_query(&app.search_query, app.cursor_position, search_inner_width);
let search_text = if app.search_query.is_empty() && app.input_mode == InputMode::Normal {
Line::from(Span::styled(
"Press / to search...",
Style::default().fg(tc.muted),
))
} else {
Line::from(Span::styled(&app.search_query, Style::default().fg(tc.fg)))
Line::from(Span::styled(visible_query, Style::default().fg(tc.fg)))
};
let search_block = Block::default()
@@ -344,10 +438,7 @@ fn draw_search_and_filters(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeC
frame.render_widget(search, chunks[0]);
if app.input_mode == InputMode::Search {
frame.set_cursor_position((
chunks[0].x + app.cursor_position as u16 + 1,
chunks[0].y + 1,
));
frame.set_cursor_position((chunks[0].x + cursor_offset + 1, chunks[0].y + 1));
}
// Provider filter summary
@@ -686,7 +777,7 @@ fn draw_table(frame: &mut Frame, app: &mut App, area: Rect, tc: &ThemeColors) {
Constraint::Length(10), // quant (AWQ-4bit, GPTQ-Int4, GPTQ-Int8)
Constraint::Length(7), // mode
Constraint::Length(6), // mem %
Constraint::Length(5), // ctx
Constraint::Length(10), // ctx ("262k→14k" when memory-constrained)
Constraint::Length(8), // date (YYYY-MM)
Constraint::Length(10), // fit
Constraint::Min(10), // use case
@@ -816,8 +907,13 @@ fn draw_table(frame: &mut Frame, app: &mut App, area: Rect, tc: &ThemeColors) {
Cell::from(fit.run_mode_text().to_string()).style(Style::default().fg(mode_color)),
Cell::from(format!("{:.0}%", fit.utilization_pct))
.style(Style::default().fg(color)),
Cell::from(format!("{}k", fit.model.context_length / 1000))
.style(Style::default().fg(tc.muted)),
Cell::from(fit.context_display()).style(Style::default().fg(
if fit.context_severely_limited() {
tc.warning
} else {
tc.muted
},
)),
Cell::from(
fit.model
.release_date
@@ -846,7 +942,7 @@ fn draw_table(frame: &mut Frame, app: &mut App, area: Rect, tc: &ThemeColors) {
Constraint::Length(6), // disk
Constraint::Length(7), // mode
Constraint::Length(7), // mem %
Constraint::Length(5), // ctx
Constraint::Length(10), // ctx ("262k→14k" when memory-constrained)
Constraint::Length(8), // date (YYYY-MM)
Constraint::Length(10), // fit
Constraint::Min(10), // use case
@@ -1687,30 +1783,7 @@ fn draw_detail(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeColors) {
Line::from(vec![
Span::styled(" Installed: ", Style::default().fg(tc.muted)),
{
let mut installed_providers = Vec::new();
if providers::is_model_installed(&fit.model.name, &app.ollama_installed) {
installed_providers.push("Ollama");
}
if providers::is_model_installed_mlx(&fit.model.name, &app.mlx_installed) {
installed_providers.push("MLX");
}
if providers::is_model_installed_llamacpp(&fit.model.name, &app.llamacpp_installed)
{
installed_providers.push("llama.cpp");
}
if providers::is_model_installed_docker_mr(
&fit.model.name,
&app.docker_mr_installed,
) {
installed_providers.push("Docker");
}
if providers::is_model_installed_lmstudio(&fit.model.name, &app.lmstudio_installed)
{
installed_providers.push("LM Studio");
}
if providers::is_model_installed_vllm(&fit.model.name, &app.vllm_installed) {
installed_providers.push("vLLM");
}
let installed_providers = app.installed.installed_providers(&fit.model.name);
let any_available = app.ollama_available
|| app.mlx_available
|| app.llamacpp_available
@@ -2439,9 +2512,16 @@ fn draw_plan(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeColors) {
fn draw_provider_popup(frame: &mut Frame, app: &App, tc: &ThemeColors) {
let area = frame.area();
let filtered = app.provider_filtered_indices();
let max_name_len = app.providers.iter().map(|p| p.len()).max().unwrap_or(10);
let popup_width = (max_name_len as u16 + 10).min(area.width.saturating_sub(4));
let popup_height = (app.providers.len() as u16 + 2).min(area.height.saturating_sub(4));
// Width must also fit the search box / hint line.
let popup_width = (max_name_len as u16 + 10)
.max(28)
.min(area.width.saturating_sub(4));
// +2 borders, +1 search row. List body shows at most all matches.
let list_rows = (filtered.len().max(1) as u16).min(area.height.saturating_sub(6));
let popup_height = (list_rows + 3).min(area.height.saturating_sub(4));
let x = area.x + (area.width.saturating_sub(popup_width)) / 2;
let y = area.y + (area.height.saturating_sub(popup_height)) / 2;
@@ -2449,28 +2529,64 @@ fn draw_provider_popup(frame: &mut Frame, app: &App, tc: &ThemeColors) {
frame.render_widget(Clear, popup_area);
let inner_height = popup_height.saturating_sub(2) as usize;
// The list body is the popup height minus borders (2) minus the search row (1).
let inner_height = popup_height.saturating_sub(3) as usize;
let total = app.providers.len();
let scroll_offset = if app.provider_cursor >= inner_height {
app.provider_cursor - inner_height + 1
app.provider_cursor + 1 - inner_height
} else {
0
};
let lines: Vec<Line> = app
.providers
.iter()
.enumerate()
.skip(scroll_offset)
.take(inner_height)
.map(|(i, name)| {
// Search input row.
let mut lines: Vec<Line> = Vec::with_capacity(inner_height + 1);
let search_prefix = " / ";
let search_inner_width = popup_width.saturating_sub(2) as usize;
let search_query_width = search_inner_width.saturating_sub(search_prefix.len());
let (visible_provider_search, provider_cursor_offset) = visible_search_query(
&app.provider_search,
app.provider_search_cursor_position,
search_query_width,
);
let search_display = if app.provider_search.is_empty() {
Line::from(vec![
Span::styled(search_prefix, Style::default().fg(tc.fg)),
Span::styled(
"type to filter",
Style::default().fg(tc.muted).add_modifier(Modifier::ITALIC),
),
])
} else {
Line::from(vec![
Span::styled(search_prefix, Style::default().fg(tc.fg)),
Span::styled(
visible_provider_search,
Style::default().fg(tc.fg).add_modifier(Modifier::BOLD),
),
])
};
lines.push(search_display);
if filtered.is_empty() {
lines.push(Line::from(Span::styled(
" (no matching providers)",
Style::default().fg(tc.muted),
)));
} else {
for (pos, &i) in filtered
.iter()
.enumerate()
.skip(scroll_offset)
.take(inner_height)
{
let name = &app.providers[i];
let checkbox = if app.selected_providers[i] {
"[x]"
} else {
"[ ]"
};
let is_cursor = i == app.provider_cursor;
let is_cursor = pos == app.provider_cursor;
let style = if is_cursor {
if app.selected_providers[i] {
@@ -2490,12 +2606,24 @@ fn draw_provider_popup(frame: &mut Frame, app: &App, tc: &ThemeColors) {
Style::default().fg(tc.muted)
};
Line::from(Span::styled(format!(" {} {}", checkbox, name), style))
})
.collect();
lines.push(Line::from(Span::styled(
format!(" {} {}", checkbox, name),
style,
)));
}
}
let active_count = app.selected_providers.iter().filter(|&&s| s).count();
let title = format!(" Providers ({}/{}) ", active_count, total);
let title = if app.provider_search.is_empty() {
format!(" Providers ({}/{}) ", active_count, total)
} else {
format!(
" Providers ({}/{}) — {} match ",
active_count,
total,
filtered.len()
)
};
let block = Block::default()
.borders(Borders::ALL)
@@ -2510,14 +2638,14 @@ fn draw_provider_popup(frame: &mut Frame, app: &App, tc: &ThemeColors) {
.title_bottom(
Line::from(vec![
Span::styled(
" a",
" ^a",
Style::default()
.fg(tc.accent_secondary)
.add_modifier(Modifier::BOLD),
),
Span::styled(": all | ", Style::default().fg(tc.muted)),
Span::styled(
"c",
"^n",
Style::default()
.fg(tc.accent_secondary)
.add_modifier(Modifier::BOLD),
@@ -2529,6 +2657,15 @@ fn draw_provider_popup(frame: &mut Frame, app: &App, tc: &ThemeColors) {
let paragraph = Paragraph::new(lines).block(block);
frame.render_widget(paragraph, popup_area);
let cursor_x = popup_area.x
+ 1
+ search_prefix.len() as u16
+ provider_cursor_offset.min(search_query_width as u16);
let cursor_y = popup_area.y + 1;
if cursor_x < popup_area.x + popup_area.width.saturating_sub(1) {
frame.set_cursor_position((cursor_x, cursor_y));
}
}
fn draw_use_case_popup(frame: &mut Frame, app: &App, tc: &ThemeColors) {
@@ -2845,7 +2982,7 @@ fn status_keys_and_mode(app: &App) -> (String, String) {
"PLAN".to_string(),
),
InputMode::ProviderPopup => (
" ↑↓/jk:navigate Space:toggle a:all/none Esc:close".to_string(),
" ↑↓:navigate (+Shift:speed up) Space:toggle a:all/none Esc:close".to_string(),
"PROVIDERS".to_string(),
),
InputMode::UseCasePopup => (
@@ -3781,7 +3918,10 @@ fn draw_downloads(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeColors) {
vertical: 1,
horizontal: 1,
});
let cursor_x = inner.x + 14 + app.dm_dir_cursor as u16;
let label_width = UnicodeWidthStr::width(DM_MODELS_DIR_LABEL) as u16;
let (_, cursor_offset) =
visible_dm_dir_input(&app.dm_dir_input, app.dm_dir_cursor, inner.width);
let cursor_x = inner.x + label_width + cursor_offset;
let cursor_y = inner.y;
if cursor_x < inner.x + inner.width {
frame.set_cursor_position((cursor_x, cursor_y));
@@ -3852,23 +3992,22 @@ fn draw_dm_config(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeColors) {
.border_style(border_style)
.title(" Config ");
let dir_display = if app.dm_editing_dir {
app.dm_dir_input.as_str()
let visible_dir = if app.dm_editing_dir {
let inner_width = area.width.saturating_sub(2);
visible_dm_dir_input(&app.dm_dir_input, app.dm_dir_cursor, inner_width).0
} else {
// Show current models dir from the llamacpp provider
// We use the public function as a fallback display
""
String::new()
};
let line = if app.dm_editing_dir {
Line::from(vec![
Span::styled(" Models dir: ", Style::default().fg(tc.muted)),
Span::styled(dir_display, Style::default().fg(tc.fg)),
Span::styled(DM_MODELS_DIR_LABEL, Style::default().fg(tc.muted)),
Span::styled(visible_dir, Style::default().fg(tc.fg)),
Span::styled("█", Style::default().fg(tc.accent)),
])
} else {
Line::from(vec![
Span::styled(" Models dir: ", Style::default().fg(tc.muted)),
Span::styled(DM_MODELS_DIR_LABEL, Style::default().fg(tc.muted)),
Span::styled(
app.llamacpp_models_dir().display().to_string(),
Style::default().fg(tc.fg),
@@ -4829,204 +4968,196 @@ fn draw_bench(frame: &mut Frame, app: &App, area: Rect, tc: &ThemeColors) {
}
// ── Detail pane (when open) ──
if let Some(det_area) = detail_area {
if let Some(ms) = app.bench_model_status.get(app.bench_selected_row) {
let result = app.bench_results.iter().find(|r| r.model == ms.name);
if let Some(det_area) = detail_area
&& let Some(ms) = app.bench_model_status.get(app.bench_selected_row)
{
let result = app.bench_results.iter().find(|r| r.model == ms.name);
let mut detail_lines: Vec<Line> = Vec::new();
let mut detail_lines: Vec<Line> = Vec::new();
if let Some(result) = result {
detail_lines.push(Line::from(vec![
Span::styled(" Model: ", Style::default().fg(tc.muted)),
Span::styled(
&result.model,
Style::default().fg(tc.accent).add_modifier(Modifier::BOLD),
),
]));
detail_lines.push(Line::from(Span::styled(
format!(
" Overall: Q:{:.1} S:{:.1} t/s C:{:.1} | Tests: {} Roles: {}",
result.overall_quality,
result.overall_speed,
result.overall_composite,
result.test_results.len(),
result.roles.len()
),
Style::default().fg(tc.fg),
)));
detail_lines.push(Line::from(""));
// ── Role summary table ──
let bold_style = Style::default().fg(tc.fg).add_modifier(Modifier::BOLD);
detail_lines.push(Line::from(vec![
Span::styled(" ", Style::default()),
Span::styled(format!("{:<16}", "Role"), bold_style),
Span::styled(format!("{:>5}", "Qual"), bold_style),
Span::styled(format!("{:>9}", "Speed"), bold_style),
Span::styled(format!("{:>7}", "Comp"), bold_style),
Span::styled(format!("{:>8}", "TTFT"), bold_style),
Span::styled(" Bar", bold_style),
]));
detail_lines.push(Line::from(Span::styled(
format!(" {}", "─".repeat(60)),
Style::default().fg(tc.border),
)));
for rs in &result.roles {
let q_color = bench_score_color(rs.quality, tc);
let c_color = bench_score_color(rs.composite, tc);
let bar = bench_bar(rs.composite, 15);
let role_tests: Vec<&llmfit_core::quality::QualityResult> = result
.test_results
.iter()
.filter(|t| t.role == rs.role)
.collect();
let avg_ttft = if role_tests.is_empty() {
0.0
} else {
role_tests.iter().filter_map(|t| t.ttft_ms).sum::<f64>()
/ role_tests
.iter()
.filter(|t| t.ttft_ms.is_some())
.count()
.max(1) as f64
};
if let Some(result) = result {
detail_lines.push(Line::from(vec![
Span::styled(" Model: ", Style::default().fg(tc.muted)),
Span::styled(format!(" {:<16}", rs.role), Style::default().fg(tc.fg)),
Span::styled(
&result.model,
Style::default().fg(tc.accent).add_modifier(Modifier::BOLD),
format!("{:>5.1}", rs.quality),
Style::default().fg(q_color),
),
Span::styled(
format!("{:>7.1}t/s", rs.speed),
Style::default().fg(tc.accent_secondary),
),
Span::styled(
format!("{:>7.1}", rs.composite),
Style::default().fg(c_color),
),
Span::styled(
if avg_ttft > 0.0 {
format!("{:>6.0}ms", avg_ttft)
} else {
format!("{:>8}", "—")
},
Style::default().fg(tc.muted),
),
Span::styled(format!(" {}", bar), Style::default().fg(c_color)),
]));
}
// ── Full test rubric grouped by role ──
if !result.test_results.is_empty() {
detail_lines.push(Line::from(""));
detail_lines.push(Line::from(Span::styled(
format!(
" Overall: Q:{:.1} S:{:.1} t/s C:{:.1} | Tests: {} Roles: {}",
result.overall_quality,
result.overall_speed,
result.overall_composite,
result.test_results.len(),
result.roles.len()
),
Style::default().fg(tc.fg),
" ── Full Test Rubric ──",
Style::default().fg(tc.title).add_modifier(Modifier::BOLD),
)));
detail_lines.push(Line::from(""));
// ── Role summary table ──
let bold_style = Style::default().fg(tc.fg).add_modifier(Modifier::BOLD);
detail_lines.push(Line::from(vec![
Span::styled(" ", Style::default()),
Span::styled(format!("{:<16}", "Role"), bold_style),
Span::styled(format!("{:>5}", "Qual"), bold_style),
Span::styled(format!("{:>9}", "Speed"), bold_style),
Span::styled(format!("{:>7}", "Comp"), bold_style),
Span::styled(format!("{:>8}", "TTFT"), bold_style),
Span::styled(" Bar", bold_style),
]));
detail_lines.push(Line::from(Span::styled(
format!(" {}", "─".repeat(60)),
Style::default().fg(tc.border),
)));
let mut current_role = String::new();
for t in &result.test_results {
if t.role != current_role {
if !current_role.is_empty() {
detail_lines.push(Line::from(Span::styled(
" └────────────────────────────────────────────────",
Style::default().fg(tc.border),
)));
}
current_role = t.role.clone();
detail_lines.push(Line::from(vec![
Span::styled(
format!(" ┌─ {} ", t.role.to_uppercase()),
Style::default().fg(tc.accent).add_modifier(Modifier::BOLD),
),
Span::styled("─".repeat(50), Style::default().fg(tc.border)),
]));
}
for rs in &result.roles {
let q_color = bench_score_color(rs.quality, tc);
let c_color = bench_score_color(rs.composite, tc);
let bar = bench_bar(rs.composite, 15);
let role_tests: Vec<&llmfit_core::quality::QualityResult> = result
.test_results
.iter()
.filter(|t| t.role == rs.role)
.collect();
let avg_ttft = if role_tests.is_empty() {
0.0
let q_color = bench_score_color(t.quality, tc);
let status = if t.error.is_some() {
"ERR"
} else if t.quality >= 7.0 {
" ✓ "
} else if t.quality >= 4.0 {
" ~ "
} else {
role_tests.iter().filter_map(|t| t.ttft_ms).sum::<f64>()
/ role_tests
.iter()
.filter(|t| t.ttft_ms.is_some())
.count()
.max(1) as f64
" ✗ "
};
let status_color = if t.error.is_some() {
tc.error
} else if t.quality >= 7.0 {
tc.good
} else if t.quality >= 4.0 {
tc.warning
} else {
tc.error
};
detail_lines.push(Line::from(vec![
Span::styled(
format!(" {:<16}", rs.role),
format!(" │ {:<28}", t.test_name),
Style::default().fg(tc.fg),
),
Span::styled(status, Style::default().fg(status_color)),
Span::styled(
format!("{:>5.1}", rs.quality),
format!(" Q:{:>4.1}", t.quality),
Style::default().fg(q_color),
),
Span::styled(
format!("{:>7.1}t/s", rs.speed),
Style::default().fg(tc.accent_secondary),
),
Span::styled(
format!("{:>7.1}", rs.composite),
Style::default().fg(c_color),
),
Span::styled(
if avg_ttft > 0.0 {
format!("{:>6.0}ms", avg_ttft)
} else {
format!("{:>8}", "—")
},
format!(" {:>6.1}t/s", t.tok_per_sec),
Style::default().fg(tc.muted),
),
Span::styled(
format!(" {:>5.1}s", t.wall_time_sec),
Style::default().fg(tc.muted),
),
Span::styled(format!(" {}", bar), Style::default().fg(c_color)),
]));
}
// ── Full test rubric grouped by role ──
if !result.test_results.is_empty() {
detail_lines.push(Line::from(""));
detail_lines.push(Line::from(Span::styled(
" ── Full Test Rubric ──",
Style::default().fg(tc.title).add_modifier(Modifier::BOLD),
)));
detail_lines.push(Line::from(""));
let mut current_role = String::new();
for t in &result.test_results {
if t.role != current_role {
if !current_role.is_empty() {
detail_lines.push(Line::from(Span::styled(
" └────────────────────────────────────────────────",
Style::default().fg(tc.border),
)));
}
current_role = t.role.clone();
detail_lines.push(Line::from(vec![
Span::styled(
format!(" ┌─ {} ", t.role.to_uppercase()),
Style::default()
.fg(tc.accent)
.add_modifier(Modifier::BOLD),
),
Span::styled(
"─".repeat(50),
Style::default().fg(tc.border),
),
]));
}
let q_color = bench_score_color(t.quality, tc);
let status = if t.error.is_some() {
"ERR"
} else if t.quality >= 7.0 {
" ✓ "
} else if t.quality >= 4.0 {
" ~ "
} else {
" ✗ "
};
let status_color = if t.error.is_some() {
tc.error
} else if t.quality >= 7.0 {
tc.good
} else if t.quality >= 4.0 {
tc.warning
} else {
tc.error
};
detail_lines.push(Line::from(vec![
Span::styled(
format!(" │ {:<28}", t.test_name),
Style::default().fg(tc.fg),
),
Span::styled(status, Style::default().fg(status_color)),
Span::styled(
format!(" Q:{:>4.1}", t.quality),
Style::default().fg(q_color),
),
Span::styled(
format!(" {:>6.1}t/s", t.tok_per_sec),
Style::default().fg(tc.muted),
),
Span::styled(
format!(" {:>5.1}s", t.wall_time_sec),
Style::default().fg(tc.muted),
),
]));
if let Some(e) = &t.error {
detail_lines.push(Line::from(Span::styled(
format!(" │ Error: {}", e),
Style::default().fg(tc.error),
)));
} else if !t.response_preview.is_empty() {
detail_lines.push(Line::from(Span::styled(
format!(" │ Preview: {}…", &t.response_preview),
Style::default().fg(tc.muted),
)));
}
}
if !result.test_results.is_empty() {
if let Some(e) = &t.error {
detail_lines.push(Line::from(Span::styled(
" └────────────────────────────────────────────────",
Style::default().fg(tc.border),
format!(" │ Error: {}", e),
Style::default().fg(tc.error),
)));
} else if !t.response_preview.is_empty() {
detail_lines.push(Line::from(Span::styled(
format!(" │ Preview: {}…", &t.response_preview),
Style::default().fg(tc.muted),
)));
}
}
} else {
detail_lines.push(Line::from(Span::styled(
format!(" {} — pending or no results yet.", ms.name),
Style::default().fg(tc.muted),
)));
if !result.test_results.is_empty() {
detail_lines.push(Line::from(Span::styled(
" └────────────────────────────────────────────────",
Style::default().fg(tc.border),
)));
}
}
let scroll = app.live_bench_scroll as u16;
let paragraph = Paragraph::new(detail_lines)
.scroll((scroll, 0))
.wrap(Wrap { trim: false });
frame.render_widget(paragraph, det_area);
} else {
detail_lines.push(Line::from(Span::styled(
format!(" {} — pending or no results yet.", ms.name),
Style::default().fg(tc.muted),
)));
}
let scroll = app.live_bench_scroll as u16;
let paragraph = Paragraph::new(detail_lines)
.scroll((scroll, 0))
.wrap(Wrap { trim: false });
frame.render_widget(paragraph, det_area);
}
}
@@ -5184,4 +5315,64 @@ mod tests {
// Exact max length — no truncation
assert_eq!(truncate_str("abc", 3), "abc");
}
#[test]
fn visible_search_query_keeps_short_query_unchanged() {
assert_eq!(
visible_search_query("hello", 3, 10),
("hello".to_string(), 3)
);
}
#[test]
fn visible_search_query_scrolls_to_keep_end_cursor_visible() {
assert_eq!(
visible_search_query("abcdefghijklmnopqrstuvwxyz", 26, 8),
("tuvwxyz".to_string(), 7)
);
}
#[test]
fn visible_search_query_keeps_middle_cursor_visible() {
assert_eq!(
visible_search_query("abcdefghijklmnopqrstuvwxyz", 13, 8),
("ghijklm".to_string(), 7)
);
}
#[test]
fn visible_search_query_handles_multibyte_cursor_boundaries() {
assert_eq!(
visible_search_query("你好世界abc", "你好世界abc".len(), 5),
("abc".to_string(), 3)
);
assert_eq!(
visible_search_query("你好世界abc", 1, 5),
("你好".to_string(), 0)
);
}
#[test]
fn visible_search_query_uses_terminal_cell_width() {
assert_eq!(
visible_search_query("ab😀cd", "ab😀cd".len(), 5),
("😀cd".to_string(), 4)
);
assert_eq!(
visible_search_query("你好世界abc", "你好世界abc".len(), 6),
("界abc".to_string(), 5)
);
}
#[test]
fn visible_dm_dir_input_keeps_unicode_cursor_visible() {
let input = "/tmp/模型/一二三四";
assert_eq!(
visible_dm_dir_input(input, input.len(), (DM_MODELS_DIR_LABEL.len() + 8) as u16),
("二三四".to_string(), 6)
);
}
}
llmfit-tui/tests/cli_smoke.rs
+51
View File
@@ -14,6 +14,12 @@ fn run_json_command(args: &[&str]) -> Value {
serde_json::from_slice(&output).expect("command did not emit valid JSON")
}
fn models_array(json: &Value) -> &[Value] {
json.get("models")
.and_then(Value::as_array)
.expect("JSON output missing models array")
}
#[test]
fn help_includes_project_description() {
let output = Command::cargo_bin("llmfit")
@@ -105,6 +111,51 @@ fn fit_json_obeys_limit_and_contains_models_field() {
}
}
#[test]
fn recommend_capability_filter_does_not_ignore_unknown_or_tts() {
let tts_json = run_json_command(&[
"--no-dashboard",
"--json",
"--memory",
"8G",
"--ram",
"16G",
"--cpu-cores",
"4",
"recommend",
"--capability",
"tts",
"-n",
"5",
]);
assert!(models_array(&tts_json).iter().all(|model| {
model
.get("capability_ids")
.and_then(Value::as_array)
.is_some_and(|caps| caps.iter().any(|cap| cap.as_str() == Some("tts")))
}));
let unknown_json = run_json_command(&[
"--no-dashboard",
"--json",
"--memory",
"8G",
"--ram",
"16G",
"--cpu-cores",
"4",
"recommend",
"--capability",
"not_a_capability",
"-n",
"5",
]);
assert!(
models_array(&unknown_json).is_empty(),
"unknown capability should not match every model"
);
}
#[test]
fn fit_json_returns_empty_models_when_no_perfect_matches() {
let json = run_json_command(&[
scripts/scrape_benchmarks.py
+1 -3
View File
@@ -11,7 +11,6 @@ Usage:
python3 scrape_benchmarks.py --presets "RTX 4090,M4 Max" # Specific presets only
Output:
data/benchmark_cache.json (repo root, for reference)
llmfit-core/data/benchmark_cache.json (compiled into binary)
The cache format is:
@@ -168,8 +167,7 @@ def main():
if i < len(presets) - 1:
time.sleep(0.5)
# Write to both locations
output_paths = ["data/benchmark_cache.json", "llmfit-core/data/benchmark_cache.json"]
output_paths = ["llmfit-core/data/benchmark_cache.json"]
for path in output_paths:
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w") as f:
scripts/scrape_hf_models.py
+255 -30
View File
@@ -107,6 +107,7 @@ TARGET_MODELS = [
# Qwen 3.6 (native multimodal + hybrid attention, Apr 2026)
"Qwen/Qwen3.6-27B",
"Qwen/Qwen3.6-35B-A3B",
"huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated",
# Microsoft Phi
"microsoft/phi-3-mini-4k-instruct",
"microsoft/Phi-3-medium-14b-instruct",
@@ -219,6 +220,12 @@ TARGET_MODELS = [
# Liquid AI LFM2 Audio
"LiquidAI/LFM2-Audio-1.5B",
"LiquidAI/LFM2.5-Audio-1.5B",
# Text-to-speech models
"hexgrad/Kokoro-82M",
"microsoft/speecht5_tts",
"facebook/mms-tts-eng",
"suno/bark",
"coqui/XTTS-v2",
# Liquid AI Liquid Nanos (task-specific fine-tunes)
"LiquidAI/LFM2-1.2B-Tool",
"LiquidAI/LFM2-1.2B-RAG",
@@ -240,9 +247,9 @@ TARGET_MODELS = [
"zai-org/GLM-5",
# Moonshot Kimi K2.5
"moonshotai/Kimi-K2.5",
# MiniMax M2.7 / M2.5
# MiniMax M3 / M2.7
"MiniMaxAI/MiniMax-M3",
"MiniMaxAI/MiniMax-M2.7",
"MiniMaxAI/MiniMax-M2.5",
# Xiaomi MiMo
"XiaomiMiMo/MiMo-V2-Flash",
"XiaomiMiMo/MiMo-7B-RL",
@@ -270,6 +277,10 @@ TARGET_MODELS = [
"shoumenchougou/RWKV7-G1f-2.9B-GGUF",
"shoumenchougou/RWKV7-G1f-7.2B-GGUF",
"shoumenchougou/RWKV7-G1f-13.3B-GGUF",
# NCAI VAETKI
"nc-ai-consortium/VAETKI-7B-A1B",
"nc-ai-consortium/VAETKI-20B-A2B",
"nc-ai-consortium/VAETKI-VL-7B-A1B",
]
# Bytes-per-parameter for different quantization levels
@@ -332,19 +343,23 @@ MOE_ACTIVE_PARAMS = {
"Qwen/Qwen3.5-122B-A10B": 10_000_000_000,
"Qwen/Qwen3.5-397B-A17B": 17_000_000_000,
"Qwen/Qwen3.6-35B-A3B": 3_000_000_000,
"huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated": 3_000_000_000, # Qwen3.6-35B-A3B finetune
"meta-llama/Llama-4-Scout-17B-16E-Instruct": 17_000_000_000,
"meta-llama/Llama-4-Maverick-17B-128E-Instruct": 17_000_000_000,
"xai-org/grok-1": 86_000_000_000,
"moonshotai/Kimi-K2-Instruct": 32_000_000_000,
"moonshotai/Kimi-K2.5": 32_000_000_000,
"zai-org/GLM-5": 40_000_000_000,
"MiniMaxAI/MiniMax-M3": 10_000_000_000,
"MiniMaxAI/MiniMax-M2.7": 10_000_000_000,
"MiniMaxAI/MiniMax-M2.5": 10_000_000_000,
"XiaomiMiMo/MiMo-V2-Flash": 15_000_000_000,
"nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16": 3_000_000_000,
"LiquidAI/LFM2-8B-A1B": 1_500_000_000,
"LiquidAI/LFM2-24B-A2B": 2_300_000_000, # 23.8B total, 2.3B active
"google/gemma-4-26B-A4B-it": 4_000_000_000,
"nc-ai-consortium/VAETKI-7B-A1B": 1_200_000_000,
"nc-ai-consortium/VAETKI-20B-A2B": 2_200_000_000,
"nc-ai-consortium/VAETKI-VL-7B-A1B": 1_200_000_000,
}
@@ -367,6 +382,37 @@ def fetch_model_info(repo_id: str) -> dict | None:
return None
def extract_license(info: dict | None) -> str | None:
"""Extract normalized license metadata from HuggingFace model info."""
if not info:
return None
card_data = info.get("cardData") or {}
license_value = card_data.get("license")
license_name = card_data.get("license_name")
if isinstance(license_name, str) and license_name.strip():
license_name = license_name.strip().lower()
else:
license_name = None
if isinstance(license_value, str) and license_value.strip():
license_value = license_value.strip().lower()
return license_name if license_value == "other" and license_name else license_value
if isinstance(license_value, list):
licenses = [str(item).strip().lower() for item in license_value if str(item).strip()]
if licenses:
return ",".join(licenses)
for tag in info.get("tags", []):
if isinstance(tag, str) and tag.startswith("license:"):
license_tag = tag.removeprefix("license:").strip().lower()
if license_tag:
return license_name if license_tag == "other" and license_name else license_tag
return None
def format_param_count(total_params: int) -> str:
"""Convert raw parameter count into human-readable string."""
if total_params >= 1_000_000_000:
@@ -590,12 +636,16 @@ def estimate_params_from_arch(config: dict | None) -> int | None:
def infer_use_case(repo_id: str, pipeline_tag: str | None, config: dict | None) -> str:
"""Infer a brief use-case description from model metadata."""
rid = repo_id.lower()
if pipeline_tag == "text-to-speech":
return "Text-to-speech"
if "embed" in rid or "bge" in rid:
return "Text embeddings for RAG"
if "coder" in rid or "starcoder" in rid or "code" in rid:
return "Code generation and completion"
if "r1" in rid or "reason" in rid:
return "Advanced reasoning, chain-of-thought"
if pipeline_tag in ("image-text-to-text", "any-to-any") or "-vl-" in rid:
return "Multimodal, vision and text"
if "instruct" in rid or "chat" in rid:
return "Instruction following, chat"
if "tiny" in rid or "small" in rid or "mini" in rid:
@@ -689,6 +739,7 @@ def extract_provider(repo_id: str) -> str:
"nousresearch": "NousResearch", # NEW
"wizardlmteam": "WizardLM", # NEW
"liquidai": "Liquid AI",
"nc-ai-consortium": "NCAI",
}
return mapping.get(org, org)
@@ -699,6 +750,9 @@ def infer_capabilities(repo_id: str, pipeline_tag: str | None, use_case: str) ->
rid = repo_id.lower()
uc = use_case.lower()
if pipeline_tag == "text-to-speech":
caps.extend(["audio", "tts"])
# Vision
if (
pipeline_tag == "image-text-to-text"
@@ -732,6 +786,64 @@ def infer_capabilities(repo_id: str, pipeline_tag: str | None, use_case: str) ->
return caps
def _looks_like_language_tag(value: str, allow_bare_iso3: bool) -> bool:
parts = value.split("-")
primary = parts[0]
if not primary.isalpha():
return False
if len(primary) == 3 and not allow_bare_iso3:
return False
if len(primary) not in (2, 3):
return False
return all(2 <= len(part) <= 8 and part.isalnum() for part in parts[1:])
def _normalize_language(value: object, explicit_field: bool = False) -> str | None:
"""Return an explicit HF language tag, or None for non-language metadata."""
if not isinstance(value, str):
return None
lang = value.strip().lower().replace("_", "-")
prefixed = False
for prefix in ("language:", "languages:", "lang:"):
if lang.startswith(prefix):
lang = lang[len(prefix):]
prefixed = True
break
if _looks_like_language_tag(lang, allow_bare_iso3=prefixed or explicit_field):
return lang
return None
def infer_languages(info: dict | None, config: dict | None) -> list[str]:
"""Extract explicitly declared language metadata from HF fields."""
values: list[object] = []
for source in (info or {}, config or {}):
for key in ("language", "languages", "language_code", "language_codes"):
val = source.get(key)
if isinstance(val, list):
values.extend((item, True) for item in val)
elif val is not None:
values.append((val, True))
values.extend((tag, False) for tag in (info or {}).get("tags", []))
# Meta MMS per-language models (facebook/mms-tts-eng, facebook/mms-tts-deu,
# ...) declare no language metadata via the API; the target language is
# only encoded as an ISO-639-3 suffix in the repo name.
repo_id = (info or {}).get("id", "") or (info or {}).get("modelId", "")
repo_lower = repo_id.lower()
if "/mms-tts-" in repo_lower:
suffix = repo_lower.rsplit("mms-tts-", 1)[1]
if suffix:
values.append((suffix, True))
languages: list[str] = []
for value, explicit_field in values:
lang = _normalize_language(value, explicit_field=explicit_field)
if lang and lang not in languages:
languages.append(lang)
return languages
def detect_quant_format(repo_id: str, config: dict | None) -> tuple[str, str]:
"""Detect quantization format and label from config.json.
@@ -835,6 +947,8 @@ def scrape_model(repo_id: str) -> dict | None:
# Detect quantization format from config.json
model_format, default_quant = detect_quant_format(repo_id, full_config)
if pipeline_tag == "text-to-speech":
model_format, default_quant = ("safetensors", "F16")
context_length = infer_context_length(full_config) if full_config else infer_context_length(config)
# Correct parameters_raw when safetensors reports quantized element counts
@@ -857,6 +971,8 @@ def scrape_model(repo_id: str) -> dict | None:
# absent fields cause the Rust side to fall back to the linear approx.
arch_meta = extract_arch_metadata(full_config)
license_name = extract_license(info)
result = {
"name": repo_id,
"provider": extract_provider(repo_id),
@@ -870,6 +986,7 @@ def scrape_model(repo_id: str) -> dict | None:
"context_length": context_length,
"use_case": use_case_str,
"capabilities": infer_capabilities(repo_id, pipeline_tag, use_case_str),
"languages": infer_languages(info, full_config or config),
"pipeline_tag": pipeline_tag or "unknown",
"architecture": architecture,
"hf_downloads": info.get("downloads", 0),
@@ -878,6 +995,9 @@ def scrape_model(repo_id: str) -> dict | None:
**arch_meta,
}
if license_name:
result["license"] = license_name
# Add MoE fields if detected
if moe_info["is_moe"]:
result["is_moe"] = True
@@ -1100,8 +1220,11 @@ DISCOVER_PIPELINES = [
"text2text-generation",
"image-text-to-text",
"feature-extraction", # Embedding models (useful for RAG sizing)
"text-to-speech",
]
PRIMARY_DISCOVER_PIPELINE = "text-generation"
# Orgs to skip — test fixtures and legacy mirrors only.
# Quantization/repack orgs (TheBloke, bartowski, unsloth, etc.) are kept
# because they provide popular quantised variants users actually run.
@@ -1145,7 +1268,8 @@ def _build_first_page_url(pipeline: str, sort: str, page_size: int) -> str:
f"direction=-1&"
f"limit={page_size}&"
f"expand[]=safetensors&"
f"expand[]=config"
f"expand[]=config&"
f"expand[]=cardData"
)
@@ -1180,9 +1304,11 @@ def _estimate_params_from_config(config: dict) -> int | None:
def _process_listing(
m: dict,
pipeline: str,
curated: set[str],
seen_ids: set[str],
min_downloads: int,
require_downloads_floor: bool,
stats: dict,
) -> dict | None:
"""Check a single model listing against filters.
@@ -1209,8 +1335,9 @@ def _process_listing(
stats["skip_org"] += 1
return None
downloads = m.get("downloads", 0)
if downloads < min_downloads:
downloads_raw = m.get("downloads")
downloads = downloads_raw or 0
if downloads < min_downloads and (require_downloads_floor or downloads_raw is not None):
stats["skip_downloads"] += 1
return None
@@ -1248,6 +1375,7 @@ def _process_listing(
stats["params_from_config"] += 1
m["_total_params"] = total_params
m["_pipeline_tag"] = m.get("pipeline_tag") or pipeline
stats["accepted"] += 1
return m
@@ -1270,6 +1398,19 @@ def discover_trending_models(limit: int = 30, min_downloads: int = 10000) -> lis
discovered = []
seen_ids = set()
# Keep --discover-limit as the mainstream LLM discovery budget. Other
# pipelines are additive so TTS/audio discovery does not take slots away
# from text-generation coverage.
side_quota = max(1, limit // len(DISCOVER_PIPELINES))
pipeline_limits = {
pipeline: limit if pipeline == PRIMARY_DISCOVER_PIPELINE else side_quota
for pipeline in DISCOVER_PIPELINES
}
pipeline_counts = {pipeline: 0 for pipeline in DISCOVER_PIPELINES}
def _quotas_full() -> bool:
return all(pipeline_counts[p] >= pipeline_limits[p] for p in DISCOVER_PIPELINES)
PAGE_SIZE = 1000
stats = {
@@ -1296,6 +1437,9 @@ def discover_trending_models(limit: int = 30, min_downloads: int = 10000) -> lis
max_pages = 50 if sort_strategy == "downloads" else 5
for pipeline in DISCOVER_PIPELINES:
if pipeline_counts[pipeline] >= pipeline_limits[pipeline]:
continue
next_url: str | None = _build_first_page_url(
pipeline, sort_strategy, PAGE_SIZE
)
@@ -1303,7 +1447,8 @@ def discover_trending_models(limit: int = 30, min_downloads: int = 10000) -> lis
hit_floor = False
page_num = 0
while len(discovered) < limit and next_url and page_num < max_pages:
while (pipeline_counts[pipeline] < pipeline_limits[pipeline]
and next_url and page_num < max_pages):
page_num += 1
try:
models, next_url = _fetch_models_page(next_url)
@@ -1319,22 +1464,30 @@ def discover_trending_models(limit: int = 30, min_downloads: int = 10000) -> lis
for m in models:
result = _process_listing(
m, curated, seen_ids, effective_min, stats
m,
pipeline,
curated,
seen_ids,
effective_min,
sort_strategy == "downloads",
stats,
)
if result is None:
# Track download-floor hits for early stop
downloads = m.get("downloads", 0)
downloads = m.get("downloads")
repo_id = m.get("id", "")
if (repo_id and "/" in repo_id
and repo_id not in curated
and downloads is not None
and downloads < effective_min):
below_min_this_page += 1
continue
discovered.append(result)
pipeline_counts[pipeline] += 1
pipeline_accepted += 1
strategy_accepted += 1
if len(discovered) >= limit:
if pipeline_counts[pipeline] >= pipeline_limits[pipeline]:
break
# For download-sorted queries, stop when most results are
@@ -1355,13 +1508,13 @@ def discover_trending_models(limit: int = 30, min_downloads: int = 10000) -> lis
print(f" {pipeline}: +{pipeline_accepted}"
f" (pages: {page_num}{suffix})")
if len(discovered) >= limit:
if _quotas_full():
break
print(f" sort={sort_strategy} (min_dl={effective_min:,}): "
f"+{strategy_accepted} new models")
if len(discovered) >= limit:
if _quotas_full():
break
# Print filter statistics
@@ -1376,24 +1529,37 @@ def discover_trending_models(limit: int = 30, min_downloads: int = 10000) -> lis
print(f" Params from safetensors: {stats['params_from_safetensors']:>6}")
print(f" Params from config est.: {stats['params_from_config']:>6}")
print(f" Accepted: {stats['accepted']:>6}")
print(f" Pipeline quotas: {pipeline_limits}")
return discovered[:limit]
return discovered
def _build_discovered_model(listing: dict) -> dict | None:
"""Build model dict from a listing returned by discover_trending_models.
Only fetches config.json for accurate context length; all other metadata
comes from the listing data already obtained via expand=safetensors.
comes from the listing data already obtained via expand fields.
"""
repo_id = listing["id"]
total_params = listing["_total_params"]
config = listing.get("config", {})
pipeline_tag = listing.get("pipeline_tag")
pipeline_tag = listing.get("pipeline_tag") or listing.get("_pipeline_tag")
# Listings from non-download sort strategies (trending, likes) omit
# downloads/likes/tags, which would zero out popularity metadata and
# lose language tags. Backfill those fields with a full info fetch.
if listing.get("downloads") is None or listing.get("likes") is None:
info = fetch_model_info(repo_id)
if info:
for key in ("downloads", "likes", "createdAt", "tags"):
if listing.get(key) is None and info.get(key) is not None:
listing[key] = info[key]
full_config = fetch_config_json(repo_id)
model_format, default_quant = detect_quant_format(repo_id, full_config)
if pipeline_tag == "text-to-speech":
model_format, default_quant = ("safetensors", "F16")
context_length = (infer_context_length(full_config) if full_config
else infer_context_length(config))
@@ -1412,6 +1578,8 @@ def _build_discovered_model(listing: dict) -> dict | None:
# Architecture metadata for the precise KV cache formula.
arch_meta = extract_arch_metadata(full_config)
license_name = extract_license(listing)
model = {
"name": repo_id,
"provider": extract_provider(repo_id),
@@ -1425,6 +1593,7 @@ def _build_discovered_model(listing: dict) -> dict | None:
"context_length": context_length,
"use_case": use_case_str,
"capabilities": infer_capabilities(repo_id, pipeline_tag, use_case_str),
"languages": infer_languages(listing, full_config or config),
"pipeline_tag": pipeline_tag or "unknown",
"architecture": architecture,
"hf_downloads": listing.get("downloads", 0),
@@ -1434,6 +1603,9 @@ def _build_discovered_model(listing: dict) -> dict | None:
"_discovered": True,
}
if license_name:
model["license"] = license_name
if moe_info["is_moe"]:
model["is_moe"] = True
model["num_experts"] = moe_info["num_experts"]
@@ -1980,30 +2152,30 @@ def main():
"pipeline_tag": "image-text-to-text", "architecture": "kimi",
"hf_downloads": 0, "hf_likes": 0, "release_date": "2026-01-26",
},
{
"name": "MiniMaxAI/MiniMax-M3",
"provider": "MiniMax", "parameter_count": "230B",
"parameters_raw": 230000000000,
"min_ram_gb": 128.6, "recommended_ram_gb": 214.4, "min_vram_gb": 117.9,
"quantization": "Q4_K_M", "context_length": 524288,
"use_case": "Latest flagship: 512K context, 128K max output, image input",
"pipeline_tag": "text-generation", "architecture": "minimax",
"is_moe": True, "num_experts": 32, "active_experts": 2,
"active_parameters": 10000000000,
"hf_downloads": 0, "hf_likes": 0, "release_date": "2026-06-03",
},
{
"name": "MiniMaxAI/MiniMax-M2.7",
"provider": "MiniMax", "parameter_count": "230B",
"parameters_raw": 230000000000,
"min_ram_gb": 128.6, "recommended_ram_gb": 214.4, "min_vram_gb": 117.9,
"quantization": "Q4_K_M", "context_length": 131072,
"use_case": "Latest flagship with enhanced reasoning and coding",
"use_case": "Previous flagship with enhanced reasoning and coding",
"pipeline_tag": "text-generation", "architecture": "minimax",
"is_moe": True, "num_experts": 32, "active_experts": 2,
"active_parameters": 10000000000,
"hf_downloads": 0, "hf_likes": 0, "release_date": "2026-03-18",
},
{
"name": "MiniMaxAI/MiniMax-M2.5",
"provider": "MiniMax", "parameter_count": "230B",
"parameters_raw": 230000000000,
"min_ram_gb": 128.6, "recommended_ram_gb": 214.4, "min_vram_gb": 117.9,
"quantization": "Q4_K_M", "context_length": 131072,
"use_case": "Coding, agentic tool use",
"pipeline_tag": "text-generation", "architecture": "minimax",
"is_moe": True, "num_experts": 32, "active_experts": 2,
"active_parameters": 10000000000,
"hf_downloads": 0, "hf_likes": 0, "release_date": "2026-02-11",
},
{
"name": "XiaomiMiMo/MiMo-V2-Flash",
"provider": "Xiaomi", "parameter_count": "309B",
@@ -2500,6 +2672,39 @@ def main():
"pipeline_tag": "text-generation", "architecture": "lfm2",
"hf_downloads": 0, "hf_likes": 0, "release_date": "2025-11-28",
},
{
"name": "hexgrad/Kokoro-82M",
"provider": "hexgrad", "parameter_count": "82M",
"parameters_raw": 82_000_000,
"min_ram_gb": 1.0, "recommended_ram_gb": 2.0, "min_vram_gb": 0.5,
"quantization": "F16", "format": "safetensors", "context_length": 4096,
"use_case": "Text-to-speech",
"capabilities": ["audio", "tts"], "languages": [],
"pipeline_tag": "text-to-speech", "architecture": "unknown",
"hf_downloads": 0, "hf_likes": 0, "release_date": None,
},
{
"name": "microsoft/speecht5_tts",
"provider": "Microsoft", "parameter_count": "144M",
"parameters_raw": 144_000_000,
"min_ram_gb": 1.0, "recommended_ram_gb": 2.0, "min_vram_gb": 0.5,
"quantization": "F16", "format": "safetensors", "context_length": 4096,
"use_case": "Text-to-speech",
"capabilities": ["audio", "tts"], "languages": [],
"pipeline_tag": "text-to-speech", "architecture": "speecht5",
"hf_downloads": 0, "hf_likes": 0, "release_date": None,
},
{
"name": "facebook/mms-tts-eng",
"provider": "Meta", "parameter_count": "36M",
"parameters_raw": 36_000_000,
"min_ram_gb": 1.0, "recommended_ram_gb": 2.0, "min_vram_gb": 0.5,
"quantization": "F16", "format": "safetensors", "context_length": 4096,
"use_case": "Text-to-speech",
"capabilities": ["audio", "tts"], "languages": [],
"pipeline_tag": "text-to-speech", "architecture": "vits",
"hf_downloads": 0, "hf_likes": 0, "release_date": None,
},
# RWKV v7 G1f: GGUF-native repos — no safetensors metadata, fallback required
{
"name": "shoumenchougou/RWKV7-G1f-1.5B-GGUF",
@@ -2608,7 +2813,7 @@ def main():
# The database is additive: models from previous runs are preserved.
# Freshly scraped models update existing entries; historical models
# that are no longer in the top discovered set are kept as-is.
output_paths = ["data/hf_models.json", "llmfit-core/data/hf_models.json"]
output_paths = ["llmfit-core/data/hf_models.json"]
# Build a map of freshly scraped models (name -> model dict)
fresh_by_name = {m["name"]: m for m in results}
@@ -2626,6 +2831,17 @@ def main():
for old_model in existing:
name = old_model.get("name", "")
if name in fresh_by_name:
fresh_model = fresh_by_name[name]
if old_model.get("license") and not fresh_model.get("license"):
fresh_model["license"] = old_model["license"]
if old_model.get("gguf_sources") and not fresh_model.get("gguf_sources"):
fresh_model["gguf_sources"] = old_model["gguf_sources"]
# Fallback stubs and trending listings carry no
# popularity/date/language metadata — never let them
# clobber real values from a previous scrape.
for key in ("hf_downloads", "hf_likes", "release_date", "languages"):
if old_model.get(key) and not fresh_model.get(key):
fresh_model[key] = old_model[key]
updated_count += 1
elif name:
# Historical model not in current scrape — keep it
@@ -2641,6 +2857,13 @@ def main():
print(f"\nMerged with existing database ({existing_count} models):")
print(f" Updated: {updated_count}, Retained historical: {retained_count}")
# Keep additive/retained entries on the current schema even if they were
# produced by an older scraper version.
for model in results:
model.setdefault("capabilities", [])
if not model.get("languages"):
model.pop("languages", None)
# Sort by parameter count
results.sort(key=lambda m: m["parameters_raw"])
@@ -2651,7 +2874,7 @@ def main():
gguf_enriched = enrich_gguf_sources(results, threads=args.threads)
print(f" Found GGUF sources for {gguf_enriched} models")
# Write to both locations: repo root (for reference) and llmfit-core (compiled into binary)
# Write to llmfit-core/data (compiled into the binary via include_str!)
for output_path in output_paths:
os.makedirs(os.path.dirname(output_path), exist_ok=True)
with open(output_path, "w") as f:
@@ -2673,3 +2896,5 @@ def main():
if __name__ == "__main__":
main()
scripts/update_models.sh
+2 -2
View File
@@ -10,7 +10,7 @@ set -e # Exit on error
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
DATA_FILE="$PROJECT_ROOT/data/hf_models.json"
DATA_FILE="$PROJECT_ROOT/llmfit-core/data/hf_models.json"
# Colors for output
RED='\033[0;31m'
@@ -103,7 +103,7 @@ echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━
echo
echo -e "${BLUE}Next steps:${NC}"
echo " • Run './target/release/llmfit' to test the updated binary"
echo " • Check 'data/hf_models.json' for the updated model list"
echo " • Check 'llmfit-core/data/hf_models.json' for the updated model list"
echo " • Example: ./scripts/update_models.sh --threads 8 --gguf-sources"
if [ ! -z "$BACKUP_FILE" ]; then
echo " • Delete backup file if satisfied: rm $BACKUP_FILE"
scripts/validate_generation_scoring.py
+1 -1
View File
@@ -239,7 +239,7 @@ def quality_score_new(model: dict) -> float:
def main():
data_path = "data/hf_models.json"
data_path = "llmfit-core/data/hf_models.json"
with open(data_path) as f:
models = json.load(f)
version.txt
+1 -1
View File
@@ -1 +1 @@
0.9.29
0.9.38