alexsjones--llmfit/AGENTS.md

AGENTS.md

Instructions for AI agents contributing to this codebase.

---

Project overview

llmfit is a Rust CLI/TUI tool that matches LLM models against local system hardware (RAM, CPU, GPU). It detects system specs, loads a model database from embedded JSON, scores each model's fit, and presents results in an interactive terminal UI or classic table output.

Language and toolchain

• Rust, edition 2024.
• Build with cargo build. Run with cargo run.
• No nightly features required. Stable toolchain only.
• Minimum supported Rust version: whatever edition 2024 requires (1.85+).

Architecture

llmfit-core/      Shared Rust library. It owns hardware detection, model data,
                 fit analysis, planning, providers, benchmarks, quality checks,
                 model updates, diagnostics, claims, and result sharing.

llmfit-tui/       Main `llmfit` binary. It provides the CLI, ratatui TUI,
                 Axum HTTP API, embedded Web dashboard, and stdio MCP server.
                 `main.rs` parses all clap flags and selects an interface.

llmfit-desktop/   Tauri desktop application. Tauri commands call llmfit-core
                 for hardware detection, fit analysis, and Ollama downloads.

llmfit-web/       React 18 and Vite dashboard. It calls `/api/v1/*` endpoints
                 from llmfit-tui. The llmfit-tui build script embeds `dist/`.
                 This directory is not a Cargo workspace member.

llmfit-python/    Python package wrapper. Its wheel includes the compiled Rust
                 binary. `python -m llmfit` forwards arguments to that binary.
                 It does not expose llmfit-core through a native Python API.

The Cargo workspace contains llmfit-core, llmfit-tui, and llmfit-desktop. The default members are llmfit-core and llmfit-tui.

Source modules in llmfit-core/src/:

• analysis.rs: Builds model-fit results. It marks installed models and applies local, community, and measured benchmark calibration.
• bench.rs: Runs throughput benchmarks against Ollama and OpenAI-compatible endpoints. It also discovers available benchmark targets.
• benchmarks.rs: Loads embedded and remote benchmark data. It builds measured throughput indexes and hardware leaderboard queries.
• claim.rs: Calculates model resource bounds. It renders Kubernetes DRA ResourceClaim and ResourceClaimTemplate manifests.
• doctor.rs: Collects installation, hardware, runtime, and model diagnostics.
• fit.rs: Calculates fit level, run mode, runtime, quantization, score, and estimated throughput.
• hardware.rs: Detects RAM, CPU, GPUs, unified memory, clusters, and memory bandwidth.
• models.rs: Defines model metadata. It loads embedded HF and ONNX catalogs, custom models, and the update cache.
• plan.rs: Estimates memory, throughput, run paths, and hardware upgrade needs for a requested model setup.
• providers.rs: Integrates Ollama, MLX, llama.cpp, Docker Model Runner, LM Studio, vLLM, and RamaLama.
• quality.rs: Runs response quality tests. It scores roles, builds routing recommendations, and compares results with baselines.
• share.rs: Stores local benchmark results. It handles GitHub authentication and submits community benchmark data.
• task_bench.rs: Provides task benchmark scores for model and task pairs.
• update.rs: Fetches model metadata and manages the local model update cache.

Source modules in llmfit-tui/src/:

• main.rs: Owns CLI parsing, hardware overrides, command execution, and interface dispatch.
• display.rs: Renders classic CLI tables, model plans, JSON, and CSV output.
• download_history.rs: Stores persistent model download records.
• events.rs: Publishes optional NATS system events and periodic snapshots.
• filter_config.rs: Loads and saves persistent TUI filter settings.
• mcp_server.rs: Exposes hardware, model, runtime, and planning MCP tools.
• serve_api.rs: Serves the embedded Web dashboard and JSON API with Axum.
• serve_shared.rs: Converts shared core types into API and MCP JSON values.
• theme.rs: Defines TUI color themes and stores the selected theme.
• tui_app.rs: Owns TUI state, model results, filters, downloads, and selection.
• tui_events.rs: Handles crossterm input and mutates TUI state.
• tui_ui.rs: Renders TUI views, tables, details, plans, and popups with ratatui.

Data flow

All interfaces use the same core analysis flow:

1. SystemSpecs::detect() detects CPU, RAM, GPU, unified-memory, and cluster information. CLI hardware overrides can replace detected values.
2. ModelDatabase::new() loads the embedded HF and ONNX catalogs.
3. Custom models replace matching embedded models. The update cache appends models that are not already present.
4. build_model_fits() removes backend-incompatible models. It calls ModelFit::analyze_with_forced_runtime() for each remaining model.
5. Fit analysis selects a runtime, quantization, and run mode. It calculates memory use, throughput, fit level, score components, and notes.
6. Local benchmark results, community results, and measured presets can replace or calibrate formula-based throughput estimates.
7. Each interface applies its own filters, sorting, limits, and presentation.

ModelFit::analyze() is the default analysis wrapper. Use analyze_with_context_limit() for a context cap. Use analyze_with_forced_runtime() for runtime selection. Use analyze_with_config() for custom calculation parameters. These methods share the private analyze_inner() implementation.

Interface-specific flow:

• CLI: main.rs dispatches a subcommand. The command calls llmfit-core and writes a table, JSON, or CSV result.
• TUI: App owns model and filter state. tui_events changes that state. apply_filters() updates visible indices. tui_ui renders the current state.
• Web: React calls /api/v1/*. Axum handlers in serve_api.rs call llmfit-core and return JSON. The same server returns the embedded React assets.
• MCP: LlmfitMcpServer receives stdio tool calls. Each tool calls shared core analysis or planning logic and returns JSON text.
• Desktop: Tauri commands call llmfit-core and serialize results for the desktop UI. Ollama pull state stays in the Tauri application state.
• Python: The Python entry point locates the installed llmfit binary. It then replaces the process on Unix or starts a subprocess on Windows.

Model database

• Source: llmfit-core/data/hf_models.json (33 models).
• Generated by scripts/scrape_hf_models.py (Python, stdlib only, no pip deps).
• Embedded at compile time via include_str!("../data/hf_models.json").
• Schema per entry: name, provider, parameter_count, min_ram_gb, recommended_ram_gb, min_vram_gb, quantization, context_length, use_case.
• min_vram_gb is VRAM needed for GPU inference. min_ram_gb is system RAM needed for CPU inference. Both are derived from the same parameter count.
• RAM formula: params * 0.5 bytes (Q4_K_M) / 1024^3 * 1.2 overhead.
• VRAM formula: params * 0.5 bytes (Q4_K_M) / 1024^3 * 1.1 activation overhead.
• Recommended RAM: model_size * 2.0.

Do not manually edit hf_models.json. Regenerate it by running the scraper:

python3 scripts/scrape_hf_models.py

The scraper has hardcoded fallback entries for gated models that require authentication.

Conventions

• No unsafe code.
• No .unwrap() on user-facing paths. Use proper error handling or expect() with a descriptive message for internal invariants only.
• Fit levels are ordered: Perfect > Good > Marginal > TooTight. Do not add levels without updating rank_models_by_fit() sort logic.
• Fit is VRAM-first. RunMode has five execution paths: Gpu, MoeOffload, CpuOffload, CpuOnly, and TensorParallel.
• Gpu keeps the model in VRAM. MoeOffload keeps active experts in VRAM and inactive experts in RAM. CpuOffload splits work between VRAM and RAM. CpuOnly uses system RAM. TensorParallel distributes work across nodes.
• min_vram_gb is the VRAM needed to load model weights on GPU. min_ram_gb is the system RAM needed for CPU-only inference (same weights, loaded into RAM instead). They represent the same workload on different hardware paths.
• On Apple Silicon (unified memory), VRAM = system RAM. The CpuOffload path is skipped because there is no separate RAM pool to spill to. SystemSpecs::unified_memory tracks this.
• TUI rendering is stateless. tui_ui::draw() must not mutate App. Pass &mut App only for TableState widget requirements -- do not use it to change application state.
• Event handling in tui_events.rs is the sole place that mutates App in the TUI loop.
• Keep display.rs and tui_*.rs independent. The CLI path must work without initializing any TUI state.

Adding a new model to the database

1. Add the model's HuggingFace repo ID to TARGET_MODELS in scripts/scrape_hf_models.py.
2. If the model is gated (requires HF auth), add a fallback entry to the FALLBACK dict in the same script.
3. Run python3 scripts/scrape_hf_models.py.
4. Verify the output in llmfit-core/data/hf_models.json.
5. Run cargo build to verify compilation.

Adding a new filter

1. Add the filter state to App in tui_app.rs.
2. Add filtering logic inside apply_filters().
3. Add the keybinding in tui_events.rs (Normal mode handler).
4. Add the UI widget in tui_ui.rs (draw_search_and_filters() function).
5. Update the status bar help text in draw_status_bar().

Adding a new CLI subcommand

1. Add a variant to the Commands enum in main.rs.
2. Add the match arm in the main() function's command dispatch.
3. Use display.rs functions for output, or add new ones as needed.

Testing

The project has Rust, Web, and Python test suites.

• Rust unit tests live beside code in llmfit-core/src/ and llmfit-tui/src/.
• Core integration tests in llmfit-core/tests/ validate catalog schemas and ONNX model data.
• CLI integration tests in llmfit-tui/tests/ use assert_cmd against the compiled llmfit binary.
• HTTP API tests exercise Axum routers and JSON responses in serve_api.rs.
• TUI tests focus on state transitions, filters, event handling, and render output. Keep production rendering stateless.
• Web tests use Vitest, jsdom, and Testing Library. They cover API query construction, localization, filtering, and dashboard interactions.
• Python tests use pytest. They cover binary discovery, package versioning, and invocation of the packaged Rust binary.

Run the default Rust test set:

cargo test

Run all Rust workspace members, including the desktop crate:

cargo test --workspace

Run one Rust package:

cargo test -p llmfit-core
cargo test -p llmfit

Run the Web tests:

npm --prefix llmfit-web test

Run the Python tests and quality checks:

uv run --project llmfit-python pytest llmfit-python/tests
make -C llmfit-python check

Dependencies policy

• Prefer crates that are well-maintained and have minimal transitive dependencies.
• sysinfo is the system detection crate. Do not replace it with raw platform calls.
• ureq is the blocking HTTP client for providers, benchmarks, updates, quality tests, and sharing. Do not add a second core HTTP client without a concrete need.
• which locates installed runtime binaries. Keep runtime discovery in providers.rs instead of adding manual PATH parsing.
• regex supports response scoring and text parsing. serde_yml parses quality test configuration. base64 encodes benchmark submissions for GitHub.
• objc2-metal reads the effective Metal working-set limit on macOS. Keep it a macOS-only dependency. Do not replace it with raw platform calls.
• ratatui + crossterm is the TUI stack. Do not mix in termion or ncurses.
• clap with derive feature for CLI parsing. Do not use manual arg parsing.
• The Python scraper uses only stdlib (urllib, json). Do not add pip dependencies.

Common tasks

# Build
cargo build

# Run TUI
cargo run

# Run CLI mode
cargo run -- --cli

# Run specific subcommand
cargo run -- system
cargo run -- fit --perfect -n 5
cargo run -- search "llama"

# Refresh model database
python3 scripts/scrape_hf_models.py && cargo build

# Check for compilation issues
cargo check

# Format code
cargo fmt

# Lint
cargo clippy

Platform notes

• GPU detection shells out to nvidia-smi (NVIDIA) and rocm-smi (AMD). These are best-effort and fail silently if unavailable.
• Apple Silicon detection uses system_profiler SPDisplaysDataType. On unified memory Macs, VRAM is reported as available system RAM (same pool).
• sysinfo handles cross-platform RAM/CPU. No conditional compilation needed.
• The TUI uses crossterm which works on Linux, macOS, and Windows terminals.