How alexsjones/llmfit works
1. What this repo is
- llmfit is a multi-surface Rust application for recommending LLMs for a machine’s hardware.
- The same recommendation engine is exposed through several front ends: terminal TUI, classic CLI, HTTP API, background web dashboard, and a Tauri desktop app.
2. Core architecture boundary
- llmfit-core is the central engine and is the main architectural boundary.
- It owns model metadata, hardware detection, fit/scoring logic, provider integration, planning, benchmark handling, and update logic.
- Every user-facing surface delegates to this crate rather than re-implementing model selection rules.
3. Primary data flow
- Startup path is: detect hardware → load embedded model database → compute fit/speed/quality scores → rank models → render in UI/API output.
- User actions such as filters, sort changes, hardware simulation, and advanced scoring changes all re-run the same core scoring pipeline against the current hardware state.
4. Model knowledge layer
- The model catalog is a generated JSON dataset (data/hf_models.json, plus related baseline/cache files) that is embedded into the binary at build time.
- A model update pipeline in scripts/ scrapes Hugging Face and enriches entries with GGUF/provider metadata before regenerating the compiled-in database.
- This means end users consume a static, versioned model database, while contributors refresh it offline.
5. Hardware detection and estimation
- hardware.rs detects RAM, CPU cores, GPU presence, GPU memory, and backend type.
- The engine supports multiple hardware families and uses that state to estimate whether a model fits and how fast it will run.
- Hardware overrides and simulation are first-class inputs, so the recommendation engine can be run against synthetic targets as well as the real machine.
6. Scoring / planning pipeline
- fit.rs, quality.rs, and plan.rs implement the main decision logic.
- Models are scored across fit, speed, quality, and context dimensions, then ranked with use-case-specific weights.
- The planning path inverts the analysis: given a model and target throughput/context, it estimates required hardware and upgrade deltas.
7. Runtime/provider integration
- providers.rs is the adapter layer to local model runtimes and download sources.
- It integrates with Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio for both installed-model detection and downloads.
- Provider status feeds back into the table/UI, so “installed” and “download” actions are based on live runtime state, not just the static model catalog.
8. Terminal product surface
- llmfit-tui is the main interactive app and uses ratatui + crossterm.
- It owns TUI state, keyboard/event handling, filters, themeing, history, comparison views, download manager, and the in-app benchmark/leaderboard/plan views.
- The TUI is essentially a stateful presentation layer over the core engine.
9. CLI and HTTP API surface
- The same core logic is exposed in classic CLI mode via table/JSON output, and in serve mode as an HTTP API for automation/schedulers.
- The API is meant for machine consumption: system info, filtered model lists, top recommendations, and planning data all come from the same scoring path used by the TUI.
- This creates a clear separation between computation in llmfit-core and delivery formats in the front ends.
10. Web dashboard
- The repo includes a separate web client in llmfit-web built with React/Vite.
- It mirrors the model/system views of the terminal app and consumes the API layer rather than owning recommendation logic itself.
- In architecture terms, it is a thin browser UI over the same backend data.
11. Desktop packaging
- llmfit-desktop wraps the application in Tauri with a native shell and a small JS UI.
- This is another presentation boundary, not a separate engine: the desktop app reuses the same core model/hardware/provider behavior through the Rust backend.
12. Benchmarking and external data
- bench.rs and related files support live inference benchmarking against local providers, while the community leaderboard pulls real-world measurements from the external localmaxxing.com API.
- These are distinct from the static model-fit estimator: one measures actual runtime behavior, the other queries crowd-sourced benchmark data.
- Cached benchmark files under data/ and llmfit-core/data/ reduce repeated network calls and allow faster startup.
13. Build, release, and packaging pipeline
- The repo is heavily automated with GitHub Actions, Docker, and release-please.
- Rust artifacts are produced for CLI/TUI/core; the Python package in llmfit-python provides a separate distribution surface; Docker images expose JSON/CLI usage; Tauri builds the desktop app.
- This is a multi-distribution project with one shared recommendation engine and several delivery targets.
14. Auxiliary integrations
- The skills/llmfit-advisor directory packages an OpenClaw skill that calls the CLI as an external advisor.
- scripts/ also contains verification and scraping utilities, which are part of the maintenance/data-refresh boundary rather than runtime code.
- These pieces reinforce that llmfit is both an application and a model-data pipeline.
Main entry points: lib.rs llmfit-core · serve_api.rs CLI/Serve · scrape_hf_models.py Scripts · hf_models.json Model data
Components 21
Core engine
Shared recommendation logic and model knowledge.
- llmfit-core · Central recommendation engine shared by every surface.
llmfit-core/src/lib.rs - Models · Embedded model metadata and dataset loading.
llmfit-core/src/models.rs - Hardware · Detects RAM, CPU, GPU, backend, and override/simulated state.
llmfit-core/src/hardware.rs - Fit scoring · Computes fit and speed scores against current hardware.
llmfit-core/src/fit.rs - Quality · Adds quality and context-oriented scoring signals.
llmfit-core/src/quality.rs - Planning · Inverts scoring to estimate required hardware and upgrade deltas.
llmfit-core/src/plan.rs - Providers · Integrates Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio.
llmfit-core/src/providers.rs - Benchmarking · Live inference benchmarking and leaderboard support.
llmfit-core/src/bench.rs - Updates · Model refresh and update logic for the embedded database.
llmfit-core/src/update.rs
User surfaces
Interfaces that present the same engine in different forms.
- llmfit-tui · Interactive TUI shell over the shared engine.
llmfit-tui/src/tui_app.rs - CLI/Serve · Table/JSON output and HTTP API mode for automation.
llmfit-tui/src/serve_api.rs - MCP server · External assistant-facing control surface.
llmfit-tui/src/mcp_server.rs - llmfit-web · React/Vite dashboard that consumes the HTTP API.
llmfit-web/src/App.jsx - llmfit-desktop · Native desktop shell around the shared Rust backend.
llmfit-desktop/src/main.rs - llmfit-python · Separate Python distribution surface for the same toolchain.
llmfit-python/src/llmfit/__init__.py
Data pipeline
Generated model datasets, benchmarks, and refresh tooling.
- Model data · Generated Hugging Face, benchmark, and baseline JSON/YAML inputs.
data/hf_models.json - Core data · Crate-local datasets used by the engine and startup cache paths.
llmfit-core/data/hf_models.json - Scripts · Scrapers, validators, and model-generation utilities.
scripts/scrape_hf_models.py - Community data · Crowd-sourced benchmark source used for leaderboard views.
llmfit-core/src/benchmarks.rs
Distribution
Packaging and release targets around the shared Rust workspace.
- Docker/Release · Container, release-please, and CI automation for multiple targets.
Dockerfile - Advisor skill · OpenClaw skill that shells out to the CLI as an advisor.
skills/llmfit-advisor/SKILL.md
Connections 21
- llmfit-tui → llmfit-core: uses engine · Interactive presentation delegates recommendation work to the core crate.
- CLI/Serve → llmfit-core: queries · CLI and HTTP API reuse the same scoring and planning pipeline.
- MCP server → llmfit-core: advises via core · Assistant integration accesses the same shared backend logic.
- llmfit-web → CLI/Serve: calls API · Browser UI consumes the serve mode rather than reimplementing logic.
- llmfit-desktop → llmfit-core: backs UI · Tauri shell wraps the Rust engine and presentation bridge.
- llmfit-python → llmfit-core: wraps core · Python distribution mirrors the same engine as a separate package.
- llmfit-core → Models: loads catalog · Engine starts by loading embedded model metadata.
- llmfit-core → Hardware: detects target · Hardware state is the main input to fit and planning.
- llmfit-core → Fit scoring: ranks fit · Fit scoring drives ranked recommendations.
- llmfit-core → Quality: weights quality · Quality signals contribute to overall ranking.
- llmfit-core → Planning: inverts analysis · Planning estimates hardware needs from a target model and throughput.
- llmfit-core → Providers: checks runtimes · Runtime integration informs installed/download state.
- llmfit-core → Benchmarking: benchmarks · Live benchmark support feeds benchmark and leaderboard views.
- llmfit-core → Updates: refreshes data · Update logic regenerates or refreshes the embedded database.
- Scripts → Model data: regenerates · Scrapers and validators produce the versioned model datasets.
- Scripts → Core data: updates caches · Maintenance scripts refresh crate-local cache and baseline files.
- Model data → llmfit-core: embedded into · Generated datasets are compiled into the application.
- Core data → llmfit-core: embedded into · Core crate consumes local cached datasets at build/runtime.
- Community data → Benchmarking: supplies results · External benchmark source powers community leaderboard views.
- Docker/Release → CLI/Serve: packages · Container and release automation target the CLI/API surface.
- Advisor skill → CLI/Serve: invokes · The advisor skill shells out to the CLI as an external tool.