How ldb0071/lemonmatrix works
LemonMatrix is a local benchmarking control plane for Lemonade server instances. Users discover or register an instance as a hardware/environment profile, select a benchmark, run trials through the Lemonade API, validate and save results, then inspect them locally or submit them to the public leaderboard. It also supports router-cost and Omni workflows, importing third-party results, and a local dashboard. The CLI, README, and selected modules establish these connections; several benchmark internals and dashboard routes were not sampled, so their internal call paths are not expanded.
Main entry points: cli.py CLI · sweep_batch.py Sweep batches · client.py API client · webapp.py Local dashboard
Components 18
User interfaces
- CLI
src/lemonmatrix/cli.py - Local dashboard
src/lemonmatrix/webapp.py
Instance profiles
- Instance discovery
src/lemonmatrix/discover.py - Profile management
src/lemonmatrix/profile.py - Host capabilities
src/lemonmatrix/capabilities.py
Benchmark execution
- API client
src/lemonmatrix/client.py - Benchmark runs
src/lemonmatrix/bench.py - Omni runs
src/lemonmatrix/omni.py - Prompt sets
src/lemonmatrix/prompt_sets.py - Router cost
src/lemonmatrix/routing_cost.py - Sweep batches
src/lemonmatrix/sweep_batch.py
Results and exchange
- Sweep state
src/lemonmatrix/sweep_store.py - Result store
src/lemonmatrix/results_store.py - Schema validation
src/lemonmatrix/validate.py - Result import
src/lemonmatrix/importers.py - Leaderboard submission
src/lemonmatrix/submit.py
Other components
- Operator
- Lemonade server
Connections 27
- Operator → CLI: runs commands
README.md - Operator → Local dashboard: uses dashboard
README.md - CLI → Instance discovery: detects instances
README.md - CLI → Profile management: manages profiles
README.md - Profile management → API client: queries server
src/lemonmatrix/profile.py - CLI → Host capabilities: checks options
src/lemonmatrix/cli.py - Host capabilities → Profile management: uses environment
src/lemonmatrix/capabilities.py - CLI → Benchmark runs: runs benchmarks
README.md - CLI → Omni runs: runs Omni tasks
README.md - Omni runs → API client: sends chat requests
src/lemonmatrix/omni.py - Omni runs → Prompt sets: uses task sets
- Benchmark runs → API client: sends benchmark requests
- CLI → Router cost: calculates routing cost
src/lemonmatrix/cli.py - CLI → Sweep batches: starts sweeps
src/lemonmatrix/cli.py - Sweep batches → Benchmark runs: runs each configuration
src/lemonmatrix/sweep_batch.py - Sweep batches → API client: uses server client
src/lemonmatrix/sweep_batch.py - Sweep batches → Schema validation: validates results
src/lemonmatrix/sweep_batch.py - Sweep batches → Result store: saves trials
src/lemonmatrix/sweep_batch.py - Local dashboard → Sweep batches: launches sweeps
src/lemonmatrix/webapp.py - Local dashboard → Sweep state: persists sweep state
src/lemonmatrix/webapp.py - Local dashboard → Result store: reads results
src/lemonmatrix/webapp.py - CLI → Result store: lists and saves results
src/lemonmatrix/cli.py - CLI → Schema validation: validates submissions
README.md - CLI → Result import: imports external results
README.md - CLI → Leaderboard submission: submits result files
README.md - API client → Lemonade server: calls REST API
src/lemonmatrix/client.py - Instance discovery → Lemonade server: probes liveness
src/lemonmatrix/discover.py