How mohinvinayak/arbiter works
1. Repository shape
- This is a two-tier monorepo: a React/Vite frontend in frontend/ and a FastAPI backend in backend/.
- The product is an LLM evaluation platform: users define suites, choose models, run evaluations, and inspect historical results.
2. Frontend boundary
- The frontend is the user-facing control plane, built from frontend/src/App.jsx, main.jsx, and styling/assets.
- It is responsible for suite editing, model selection, settings, run history, and displaying evaluation results/graphs.
- It does not own business logic for evaluation; it sends requests to the backend API.
3. BYOK key handling
- A key architectural boundary is Bring Your Own Key: API keys are stored in the browser’s localStorage, not in the server database.
- When an evaluation is started, the frontend injects those keys into request headers so the backend can call the model providers.
- The backend can also use fallback credentials from environment configuration when present.
4. Backend API layer
- The FastAPI entry point is backend/app/main.py.
- API surface is split into route modules:
- routes/runs.py for evaluation execution and run history
- routes/suites.py for CRUD on test suites
- routes/settings.py for settings/model-related configuration
- This separation suggests a clean boundary between transport concerns and evaluation/business logic.
5. Persistence layer
- Data is modeled with SQLAlchemy-style entities in backend/app/models/:
- run.py for evaluation run records
- test_suite.py for test suite definitions
- settings.py for persisted app configuration
- backend/app/database.py likely centralizes engine/session setup.
- The README indicates SQLite locally and PostgreSQL in production, so the persistence layer is portable across those environments.
6. Evaluation orchestration
- The core evaluation pipeline lives in backend/app/services/.
- llm_runner.py appears to orchestrate concurrent model calls across providers.
- evaluator.py applies the scoring logic after model outputs are collected.
- This makes the backend a pipeline: suite definition → model inference → scoring → persisted run result.
7. Three-tier scoring model
- The architecture supports three distinct evaluation modes:
- Deterministic checks: exact match, JSON compliance, length rules, regex rules
- Semantic similarity: embedding-based scoring using sentence-transformers
- LLM-as-a-judge: one model is selected to critique and score other outputs
- This is important for diagramming because scoring is not a single step; it is a layered decision engine.
8. Model/provider routing
- backend/app/utils/keys.py and backend/app/config.py likely manage provider-specific key lookup and environment-driven defaults.
- The README names multiple provider families: Gemini, Groq/LLaMA, OpenAI, Anthropic, Deepseek, Mistral, OpenRouter.
- The backend’s model list is therefore dynamic: available models depend on which provider keys are present.
9. Data returned to the UI
- Evaluation runs are not just pass/fail summaries; the product emphasizes raw inference logging.
- That means the backend likely stores or returns per-model, per-test-case outputs alongside aggregate scores.
- The UI can then show detailed traces, judge reasoning, and score comparisons for each run.
10. Suite management workflow
- Suites are first-class objects and have full CRUD through the API.
- A suite likely contains prompts, expected outputs, and deterministic validation rules that are later consumed by the evaluator.
- This is the user’s main authoring boundary: they define tests once, then reuse them across many model sets and judge choices.
11. Run history workflow
- Runs are persisted and later listed through GET /api/runs.
- This creates a historical analysis loop: execute evaluation, store artifacts, revisit earlier runs, compare outcomes, and inspect raw logs.
- The run model is therefore the audit/history backbone of the system.
12. Deployment and runtime boundaries
- The repo is designed for split deployment:
- Frontend on Vercel
- Backend and DB on Railway
- Local development uses Docker Compose at the repo root, plus a backend Dockerfile and a frontend Dockerfile.
- The backend’s Railway config and the frontend’s Vercel config reflect that this is meant to run as separate deployable units.
13. Testing and quality
- There is a small backend test surface in backend/tests/test_app.py, suggesting API-level smoke testing.
- Frontend build/lint tooling is standard Vite/React tooling, while backend dependencies are managed via requirements.txt.
14. How to draw the architecture
- Draw three main zones:
1. Browser/UI: suite editor, settings, model picker, run history, results viewer
2. FastAPI backend: routes → services → models/database
3. External model providers + DB: provider APIs for inference, SQLite/PostgreSQL for persistence
- The critical flows are:
- UI stores keys locally and sends them on demand
- UI posts suite/model/judge selections to backend
- backend fans out to multiple model APIs
- evaluator scores outputs
- run artifacts are written to the database and later read back by the UI
Main entry points: App.jsx App · llm_runner.py LLM runner · runs.py Runs routes · settings.py Settings routes
Components 19
Browser UI
React/Vite control plane for authoring suites, starting runs, and reviewing results.
- App · Top-level UI composition and client-side workflow entrypoint.
frontend/src/App.jsx - Main · Mounts the app and initializes the frontend runtime.
frontend/src/main.jsx - Browser state · Holds user-managed API keys and settings in the browser.
frontend/src/index.css - Assets · Visual assets used by the interface and results views.
frontend/src/assets
FastAPI Backend
HTTP API, orchestration, and persistence layer for evaluations.
- API app · Application startup and API wiring for the backend service.
backend/app/main.py - Config · Environment-driven defaults, provider settings, and deployment configuration.
backend/app/config.py - Database · Engine and session management for app persistence.
backend/app/database.py - Runs routes · Execution endpoints and historical run retrieval.
backend/app/routes/runs.py - Suites routes · CRUD endpoints for test suite authoring and management.
backend/app/routes/suites.py - Settings routes · Endpoints for model and application settings.
backend/app/routes/settings.py - LLM runner · Fans out requests to model providers and collects outputs.
backend/app/services/llm_runner.py - Evaluator · Applies deterministic, semantic, and judge-based scoring to model outputs.
backend/app/services/evaluator.py - Run record · Persisted evaluation run metadata and artifacts.
backend/app/models/run.py - Suite model · Stored test suite definitions consumed by the evaluator.
backend/app/models/test_suite.py - Settings model · Persisted configuration for model selection and app preferences.
backend/app/models/settings.py - Key utils · Resolves provider keys from request headers and fallback environment values.
backend/app/utils/keys.py - Settings schema · API contract for settings payloads.
backend/app/schemas/settings.py
External Systems
Model providers and database systems used at runtime.
- Model APIs · External LLM providers such as OpenAI, Anthropic, Gemini, Groq, Deepseek, Mistral, and OpenRouter.
- SQLite/Postgres · Local SQLite and production PostgreSQL storage for runs, suites, and settings.
Connections 22
- Main → App: mounts · Bootstraps the React application.
- App → Browser state: reads keys · Uses browser-stored BYOK credentials and user settings.
- App → Suites routes: suite CRUD · Creates, edits, and loads test suites through the API.
- App → Settings routes: settings · Loads and updates model/app configuration.
- App → Runs routes: run history · Starts evaluations and fetches historical runs/results.
- App → Assets: renders · Uses assets in the UI.
- API app → Runs routes: registers · Wires run endpoints into the FastAPI app.
- API app → Suites routes: registers · Wires suite endpoints into the FastAPI app.
- API app → Settings routes: registers · Wires settings endpoints into the FastAPI app.
- Runs routes → LLM runner: executes · Dispatches model calls for evaluation runs.
- Runs routes → Run record: persists · Stores run records and artifacts.
- Suites routes → Suite model: stores · Persists suite definitions.
- Settings routes → Settings schema: validates · Uses API schemas for settings payloads.
- Settings routes → Settings model: persists · Stores application configuration.
- LLM runner → Key utils: auth keys · Resolves provider credentials for model calls.
- LLM runner → Model APIs: calls · Invokes external model providers.
- LLM runner → Evaluator: hands off · Passes collected outputs to scoring.
- Evaluator → Run record: writes results · Attaches scores and traces to run artifacts.
- Database → SQLite/Postgres: connects · Backs persistence with SQLite or PostgreSQL.
- Run record → Database: via session · Uses the shared database layer.
- Suite model → Database: via session · Uses the shared database layer.
- Settings model → Database: via session · Uses the shared database layer.