# mohinvinayak/arbiter architecture

> Architecture diagram of the GitHub repository mohinvinayak/arbiter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mohinvinayak/arbiter
- Repository: https://github.com/mohinvinayak/arbiter

## Overview

1. **Repository shape**
   - This is a two-tier monorepo: a **React/Vite frontend** in `frontend/` and a **FastAPI backend** in `backend/`.
   - The product is an **LLM evaluation platform**: users define suites, choose models, run evaluations, and inspect historical results.

2. **Frontend boundary**
   - The frontend is the user-facing control plane, built from `frontend/src/App.jsx`, `main.jsx`, and styling/assets.
   - It is responsible for suite editing, model selection, settings, run history, and displaying evaluation results/graphs.
   - It does not own business logic for evaluation; it sends requests to the backend API.

3. **BYOK key handling**
   - A key architectural boundary is **Bring Your Own Key**: API keys are stored in the browser’s `localStorage`, not in the server database.
   - When an evaluation is started, the frontend injects those keys into request headers so the backend can call the model providers.
   - The backend can also use fallback credentials from environment configuration when present.

4. **Backend API layer**
   - The FastAPI entry point is `backend/app/main.py`.
   - API surface is split into route modules:
     - `routes/runs.py` for evaluation execution and run history
     - `routes/suites.py` for CRUD on test suites
     - `routes/settings.py` for settings/model-related configuration
   - This separation suggests a clean boundary between transport concerns and evaluation/business logic.

5. **Persistence layer**
   - Data is modeled with SQLAlchemy-style entities in `backend/app/models/`:
     - `run.py` for evaluation run records
     - `test_suite.py` for test suite definitions
     - `settings.py` for persisted app configuration
   - `backend/app/database.py` likely centralizes engine/session setup.
   - The README indicates **SQLite locally** and **PostgreSQL in production**, so the persistence layer is portable across those environments.

6. **Evaluation orchestration**
   - The core evaluation pipeline lives in `backend/app/services/`.
   - `llm_runner.py` appears to orchestrate concurrent model calls across providers.
   - `evaluator.py` applies the scoring logic after model outputs are collected.
   - This makes the backend a pipeline: **suite definition → model inference → scoring → persisted run result**.

7. **Three-tier scoring model**
   - The architecture supports three distinct evaluation modes:
     - **Deterministic checks**: exact match, JSON compliance, length rules, regex rules
     - **Semantic similarity**: embedding-based scoring using `sentence-transformers`
     - **LLM-as-a-judge**: one model is selected to critique and score other outputs
   - This is important for diagramming because scoring is not a single step; it is a layered decision engine.

8. **Model/provider routing**
   - `backend/app/utils/keys.py` and `backend/app/config.py` likely manage provider-specific key lookup and environment-driven defaults.
   - The README names multiple provider families: Gemini, Groq/LLaMA, OpenAI, Anthropic, Deepseek, Mistral, OpenRouter.
   - The backend’s model list is therefore dynamic: available models depend on which provider keys are present.

9. **Data returned to the UI**
   - Evaluation runs are not just pass/fail summaries; the product emphasizes **raw inference logging**.
   - That means the backend likely stores or returns per-model, per-test-case outputs alongside aggregate scores.
   - The UI can then show detailed traces, judge reasoning, and score comparisons for each run.

10. **Suite management workflow**
    - Suites are first-class objects and have full CRUD through the API.
    - A suite likely contains prompts, expected outputs, and deterministic validation rules that are later consumed by the evaluator.
    - This is the user’s main authoring boundary: they define tests once, then reuse them across many model sets and judge choices.

11. **Run history workflow**
    - Runs are persisted and later listed through `GET /api/runs`.
    - This creates a historical analysis loop: execute evaluation, store artifacts, revisit earlier runs, compare outcomes, and inspect raw logs.
    - The `run` model is therefore the audit/history backbone of the system.

12. **Deployment and runtime boundaries**
    - The repo is designed for split deployment:
      - **Frontend on Vercel**
      - **Backend and DB on Railway**
    - Local development uses **Docker Compose** at the repo root, plus a backend `Dockerfile` and a frontend `Dockerfile`.
    - The backend’s Railway config and the frontend’s Vercel config reflect that this is meant to run as separate deployable units.

13. **Testing and quality**
    - There is a small backend test surface in `backend/tests/test_app.py`, suggesting API-level smoke testing.
    - Frontend build/lint tooling is standard Vite/React tooling, while backend dependencies are managed via `requirements.txt`.

14. **How to draw the architecture**
    - Draw three main zones:
      1. **Browser/UI**: suite editor, settings, model picker, run history, results viewer
      2. **FastAPI backend**: routes → services → models/database
      3. **External model providers + DB**: provider APIs for inference, SQLite/PostgreSQL for persistence
    - The critical flows are:
      - UI stores keys locally and sends them on demand
      - UI posts suite/model/judge selections to backend
      - backend fans out to multiple model APIs
      - evaluator scores outputs
      - run artifacts are written to the database and later read back by the UI

Main entry points: [`frontend/src/App.jsx`](https://github.com/mohinvinayak/arbiter/blob/main/frontend/src/App.jsx) (App), [`backend/app/services/llm_runner.py`](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/services/llm_runner.py) (LLM runner), [`backend/app/routes/runs.py`](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/routes/runs.py) (Runs routes), [`backend/app/routes/settings.py`](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/routes/settings.py) (Settings routes)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["Browser UI"]
  node_frontend_app["App<br/>React shell<br/>[App.jsx]"]
  node_frontend_main["Main<br/>React bootstrap<br/>[main.jsx]"]
  node_frontend_state["Browser state<br/>LocalStorage BYOK<br/>[index.css]"]
  node_frontend_assets["Assets"]
end

subgraph group_api["FastAPI Backend"]
  node_backend_main["API app<br/>FastAPI entry<br/>[main.py]"]
  node_backend_config["Config<br/>[config.py]"]
  node_backend_db[("Database<br/>SQLAlchemy setup<br/>[database.py]")]
  node_routes_runs["Runs routes<br/>API routes<br/>[runs.py]"]
  node_routes_suites["Suites routes<br/>API routes<br/>[suites.py]"]
  node_routes_settings["Settings routes<br/>API routes<br/>[settings.py]"]
  node_service_runner["LLM runner<br/>Inference orchestration<br/>[llm_runner.py]"]
  node_service_eval["Evaluator<br/>Scoring pipeline<br/>[evaluator.py]"]
  node_model_runs[("Run record<br/>Run model<br/>[run.py]")]
  node_model_suites[("Suite model<br/>[test_suite.py]")]
  node_model_settings[("Settings model<br/>App model<br/>[settings.py]")]
  node_utils_keys["Key utils<br/>Key routing<br/>[keys.py]"]
  node_schemas_settings["Settings schema<br/>Pydantic schema<br/>[settings.py]"]
end

subgraph group_external["External Systems"]
  node_provider_apis{{"Model APIs<br/>Provider endpoints"}}
  node_sql_db[("SQLite/Postgres<br/>Persistent store")]
end

node_frontend_main -->|"mounts"| node_frontend_app
node_frontend_app -->|"reads keys"| node_frontend_state
node_frontend_app -->|"suite CRUD"| node_routes_suites
node_frontend_app -->|"settings"| node_routes_settings
node_frontend_app -->|"run history"| node_routes_runs
node_frontend_app -.->|"renders"| node_frontend_assets
node_backend_main -->|"registers"| node_routes_runs
node_backend_main -->|"registers"| node_routes_suites
node_backend_main -->|"registers"| node_routes_settings
node_routes_runs -->|"executes"| node_service_runner
node_routes_runs -->|"persists"| node_model_runs
node_routes_suites -->|"stores"| node_model_suites
node_routes_settings -->|"validates"| node_schemas_settings
node_routes_settings -->|"persists"| node_model_settings
node_service_runner -->|"auth keys"| node_utils_keys
node_service_runner -->|"calls"| node_provider_apis
node_service_runner -->|"hands off"| node_service_eval
node_service_eval -->|"writes results"| node_model_runs
node_backend_db -->|"connects"| node_sql_db
node_model_runs -.->|"via session"| node_backend_db
node_model_suites -.->|"via session"| node_backend_db
node_model_settings -.->|"via session"| node_backend_db

click node_frontend_app "https://github.com/mohinvinayak/arbiter/blob/main/frontend/src/App.jsx"
click node_frontend_main "https://github.com/mohinvinayak/arbiter/blob/main/frontend/src/main.jsx"
click node_frontend_state "https://github.com/mohinvinayak/arbiter/blob/main/frontend/src/index.css"
click node_frontend_assets "https://github.com/mohinvinayak/arbiter/tree/main/frontend/src/assets"
click node_backend_main "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/main.py"
click node_backend_config "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/config.py"
click node_backend_db "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/database.py"
click node_routes_runs "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/routes/runs.py"
click node_routes_suites "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/routes/suites.py"
click node_routes_settings "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/routes/settings.py"
click node_service_runner "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/services/llm_runner.py"
click node_service_eval "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/services/evaluator.py"
click node_model_runs "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/models/run.py"
click node_model_suites "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/models/test_suite.py"
click node_model_settings "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/models/settings.py"
click node_utils_keys "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/utils/keys.py"
click node_schemas_settings "https://github.com/mohinvinayak/arbiter/blob/main/backend/app/schemas/settings.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend_app,node_frontend_main,node_frontend_state,node_frontend_assets toneBlue
class node_backend_main,node_backend_config,node_backend_db,node_routes_runs,node_routes_suites,node_routes_settings,node_service_runner,node_service_eval,node_model_runs,node_model_suites,node_model_settings,node_utils_keys,node_schemas_settings toneAmber
class node_provider_apis,node_sql_db toneMint
```

## Components

### Browser UI

React/Vite control plane for authoring suites, starting runs, and reviewing results.

- [App](https://github.com/mohinvinayak/arbiter/blob/main/frontend/src/App.jsx): React shell · Top-level UI composition and client-side workflow entrypoint. · `frontend/src/App.jsx`
- [Main](https://github.com/mohinvinayak/arbiter/blob/main/frontend/src/main.jsx): React bootstrap · Mounts the app and initializes the frontend runtime. · `frontend/src/main.jsx`
- [Browser state](https://github.com/mohinvinayak/arbiter/blob/main/frontend/src/index.css): LocalStorage BYOK · Holds user-managed API keys and settings in the browser. · `frontend/src/index.css`
- [Assets](https://github.com/mohinvinayak/arbiter/tree/main/frontend/src/assets): UI assets · Visual assets used by the interface and results views. · `frontend/src/assets`

### FastAPI Backend

HTTP API, orchestration, and persistence layer for evaluations.

- [API app](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/main.py): FastAPI entry · Application startup and API wiring for the backend service. · `backend/app/main.py`
- [Config](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/config.py): env config · Environment-driven defaults, provider settings, and deployment configuration. · `backend/app/config.py`
- [Database](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/database.py): SQLAlchemy setup · Engine and session management for app persistence. · `backend/app/database.py`
- [Runs routes](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/routes/runs.py): API routes · Execution endpoints and historical run retrieval. · `backend/app/routes/runs.py`
- [Suites routes](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/routes/suites.py): API routes · CRUD endpoints for test suite authoring and management. · `backend/app/routes/suites.py`
- [Settings routes](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/routes/settings.py): API routes · Endpoints for model and application settings. · `backend/app/routes/settings.py`
- [LLM runner](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/services/llm_runner.py): Inference orchestration · Fans out requests to model providers and collects outputs. · `backend/app/services/llm_runner.py`
- [Evaluator](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/services/evaluator.py): Scoring pipeline · Applies deterministic, semantic, and judge-based scoring to model outputs. · `backend/app/services/evaluator.py`
- [Run record](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/models/run.py): Run model · Persisted evaluation run metadata and artifacts. · `backend/app/models/run.py`
- [Suite model](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/models/test_suite.py): Suite model · Stored test suite definitions consumed by the evaluator. · `backend/app/models/test_suite.py`
- [Settings model](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/models/settings.py): App model · Persisted configuration for model selection and app preferences. · `backend/app/models/settings.py`
- [Key utils](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/utils/keys.py): Key routing · Resolves provider keys from request headers and fallback environment values. · `backend/app/utils/keys.py`
- [Settings schema](https://github.com/mohinvinayak/arbiter/blob/main/backend/app/schemas/settings.py): Pydantic schema · API contract for settings payloads. · `backend/app/schemas/settings.py`

### External Systems

Model providers and database systems used at runtime.

- **Model APIs**: Provider endpoints · External LLM providers such as OpenAI, Anthropic, Gemini, Groq, Deepseek, Mistral, and OpenRouter.
- **SQLite/Postgres**: Persistent store · Local SQLite and production PostgreSQL storage for runs, suites, and settings.

## Connections

- Main → App: mounts · Bootstraps the React application.
- App → Browser state: reads keys · Uses browser-stored BYOK credentials and user settings.
- App → Suites routes: suite CRUD · Creates, edits, and loads test suites through the API.
- App → Settings routes: settings · Loads and updates model/app configuration.
- App → Runs routes: run history · Starts evaluations and fetches historical runs/results.
- App → Assets: renders · Uses assets in the UI.
- API app → Runs routes: registers · Wires run endpoints into the FastAPI app.
- API app → Suites routes: registers · Wires suite endpoints into the FastAPI app.
- API app → Settings routes: registers · Wires settings endpoints into the FastAPI app.
- Runs routes → LLM runner: executes · Dispatches model calls for evaluation runs.
- Runs routes → Run record: persists · Stores run records and artifacts.
- Suites routes → Suite model: stores · Persists suite definitions.
- Settings routes → Settings schema: validates · Uses API schemas for settings payloads.
- Settings routes → Settings model: persists · Stores application configuration.
- LLM runner → Key utils: auth keys · Resolves provider credentials for model calls.
- LLM runner → Model APIs: calls · Invokes external model providers.
- LLM runner → Evaluator: hands off · Passes collected outputs to scoring.
- Evaluator → Run record: writes results · Attaches scores and traces to run artifacts.
- Database → SQLite/Postgres: connects · Backs persistence with SQLite or PostgreSQL.
- Run record → Database: via session · Uses the shared database layer.
- Suite model → Database: via session · Uses the shared database layer.
- Settings model → Database: via session · Uses the shared database layer.