# mattpocock/evalite architecture

> Architecture diagram of the GitHub repository mattpocock/evalite, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mattpocock/evalite
- Repository: https://github.com/mattpocock/evalite

## Overview

##### Project type and purpose
- This is a TypeScript-native, local-first testing tool (Evalite) for LLM-powered apps.
- It’s organized as a pnpm/Turbo monorepo with:
– A CLI & core library (`packages/evalite`)
– A test harness package (`packages/evalite-tests`)
– Example usage (`packages/example`)
– A React/Vite UI app (`apps/evalite-ui`)
– A static docs site built with Astro (`apps/evalite-docs`)
– CI/CD workflows under `.github/workflows`

##### File structure
- Top-level “apps” and “packages” folders show separate deployable applications and shareable libraries.
- Each “apps” entry has its own build config (Vite/Astro).
- Shared infra: `pnpm-workspace.yaml`, `turbo.json`, TypeScript configs, Husky hooks, GitHub workflows.

##### Key insights from README
- Dev mode concurrently runs: TS type-checking, tests, UI dev server, and the CLI in “watch” mode on example code.
- The CLI („evalite“) invokes AI via the OpenAI API, writes traces locally, and emits updates to the UI over a socket.

##### Main components
a. Developer Machine / CI Agent
– Runs pnpm/Turbo tasks, Vitest, type checks, build jobs
b. Monorepo Workspace
– Core library & CLI (`packages/evalite`)
– Test suite (`packages/evalite-tests`)
– Example project (`packages/example`)
– UI app (`apps/evalite-ui`)
– Docs site (`apps/evalite-docs`)
c. Runtime Services
– OpenAI API (external service)
– Local file-queue storage (for traces)
– WebSocket server (embedded in CLI/server)
d. User Interfaces
– Command-line interface (`evalite` command)
– Web UI (React app subscribing to traces)
– Static documentation site

##### Relationships and data flows
- CLI → OpenAI API: sends prompts, receives responses
- CLI → Local file storage: appends trace records
- CLI/server → WebSocket → UI app: pushes real-time updates
- UI app → SDK & subscription hooks: fetches and renders test results
- Example code under `packages/example` → invoked by CLI in watch mode
- Test harness (`evalite-tests`) → depends on core library, runs Vitest against fixtures

##### Architectural patterns & design principles
- Monorepo with strong package boundaries and internal versioning
- Separation of concerns: CLI orchestration vs. UI presentation vs. docs
- Local-first storage pattern (no external DB)
- Real-time streaming via WebSockets for live feedback
- Static site generation for docs

##### Essential elements
- Clear component labels (e.g., “Evalite CLI”, “OpenAI API”, “Local Trace Store”)
- A note on CI/CD flows invoking tests and builds via GitHub Actions

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Developer and CI
    subgraph "Developer / CI Environment"
        DM["Developer Machine"]:::dev
        CI["CI Agent"]:::ci
        DM -->|"pnpm/Turbo tasks"| Monorepo
        CI -->|"GitHub Actions"| Monorepo
    end

    %% Monorepo Workspace
    subgraph "Monorepo Workspace"
        direction TB
        subgraph "Packages" 
            direction TB
            CLI["Evalite CLI & SDK"]:::package
            TH["Test Harness (Vitest)"]:::package
            EX["Example Project"]:::package
        end
        subgraph "Apps"
            direction TB
            UI["Evalite UI App"]:::app
            DOC["Evalite Docs Site"]:::app
        end
        Monorepo(("pnpm-workspace.yaml<br>turbo.json<br>tooling configs")):::infra
    end

    Monorepo --> CLI
    Monorepo --> TH
    Monorepo --> EX
    Monorepo --> UI
    Monorepo --> DOC

    %% Runtime Services
    subgraph "Runtime Services"
        direction TB
        OA["OpenAI API"]:::external
        FS["Local Trace Store"]:::datastore
        WS["WebSocket Server"]:::external
    end

    %% Data Flows
    CLI -->|HTTP| OA
    CLI -->|write trace.json| FS
    CLI -->|WS publish| WS
    WS -->|WS subscribe| UI
    UI -->|SDK & hooks| CLI
    EX -->|watch run| CLI
    TH -->|runs tests via SDK| CLI

    DOC -.->|static site| Monorepo

    %% CI/CD Workflows
    subgraph "GitHub Workflows"
        CI1["ci.yml"]:::workflow
        CI2["preview.yml"]:::workflow
        CI3["publish.yml"]:::workflow
    end
    CI --> CI1
    CI --> CI2
    CI --> CI3

    %% Click Events
    click CLI "https://github.com/mattpocock/evalite/tree/main/packages/evalite"
    click TH "https://github.com/mattpocock/evalite/tree/main/packages/evalite-tests"
    click EX "https://github.com/mattpocock/evalite/tree/main/packages/example"
    click UI "https://github.com/mattpocock/evalite/tree/main/apps/evalite-ui"
    click DOC "https://github.com/mattpocock/evalite/tree/main/apps/evalite-docs"
    click CI1 "https://github.com/mattpocock/evalite/blob/main/.github/workflows/ci.yml"
    click CI2 "https://github.com/mattpocock/evalite/blob/main/.github/workflows/preview.yml"
    click CI3 "https://github.com/mattpocock/evalite/blob/main/.github/workflows/publish.yml"
    click CLI "https://github.com/mattpocock/evalite/blob/main/packages/evalite/src/bin.ts"
    click CLI "https://github.com/mattpocock/evalite/blob/main/packages/evalite/src/command.ts"
    click CLI "https://github.com/mattpocock/evalite/blob/main/packages/evalite/src/command.test.ts"
    click CLI "https://github.com/mattpocock/evalite/blob/main/packages/evalite/src/sdk.ts"
    click CLI "https://github.com/mattpocock/evalite/blob/main/packages/evalite/src/ai-sdk.ts"
    click CLI "https://github.com/mattpocock/evalite/blob/main/packages/evalite/src/write-file-queue-local-storage.ts"
    click CLI "https://github.com/mattpocock/evalite/blob/main/packages/evalite/src/server.ts"
    click CLI "https://github.com/mattpocock/evalite/blob/main/packages/evalite/src/traces.ts"
    click CLI "https://github.com/mattpocock/evalite/blob/main/packages/evalite/src/reporter.ts"
    click CLI "https://github.com/mattpocock/evalite/blob/main/packages/evalite/src/run-vitest.ts"
    click EX "https://github.com/mattpocock/evalite/tree/main/packages/example/src"
    click UI "https://github.com/mattpocock/evalite/blob/main/apps/evalite-ui/app/data/use-subscribe-to-socket.ts"
    click UI "https://github.com/mattpocock/evalite/blob/main/apps/evalite-ui/app/data/queries.ts"
    click UI "https://github.com/mattpocock/evalite/blob/main/apps/evalite-ui/app/sdk.ts"
    click UI "https://github.com/mattpocock/evalite/tree/main/apps/evalite-ui/app/components"
    click DOC "https://github.com/mattpocock/evalite/tree/main/apps/evalite-docs/src/content"
    click DOC "https://github.com/mattpocock/evalite/blob/main/apps/evalite-docs/astro.config.mts"
    click Monorepo "https://github.com/mattpocock/evalite/blob/main/pnpm-workspace.yaml"
    click Monorepo "https://github.com/mattpocock/evalite/blob/main/turbo.json"
    click Monorepo "https://github.com/mattpocock/evalite/blob/main/vitest.workspace.ts"
    click Monorepo "https://github.com/mattpocock/evalite/tree/main/.husky/pre-commit"

    %% Styles
    classDef package fill:#b7e4c7,stroke:#2d6a4f,color:#1b4332
    classDef app fill:#cfe2f3,stroke:#1e3a8a,color:#1e3a8a
    classDef external fill:#e0e0e0,stroke:#6b7280,color:#374151
    classDef datastore fill:#ffe5b4,stroke:#d97706,color:#92400e
    classDef infra fill:#f0efeb,stroke:#a799b7,color:#5b375a
    classDef workflow fill:#fde68a,stroke:#b45309,color:#78350f
    classDef dev fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
    classDef ci fill:#fee2e2,stroke:#991b1b,color:#7f1d1d
```
