How mattpocock/evalite works
Project type and purpose
• This is a TypeScript-native, local-first testing tool (Evalite) for LLM-powered apps.
• It’s organized as a pnpm/Turbo monorepo with:
– A CLI & core library (packages/evalite)
– A test harness package (packages/evalite-tests)
– Example usage (packages/example)
– A React/Vite UI app (apps/evalite-ui)
– A static docs site built with Astro (apps/evalite-docs)
– CI/CD workflows under .github/workflows
File structure
• Top-level “apps” and “packages” folders show separate deployable applications and shareable libraries.
• Each “apps” entry has its own build config (Vite/Astro).
• Shared infra: pnpm-workspace.yaml, turbo.json, TypeScript configs, Husky hooks, GitHub workflows.
Key insights from README
• Dev mode concurrently runs: TS type-checking, tests, UI dev server, and the CLI in “watch” mode on example code.
• The CLI („evalite“) invokes AI via the OpenAI API, writes traces locally, and emits updates to the UI over a socket.
Main components
a. Developer Machine / CI Agent
– Runs pnpm/Turbo tasks, Vitest, type checks, build jobs
b. Monorepo Workspace
– Core library & CLI (packages/evalite)
– Test suite (packages/evalite-tests)
– Example project (packages/example)
– UI app (apps/evalite-ui)
– Docs site (apps/evalite-docs)
c. Runtime Services
– OpenAI API (external service)
– Local file-queue storage (for traces)
– WebSocket server (embedded in CLI/server)
d. User Interfaces
– Command-line interface (evalite command)
– Web UI (React app subscribing to traces)
– Static documentation site
Relationships and data flows
• CLI → OpenAI API: sends prompts, receives responses
• CLI → Local file storage: appends trace records
• CLI/server → WebSocket → UI app: pushes real-time updates
• UI app → SDK & subscription hooks: fetches and renders test results
• Example code under packages/example → invoked by CLI in watch mode
• Test harness (evalite-tests) → depends on core library, runs Vitest against fixtures
Architectural patterns & design principles
• Monorepo with strong package boundaries and internal versioning
• Separation of concerns: CLI orchestration vs. UI presentation vs. docs
• Local-first storage pattern (no external DB)
• Real-time streaming via WebSockets for live feedback
• Static site generation for docs
Essential elements
• Clear component labels (e.g., “Evalite CLI”, “OpenAI API”, “Local Trace Store”)
• A note on CI/CD flows invoking tests and builds via GitHub Actions