# vtempest/ai-research-agent architecture

> Architecture diagram of the GitHub repository vtempest/ai-research-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vtempest/ai-research-agent
- Repository: https://github.com/vtempest/ai-research-agent

## Overview

##### Project type and purpose
- This is a monorepo (Turborepo-managed) containing:
- Shared TypeScript/JavaScript packages (core “ai-research-agent,” crawler, neural-net, API clients, editor modules)
- Multiple applications (“apps”): a web app (SvelteKit + serverless APIs), a desktop app (Tauri & Svelte UI), a browser extension, and a Docusaurus-powered docs site
- The README describes a research-agent library (vector-space text extraction, search streaming, summarization, web crawling, etc.) plus front-ends that consume it.

##### File structure
- Top-level folders:
- /packages – pluggable modules published to NPM
- /apps – four distinct front-end and docs applications
- compose.yml – for local Docker services (e.g., crawler proxy)
- turbo.json – build orchestration
- Configuration files:
- package.json, pnpm-lock, tsconfig, vite.config, Rust Cargo.toml (Tauri)
- Apps:
- apps/web – SvelteKit UI + server-side API routes under /src/routes/api
- apps/desktop – Svelte UI + Tauri Rust backend for system features
- apps/extension – multi-page browser extension (background, content, sidepanel)
- apps/docs – Docusaurus documentation site

##### README for additional insights
- Core functions: extractor (HTML/PDF/YouTube), search-STREAM, topic modeling, weighting (WRITEFAT), autocomplete
- Data flow: Web UI → serverless API → core packages → external services (web, LLMs)
- Deployment targets: NPM packages, Cloudflare Workers, Docker proxy, GitHub Pages for docs

##### Main components
a. Monorepo orchestrator
- Turborepo / package manager / CI (GitHub Actions)
b. Shared libraries (NPM packages)
- ai-research-agent (core functions)
- crawler (web scraping Docker proxy)
- neural-net (ML wrappers)
- qwksearch-api-client, reason-editor
c. Applications
- Web App
– SvelteKit UI (Browser)
– SvelteKit Serverless API (Cloudflare Workers or Node)
- Desktop App
– Svelte UI (WebView)
– Tauri Rust backend (system capabilities)
- Browser Extension
– Content scripts / background service worker
- Documentation Site
– Docusaurus static site
d. External services and infrastructure
- NPM Registry (publish/consume packages)
- Docker crawler proxy (Puppeteer for Cloudflare bypass)
- LLM providers (OpenAI, Groq, Claude, etc.)
- Cloudflare Workers / Serverless runtime

##### Relationships and interactions
- All apps import and call shared NPM packages
- Web App’s /api routes invoke ai-research-agent functions
- Desktop UI calls core library in JS and Tauri RPC for OS integration
- Extension scripts call core library directly in the browser
- Crawler package runs in Docker to bypass bot checks—Web/Agent code routes scraping requests through it
- CI pipeline builds, tests, publishes packages; deploys web & docs to hosting

##### Architectural patterns and principles
- Monorepo with Turborepo for build-and-test orchestration
- Code reusability via published packages
- Serverless micro-API pattern for Web App
- Polyglot runtime: TypeScript for logic, Rust for system calls (desktop)
- Dockerized sidecar for complex scraping

##### Final advice
- Be exhaustive: include every “apps/…” and “packages/…” component at a high level
- Essential architectural layers: monorepo orchestration, shared libraries, UIs, APIs, external services, infrastructure
- Avoid clutter: group similar packages (e.g., all AI functions) into a single “Core AI Library” block, unless a deeper dive is needed

## Diagram (Mermaid)

```mermaid
graph TB
    %% Monorepo Orchestrator
    subgraph "Monorepo Orchestrator"
        Turbo["Turborepo Orchestrator"]:::orchestrator
        CI["CI Pipeline"]:::infra
    end

    %% Shared Packages
    subgraph "Shared Packages"
        Core["Core AI Library"]:::core
        CrawlerPkg["Web Crawler Package"]:::core
        NeuralNet["Neural-Net Library"]:::core
        QwkClient["QwkSearch API Client"]:::core
        EditorMod["Reason Editor Module"]:::core
    end

    %% Applications
    subgraph "Applications"
        subgraph "Web App"
            WebUI["SvelteKit UI"]:::frontend
            WebAPI["Serverless API"]:::backend
        end
        subgraph "Desktop App"
            DesktopUI["Svelte UI"]:::frontend
            DesktopBE["Tauri Rust Backend"]:::backend
        end
        subgraph "Browser Extension"
            Extension["Extension Scripts"]:::frontend
        end
        subgraph "Documentation Site"
            Docs["Docusaurus Site"]:::frontend
        end
    end

    %% Infrastructure / External
    subgraph "Infrastructure / External"
        DockerProxy["Docker Crawler Proxy"]:::infra
        NPM["NPM Registry"]:::external
        LLM["LLM Providers"]:::external
        Internet["Internet / Web"]:::external
        Cloudflare["Cloudflare Workers"]:::infra
    end

    %% Relationships
    Turbo -->|orchestrates builds/tests| Core
    Turbo --> CrawlerPkg
    Turbo --> NeuralNet
    Turbo --> QwkClient
    Turbo --> EditorMod

    CI -->|publishes| NPM
    CI -->|deploys| Cloudflare

    WebUI -->|calls| WebAPI
    WebAPI -->|imports/executes| Core
    WebAPI -->|routes scraping via| DockerProxy
    DesktopUI -->|uses| Core
    DesktopUI -->|RPC calls| DesktopBE
    DesktopBE -->|integrates OS via| Core
    Extension -->|imports| Core

    Core -->|fetch/scrape| DockerProxy
    CrawlerPkg -->|runs in| DockerProxy
    Core -->|queries| LLM
    CrawlerPkg -->|scrapes| Internet

    Docs -->|static content| NPM

    subgraph Deployments
        Cloudflare
    end

    %% Click Events
    click Turbo "https://github.com/vtempest/ai-research-agent/blob/master/turbo.json"
    click CI "https://github.com/vtempest/ai-research-agent/blob/master/.github/workflows/docs.yml"
    click DockerProxy "https://github.com/vtempest/ai-research-agent/blob/master/compose.yml"
    click DockerProxy "https://github.com/vtempest/ai-research-agent/tree/master/packages/crawler/Dockerfile"
    click Core "https://github.com/vtempest/ai-research-agent/tree/master/packages/ai-research-agent"
    click CrawlerPkg "https://github.com/vtempest/ai-research-agent/tree/master/packages/crawler"
    click NeuralNet "https://github.com/vtempest/ai-research-agent/tree/master/packages/neural-net"
    click QwkClient "https://github.com/vtempest/ai-research-agent/tree/master/packages/qwksearch-api-client"
    click EditorMod "https://github.com/vtempest/ai-research-agent/tree/master/packages/reason-editor"
    click WebUI "https://github.com/vtempest/ai-research-agent/tree/master/apps/web"
    click DesktopUI "https://github.com/vtempest/ai-research-agent/tree/master/apps/desktop"
    click Extension "https://github.com/vtempest/ai-research-agent/tree/master/apps/extension"
    click Docs "https://github.com/vtempest/ai-research-agent/tree/master/apps/docs"

    %% Styles
    classDef orchestrator fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef core fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef frontend fill:#d4edda,stroke:#333,stroke-width:1px
    classDef backend fill:#ffe5b4,stroke:#333,stroke-width:1px
    classDef infra fill:#f0f0f0,stroke:#333,stroke-width:1px
    classDef external fill:#e2e3e5,stroke:#333,stroke-width:1px
```
