# amaghzaz-y/webscout architecture

> Architecture diagram of the GitHub repository amaghzaz-y/webscout, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amaghzaz-y/webscout
- Repository: https://github.com/amaghzaz-y/webscout

## Overview

##### Project type and purpose
- This is a full-stack, serverless search engine monorepo (WebScout) composed of three sub-projects:
- core (Rust library + TS binding) – implements indexing, tokenization, query logic
- api (TypeScript/Node.js – Cloudflare Workers) – provides HTTP routes (crawl, parse, index, search, project) and background consumers (crawler, parser, indexer)
- dash (Next.js + React + Tailwind CSS) – user-facing SPA/SSR frontend

##### File structure
- Top-level “api”, “core”, and “dash” folders map to the main architecture components.
- “api/src/routes” holds REST endpoints; “api/src/consumers” are background jobs triggered via a queue.
- “api/src/lib” abstracts Cloudflare KV, Queue bindings, and the spider logic.
- “core/src” contains Rust modules (document, index, tokenizer, query) and “core/tokenizers” houses language packs.
- “dash/app” contains Next.js pages (/, /search, /auth) and components.
- Configs: tsconfig.json in each TS project, Cargo.toml/Cargo.lock in core, turbo.json orchestrates the monorepo.

##### Key insights from README
- Designed for Cloudflare Workers (edge), uses KV for index storage, R2 for document storage, and a queue for task orchestration.
- The search algorithm is Rust-based (compiled to WASM or Workers native).
- Lightweight, serverless, distributed – optimized for low latency.

##### Main components & interactions
a. Frontend (dash)
- Next.js runs on Cloudflare Pages or any edge platform.
- UI pages call the API’s /search route via HTTPS.
b. API layer (api)
- Exposes HTTP endpoints: crawl, parse, index, search, project.
- Pushes tasks to a Cloudflare Queue (consumer pattern).
- Uses core’s WASM functions (index/query) via TS bindings or native Rust Workers.
- Reads/writes indexes in KV; stores raw documents in R2.
c. Consumers
- Crawler: fetches URLs, writes raw HTML to R2, enqueues parse jobs.
- Parser: reads HTML from R2, extracts text, enqueues index jobs.
- Indexer: tokenizes with core library, updates postings in KV.
d. Storage & External Services
- Cloudflare KV – inverted index shards, caches
- R2 Buckets – raw/processed content storage
- Queue – task orchestration
- (Optional) Cloudflare Workers environment

##### Architectural patterns & design principles
- “Function-per-responsibility”: separate consumers for crawl, parse, index
- Serverless micro-services: each route/consumer can be a separate Worker
- Publish-subscribe or queue-based decoupling for resilience and horizontal scalability
- Shared core library in Rust for performance, compiled once and used by multiple Workers

## Diagram (Mermaid)

```mermaid
graph TB
    %% Client
    subgraph "Client"
        User["User"]:::client
    end

    %% Frontend Layer
    subgraph "Frontend (dash)" 
        Landing["Landing Page"]:::frontend
        SearchPg["Search Page"]:::frontend
        AuthPg["Auth Page"]:::frontend
        IndexPg["Index Namespace Page"]:::frontend
        NavBar["NavBar Component"]:::frontend
        SearchRes["SearchResult Component"]:::frontend
        Layout["Global Layout"]:::frontend
        Styles["Global CSS"]:::frontend
    end

    %% API Layer
    subgraph "API Workers (TypeScript/Node.js)" 
        APIIndex["API Entry Point"]:::compute
        subgraph "HTTP Routes"
            CrawlRoute["/crawl Route"]:::compute
            ParseRoute["/parse Route"]:::compute
            IndexRoute["/index Route"]:::compute
            SearchRoute["/search Route"]:::compute
            ProjectRoute["/project Route"]:::compute
        end
        subgraph "Consumers"
            Crawler["Crawler Consumer"]:::compute
            Parser["Parser Consumer"]:::compute
            Indexer["Indexer Consumer"]:::compute
        end
        subgraph "Lib Abstractions"
            KVLib["KV Binding"]:::compute
            QueueLib["Queue Binding"]:::compute
            SpiderLib["Spider Logic"]:::compute
            TypesLib["Shared Types"]:::compute
        end
    end

    %% Core WASM Module
    subgraph "Core (Rust WASM)" 
        CoreLib["Core Library (lib.rs)"]:::compute
        subgraph "Modules"
            DocMod["Document Module"]:::compute
            IndexMod["Index Module"]:::compute
            TokenMod["Tokenizer Module"]:::compute
            QueryMod["Query Module"]:::compute
            UtilMod["Utilities"]:::compute
        end
        Tokenizers["Language Packs"]:::compute
        TSBinding["TS Binding (webscout.ts)"]:::compute
    end

    %% Storage & Messaging
    Queue["Cloudflare Queue"]:::queue
    KVStore["Cloudflare KV"]:::storage
    R2Bucket["R2 Bucket"]:::storage

    %% Monorepo Config
    subgraph "Monorepo Config"
        Turbo["turbo.json"]:::config
        RootPkg["root package.json"]:::config
    end

    %% Frontend Config
    subgraph "dash Config"
        DashPkg["dash/package.json"]:::config
        DashTS["dash/tsconfig.json"]:::config
        NextCfg["dash/next.config.js"]:::config
        TailwindCfg["dash/tailwind.config.js"]:::config
        PostCSS["dash/postcss.config.js"]:::config
    end

    %% API Config
    subgraph "api Config"
        ApiPkg["api/package.json"]:::config
        ApiTS["api/tsconfig.json"]:::config
        WorkerCfg["api/worker-configuration.d.ts"]:::config
    end

    %% Core Config
    subgraph "core Config"
        CargoToml["core/Cargo.toml"]:::config
        CargoLock["core/Cargo.lock"]:::config
        CoreTSConfig["core/tsconfig.json"]:::config
    end

    %% Data Flows
    User -->|visits| Landing
    Landing -->|navigates| SearchPg
    SearchPg -->|"HTTP/JSON"| SearchRoute
    AuthPg -->|"HTTP/JSON"| ProjectRoute
    SearchRoute -->|"uses WASM"| CoreLib
    CoreLib -->|"reads/writes"| KVStore
    APIIndex --> CrawlRoute
    APIIndex --> ParseRoute
    APIIndex --> IndexRoute
    APIIndex --> SearchRoute
    APIIndex --> ProjectRoute

    CrawlRoute -->|enqueue| Queue
    ParseRoute -->|enqueue| Queue
    IndexRoute -->|enqueue| Queue

    Queue -->|triggers| Crawler
    Crawler -->|stores HTML| R2Bucket
    Crawler -->|enqueue parse| Queue
    Queue -->|triggers| Parser
    Parser -->|stores text| R2Bucket
    Parser -->|enqueue index| Queue
    Queue -->|triggers| Indexer
    Indexer -->|"token lists"| CoreLib
    Indexer -->|"updates postings"| KVStore

    SearchRoute -->|"queries"| CoreLib
    ProjectRoute -->|"manages projects"| KVStore

    %% Legend
    classDef client fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef frontend fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef compute fill:#C6F6D5,stroke:#333,stroke-width:1px
    classDef storage fill:#FFD59E,stroke:#333,stroke-width:1px
    classDef queue fill:#E9D8FD,stroke:#333,stroke-width:1px
    classDef config fill:#F0F0F0,stroke:#333,stroke-width:1px

    %% Click Events - Core
    click CoreLib "https://github.com/amaghzaz-y/webscout/blob/dev/core/src/lib.rs"
    click DocMod "https://github.com/amaghzaz-y/webscout/blob/dev/core/src/document.rs"
    click IndexMod "https://github.com/amaghzaz-y/webscout/blob/dev/core/src/index.rs"
    click TokenMod "https://github.com/amaghzaz-y/webscout/blob/dev/core/src/tokenizer.rs"
    click QueryMod "https://github.com/amaghzaz-y/webscout/blob/dev/core/src/query.rs"
    click UtilMod "https://github.com/amaghzaz-y/webscout/blob/dev/core/src/utils.rs"
    click Tokenizers "https://github.com/amaghzaz-y/webscout/tree/dev/core/tokenizers"
    click TSBinding "https://github.com/amaghzaz-y/webscout/blob/dev/core/ts/webscout.ts"
    click CargoToml "https://github.com/amaghzaz-y/webscout/blob/dev/core/Cargo.toml"
    click CargoLock "https://github.com/amaghzaz-y/webscout/blob/dev/core/Cargo.lock"
    click CoreTSConfig "https://github.com/amaghzaz-y/webscout/blob/dev/core/tsconfig.json"

    %% Click Events - API
    click APIIndex "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/index.ts"
    click CrawlRoute "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/routes/crawl.ts"
    click ParseRoute "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/routes/parse.ts"
    click IndexRoute "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/routes/index.ts"
    click SearchRoute "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/routes/search.ts"
    click ProjectRoute "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/routes/project.ts"
    click Crawler "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/consumers/crawler.ts"
    click Parser "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/consumers/parser.ts"
    click Indexer "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/consumers/indexer.ts"
    click KVLib "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/lib/kv.ts"
    click QueueLib "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/lib/queue.ts"
    click SpiderLib "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/lib/spider.ts"
    click TypesLib "https://github.com/amaghzaz-y/webscout/blob/dev/api/src/lib/types.ts"
    click ApiPkg "https://github.com/amaghzaz-y/webscout/blob/dev/api/package.json"
    click ApiTS "https://github.com/amaghzaz-y/webscout/blob/dev/api/tsconfig.json"
    click WorkerCfg "https://github.com/amaghzaz-y/webscout/blob/dev/api/worker-configuration.d.ts"

    %% Click Events - Frontend
    click Landing "https://github.com/amaghzaz-y/webscout/blob/dev/dash/app/page.tsx"
    click SearchPg "https://github.com/amaghzaz-y/webscout/blob/dev/dash/app/search/page.tsx"
    click AuthPg "https://github.com/amaghzaz-y/webscout/blob/dev/dash/app/auth/page.tsx"
    click IndexPg "https://github.com/amaghzaz-y/webscout/blob/dev/dash/app/index/page.tsx"
    click Layout "https://github.com/amaghzaz-y/webscout/blob/dev/dash/app/layout.tsx"
    click Styles "https://github.com/amaghzaz-y/webscout/blob/dev/dash/app/globals.css"
    click NavBar "https://github.com/amaghzaz-y/webscout/blob/dev/dash/app/components/NavBar.tsx"
    click SearchRes "https://github.com/amaghzaz-y/webscout/blob/dev/dash/app/components/SearchResult.tsx"
    click DashPkg "https://github.com/amaghzaz-y/webscout/blob/dev/dash/package.json"
    click DashTS "https://github.com/amaghzaz-y/webscout/blob/dev/dash/tsconfig.json"
    click NextCfg "https://github.com/amaghzaz-y/webscout/blob/dev/dash/next.config.js"
    click TailwindCfg "https://github.com/amaghzaz-y/webscout/blob/dev/dash/tailwind.config.js"
    click PostCSS "https://github.com/amaghzaz-y/webscout/blob/dev/dash/postcss.config.js"

    %% Click Events - Monorepo
    click Turbo "https://github.com/amaghzaz-y/webscout/blob/dev/turbo.json"
    click RootPkg "https://github.com/amaghzaz-y/webscout/blob/dev/package.json"
```
