# outpoot/vyntr architecture

> Architecture diagram of the GitHub repository outpoot/vyntr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/outpoot/vyntr
- Repository: https://github.com/outpoot/vyntr

## Overview

##### Main Components
- Genesis (Rust crawler & analyzer)
– Crawls web pages, fingerprints, parses HTML, generates metrics
– Sources: `genesis/src`
- Genesis Tools (Python & JS scripts)
– Dataset cleaning (`content_cleaner.js`, `remove_largest.js`)
– Embedding generation & benchmarks (`generate_embeddings.py`, `benchmark.py`)
– Database init & migration (`init.sql`, `migration.sql`)
- Storage
– AWS S3 (or S3-compatible) for JSONL data partitions
– PostgreSQL with pgvector for metadata, embeddings, user data
- Pulse (Rust indexing service)
– Reads cleaned JSONL/embeddings from S3
– Builds & serves full-text search indices via Tantivy
– CLI binaries: `pulse/src/bin/search.rs`, `server.rs`
- Lexicon (Bun/Node WordNet service)
– Loads WordNet into DB
– Provides dictionary lookup API
– Files: `lexicon/tools/src`
- Website (SvelteKit frontend & server)
– UI components under `website/src/lib/components/...`
– Server routes under `website/src/routes/api/...`
– Auth, search, chat, domain-management, user preferences
- External Services
– AWS S3, PostgreSQL, (optionally) Redis or monitoring

##### Architectural Patterns
- Microservices: discrete services for crawling, processing, indexing, dictionary, frontend
- Event-driven pipeline: S3 serves as the event bus between ingestion & processing
- CQRS: separate write path (Genesis & Tools) from read path (Pulse & Website)
- Polyglot stack: Rust for high-performance services, Python/JS for tooling, SvelteKit for UI

##### Key Technologies & Frameworks
- Rust & Cargo (Genesis, Pulse)
- Python (Genesis dataset & embedding scripts)
- Bun/Node (Lexicon service)
- SvelteKit, TypeScript (Website)
- Docker & Docker Compose (local environments)
- PostgreSQL + pgvector, Tantivy (search index)
- AWS S3 (storage), optional Redis/metrics

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers and Services

    subgraph "Ingestion & Processing"
        style Ingestion fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
        GC["Genesis Crawler\n(Rust)"]:::compute
        GT["Genesis Tools"]:::compute
        subgraph "Genesis Tools"
            DT["Dataset Cleaning\n(JS)"]:::compute
            ET["Embedding & Benchmarks\n(Python)"]:::compute
            DBM["DB Init & Migration\n(SQL)"]:::compute
        end
    end

    subgraph "Storage Layer"
        style Storage fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
        S3["AWS S3\n(JSONL Data)"]:::storage
        PG["PostgreSQL\n(pgvector)"]:::storage
    end

    subgraph "Indexing & Search"
        style Indexing fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
        PI["Pulse Indexer\n(Rust)"]:::compute
        PS["Pulse Server\n(Search API)"]:::compute
    end

    subgraph "Lexicon Service"
        style Lexicon fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
        LX["Lexicon API\n(Node/Bun)"]:::compute
    end

    subgraph "Frontend & API"
        style FrontendLayer fill:#fff3e0,stroke:#ef6c00,stroke-width:2px
        WS["Website Service\n(SvelteKit)"]:::frontend
        BR["Browser Client"]:::frontend
        DBMIG["Website DB Migrations\n(Drizzle/SQL)"]:::compute
    end

    subgraph "Infrastructure Config"
        DC["Docker & Compose\nConfigurations"]:::external
    end

    %% Data Flows
    GC -->|"raw HTML → JSONL"| S3
    GC -->|"crawl metadata"| PG

    DT -->|"reads raw JSONL"| S3
    DT -->|"writes cleaned JSONL"| S3
    ET -->|"reads cleaned JSONL"| S3
    ET -->|"writes embeddings"| PG
    DBM -->|"init / migrate"| PG

    PI -->|"reads cleaned JSONL & embeddings"| S3
    PI -->|"builds index"| PS
    PS -->|"exposes Search API"| WS

    LX -->|"reads WordNet data"| PG
    LX -->|"exposes Lexicon API"| WS

    WS -->|"queries Search API"| PS
    WS -->|"queries Lexicon API"| LX
    WS -->|"reads/writes domains & users"| PG
    BR -->|"HTTP requests"| WS

    %% Click Events
    click GC "https://github.com/outpoot/vyntr/blob/main/genesis/src/main.rs"
    click DT "https://github.com/outpoot/vyntr/tree/main/genesis/tools/dataset/"
    click ET "https://github.com/outpoot/vyntr/tree/main/genesis/tools/embedding/"
    click DBM "https://github.com/outpoot/vyntr/tree/main/genesis/tools/database/"
    click PI "https://github.com/outpoot/vyntr/blob/main/pulse/src/main.rs"
    click PS "https://github.com/outpoot/vyntr/blob/main/pulse/src/bin/server.rs"
    click LX "https://github.com/outpoot/vyntr/tree/main/lexicon/tools/src/"
    click WS "https://github.com/outpoot/vyntr/tree/main/website/src/"
    click DBMIG "https://github.com/outpoot/vyntr/tree/main/website/drizzle/"
    click DC "https://github.com/outpoot/vyntr/blob/main/genesis/docker-compose.yml"

    %% Styles
    classDef compute fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
    classDef storage fill:#bbdefb,stroke:#1565c0,stroke-width:1px
    classDef frontend fill:#ffe0b2,stroke:#ef6c00,stroke-width:1px
    classDef external fill:#f5f5f5,stroke:#9e9e9e,stroke-width:1px
```
