# krljakob/ragnificent architecture

> Architecture diagram of the GitHub repository krljakob/ragnificent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/krljakob/ragnificent
- Repository: https://github.com/krljakob/ragnificent

## Overview

##### Project Type and Purpose
- This is a **full-stack CLI/Library tool** (with an optional minimal web UI) that scrapes web pages (via HTTP and optional sitemap discovery), converts HTML to Markdown/JSON/XML, performs semantic chunking for RAG, and writes outputs to the file system.
- It uses **Python** for orchestration (CLI, HTTP client, caching, throttling, sitemap parsing) and **Rust** for performance-critical tasks (HTML parsing, conversion, chunking, optional JS rendering via headless Chrome).

##### Main Components
A. **User Interfaces**
1. **Command-Line Interface** (`python -m RAGnificent` entry point in `__main__.py`)
2. **Optional Web UI** (`webui.py` + `static/index.html`)

B. **Python Orchestration Layer**
- **HTTP Client & Scraper** (`core/scraper.py`)
- **Rate Limiting** (`core/throttle.py`)
- **Request Caching** (`core/cache.py`)
- **Sitemap Discovery & Filtering** (`utils/sitemap.py`, `utils/sitemap_utils.py`)
- **Chunk Utilities** (`utils/chunk_utils.py`)
- **Versioning & Config** (`utils/version.py`, `.env` support)

C. **Rust Performance Layer** (built as a Python extension via `maturin`)
- **HTML Parser** (`src/html_parser.rs`)
- **Markdown Converter** (`src/markdown_converter.rs`)
- **Chunker** (`src/chunker.rs`)
- **JS Renderer** (`src/js_renderer.rs`) – optional headless Chrome integration

D. **Data Storage & Outputs**
- **Filesystem** (output directory or single file: `.md`, `.json`, `.xml`)
- **Chunk Directory** (JSON/JSONL chunk files when `--save-chunks` is enabled)
- **links.txt** or other URL list inputs

E. **External Services**
- **Web Servers** (target websites)
- **Sitemaps** (`robots.txt` discovery, direct sitemap URLs)
- **Headless Chrome** (for JS rendering, optional)

F. **Utilities, Tests & Benchmarks**
- **Unit/Integration Tests** (Python `pytest`, Rust `cargo test`)
- **Benchmarks** (`benches/`, scripts under `scripts/`)
- **Examples** (`examples/`)
- **Documentation** (`docs/`, `.md` guides, flowchart assets)

##### Architectural Patterns & Design Principles
- **Layered Architecture**: UI → Python Orchestration → Rust Engine → Filesystem/External
- **Pipeline Pattern**: Scrape → Parse → Convert → Chunk → Persist
- **Separation of Concerns**:
- Python handles I/O, orchestration, high-level logic
- Rust handles CPU-intensive transformations

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Inputs/Services
    WebSite["Web Servers"]:::external
    SitemapsExt["Sitemaps"]:::external
    LinksTxt["links.txt"]:::external
    click LinksTxt "https://github.com/krljakob/ragnificent/blob/main/links.txt"

    %% User Interfaces
    CLI["CLI\n__main__.py"]:::python
    click CLI "https://github.com/krljakob/ragnificent/blob/main/RAGnificent/__main__.py"
    subgraph Optional_Web_UI
        direction TB
        WebUI["Web UI\nwebui.py"]:::pythonOpt
        StaticUI["Static Assets\nstatic/index.html"]:::pythonOpt
        click WebUI "https://github.com/krljakob/ragnificent/blob/main/webui.py"
        click StaticUI "https://github.com/krljakob/ragnificent/blob/main/static/index.html"
    end

    %% Python Orchestration Layer
    subgraph "Python Orchestration Layer"
        direction TB
        PythonAppNode["Python App"]:::python
        Scraper["Scraper\ncore/scraper.py"]:::python
        click Scraper "https://github.com/krljakob/ragnificent/blob/main/RAGnificent/core/scraper.py"
        Throttle["Throttle\ncore/throttle.py"]:::python
        click Throttle "https://github.com/krljakob/ragnificent/blob/main/RAGnificent/core/throttle.py"
        Cache["Cache\ncore/cache.py"]:::python
        click Cache "https://github.com/krljakob/ragnificent/blob/main/RAGnificent/core/cache.py"
        SitemapCore["Sitemap\nutils/sitemap.py"]:::python
        click SitemapCore "https://github.com/krljakob/ragnificent/blob/main/RAGnificent/utils/sitemap.py"
        SitemapUtil["Sitemap Utils\nutils/sitemap_utils.py"]:::python
        click SitemapUtil "https://github.com/krljakob/ragnificent/blob/main/RAGnificent/utils/sitemap_utils.py"
        ChunkUtils["Chunk Utils\nutils/chunk_utils.py"]:::python
        click ChunkUtils "https://github.com/krljakob/ragnificent/blob/main/RAGnificent/utils/chunk_utils.py"
        Version["Version & Config\nutils/version.py"]:::python
        click Version "https://github.com/krljakob/ragnificent/blob/main/RAGnificent/utils/version.py"
        Binding["Python-Rust Binding\nragnificent_rs.py"]:::python
        click Binding "https://github.com/krljakob/ragnificent/blob/main/RAGnificent/ragnificent_rs.py"
    end

    %% Rust Performance Layer
    subgraph "Rust Engine"
        direction TB
        HTMLParser["HTML Parser\nsrc/html_parser.rs"]:::rust
        click HTMLParser "https://github.com/krljakob/ragnificent/blob/main/src/html_parser.rs"
        MarkdownConv["Markdown Converter\nsrc/markdown_converter.rs"]:::rust
        click MarkdownConv "https://github.com/krljakob/ragnificent/blob/main/src/markdown_converter.rs"
        Chunker["Chunker\nsrc/chunker.rs"]:::rust
        click Chunker "https://github.com/krljakob/ragnificent/blob/main/src/chunker.rs"
        JSRenderer["JS Renderer\nsrc/js_renderer.rs"]:::rustOpt
        click JSRenderer "https://github.com/krljakob/ragnificent/blob/main/src/js_renderer.rs"
    end

    %% Data Storage
    subgraph "Data Storage & Outputs"
        direction TB
        FS["Filesystem\n(.md, .json, .xml)\nChunk Dir"]:::storage
    end

    %% Connections
    CLI -->|"invoke"| PythonAppNode
    WebUI -->|"invoke"| PythonAppNode

    PythonAppNode --> Scraper
    PythonAppNode --> Throttle
    PythonAppNode --> Cache
    PythonAppNode --> SitemapCore
    PythonAppNode --> SitemapUtil
    PythonAppNode --> ChunkUtils
    PythonAppNode --> Version
    PythonAppNode --> Binding

    Scraper -->|"HTTP GET"| WebSite
    SitemapCore -->|"fetch/parse"| SitemapsExt
    SitemapUtil -->|"filter"| SitemapsExt
    LinksTxt -->|"read URLs"| PythonAppNode

    Binding --> HTMLParser
    Binding --> MarkdownConv
    Binding --> Chunker
    Binding --> JSRenderer

    HTMLParser -->|returns HTML AST| Binding
    MarkdownConv -->|returns Markdown| Binding
    Chunker -->|returns Chunks| Binding
    JSRenderer -->|returns Rendered HTML| Binding

    PythonAppNode -->|write outputs| FS
    ChunkUtils -->|writes chunks| FS

    %% Legend
    subgraph Legend
        direction TB
        A["Python Modules"]:::python
        B["Rust Modules"]:::rust
        C["External Services"]:::external
        D["Storage"]:::storage
        E["Optional"]:::pythonOpt
        F["Optional"]:::rustOpt
    end

    %% Styles
    classDef python fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef rust fill:#FFA500,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#333,stroke-width:1px
    classDef pythonOpt fill:#ADD8E6,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef rustOpt fill:#FFA500,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
```
