How krljakob/ragnificent works
Project Type and Purpose
• This is a full-stack CLI/Library tool (with an optional minimal web UI) that scrapes web pages (via HTTP and optional sitemap discovery), converts HTML to Markdown/JSON/XML, performs semantic chunking for RAG, and writes outputs to the file system.
• It uses Python for orchestration (CLI, HTTP client, caching, throttling, sitemap parsing) and Rust for performance-critical tasks (HTML parsing, conversion, chunking, optional JS rendering via headless Chrome).
Main Components
A. User Interfaces
1. Command-Line Interface (python -m RAGnificent entry point in __main__.py)
2. Optional Web UI (webui.py + static/index.html)
B. Python Orchestration Layer
• HTTP Client & Scraper (core/scraper.py)
• Rate Limiting (core/throttle.py)
• Request Caching (core/cache.py)
• Sitemap Discovery & Filtering (utils/sitemap.py, utils/sitemap_utils.py)
• Chunk Utilities (utils/chunk_utils.py)
• Versioning & Config (utils/version.py, .env support)
C. Rust Performance Layer (built as a Python extension via maturin)
• HTML Parser (src/html_parser.rs)
• Markdown Converter (src/markdown_converter.rs)
• Chunker (src/chunker.rs)
• JS Renderer (src/js_renderer.rs) – optional headless Chrome integration
D. Data Storage & Outputs
• Filesystem (output directory or single file: .md, .json, .xml)
• Chunk Directory (JSON/JSONL chunk files when --save-chunks is enabled)
• links.txt or other URL list inputs
E. External Services
• Web Servers (target websites)
• Sitemaps (robots.txt discovery, direct sitemap URLs)
• Headless Chrome (for JS rendering, optional)
F. Utilities, Tests & Benchmarks
• Unit/Integration Tests (Python pytest, Rust cargo test)
• Benchmarks (benches/, scripts under scripts/)
• Examples (examples/)
• Documentation (docs/, .md guides, flowchart assets)
Architectural Patterns & Design Principles
• Layered Architecture: UI → Python Orchestration → Rust Engine → Filesystem/External
• Pipeline Pattern: Scrape → Parse → Convert → Chunk → Persist
• Separation of Concerns:
• Python handles I/O, orchestration, high-level logic
• Rust handles CPU-intensive transformations