# lumina-ai-inc/chunkr architecture

> Architecture diagram of the GitHub repository lumina-ai-inc/chunkr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lumina-ai-inc/chunkr
- Repository: https://github.com/lumina-ai-inc/chunkr

## Overview

##### Main Components
- Frontend (apps/web): A modern, Vite/React-based web user interface that includes assets, authentication components, various UI components (like CheckoutOverlay, CodeBlock, Viewer, etc.), and configuration files (e.g., tailwind.config.js, vite.config.ts). This represents the entry point for users accessing the web UI.
- Core API Server (core): Written in Rust (note the Cargo.toml, build.rs, and Rust source files). It contains models, routes, middleware, jobs, workers, and service integrations (e.g., authentication, Azure, GitHub, Stripe, OCR, and LLN integration). This service handles the main business logic for document processing, OCR, layout analysis, and chunking.
- Background Job/Worker System: Within the API core (directories like core/src/jobs, core/src/workers, and core/src/models/rrq) there is a clear implementation of job processing, queuing (with RRQ patterns), and scheduled tasks (monthly usage, timeouts) that run asynchronously.
- Database: The presence of Docker deployments for Postgres (docker/postgres) and detailed migration scripts under core/migrations indicates a relational database layer that stores application data (users, tasks, usage statistics, invoices, etc.).
- Client SDKs: Located under clients (both node-client and python-client). They offer programmatic access to the API and are an important integration point for external consumption.
- Additional Services: Under services (for example, services/doctr and services/vgt) you have specialized processing tools for document processing (OCR, segmentation, object detection) that support the core API.
- Deployment Infrastructure: Docker, Kubernetes (kube directory with Helm charts, deployment YAMLs), and configuration files for both Docker Compose and Kubernetes deployments play an essential role in orchestrating the entire stack.

##### Relationships and Data Flow
- Users interact with the Frontend (web UI) which communicates with the Core API server via REST or WebSocket calls.
- The Core API orchestrates document processing tasks: It receives file uploads and API requests, then dispatches jobs to the background worker system.
- The worker system performs tasks like OCR, layout analysis, and semantic chunking, possibly calling extra services (e.g., VGT for object detection, doctr for document conversion as separate microservices).
- Processed data (results in HTML, Markdown, JSON, etc.) gets stored in the database and is eventually returned to and rendered by the UI or sent back through the respective client SDK.
- External API integrations (OpenAI’s LLM endpoints, Azure services, GitHub, Stripe) are accessed via configured clients and are represented as external dependencies.
- Docker and Kubernetes deployment files illustrate how each component (API, workers, frontend, database, supporting services) is containerized and orchestrated.

##### Architectural Patterns and Design Principles
- Modular structure: Separation into distinct “apps”, “core”, “clients”, and “services” directories shows a clean separation of concerns.
- Microservices-like Elements: With separate services (doctr, vgt) and clearly separated clients/SDKs, the architecture allows for independent deployment and scaling.
- Job Queues and Asynchronous Processing: The core uses RRQ-based queue management and multiple workers to process tasks asynchronously.
- Containerization and Orchestration: Docker Compose and Kubernetes deployment files reveal the commitment to containerized, scalable, production-ready deployment.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction
    U("User"):::external
  
    %% Frontend Layer
    FW("Web UI (apps/web)<br/>(React/Vite, Authentication, UI Components)"):::frontend
  
    %% Backend Services Layer
    API("Core API (Rust)<br/>(Routes, Middleware, Job Dispatcher, Service Integrations)"):::api
    W("Workers/Job Queue (RRQ)"):::worker
    DB("PostgreSQL"):::database
  
    %% Client SDKs
    NC("Node Client SDK"):::sdk
    PC("Python Client SDK"):::sdk
  
    %% Additional Services
    DS1("Doctr Service"):::aux
    DS2("VGT Service"):::aux
  
    %% External Services
    OLLM("OpenAI LLM"):::external
    AZ("Azure"):::external
    GH("GitHub"):::external
    ST("Stripe"):::external
    S3("S3/Cloud Storage"):::external
  
    %% Deployment Infrastructure Layer
    DDEP("Docker Deployment"):::deployment
    KDEP("Kubernetes Deployment"):::deployment
  
    %% Connections - Data Flow
    U -->|"requests"| FW
    FW -->|"REST/WebSocket"| API
    API -->|"dispatches job"| W
    API -->|"CRUD operations"| DB
    W -->|"stores results"| DB
    NC -->|"calls"| API
    PC -->|"calls"| API
    API -->|"integrates"| DS1
    API -->|"integrates"| DS2
    API -->|"integrates"| OLLM
    API -->|"integrates"| AZ
    API -->|"integrates"| GH
    API -->|"integrates"| ST
    API -->|"integrates"| S3
    W -->|"utilizes"| DS1
    W -->|"utilizes"| DS2
  
    %% Deployment dashed connections to indicate container orchestration
    DDEP --- FW
    DDEP --- API
    DDEP --- W
    DDEP --- DB
    DDEP --- DS1
    DDEP --- DS2
    KDEP --- FW
    KDEP --- API
    KDEP --- W
    KDEP --- DB
    KDEP --- DS1
    KDEP --- DS2
  
    %% Click Events
    click FW "https://github.com/lumina-ai-inc/chunkr/tree/main/apps/web"
    click API "https://github.com/lumina-ai-inc/chunkr/tree/main/core"
    click W "https://github.com/lumina-ai-inc/chunkr/tree/main/core/src/workers"
    click W "https://github.com/lumina-ai-inc/chunkr/tree/main/core/src/jobs"
    click W "https://github.com/lumina-ai-inc/chunkr/tree/main/core/src/models/rrq"
    click DB "https://github.com/lumina-ai-inc/chunkr/tree/main/docker/postgres"
    click DB "https://github.com/lumina-ai-inc/chunkr/tree/main/core/migrations"
    click NC "https://github.com/lumina-ai-inc/chunkr/tree/main/clients/node-client"
    click PC "https://github.com/lumina-ai-inc/chunkr/tree/main/clients/python-client"
    click DS1 "https://github.com/lumina-ai-inc/chunkr/tree/main/services/doctr"
    click DS2 "https://github.com/lumina-ai-inc/chunkr/tree/main/services/vgt"
    click DDEP "https://github.com/lumina-ai-inc/chunkr/tree/main/docker"
    click DDEP "https://github.com/lumina-ai-inc/chunkr/blob/main/compose.yaml/compose-cpu.yaml"
    click KDEP "https://github.com/lumina-ai-inc/chunkr/tree/main/kube"
  
    %% Styles
    classDef frontend fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef api fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef worker fill:#9ACD32,stroke:#000,stroke-width:1px;
    classDef database fill:#D3D3D3,stroke:#000,stroke-width:1px;
    classDef sdk fill:#FFD700,stroke:#000,stroke-width:1px;
    classDef aux fill:#FFA500,stroke:#000,stroke-width:1px;
    classDef external fill:#FFB6C1,stroke:#000,stroke-width:1px;
    classDef deployment fill:#E0FFFF,stroke:#000,stroke-width:1px;
  
    %% Additional Connections for External Client SDKs
    NC --- U
    PC --- U
```
