How lumina-ai-inc/chunkr works
Main Components
• Frontend (apps/web): A modern, Vite/React-based web user interface that includes assets, authentication components, various UI components (like CheckoutOverlay, CodeBlock, Viewer, etc.), and configuration files (e.g., tailwind.config.js, vite.config.ts). This represents the entry point for users accessing the web UI.
• Core API Server (core): Written in Rust (note the Cargo.toml, build.rs, and Rust source files). It contains models, routes, middleware, jobs, workers, and service integrations (e.g., authentication, Azure, GitHub, Stripe, OCR, and LLN integration). This service handles the main business logic for document processing, OCR, layout analysis, and chunking.
• Background Job/Worker System: Within the API core (directories like core/src/jobs, core/src/workers, and core/src/models/rrq) there is a clear implementation of job processing, queuing (with RRQ patterns), and scheduled tasks (monthly usage, timeouts) that run asynchronously.
• Database: The presence of Docker deployments for Postgres (docker/postgres) and detailed migration scripts under core/migrations indicates a relational database layer that stores application data (users, tasks, usage statistics, invoices, etc.).
• Client SDKs: Located under clients (both node-client and python-client). They offer programmatic access to the API and are an important integration point for external consumption.
• Additional Services: Under services (for example, services/doctr and services/vgt) you have specialized processing tools for document processing (OCR, segmentation, object detection) that support the core API.
• Deployment Infrastructure: Docker, Kubernetes (kube directory with Helm charts, deployment YAMLs), and configuration files for both Docker Compose and Kubernetes deployments play an essential role in orchestrating the entire stack.
Relationships and Data Flow
• Users interact with the Frontend (web UI) which communicates with the Core API server via REST or WebSocket calls.
• The Core API orchestrates document processing tasks: It receives file uploads and API requests, then dispatches jobs to the background worker system.
• The worker system performs tasks like OCR, layout analysis, and semantic chunking, possibly calling extra services (e.g., VGT for object detection, doctr for document conversion as separate microservices).
• Processed data (results in HTML, Markdown, JSON, etc.) gets stored in the database and is eventually returned to and rendered by the UI or sent back through the respective client SDK.
• External API integrations (OpenAI’s LLM endpoints, Azure services, GitHub, Stripe) are accessed via configured clients and are represented as external dependencies.
• Docker and Kubernetes deployment files illustrate how each component (API, workers, frontend, database, supporting services) is containerized and orchestrated.
Architectural Patterns and Design Principles
• Modular structure: Separation into distinct “apps”, “core”, “clients”, and “services” directories shows a clean separation of concerns.
• Microservices-like Elements: With separate services (doctr, vgt) and clearly separated clients/SDKs, the architecture allows for independent deployment and scaling.
• Job Queues and Asynchronous Processing: The core uses RRQ-based queue management and multiple workers to process tasks asynchronously.
• Containerization and Orchestration: Docker Compose and Kubernetes deployment files reveal the commitment to containerized, scalable, production-ready deployment.