# ikueisou/mllm-rag architecture

> Architecture diagram of the GitHub repository ikueisou/mllm-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ikueisou/mllm-rag
- Repository: https://github.com/ikueisou/mllm-rag

## Overview

##### Project type and purpose
- This is a full-stack AI-powered document processing, embedding, search and visualization platform.
- The Python “core” directory implements a FastAPI-style backend with modules for parsing, embeddings, caching, vector storage, document/graph services, and background workers.
- The “ee/ui-component” directory is a Next.js/React frontend providing UI components for chat, document upload, graph visualization, search, and theme management.

##### File structure
- Top-level:
- start_server.py, quick_setup.py, init.sql, pyproject.toml, requirements.txt – server bootstrap, DB initialization, dependencies.
- core/ – backend packages (api, config, logging, models, services, database, vector_store, cache, embedding, completion, parser, reranker, storage, workers).
- ee/ui-component/ – frontend app, components, hooks, styles, build/config files.
- Architectural clues:
- “api.py” exposes HTTP endpoints.
- “workers/ingestion_worker.py” runs asynchronous ingestion pipelines.
- “vector_store” + “database” modules implement persistent Postgres + pgvector storage.
- “cache” and “completion” modules wrap ML model calls with factories.
- “services” layer orchestrates domain logic (documents, graph, rules, user, telemetry).

##### README for additional insights
- Backend relies on PostgreSQL with pgvector, plus system libs (poppler, tesseract).
- Quick-setup and start scripts spin up environment and server.
- Frontend runs via npm in “ee/ui-component”.

##### Elements
- Blue for user-facing components (UI)
- Green for API/Services
- Orange for data stores
- Purple for ML components & caches
- Grey for external dependencies (S3, LLM APIs)

##### Detail without overthinking
- Both synchronous flows (UI→API→DB) and asynchronous processes (Worker→Parser→Vector Store).
- Update and refine as you discover more internal interactions or third-party calls.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend (Next.js UI)" 
        direction TB
        Browser["Web Browser"]:::frontend
        UIApp["Application Root (app)"]:::frontend
        APIRoutes["API Routes (notebooks)"]:::frontend
        GraphComp["ForceGraphComponent"]:::frontend
        ChatComp["Chat Components"]:::frontend
        DocsComp["Document Components"]:::frontend
        SearchComp["Search Components"]:::frontend
        ThemeProv["Theme Provider & UI Primitives"]:::frontend
        UtilsUI["Hooks & Utilities"]:::frontend
    end

    Browser -->|interacts via HTTP| UIApp
    UIApp --> APIRoutes
    UIApp --> GraphComp
    UIApp --> ChatComp
    UIApp --> DocsComp
    UIApp --> SearchComp
    UIApp --> ThemeProv
    UIApp --> UtilsUI

    click UIApp "https://github.com/ikueisou/mllm-rag/tree/main/ee/ui-component/app"
    click APIRoutes "https://github.com/ikueisou/mllm-rag/blob/main/ee/ui-component/app/api/notebooks/route.ts"
    click GraphComp "https://github.com/ikueisou/mllm-rag/blob/main/ee/ui-component/components/ForceGraphComponent.tsx"
    click ChatComp "https://github.com/ikueisou/mllm-rag/tree/main/ee/ui-component/components/chat/"
    click DocsComp "https://github.com/ikueisou/mllm-rag/tree/main/ee/ui-component/components/documents/"
    click SearchComp "https://github.com/ikueisou/mllm-rag/tree/main/ee/ui-component/components/search/"
    click ThemeProv "https://github.com/ikueisou/mllm-rag/blob/main/ee/ui-component/components/theme-provider.tsx"
    click UtilsUI "https://github.com/ikueisou/mllm-rag/blob/main/ee/ui-component/hooks/useMorphikChat.ts"

    %% Backend Layer
    subgraph "Backend"
        direction TB
        subgraph "API Layer" 
            direction TB
            APIServer["FastAPI Server (start_server.py)"]:::api
            APIEndpoints["HTTP Endpoints (core/api.py)"]:::api
        end

        subgraph "Services Layer"
            direction TB
            DocService["DocumentService"]:::service
            GraphService["GraphService"]:::service
            UserService["UserService"]:::service
            TelemetryService["Telemetry"]:::service
            RulesProcessor["RulesProcessor"]:::service
            EntityResolution["EntityResolution"]:::service
        end

        subgraph "Storage & Persistence"
            direction TB
            subgraph "Relational DB (Postgres)"
                direction TB
                BaseDB["BaseDatabase"]:::db
                PostgresDB["PostgresDatabase"]:::db
                UserLimitsDB["UserLimitsDB"]:::db
            end
            subgraph "Vector Store (pgvector)"
                direction TB
                BaseVector["BaseVectorStore"]:::db
                PGVector["PgVectorStore"]:::db
                MultiVector["MultiVectorStore"]:::db
            end
            subgraph "File Storage"
                direction TB
                BaseStorage["BaseStorage"]:::db
                LocalStorage["LocalStorage"]:::db
                S3Storage["S3Storage"]:::db
                FileUtils["File Utils"]:::db
            end
        end

        subgraph "ML Components"
            direction TB
            subgraph "Embeddings"
                BaseEmbed["BaseEmbeddingModel"]:::ml
                ColpaliEmbed["ColpaliAdapter"]:::ml
                LitellmEmbed["LitellmAdapter"]:::ml
            end
            subgraph "Completion"
                BaseComp["BaseCompletion"]:::ml
                LitellmComp["LitellmCompletion"]:::ml
            end
            subgraph "Reranker"
                BaseRank["BaseReranker"]:::ml
                FlagRank["FlagReranker"]:::ml
            end
            subgraph "Cache"
                BaseCache["BaseCache"]:::ml
                HFCache["HFCache"]:::ml
                LlamaCache["LlamaCache"]:::ml
                CacheFactory["Cache Factories"]:::ml
            end
        end

        subgraph "Parsing & Ingestion"
            direction TB
            BaseParser["BaseParser"]:::worker
            MorphikParser["MorphikParser"]:::worker
            VideoParser["VideoParser"]:::worker
            IngestionWorker["Ingestion Worker"]:::worker
        end

        %% External Systems
        subgraph "External Dependencies"
            direction TB
            S3["S3 / Object Storage"]:::external
            LLMAPI["Third-Party LLM APIs"]:::external
        end
    end

    %% Connections
    Browser -->|HTTP request| APIServer
    APIServer -->|routes to| APIEndpoints
    APIEndpoints -->|calls| DocService
    APIEndpoints -->|calls| GraphService
    APIEndpoints -->|calls| UserService
    APIEndpoints -->|calls| TelemetryService
    APIEndpoints -->|calls| RulesProcessor

    DocService -->|reads/writes| BaseDB
    DocService -->|reads/writes| PostgresDB
    DocService -->|stores vectors| BaseVector
    DocService -->|stores vectors| PGVector
    DocService -->|stores files| BaseStorage
    DocService -->|stores files| LocalStorage
    DocService -->|stores files| S3Storage
    DocService -->|uses| BaseEmbed
    DocService -->|uses| ColpaliEmbed
    DocService -->|uses| LitellmEmbed
    DocService -->|uses| BaseCache
    DocService -->|uses| HFCache
    DocService -->|uses| LlamaCache
    DocService -->|uses| CacheFactory

    GraphService -->|uses| EntityResolution
    RulesProcessor -->|uses| UserService

    IngestionWorker -->|parses docs| BaseParser
    IngestionWorker -->|parses docs| MorphikParser
    IngestionWorker -->|parses videos| VideoParser
    IngestionWorker -->|writes| PGVector
    IngestionWorker -->|writes| PostgresDB
    IngestionWorker -->|stores files| LocalStorage
    IngestionWorker -->|stores files| S3Storage

    TelemetryService -->|logs to| PostgresDB

    DocService -->|calls| LLMAPI
    DocService -->|uploads to| S3

    %% Click Events Backend
    click APIServer "https://github.com/ikueisou/mllm-rag/blob/main/start_server.py"
    click APIEndpoints "https://github.com/ikueisou/mllm-rag/blob/main/core/api.py"

    click DocService "https://github.com/ikueisou/mllm-rag/blob/main/core/services/document_service.py"
    click GraphService "https://github.com/ikueisou/mllm-rag/blob/main/core/services/graph_service.py"
    click UserService "https://github.com/ikueisou/mllm-rag/blob/main/core/services/user_service.py"
    click TelemetryService "https://github.com/ikueisou/mllm-rag/blob/main/core/services/telemetry.py"
    click RulesProcessor "https://github.com/ikueisou/mllm-rag/blob/main/core/services/rules_processor.py"
    click EntityResolution "https://github.com/ikueisou/mllm-rag/blob/main/core/services/entity_resolution.py"

    click BaseDB "https://github.com/ikueisou/mllm-rag/blob/main/core/database/base_database.py"
    click PostgresDB "https://github.com/ikueisou/mllm-rag/blob/main/core/database/postgres_database.py"
    click UserLimitsDB "https://github.com/ikueisou/mllm-rag/blob/main/core/database/user_limits_db.py"

    click BaseVector "https://github.com/ikueisou/mllm-rag/blob/main/core/vector_store/base_vector_store.py"
    click PGVector "https://github.com/ikueisou/mllm-rag/blob/main/core/vector_store/pgvector_store.py"
    click MultiVector "https://github.com/ikueisou/mllm-rag/blob/main/core/vector_store/multi_vector_store.py"

    click BaseStorage "https://github.com/ikueisou/mllm-rag/blob/main/core/storage/base_storage.py"
    click LocalStorage "https://github.com/ikueisou/mllm-rag/blob/main/core/storage/local_storage.py"
    click S3Storage "https://github.com/ikueisou/mllm-rag/blob/main/core/storage/s3_storage.py"
    click FileUtils "https://github.com/ikueisou/mllm-rag/blob/main/core/storage/utils_file_extensions.py"

    click BaseEmbed "https://github.com/ikueisou/mllm-rag/blob/main/core/embedding/base_embedding_model.py"
    click ColpaliEmbed "https://github.com/ikueisou/mllm-rag/blob/main/core/embedding/colpali_embedding_model.py"
    click LitellmEmbed "https://github.com/ikueisou/mllm-rag/blob/main/core/embedding/litellm_embedding.py"

    click BaseComp "https://github.com/ikueisou/mllm-rag/blob/main/core/completion/base_completion.py"
    click LitellmComp "https://github.com/ikueisou/mllm-rag/blob/main/core/completion/litellm_completion.py"

    click BaseRank "https://github.com/ikueisou/mllm-rag/blob/main/core/reranker/base_reranker.py"
    click FlagRank "https://github.com/ikueisou/mllm-rag/blob/main/core/reranker/flag_reranker.py"

    click BaseCache "https://github.com/ikueisou/mllm-rag/blob/main/core/cache/base_cache.py"
    click HFCache "https://github.com/ikueisou/mllm-rag/blob/main/core/cache/hf_cache.py"
    click LlamaCache "https://github.com/ikueisou/mllm-rag/blob/main/core/cache/llama_cache.py"
    click CacheFactory "https://github.com/ikueisou/mllm-rag/blob/main/core/cache/base_cache_factory.py"
    click CacheFactory "https://github.com/ikueisou/mllm-rag/blob/main/core/cache/llama_cache_factory.py"

    click BaseParser "https://github.com/ikueisou/mllm-rag/blob/main/core/parser/base_parser.py"
    click MorphikParser "https://github.com/ikueisou/mllm-rag/blob/main/core/parser/morphik_parser.py"
    click VideoParser "https://github.com/ikueisou/mllm-rag/blob/main/core/parser/video/parse_video.py"
    click IngestionWorker "https://github.com/ikueisou/mllm-rag/blob/main/core/workers/ingestion_worker.py"

    %% Styles
    classDef frontend fill:#edf5ff,stroke:#0366d6,stroke-width:2px
    classDef api fill:#e6ffed,stroke:#28a745,stroke-width:2px
    classDef service fill:#e8f0fe,stroke:#1967d2,stroke-width:2px
    classDef db fill:#fff5e6,stroke:#d48806,stroke-width:2px
    classDef ml fill:#faf0ff,stroke:#9b59b6,stroke-width:2px
    classDef worker fill:#f0f0f0,stroke:#737373,stroke-width:2px
    classDef external fill:#f5f5f5,stroke:#999999,stroke-width:2px
```
