# morphik-org/morphik-core architecture

> Architecture diagram of the GitHub repository morphik-org/morphik-core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/morphik-org/morphik-core
- Repository: https://github.com/morphik-org/morphik-core

## Overview

##### Project type and purpose
- This is an open-source, self-hostable backend service for multimodal document ingestion, search, and RAG-style generation.
- Exposes a REST API (core/api.py) and a Python SDK (sdks/python) for programmatic access.
- Offers an enterprise-only UI component (ee/ui-component) built in Next.js/React.

##### File structure
- Top-level “core” directory holds the service logic, subdivided into:
- api.py – endpoint definitions
- cache/ – plugin caches (base + HF, LLaMa factories)
- completion/ – LLM invocation (base + Litellm)
- embedding/ – vector embedding engines (ColPali, Litellm)
- database/ – Postgres integration and user limits
- vector_store/ – vector index implementations (PGVector, multi)
- parser/ – document parsers (PDF, video)
- storage/ – file stores (local, S3)
- services/ – business logic (document, graph, rules, telemetry, user)
- reranker/ – result ranking plugins
- workers/ – background ingestion tasks
- Deployment: Dockerfile, docker-compose.yml, init.sql, postgres.dockerfile.
- Client/Integration: examples/, sdks/python/, scripts/, utils/.

##### README insights
- Purpose: ingest/search multimodal docs, build knowledge graphs, cache-augmented generation.
- Tech stack: Python, Postgres + PGVector, S3/local, HuggingFace/LLaMa models, FastAPI (implied), Next.js for UI.

- Blue for internal components (API, Services, Worker)
- Green for data stores (Postgres, S3, Vector Store)
- Orange for plugin/factory modules (cache, embedding, parser, completion)
- Purple for external ML providers
- Major interactions, e.g., “Store file → S3”, “Fetch vectors → PGVector”.

##### Detail and extensibility
- Factory classes loading specific implementations at runtime.
- Both the real-time API flow and the background worker flow.
- Deployment artifacts (Docker, Compose) at the bottom or side.
- Completeness without overcomplicating: represent each folder/module as a component.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Clients
    subgraph "Clients" 
        direction TB
        SDK["Python SDK"]:::external
        CLI["CLI (start_server.py)"]:::external
        APIClient["REST API Consumer"]:::external
        UI["Enterprise UI"]:::external
    end

    %% API Layer
    subgraph "API Layer" 
        direction TB
        API["REST API Layer"]:::internal
    end

    %% Services Layer
    subgraph "Services Layer"
        direction TB
        DocumentSvc["Document Service"]:::internal
        GraphSvc["Graph Service"]:::internal
        RulesSvc["Rules Processor"]:::internal
        EntitySvc["Entity Resolution"]:::internal
        UserSvc["User Service"]:::internal
        TelemetrySvc["Telemetry Service"]:::internal
    end

    %% Plugin Layer
    subgraph "Plugin & Factory Layer"
        direction TB
        Parser["Parser Plugins"]:::plugin
        Embedding["Embedding Engines"]:::plugin
        Completion["Completion Plugins"]:::plugin
        Cache["Cache Plugins"]:::plugin
        Reranker["Reranker Plugins"]:::plugin
    end

    %% Data Stores
    subgraph "Data Stores"
        direction TB
        PostgresDB["Postgres DB"]:::datastore
        VectorStore["Vector Store"]:::datastore
        FileStorage["File Storage"]:::datastore
    end

    %% Background Worker
    subgraph "Async Worker"
        direction TB
        Worker["Ingestion Worker"]:::internal
    end

    %% External ML
    subgraph "External ML Providers"
        direction TB
        HF["HuggingFace"]:::external
        LLama["LLaMa Endpoints"]:::external
    end

    %% Deployment
    subgraph "Deployment & Config"
        direction TB
        Config["Config & Logging"]:::internal
        Models["Data Models"]:::internal
        Docker["Docker & Compose"]:::internal
    end

    %% Connections - Sync Path
    SDK -->|sync request| API
    CLI -->|sync request| API
    API -->|calls| DocumentSvc
    API -->|calls| UserSvc
    API -->|calls| TelemetrySvc

    DocumentSvc -->|invoke| Parser
    DocumentSvc -->|invoke| Embedding
    DocumentSvc -->|invoke| Completion
    DocumentSvc -->|invoke| Cache
    DocumentSvc -->|invoke| Reranker
    DocumentSvc -->|store metadata| PostgresDB
    DocumentSvc -->|query| VectorStore
    DocumentSvc -->|save file| FileStorage

    %% Background Async Path
    API -->|enqueue task| Worker
    Worker -.->|async| Parser
    Worker -.->|async| Embedding
    Worker -.->|async| Cache
    Worker -.->|async| Completion
    Worker -.->|async| VectorStore
    Worker -.->|async| PostgresDB

    %% Telemetry flow
    DocumentSvc -->|emit events| TelemetrySvc
    Worker -->|emit events| TelemetrySvc

    %% External ML integration
    Embedding -->|call API| HF
    Embedding -->|call API| LLama
    Completion -->|call API| HF
    Completion -->|call API| LLama

    %% Deployment connections
    API -->|loads| Config
    Worker -->|loads| Config

    %% Click events
    click API "https://github.com/morphik-org/morphik-core/blob/main/core/api.py"
    click SDK "https://github.com/morphik-org/morphik-core/tree/main/sdks/python/"
    click UI "https://github.com/morphik-org/morphik-core/tree/main/ee/ui-component/"
    click Cache "https://github.com/morphik-org/morphik-core/tree/main/core/cache/"
    click Completion "https://github.com/morphik-org/morphik-core/tree/main/core/completion/"
    click Embedding "https://github.com/morphik-org/morphik-core/tree/main/core/embedding/"
    click Parser "https://github.com/morphik-org/morphik-core/tree/main/core/parser/"
    click PostgresDB "https://github.com/morphik-org/morphik-core/tree/main/core/database/"
    click VectorStore "https://github.com/morphik-org/morphik-core/tree/main/core/vector_store/"
    click FileStorage "https://github.com/morphik-org/morphik-core/tree/main/core/storage/"
    click DocumentSvc "https://github.com/morphik-org/morphik-core/blob/main/core/services/document_service.py"
    click GraphSvc "https://github.com/morphik-org/morphik-core/blob/main/core/services/graph_service.py"
    click RulesSvc "https://github.com/morphik-org/morphik-core/blob/main/core/services/rules_processor.py"
    click EntitySvc "https://github.com/morphik-org/morphik-core/blob/main/core/services/entity_resolution.py"
    click UserSvc "https://github.com/morphik-org/morphik-core/blob/main/core/services/user_service.py"
    click TelemetrySvc "https://github.com/morphik-org/morphik-core/blob/main/core/services/telemetry.py"
    click Reranker "https://github.com/morphik-org/morphik-core/tree/main/core/reranker/"
    click Worker "https://github.com/morphik-org/morphik-core/blob/main/core/workers/ingestion_worker.py"
    click Config "https://github.com/morphik-org/morphik-core/blob/main/core/config.py"
    click Models "https://github.com/morphik-org/morphik-core/tree/main/core/models/"
    click Docker "https://github.com/morphik-org/morphik-core/blob/main/docker-compose.yml"

    %% Styles
    classDef internal fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef datastore fill:#DFFFD0,stroke:#333,stroke-width:1px
    classDef plugin fill:#FFE4B5,stroke:#333,stroke-width:1px
    classDef external fill:#E6D5FF,stroke:#333,stroke-width:1px
```
