# teh72/unstract architecture

> Architecture diagram of the GitHub repository teh72/unstract, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/teh72/unstract
- Repository: https://github.com/teh72/unstract

## Overview

Based on our analysis of the file structure and the README, this project is a full-scale, multi-component platform for Intelligent Document Processing (IDP 2.0) that leverages large language models to power no‐code tools such as Prompt Studio and Workflow Studio. The system is comprised of several backend microservices (organized as Django-style apps and other Python services under directories like “backend”, “platform-service”, “prompt-service”, “runner”, “unstract/…” etc.), a modern React frontend (in the “frontend” directory), as well as additional tools (under “tools/…” and various “unstract/…” submodules) that provide connector services, ETL, text extraction, and support for a variety of external LLM, vector DB, and file system vendors.

##### Main Components
- Frontend: The React-based UI (in the “frontend” directory) that hosts the Prompt Studio, Workflow Studio, and various configuration pages.
- Backend/API Layer: A Django-style backend (in “backend”) with many apps (account_v2, api_v2, connector_auth, pipeline_v2, workflow_manager, prompt, etc.) handling authentication, business logic, data models, and API endpoints.
- Microservices:
– Platform Service (“platform-service”): Provides configuration, system management, and core platform functionality.
– Prompt Service (“prompt-service”): Dedicated to prompt processing, prompt engineering, and likely interacting with LLM providers.
– Runner (“runner”): Orchestrates asynchronous job execution and possibly workflow runs.
– Additional services for specialized tasks (such as x2text-service for document conversions and tools for text extraction, classification, etc.).
- Connectors and Plugins: Under “unstract/connectors”, “unstract/core”, “tool-registry”, “tool-sandbox”, etc. which integrate external sources (like databases, cloud storages, ETL sources/destinations) and file systems (such as S3, MinIO, Google Drive, etc.).
- External Integrations: Interfaces to multiple LLM providers (OpenAI, Google Vertex, Anthropic, etc.), vector databases (Qdrant, Pinecone, Weaviate, etc.), and text extractors.
- CI/CD & DevOps: GitHub workflows, Dockerfiles (in the “docker” folder and various Dockerfiles under each service), and configuration scripts handling automated testing, builds, and deployments.

##### Relationships & Data Flow
- The frontend communicates (likely via REST or GraphQL) with the backend API layer.
- The backend is a composite of many interrelated Django apps providing core processing, authentication, workflow management, and document processing.
- Microservices (platform-service, prompt-service, runner) interact with the backend to handle specialized tasks such as prompt processing, executing document extraction pipelines and deploying workflows.
- Connector modules act as an integration layer to interface with external systems (data sources, vector DBs, file systems, etc.); they are extensible via plugins and custom authentication modules.
- External LLM APIs, vector DBs, and other third-party services are invoked through API calls from the backend/microservices, often asynchronously (using Celery tasks or runner services).
- A message and task queue system (as suggested by “celery_task”, “celery_service”, and “queue_utils” folders) supports background processing and inter-service communication.

##### Architectural Patterns & Principles
- Microservices Architecture: Separate services for prompts, platform management, and asynchronous execution.
- Modular (Django/MVC-like) structure within the backend, with a clear separation of concerns (authentication, account management, file management, workflows, etc.).
- Plugin/extensibility: The “plugins” and “unstract/connectors” sections reveal a strong emphasis on extensibility and supporting multiple external vendors.
- Containerized Deployment: Dockerfiles and Compose files indicate container-based deployments for consistency across environments.
- CI/CD Integration: Automated workflows for testing, container builds, and production deployment.

##### Relevant Technologies & Frameworks
- Python (with Django-like patterns), Celery for asynchronous tasks, and possibly other microservice frameworks.
- React.js for the frontend UI.
- Docker for containerization; CI/CD pipelines using GitHub Actions.
- Messaging/queue systems (Celery or similar), and libraries for interacting with LLMs and vector databases.
- Configuration management via files like .env, pyproject.toml, and usage of PDM for dependency management.

- Clearly labeled components for: “Frontend (React)”, “Backend API (Django Modular Apps)”, “Platform Service”, “Prompt Service”, “Runner Service”, “Connectors/Plugins”, and “External Services/Providers”.
– From Frontend to Backend APIs.
– From Backend to microservices (and vice versa for status updates or callback data).
– From backend/microservices to external providers (LLM, vector DB, file systems, etc.).
– Communication flows for asynchronous tasks (via queues and Celery components).
– Presentation layer (frontend),
– Application logic (backend, microservices),
– Integration/connector modules,
– External systems.
- Annotations about methods of communication (REST APIs, message queues, etc.) and protocols (e.g., HTTP/Docker networking).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        F["Frontend (React UI)"]:::frontend
    end

    %% Application Layer
    subgraph "Application Layer"
        B["Backend API (Django Modular Apps)"]:::app
        PS["Platform Service"]:::micro
        PRS["Prompt Service"]:::micro
        RS["Runner Service"]:::micro
        X2["Doc Conversion (X2Text)"]:::micro
    end

    %% Connector / Plugins Layer
    subgraph "Connector / Plugins"
        C1["Connector Integrations"]:::connector
        C2["Core Extensions & Utilities"]:::connector
        C3["Tool Registry"]:::connector
        C4["Tool Sandbox"]:::connector
        AT["Additional Tools (ETL, Extraction, Classification)"]:::connector
    end

    %% Messaging & Async Processing
    subgraph "Messaging & Async"
        CS["Celery Service"]:::messaging
        CT["Celery Task Manager"]:::messaging
    end

    %% CI/CD & DevOps
    subgraph "CI/CD & DevOps"
        CD["GitHub Workflows"]:::devops
        DK["Docker"]:::devops
    end

    %% External Integrations
    subgraph "External Integrations"
        LLM["LLM Providers"]:::external
        VDB["Vector Databases"]:::external
        FS["File Systems & Cloud Storage"]:::external
    end

    %% Data Flow Connections
    F -->|"HTTP/REST"| B

    B -->|"config/auth"| PS
    B -->|"prompt calls"| PRS
    B -->|"workflow execution"| RS
    B -->|"conversion request"| X2

    B -->|"integration"| C1
    B -->|"utilities"| C2
    B -->|"tool registration"| C3
    B -->|"sandbox operations"| C4
    B -->|"ETL & extraction"| AT

    B -->|"enqueue tasks"| CS
    CS -->|"process tasks"| CT

    PS -->|"status update"| LLM
    PRS -->|"LLM invocation"| LLM
    RS -->|"data storage"| VDB
    RS -->|"file processing"| FS

    CD -.-> B
    CD -.-> PS
    CD -.-> PRS
    CD -.-> RS
    DK -.-> F

    %% Styling
    classDef frontend fill:#ffcc00,stroke:#333,stroke-width:2px;
    classDef app fill:#aqua,stroke:#333,stroke-width:2px;
    classDef micro fill:#99ccff,stroke:#333,stroke-width:2px;
    classDef connector fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef messaging fill:#ff9999,stroke:#333,stroke-width:2px;
    classDef devops fill:#d9d2e9,stroke:#333,stroke-width:2px;
    classDef external fill:#ffe6cc,stroke:#333,stroke-width:2px;

    %% Click Events
    click F "https://github.com/teh72/unstract/tree/main/frontend/"
    click B "https://github.com/teh72/unstract/tree/main/backend/"
    click PS "https://github.com/teh72/unstract/tree/main/platform-service/"
    click PRS "https://github.com/teh72/unstract/tree/main/prompt-service/"
    click RS "https://github.com/teh72/unstract/tree/main/runner/"
    click X2 "https://github.com/teh72/unstract/tree/main/x2text-service/"
    click C1 "https://github.com/teh72/unstract/tree/main/unstract/connectors/"
    click C2 "https://github.com/teh72/unstract/tree/main/unstract/core/"
    click C3 "https://github.com/teh72/unstract/tree/main/unstract/tool-registry/"
    click C4 "https://github.com/teh72/unstract/tree/main/unstract/tool-sandbox/"
    click AT "https://github.com/teh72/unstract/tree/main/tools/"
    click CD "https://github.com/teh72/unstract/tree/main/.github/workflows/"
    click DK "https://github.com/teh72/unstract/tree/main/docker/"
    click CS "https://github.com/teh72/unstract/blob/main/backend/backend/celery_service.py"
    click CT "https://github.com/teh72/unstract/blob/main/backend/backend/celery_task.py"
```
