# ucbepic/docetl architecture

> Architecture diagram of the GitHub repository ucbepic/docetl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ucbepic/docetl
- Repository: https://github.com/ucbepic/docetl

## Overview

##### Main System Components
The system should be divided into three major sections:

a) Core Engine (docetl package):
- Operations Layer (docetl/operations/*)
- Optimization Layer (docetl/optimizers/*)
- Pipeline Execution Layer (docetl/runner.py, docetl/builder.py)
- Configuration Management (docetl/config_wrapper.py)

b) Web Interface (website directory):
- Next.js Frontend (website/src/app)
- API Routes (website/src/app/api)
- WebSocket Communication (website/src/contexts/WebSocketContext.tsx)

c) Backend Server (server directory):
- FastAPI Server (server/app)
- Route Handlers (server/app/routes)

##### Component Relationships

a) Data Flow:
- Document Input → Core Engine → Processed Output
- UI Components → API Routes → Backend Server → Core Engine
- WebSocket connections between Frontend and Backend for real-time updates

b) Integration Points:
- Configuration flow between UI and Core Engine
- Pipeline execution results from Core Engine to UI
- File system interactions for document storage

##### Specific Implementation Details

a) Core Processing Components (Hexagons):
- Operation Nodes (Map, Reduce, Filter, etc.)
- Optimization Nodes
- Pipeline Execution Engine

b) Interface Components (Rectangles):
- UI Components
- API Endpoints
- Configuration Management

- Document Storage
- Pipeline Configurations
- Execution State

##### Architecture Patterns to Highlight

a) Pipeline Pattern:
- Flow of operations in the core engine
- Optimization layer's interaction points

b) Client-Server Architecture:
- Clear separation between frontend, backend, and core engine
- API boundaries and communication protocols

##### Technical Stack Indicators

- Python (Core Engine)
- Next.js (Frontend)
- FastAPI (Backend)
- WebSocket (Real-time Communication)
- OpenAI/LLM Integration Points

##### Optimization Flow

- Optimizer
- Join Optimizer
- Reduce Optimizer
- Their interaction with the main pipeline

**[UI Layer]**
- ↕
**[API Layer]**
- ↕
**[Backend Server Layer]**
- ↕
**[Core Engine Layer]**
- ↕
**[Storage Layer]**

Important Considerations:

- Operation execution flow
- Optimization process
- Document processing pipeline
- UI interaction flow

This system design should capture both the high-level architecture and the detailed component interactions while maintaining clarity and readability.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "UI Layer"
        UI_Main["Frontend App"]:::frontend
        PipelineGUI["Pipeline GUI"]:::frontend
        PipelineVis["Pipeline Visualization"]:::frontend
        DocViewer["Document Viewer"]:::frontend
        DataTable["Data Table"]:::frontend
        OpCards["Operation Cards"]:::frontend
        WSContext["WebSocket Context"]:::frontend
    end

    subgraph "API Layer"
        API["API Routes"]:::api
        WSEndpoint["WebSocket Endpoint"]:::api
    end

    subgraph "Backend Server"
        FastAPI["FastAPI Server"]:::backend
        PipelineRoutes["Pipeline Routes"]:::backend
        FSRoutes["File System Routes"]:::backend
        ConvRoutes["Conversion Routes"]:::backend
    end

    subgraph "Core Engine"
        subgraph "Operations"
            BaseOps["Base Operations"]:::core
            MapOp["Map Operation"]:::core
            ReduceOp["Reduce Operation"]:::core
            FilterOp["Filter Operation"]:::core
        end

        subgraph "Optimizers"
            MapOpt["Map Optimizer"]:::optimizer
            JoinOpt["Join Optimizer"]:::optimizer
            ReduceOpt["Reduce Optimizer"]:::optimizer
        end

        subgraph "Pipeline Execution"
            Runner["Pipeline Runner"]:::core
            Builder["Pipeline Builder"]:::core
            Config["Configuration Manager"]:::core
        end

        subgraph "Integration"
            LLMUtils["LLM Utils"]:::integration
            APIUtils["API Integration"]:::integration
        end
    end

    subgraph "Storage Layer"
        Cache["Cache"]:::storage
        Dataset["Dataset Management"]:::storage
        FileSystem["File System"]:::storage
    end

    %% Connections
    UI_Main --> API
    PipelineGUI --> WSEndpoint
    WSContext --> WSEndpoint
    API --> FastAPI
    WSEndpoint --> FastAPI
    FastAPI --> PipelineRoutes
    FastAPI --> FSRoutes
    FastAPI --> ConvRoutes
    PipelineRoutes --> Runner
    Runner --> Builder
    Builder --> BaseOps
    BaseOps --> MapOp
    BaseOps --> ReduceOp
    BaseOps --> FilterOp
    MapOp --> MapOpt
    ReduceOp --> ReduceOpt
    Runner --> LLMUtils
    Runner --> APIUtils
    Runner --> Cache
    Runner --> Dataset
    FSRoutes --> FileSystem

    %% Styles
    classDef frontend fill:#3498db
    classDef api fill:#2ecc71
    classDef backend fill:#27ae60
    classDef core fill:#e67e22
    classDef optimizer fill:#9b59b6
    classDef integration fill:#f1c40f
    classDef storage fill:#95a5a6

    %% Click Events
    click UI_Main "https://github.com/ucbepic/docetl/tree/main/website/src/app/"
    click PipelineGUI "https://github.com/ucbepic/docetl/blob/main/website/src/components/PipelineGui.tsx"
    click PipelineVis "https://github.com/ucbepic/docetl/blob/main/website/src/components/PipelineVisualization.tsx"
    click DocViewer "https://github.com/ucbepic/docetl/blob/main/website/src/components/DocumentViewer.tsx"
    click DataTable "https://github.com/ucbepic/docetl/blob/main/website/src/components/ResizableDataTable.tsx"
    click OpCards "https://github.com/ucbepic/docetl/blob/main/website/src/components/OperationCard.tsx"
    click WSContext "https://github.com/ucbepic/docetl/blob/main/website/src/contexts/WebSocketContext.tsx"
    click API "https://github.com/ucbepic/docetl/tree/main/website/src/app/api/"
    click FastAPI "https://github.com/ucbepic/docetl/blob/main/server/app/main.py"
    click PipelineRoutes "https://github.com/ucbepic/docetl/blob/main/server/app/routes/pipeline.py"
    click FSRoutes "https://github.com/ucbepic/docetl/blob/main/server/app/routes/filesystem.py"
    click ConvRoutes "https://github.com/ucbepic/docetl/blob/main/server/app/routes/convert.py"
    click BaseOps "https://github.com/ucbepic/docetl/blob/main/docetl/operations/base.py"
    click MapOp "https://github.com/ucbepic/docetl/blob/main/docetl/operations/map.py"
    click ReduceOp "https://github.com/ucbepic/docetl/blob/main/docetl/operations/reduce.py"
    click FilterOp "https://github.com/ucbepic/docetl/blob/main/docetl/operations/filter.py"
    click MapOpt "https://github.com/ucbepic/docetl/tree/main/docetl/optimizers/map_optimizer/"
    click JoinOpt "https://github.com/ucbepic/docetl/blob/main/docetl/optimizers/join_optimizer.py"
    click ReduceOpt "https://github.com/ucbepic/docetl/blob/main/docetl/optimizers/reduce_optimizer.py"
    click Runner "https://github.com/ucbepic/docetl/blob/main/docetl/runner.py"
    click Builder "https://github.com/ucbepic/docetl/blob/main/docetl/builder.py"
    click Config "https://github.com/ucbepic/docetl/blob/main/docetl/config_wrapper.py"
    click LLMUtils "https://github.com/ucbepic/docetl/blob/main/docetl/operations/utils/llm.py"
    click APIUtils "https://github.com/ucbepic/docetl/blob/main/docetl/operations/utils/api.py"
    click Cache "https://github.com/ucbepic/docetl/blob/main/docetl/operations/utils/cache.py"
    click Dataset "https://github.com/ucbepic/docetl/blob/main/docetl/dataset.py"
```
