# runllm/aqueduct architecture

> Architecture diagram of the GitHub repository runllm/aqueduct, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/runllm/aqueduct
- Repository: https://github.com/runllm/aqueduct

## Overview

##### Main Components

a. Core System Components:
- Server Component (src/golang/cmd/server)
- Executor Component (src/python/aqueduct_executor)
- SDK Component (sdk/)
- UI Component (src/ui/)
- Storage Layer (src/golang/lib/storage)

b. External Integration Points:
- Cloud Infrastructure Connectors
- Database Connectors
- Computing Resources (Kubernetes, AWS Lambda, Spark)
- Authentication Systems

##### Key Architectural Patterns

- Microservices Architecture (separate components for server, executor, UI)
- Plugin Architecture (for resource connectors)
- Event-driven Architecture (for workflow execution)
- RESTful API Architecture (for client-server communication)

**[Web UI] -----> [SDK/Python Client]**

**[API Server (Go)] <---> [Workflow Engine] <---> [Executor Service]**

**[Resource Connectors] <---> [Compute Engine] <---> [Storage Engine]**

**[Cloud Infrastructure] [Databases] [Computing Resources]**

##### Specific Implementation Details

- Workflow Definition Flow
- Execution Path
- Data Movement
- Resource Management
- Authentication Flow

##### Key Components Detail

Server:
**[API Server]**
- Request Handler
- Authentication
- Workflow Manager
- Resource Controller

Executor:
**[Executor Service]**
- Function Executor
- Connector Handlers
- System Metrics
- Parameter Handler

Resource Management:
**[Resource Layer]**
- Database Connectors
- Cloud Storage
- Compute Resources
- Authentication

## Diagram (Mermaid)

```mermaid
graph TB
    %% Style Definitions
    classDef frontend fill:#D4D4D4,stroke:#333,stroke-width:2px
    classDef core fill:#3498DB,stroke:#333,stroke-width:2px
    classDef executor fill:#2ECC71,stroke:#333,stroke-width:2px
    classDef storage fill:#9B59B6,stroke:#333,stroke-width:2px
    classDef resource fill:#E67E22,stroke:#333,stroke-width:2px
    classDef external fill:#95A5A6,stroke:#333,stroke-width:2px
    classDef database fill:#34495E,stroke:#333,stroke-width:2px

    %% Client Layer
    subgraph ClientLayer
        UI["Web UI"]:::frontend
        SDK["SDK/Python Client"]:::frontend
    end

    %% Core Services Layer
    subgraph CoreServices
        APIServer["API Server"]:::core
        subgraph ServerComponents
            RequestHandler["Request Handler"]:::core
            Auth["Authentication"]:::core
            WorkflowMgr["Workflow Manager"]:::core
        end
    end

    %% Executor Layer
    subgraph ExecutorLayer
        Executor["Executor Service"]:::executor
        FuncExec["Function Executor"]:::executor
        ConnHandler["Connector Handlers"]:::executor
        SysMetrics["System Metrics"]:::executor
    end

    %% Storage Layer
    subgraph StorageLayer
        Storage["Storage Engine"]:::storage
        FileStore["File Storage"]:::storage
        S3Store["S3 Storage"]:::storage
        GCSStore["GCS Storage"]:::storage
    end

    %% Resource Layer
    subgraph ResourceLayer
        ResConnectors["Resource Connectors"]:::resource
        DBConnectors["Database Connectors"]:::resource
        CloudConn["Cloud Connectors"]:::resource
        CompRes["Computing Resources"]:::resource
    end

    %% External Services
    subgraph ExternalServices
        Cloud["Cloud Infrastructure"]:::external
        DB["Databases"]:::external
        Compute["Computing Resources"]:::external
    end

    %% Relationships
    UI --> SDK
    SDK --> APIServer
    APIServer --> RequestHandler
    RequestHandler --> Auth
    Auth --> WorkflowMgr
    WorkflowMgr --> Executor
    Executor --> FuncExec
    Executor --> ConnHandler
    Executor --> SysMetrics
    ConnHandler --> ResConnectors
    ResConnectors --> DBConnectors
    ResConnectors --> CloudConn
    ResConnectors --> CompRes
    Storage --> FileStore
    Storage --> S3Store
    Storage --> GCSStore
    DBConnectors --> DB
    CloudConn --> Cloud
    CompRes --> Compute

    %% Click Events
    click UI "https://github.com/RunLLM/aqueduct/tree/main/src/ui"
    click SDK "https://github.com/RunLLM/aqueduct/tree/main/sdk/aqueduct"
    click APIServer "https://github.com/RunLLM/aqueduct/tree/main/src/golang/cmd/server"
    click RequestHandler "https://github.com/RunLLM/aqueduct/tree/main/src/golang/cmd/server/handler"
    click Auth "https://github.com/RunLLM/aqueduct/tree/main/src/golang/cmd/server/middleware/authentication"
    click Executor "https://github.com/RunLLM/aqueduct/tree/main/src/python/aqueduct_executor"
    click FuncExec "https://github.com/RunLLM/aqueduct/tree/main/src/python/aqueduct_executor/operators/function_executor"
    click ConnHandler "https://github.com/RunLLM/aqueduct/tree/main/src/python/aqueduct_executor/operators/connectors"
    click SysMetrics "https://github.com/RunLLM/aqueduct/tree/main/src/python/aqueduct_executor/operators/system_metric_executor"
    click Storage "https://github.com/RunLLM/aqueduct/tree/main/src/golang/lib/storage"
    click FileStore "https://github.com/RunLLM/aqueduct/blob/main/src/golang/lib/storage/file.go"
    click S3Store "https://github.com/RunLLM/aqueduct/blob/main/src/golang/lib/storage/s3.go"
    click GCSStore "https://github.com/RunLLM/aqueduct/blob/main/src/golang/lib/storage/gcs.go"
    click DBConnectors "https://github.com/RunLLM/aqueduct/tree/main/src/python/aqueduct_executor/operators/connectors/data"
    click CloudConn "https://github.com/RunLLM/aqueduct/tree/main/src/golang/lib/container_registry"
    click CompRes "https://github.com/RunLLM/aqueduct/tree/main/src/golang/lib/k8s"
    click WorkflowMgr "https://github.com/RunLLM/aqueduct/tree/main/src/golang/lib/workflow"
```
