# paddlepaddle/paddleflow architecture

> Architecture diagram of the GitHub repository paddlepaddle/paddleflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/paddlepaddle/paddleflow
- Repository: https://github.com/paddlepaddle/paddleflow

## Overview

##### Project type and purpose
- PaddleFlow is a cloud-native, full-stack AI resource management and scheduling platform. It provides:
- A command-line & Python client (including a POSIX-compatible Fuse filesystem with two-layer caching)
- A Go-based API server with controllers for queues, storage, jobs, and workflows
- A Kubernetes CSI plugin for mounting PaddleFlowFS into pods
- Integration with Kubernetes/K3s and the Volcano batch-scheduler

##### File structure
- Top-level directories:
- client/      → Python CLI and Fuse filesystem code
- cmd/         → Standalone Fuse and server binaries entrypoints
- pkg/apiserver/ → Server HTTP handlers, controllers, models, routers
- pkg/fs/        → Filesystem client library, CSI plugin, cache, location-awareness
- pkg/job/, pkg/pipeline/, pkg/storage/ → Core scheduling, workflow, persistence layers
- installer/     → Kubernetes manifests, CRDs, Dockerfiles for deployment
- go-sdk/        → Go client library and examples

##### Extract architecture insights from the README
- Four main components as described in the README:
1. PaddleFlow Client + Fuse FS
2. PaddleFlow Server
3. Volcano scheduler (with elastic-quota patches)
4. paddleflow-csi-plugin

##### Details
- Every core component (cache, pipeline DSL compiler, job plugins)
- Retries/fallbacks (e.g., cache miss → fetch from S3)
- Multi-tenant aspects: queue namespacing, resourcepools
- Call out dynamic vs. static objects: CRDs vs. standard Deployments

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Clients Layer
    subgraph "Clients Layer"
        direction TB
        PFClient["PaddleFlow Python Client"]:::clients
        GoSDK["Go SDK Client Library"]:::clients
    end

    %% Filesystem Layer
    subgraph "Filesystem Layer"
        direction TB
        FuseClient["PaddleFlowFS Fuse Client"]:::fs
        CacheLayer["Two-layer Cache Manager"]:::fs
        LocationAwareness["Location-Awareness Scheduler"]:::fs
        UFSModules["External Data Source Modules\n(S3/HDFS/Local)"]:::storage
    end

    %% Control Layer (API Server)
    subgraph "Control Layer"
        direction TB
        APIHandlers["API Server Handlers"]:::server
        APIRouter["API Server Routers (v1 REST)"]:::server
        Controllers["API Server Controllers"]:::server
        Models["API Server Models & Persistence"]:::server
        Database["PostgreSQL Database"]:::database
    end

    %% Scheduling Layer
    subgraph "Scheduling Layer"
        direction TB
        K8sClient["Kubernetes/K3s Runtime Client"]:::k8s
        Volcano["Volcano Scheduler"]:::k8s
        ElasticQuota["Elastic-Quota Extension"]:::k8s
    end

    %% CSI Plugin
    subgraph "CSI Plugin"
        direction TB
        CSIPlugin["Kubernetes CSI Plugin"]:::k8s
        CSIBinary["CSI Plugin Entrypoint Binary"]:::k8s
        FuseBinary["Fuse Daemon Entrypoint Binary"]:::k8s
    end

    %% Storage & Monitoring
    subgraph "External Storage"
        direction TB
        S3["S3/BOS"]:::storage
        HDFS["HDFS"]:::storage
        LocalFS["Local FS"]:::storage
    end
    subgraph "Monitoring & Metrics"
        direction TB
        Exporter["Prometheus Exporter"]:::monitor
        Metrics["Metrics Management"]:::monitor
        TraceLogger["Trace Logging Service"]:::monitor
    end

    %% Core Engine & Storage Abstraction
    subgraph "Core Engine & Storage"
        direction TB
        Pipeline["Workflow & Pipeline Engine"]:::server
        StorageAbstraction["Storage Abstraction Layer"]:::storage
    end

    %% Connections
    PFClient -->|"HTTP/REST"| APIHandlers
    GoSDK -->|"HTTP/REST"| APIHandlers
    APIHandlers --> APIRouter
    APIRouter --> Controllers
    Controllers --> Models
    Models -->|"SQL"| Database

    Controllers -->|"K8s API"| K8sClient
    K8sClient --> Volcano
    Volcano --> ElasticQuota

    Controllers -->|"DAG compile"| Pipeline
    Pipeline --> StorageAbstraction

    FuseClient --> CacheLayer
    CacheLayer --> LocationAwareness
    CacheLayer --> UFSModules
    UFSModules --> S3
    UFSModules --> HDFS
    UFSModules --> LocalFS

    CSIPlugin --> CSIBinary
    CSIPlugin --> FuseBinary
    CSIBinary --> K8sClient
    FuseBinary --> FuseClient

    Exporter --> PFClient
    Exporter --> APIHandlers
    Exporter --> CSIPlugin
    Metrics --> Controllers
    TraceLogger --> Controllers

    %% Click Events
    click PFClient "https://github.com/paddlepaddle/paddleflow/tree/develop/client/paddleflow/"
    click GoSDK "https://github.com/paddlepaddle/paddleflow/tree/develop/go-sdk/"
    click FuseClient "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/fs/client/fuse"
    click CacheLayer "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/fs/client/cache"
    click LocationAwareness "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/fs/location-awareness"
    click UFSModules "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/fs/client/ufs"
    click CSIPlugin "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/fs/csiplugin"
    click CSIBinary "https://github.com/paddlepaddle/paddleflow/tree/develop/cmd/fs/csi-plugin"
    click FuseBinary "https://github.com/paddlepaddle/paddleflow/tree/develop/cmd/fs/fuse"
    click APIHandlers "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/apiserver/handler"
    click APIRouter "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/apiserver/router/v1"
    click Controllers "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/apiserver/controller"
    click Models "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/apiserver/models"
    click Database "https://github.com/paddlepaddle/paddleflow/blob/develop/installer/database/paddleflow-pg.sql"
    click K8sClient "https://github.com/paddlepaddle/paddleflow/blob/develop/pkg/job/runtime_v2/client/k3s_runtime_client.go"
    click Volcano "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/job/runtime_v2/queue/vcqueue"
    click ElasticQuota "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/job/runtime_v2/queue/elasticqueue"
    click Exporter "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/monitor"
    click Metrics "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/metrics"
    click TraceLogger "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/trace_logger"
    click Pipeline "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/pipeline"
    click StorageAbstraction "https://github.com/paddlepaddle/paddleflow/tree/develop/pkg/storage"

    %% Styles
    classDef clients fill:#f9f,stroke:#333,stroke-width:1px;
    classDef fs fill:#bbf,stroke:#333,stroke-width:1px;
    classDef server fill:#8fd,stroke:#333,stroke-width:1px;
    classDef k8s fill:#afa,stroke:#333,stroke-width:1px;
    classDef storage fill:#fb0,stroke:#333,stroke-width:1px;
    classDef database fill:#888,stroke:#333,stroke-width:1px,color:#fff;
    classDef monitor fill:#f66,stroke:#333,stroke-width:1px;
```
