# bytedance/fedlearner architecture

> Architecture diagram of the GitHub repository bytedance/fedlearner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bytedance/fedlearner
- Repository: https://github.com/bytedance/fedlearner

## Overview

##### Project type and purpose
- Fedlearner is a full-stack, federated machine-learning framework that spans:
- Core ML engine (Python + C++ custom TensorFlow ops)
- Orchestration on Kubernetes (Go apiserver & operator, Helm charts, Docker)
- Cross-site communication (gRPC channel, proxy)
- User interface (web_console and web_console_v2 front-ends + back-end API)
- CLI tools and scripts for data preparation, PSI, training, etc.

##### File structure
- Top-level folders:
- cc/ – C++ kernels & ops for custom TensorFlow layers
- deploy/ – Helm charts (fedlearner-stack, apiserver, operator, PVC, add-on), examples, scripts
- kubernetes_operator/ – Go code for CRD definitions, apiserver & operator controllers
- proxy/ – Lua router, patched Nginx gRPC module
- fedlearner/ – Python packages for channel, common, data_join, fedavg, model, trainer
- web_console/ and web_console_v2/ – two versions of UI (Node.js/Next.js + React + TypeScript + Python API)
- protocols/ – .proto definitions for all services
- scripts/, docker/, Makefile, README.md, integration_tests.sh, etc.

##### README
- Confirms “collaborative machine learning framework” for joint modeling of distributed data.

##### Be detailed but avoid over-splitting
- Essential microservices and their interactions
- Leader vs follower parallel structure

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface" 
        direction TB
        CLI["CLI Tools"]:::frontend
        WebV1["Web Console v1 UI"]:::frontend
        WebV2["Web Console v2 UI"]:::frontend
    end

    %% API Layer
    subgraph "API Layer"
        direction TB
        API1["web_console API Server"]:::api
        API2["web_console_v2 API Server"]:::api
    end

    %% Control Plane
    subgraph "Control Plane"
        direction TB
        K8sAPI["Kubernetes API"]:::control
        APIServer["Fedlearner API Server"]:::control
        Operator["Fedlearner Operator"]:::control
        Helm["Helm Charts"]:::control
    end

    %% Worker Pods
    subgraph "Worker Pods"
        direction TB
        subgraph "Leader Site"
            direction TB
            DJM["DataJoin Master"]:::worker
            DJW["DataJoin Worker"]:::worker
            TM["Trainer Master"]:::worker
            TW["Trainer Worker"]:::worker
        end
        subgraph "Follower Site"
            direction TB
            DJM_F["DataJoin Master"]:::worker
            DJW_F["DataJoin Worker"]:::worker
            TM_F["Trainer Master"]:::worker
            TW_F["Trainer Worker"]:::worker
        end
    end

    %% Proxy & Discovery
    subgraph "Proxy & Discovery"
        direction TB
        FLRouter["fl_router (Lua/Nginx)"]:::proxy
        EtcdResolver["etcd Resolver"]:::proxy
    end

    %% Storage & External
    subgraph "Storage & External"
        direction TB
        MySQL["MySQL/LevelDB"]:::storage
        DFS["HDFS/DFS"]:::storage
        EtcdDB["etcd"]:::storage
        Logs["Elasticsearch"]:::storage
    end

    %% Custom Ops
    CustomOps["Custom TF Ops"]:::compute

    %% Connections: UI to API
    CLI -->|invoke| API1
    CLI -->|invoke| API2
    WebV1 -->|HTTP| API1
    WebV2 -->|HTTP| API2

    %% API to K8s
    API1 -->|create CRD| K8sAPI
    API2 -->|create CRD| K8sAPI
    K8sAPI -->|watch| Operator
    Operator -->|deploy| DJM
    Operator -->|deploy| DJW
    Operator -->|deploy| TM
    Operator -->|deploy| TW
    Operator -->|deploy| DJM_F
    Operator -->|deploy| DJW_F
    Operator -->|deploy| TM_F
    Operator -->|deploy| TW_F

    %% DataJoin gRPC flows
    DJM -->|gRPC| FLRouter
    FLRouter -->|gRPC| DJM_F
    DJW -->|gRPC| FLRouter
    FLRouter -->|gRPC| DJW_F

    %% Trainer gRPC flows
    TM -->|gRPC| FLRouter
    FLRouter -->|gRPC| TM_F
    TW -->|gRPC| FLRouter
    FLRouter -->|gRPC| TW_F

    %% Storage interactions
    DJM -->|write| DFS
    DJW -->|write| DFS
    DJM -->|metadata| EtcdDB
    DJW -->|metadata| EtcdDB
    TM -->|checkpoints| MySQL
    TW -->|checkpoints| MySQL
    TM -->|metrics| EtcdDB
    TW -->|metrics| EtcdDB
    DJM -->|logs| Logs
    TM -->|logs| Logs

    %% Custom Ops usage
    TM -->|uses| CustomOps
    TW -->|uses| CustomOps

    %% Click Events
    click DJM "https://github.com/bytedance/fedlearner/blob/master/fedlearner/data_join/data_join_master.py"
    click DJW "https://github.com/bytedance/fedlearner/blob/master/fedlearner/data_join/data_join_worker.py"
    click TM "https://github.com/bytedance/fedlearner/blob/master/fedlearner/trainer/trainer_master.py"
    click TW "https://github.com/bytedance/fedlearner/blob/master/fedlearner/trainer/trainer_worker.py"
    click CustomOps "https://github.com/bytedance/fedlearner/blob/master/cc/operators/kernels/embedding_pooling.cc"
    click CustomOps "https://github.com/bytedance/fedlearner/blob/master/cc/operators/ops/embedding_pooling.cc"
    click CustomOps "https://github.com/bytedance/fedlearner/blob/master/cc/operators/utils/common.h"
    click Operator "https://github.com/bytedance/fedlearner/blob/master/kubernetes_operator/cmd/operator/main.go"
    click APIServer "https://github.com/bytedance/fedlearner/blob/master/kubernetes_operator/cmd/apiserver/main.go"
    click Helm "https://github.com/bytedance/fedlearner/blob/master/deploy/charts/fedlearner-apiserver/Chart.yaml"
    click Helm "https://github.com/bytedance/fedlearner/blob/master/deploy/charts/fedlearner-operator/Chart.yaml"
    click Helm "https://github.com/bytedance/fedlearner/blob/master/deploy/charts/fedlearner-stack/Chart.yaml"
    click FLRouter "https://github.com/bytedance/fedlearner/blob/master/deploy/proxy/fl_router/router.lua"
    click EtcdResolver "https://github.com/bytedance/fedlearner/blob/master/deploy/proxy/fl_router/etcd_resolver.lua"
    click MySQL "https://github.com/bytedance/fedlearner/blob/master/fedlearner/common/mysql_client.py"
    click EtcdDB "https://github.com/bytedance/fedlearner/blob/master/fedlearner/common/etcd_client.py"
    click API2 "https://github.com/bytedance/fedlearner/blob/master/web_console_v2/api/fedlearner_webconsole/app.py"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#004085,color:#004085
    classDef api fill:#b8daff,stroke:#004085,color:#004085
    classDef control fill:#d4edda,stroke:#155724,color:#155724
    classDef worker fill:#ffe5b4,stroke:#ff8c00,color:#ff8c00
    classDef proxy fill:#fff3cd,stroke:#856404,color:#856404
    classDef storage fill:#e2e3e5,stroke:#6c757d,color:#6c757d
    classDef compute fill:#e2d4f0,stroke:#6f42c1,color:#6f42c1
```
