# wizard1203/gossipfl architecture

> Architecture diagram of the GitHub repository wizard1203/gossipfl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wizard1203/gossipfl
- Repository: https://github.com/wizard1203/gossipfl

## Overview

##### Main Components
a. Experiment Launcher & Config Manager
b. Data Preprocessing & Loaders
c. Model Definitions
d. Algorithm Modules
e. Communication Core (fedml_core)
f. Optimizers & Compression
g. Trainers & Aggregators
h. Logging, Metrics & Checkpointing
i. External Infrastructure (MPI/MQTT broker, GPUs, Machines)

##### Relationships & Data Flows
– The **Experiment Launcher** reads configuration files (from `experiments/configs_*`) and initializes the **Topology Manager** in the Communication Core.
– For each client and/or server process:
- It invokes the corresponding **Algorithm API** (e.g. `CHOCO_SGD_API.py`, `FedAvgAPI.py`, etc.).
- The API sets up its **Trainer** (in `MyModelTrainer.py` or `model_trainer.py`) with a **Model** (from `model/…`) and **Optimizer** (from `optim/…`).
- It pulls local data via the **Data Preprocessing** module (in `data_preprocessing/…` or `data/…`).
– At each training step, the **Trainer** computes gradients, calls the **Communication Module** (in `fedml_core/distributed/communication`) to exchange messages according to the **Topology Manager** (in `fedml_core/distributed/topology`).
– Optionally, **Compression** (in `compression/…`) is applied before sending updates.
– Updates arrive at other nodes or the central aggregator (in `basePS/ps_aggregator.py` or decentralized managers), which merge them and push new parameters back out.
– Meanwhile, **Logging & Metrics** (in `utils/logger.py`, `utils/metrics.py`, `utils/checkpoint.py`, `utils/wandb_util.py`) record progress and save checkpoints.

##### Architectural Patterns
– **Plugin Architecture**: Each algorithm lives in its own folder under `algorithms/…`, sharing a common API and message definitions.
– **Separation of Concerns**:
- Data concerns live under `data_preprocessing/…` and `data/…`.
- Model architectures under `model/…`.
- Low-level communication under `fedml_core/…`.
- Experiment orchestration in `experiments/…` and `configs/…`.
– **Distributed Topology Abstraction**: Centralized PS vs. fully decentralized via interchangeable topology managers.
– **Communication Abstraction**: Support for multiple backends – MPI, MQTT, etc., all conforming to `base_com_manager`.

##### Detailed Elements to Capture
– **Experiment Launcher** reads `main_args.conf` → kicks off MPI-based job (`experiments/mpi_based/main.py`).
– **Communication Stack**: `fedml_core/distributed/communication/{mpi,mqtt}` + `fedml_core/distributed/communication/base_com_manager.py`.
– **Topology Management**: Symmetric vs. Asymmetric (`symmetric_topology_manager.py` vs. `asymmetric_topology_manager.py`).
– **Algorithm Flow**: API → Trainer → Model → Optimizer → Communicator → Aggregator/Peer.
– **Data Pipeline**: Raw datasets (`data/…`) → Preprocessor (`data_preprocessing/…`) → IID/non-IID loaders → Trainer input.
– **Metrics & Logging**: `utils/logger.py`, `utils/metrics.py`, `utils/checkpoint.py`.

– Clearly separates the major layers (experiment orchestration, communication, algorithm logic, data, models).
– Shows how control and data move through the system during a federated training run.
– Highlights the project’s extensible plugin architecture and multi-backend communication support.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Experiment Orchestration
    subgraph "Experiment Orchestration" 
        EL["Experiment Launcher"]:::orchestration
        CM["Config Manager"]:::orchestration
    end

    %% Data Layer
    subgraph "Data Layer"
        DP["Data Preprocessor"]:::data
        DL["Data Loader"]:::data
    end

    %% Model Layer
    subgraph "Model Layer"
        MB["Model Builder"]:::model
        CV["CV Models"]:::model
        MS["Model Scan"]:::model
    end

    %% Algorithm Plugins
    subgraph "Algorithm Plugins"
        CHOCO_API["CHOCO_SGD_API"]:::algorithm
        CHOCO_TR["CHOCO_SGD Trainer"]:::algorithm
        FEDAVG_API["FedAvgAPI"]:::algorithm
        FEDAVG_TR["FedAvg Trainer"]:::algorithm
        BASEPS_AGG["PS Aggregator"]:::algorithm
        BASEPS_TR["PS Client Trainer"]:::algorithm
        OTHER_API["Other Algorithm APIs..."]:::algorithm
    end

    %% Communication Core
    subgraph "Communication Core"
        BCM["BaseComManager"]:::comm
        MPI["MPI Comm Manager"]:::comm
        MQTT["MQTT Comm Manager"]:::comm
        STM["Symmetric Topology Manager"]:::comm
        ATM["Asymmetric Topology Manager"]:::comm
        CMG["Client Manager"]:::comm
        SMG["Server Manager"]:::comm
    end

    %% Optimization & Compression
    subgraph "Optimizers & Compression"
        OPT["Optimizers"]:::optimization
        COMP["Compression"]:::optimization
    end

    %% Trainer & Aggregator
    subgraph "Trainer & Aggregator"
        MT["ModelTrainer"]:::trainer
        NT["Normal Trainer"]:::trainer
    end

    %% Logging & Metrics
    subgraph "Logging & Metrics"
        LOG["Logger"]:::util
        MET["Metrics"]:::util
        CP["Checkpoint"]:::util
        WB["WandB Util"]:::util
    end

    %% External Infrastructure
    subgraph "External Infrastructure"
        MPI_BG["MPI Scheduler & Scripts"]:::infra
        MQTT_BR["MQTT Broker"]:::infra
        GPU["GPU / Machine Mapping"]:::infra
    end

    %% Connections
    EL -->|reads config| CM
    EL -->|initializes| STM
    EL -->|launches API| CHOCO_API
    EL -->|launches API| FEDAVG_API
    EL -->|launches API| OTHER_API

    DP --> DL
    DL -->|provides data| MT
    DL -->|provides data| CHOCO_TR
    DL -->|provides data| FEDAVG_TR

    MB --> CHOCO_TR
    MB --> FEDAVG_TR
    CV --> MB
    MS --> MB

    CHOCO_API --> CHOCO_TR
    FEDAVG_API --> FEDAVG_TR
    BASEPS_AGG --> MT
    BASEPS_TR --> MT

    MT -->|compute gradients| BCM
    CHOCO_TR -->|compute gradients| BCM
    FEDAVG_TR -->|compute gradients| BCM

    BCM --> MPI
    BCM --> MQTT
    MPI --> STM
    MQTT --> STM
    MPI --> ATM
    MQTT --> ATM

    STM --> CMG
    ATM --> CMG
    CMG -->|coordinates| SMG
    SMG -->|aggregates| BASEPS_AGG

    OPT --> CHOCO_TR
    OPT --> FEDAVG_TR
    COMP --> BCM

    MT --> LOG
    CHOCO_TR --> LOG
    FEDAVG_TR --> LOG
    MT --> MET
    MT --> CP
    MT --> WB

    MPI_BG --> MPI
    MQTT_BR --> MQTT
    GPU --> CMG

    %% Click Events - Experiment Orchestration
    click EL "https://github.com/wizard1203/gossipfl/blob/main/experiments/mpi_based/main.py"
    click CM "https://github.com/wizard1203/gossipfl/blob/main/configs/config.py"
    click CM "https://github.com/wizard1203/gossipfl/blob/main/configs/default.py"
    click EL "https://github.com/wizard1203/gossipfl/blob/main/experiments/main_args.conf"

    %% Click Events - Data Layer
    click DP "https://github.com/wizard1203/gossipfl/tree/main/data_preprocessing"
    click DL "https://github.com/wizard1203/gossipfl/tree/main/data"

    %% Click Events - Model Layer
    click MB "https://github.com/wizard1203/gossipfl/blob/main/model/build.py"
    click CV "https://github.com/wizard1203/gossipfl/tree/main/model/cv"
    click MS "https://github.com/wizard1203/gossipfl/blob/main/model/utils/model_scan.py"

    %% Click Events - Algorithm Plugins
    click CHOCO_API "https://github.com/wizard1203/gossipfl/blob/main/algorithms/CHOCO_SGD/CHOCO_SGD_API.py"
    click CHOCO_TR "https://github.com/wizard1203/gossipfl/blob/main/algorithms/CHOCO_SGD/MyModelTrainer.py"
    click FEDAVG_API "https://github.com/wizard1203/gossipfl/blob/main/algorithms/fedavg/FedAvgAPI.py"
    click FEDAVG_TR "https://github.com/wizard1203/gossipfl/blob/main/algorithms/fedavg/FedAVGTrainer.py"
    click BASEPS_AGG "https://github.com/wizard1203/gossipfl/blob/main/algorithms/basePS/ps_aggregator.py"
    click BASEPS_TR "https://github.com/wizard1203/gossipfl/blob/main/algorithms/basePS/ps_client_trainer.py"

    %% Click Events - Communication Core
    click BCM "https://github.com/wizard1203/gossipfl/blob/main/fedml_core/distributed/communication/base_com_manager.py"
    click MPI "https://github.com/wizard1203/gossipfl/blob/main/fedml_core/distributed/communication/mpi/com_manager.py"
    click MQTT "https://github.com/wizard1203/gossipfl/blob/main/fedml_core/distributed/communication/mqtt/mqtt_comm_manager.py"
    click STM "https://github.com/wizard1203/gossipfl/blob/main/fedml_core/distributed/topology/symmetric_topology_manager.py"
    click ATM "https://github.com/wizard1203/gossipfl/blob/main/fedml_core/distributed/topology/asymmetric_topology_manager.py"
    click CMG "https://github.com/wizard1203/gossipfl/blob/main/fedml_core/distributed/client/client_manager.py"
    click SMG "https://github.com/wizard1203/gossipfl/blob/main/fedml_core/distributed/server/server_manager.py"

    %% Click Events - Optimizers & Compression
    click OPT "https://github.com/wizard1203/gossipfl/blob/main/optim/build.py"
    click COMP "https://github.com/wizard1203/gossipfl/blob/main/compression/compression.py"

    %% Click Events - Trainer & Aggregator
    click MT "https://github.com/wizard1203/gossipfl/blob/main/fedml_core/trainer/model_trainer.py"
    click NT "https://github.com/wizard1203/gossipfl/blob/main/trainers/normal_trainer.py"

    %% Click Events - Logging & Metrics
    click LOG "https://github.com/wizard1203/gossipfl/blob/main/utils/logger.py"
    click MET "https://github.com/wizard1203/gossipfl/blob/main/utils/metrics.py"
    click CP "https://github.com/wizard1203/gossipfl/blob/main/utils/checkpoint.py"
    click WB "https://github.com/wizard1203/gossipfl/blob/main/utils/wandb_util.py"

    %% Click Events - External Infrastructure
    click MPI_BG "https://github.com/wizard1203/gossipfl/blob/main/experiments/mpi_based/launch_mpi_based.sh"
    click MPI_BG "https://github.com/wizard1203/gossipfl/tree/main/batch_experiment_scripts_for_GossipFL"
    click MQTT_BR "https://github.com/wizard1203/gossipfl/tree/main/fedml_core/distributed/communication/mqtt"
    click GPU "https://github.com/wizard1203/gossipfl/blob/main/utils/gpu_mapping.py"

    %% Styles
    classDef orchestration fill:#ffe0b2,stroke:#ff9800
    classDef data fill:#e3f2fd,stroke:#2196f3
    classDef model fill:#f3e5f5,stroke:#9c27b0
    classDef algorithm fill:#ffebee,stroke:#f44336
    classDef comm fill:#e8f5e9,stroke:#4caf50
    classDef optimization fill:#fff3e0,stroke:#ffb74d
    classDef trainer fill:#ede7f6,stroke:#673ab7
    classDef util fill:#f0f4c3,stroke:#cddc39
    classDef infra fill:#eceff1,stroke:#607d8b
```
