# moh-a-abde/fl-cml-pipeline architecture

> Architecture diagram of the GitHub repository moh-a-abde/fl-cml-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/moh-a-abde/fl-cml-pipeline
- Repository: https://github.com/moh-a-abde/fl-cml-pipeline

## Overview

##### Project type and purpose
- This is a full-stack machine-learning pipeline implementing privacy-preserving federated learning (FL) with the Flower framework.
- It ingests live/streamed network traffic, preprocesses it, partitions data among simulated clients, trains an XGBoost model in parallel (bagging) or sequentially (cyclic), and evaluates/exports results.

##### Top-level components (directory/filenames)
- Data Ingestion & Preprocessing
- data/livepreprocessing_socket.py (Kafka/Zeek stream reader)
- data/receiving_data.py (persist incoming records)
- dataset.py (batch loading, cleaning, train/test split, client partitioning)
- Federated Learning Core
- server.py + server_utils.py (FL server, strategy selection FedXgbBagging/FedXgbCyclic)
- client.py + client_utils.py (FL clients, local XGBoost training, model updates)
- Training Orchestration & Simulation
- run.py (end-to-end orchestration via Hydra)
- run_bagging.sh / run_cyclic.sh (shell wrappers)
- sim.py (local simulation harness)
- Hyperparameter Tuning
- ray_tune_xgboost*.py, tuned_params.py, tune_results/ (Ray Tune integration)
- Utilities & Reporting
- utils.py, visualization_utils.py (shared helpers, plots)
- CML workflow (.github/workflows/cml.yaml) for CI/CD and result reporting
- Outputs & Logs
- outputs/ (per-run model artifacts, predictions, .hydra config snapshots)
- results/ (aggregated metrics)

##### Key interactions & data flow
a. External Data Source (Kafka/Zeek stream) → Data Ingestion
→ data/… scripts persist raw network records
b. dataset.py reads raw CSVs → applies cleaning, feature engineering → partitions into client shards
c. FL Server (server.py) spins up on a designated port; defines federated strategy (bagging or cyclic)
d. Multiple FL Clients (client.py) connect to server via Flower RPC
- Each client loads its shard from dataset.py
- Performs local XGBoost training (client_utils.py)
- Returns weight updates or candidate trees to server
e. Server aggregates updates (FedXgbBagging or FedXgbCyclic) → updates global model → next round
f. At completion: server writes results.pkl, predictions_round_X.csv into outputs/
g. CML GitHub Action picks up outputs → generates dashboards/metrics in results/

##### Architectural patterns & principles
- Modular separation: Data layer vs ML core vs orchestration vs tuning
- Federated strategy pattern: interchangeable bagging/cyclic via Flower strategies
- Configuration-driven: Hydra for experiment parameters, .hydra snapshots under outputs
- Reproducibility: deterministic output folder structure (date/time) and CML integration

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Data Source
    ExtDS["Kafka/Zeek Stream"]:::data

    %% Data Pipeline
    subgraph "Data Ingestion & Preprocessing"
        direction TB
        LP["livepreprocessing_socket.py"]:::data
        RD["receiving_data.py"]:::data
        DS["dataset.py"]:::data
    end

    %% Federated Learning Core
    subgraph "Federated Learning Core"
        direction TB
        SV["server.py"]:::fl
        SU["server_utils.py"]:::fl
        subgraph "Clients (×N)"
            direction TB
            CL["client.py"]:::fl
            CU["client_utils.py"]:::fl
        end
    end

    %% Hyperparameter Tuner
    subgraph "Hyperparameter Tuning (Ray Tune)"
        direction TB
        RT1["ray_tune_xgboost.py"]:::tuning
        RT2["ray_tune_xgboost_updated.py"]:::tuning
        RU["update_ray_tune_xgboost.py"]:::tuning
        TP["tuned_params.py"]:::tuning
        RW["run_ray_tune.sh"]:::tuning
    end

    %% Orchestration & CI/CD
    subgraph "Orchestration & CI/CD"
        direction TB
        RUN["run.py"]:::orchestration
        RB["run_bagging.sh"]:::orchestration
        RC["run_cyclic.sh"]:::orchestration
        SIM["sim.py"]:::orchestration
        CML[".github/workflows/cml.yaml"]:::orchestration
    end

    %% Storage
    subgraph "Storage"
        direction TB
        OUT["outputs/"]:::storage
        RES["results/"]:::storage
        TR["tune_results/"]:::storage
    end

    %% Connections
    ExtDS -->|ingest| LP
    LP -->|persist raw records| RD
    RD -->|batch load & clean| DS
    DS -->|shard data| CL

    %% Federated interactions
    CL <-->|"model weights/updates"| SV
    SV -->|aggregates| SU
    SU -->|config strategy| SV

    %% Orchestration triggers
    RUN -->|start ingestion| LP
    RUN -->|launch server| SV
    RUN -->|spawn clients| CL
    RUN -->|invoke tuning| RW
    RB -->|wrapper| RUN
    RC -->|wrapper| RUN
    SIM -->|simulate| CL

    %% Hyperparameter tuning loop
    RW -->|hyperparameter search| RT1
    RT1 --> RT2
    RT2 --> RU
    RU --> TP
    TP -->|update config| SV

    %% Outputs & Reporting
    SV -->|persist results| OUT
    CL -->|persist predictions| OUT
    RW -->|store tuning outputs| TR
    OUT -->|CML picks up| CML
    CML -->|generate dashboards| RES

    %% Click Events
    click LP "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/data/livepreprocessing_socket.py"
    click RD "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/data/receiving_data.py"
    click DS "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/dataset.py"
    click SV "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/server.py"
    click SU "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/server_utils.py"
    click CL "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/client.py"
    click CU "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/client_utils.py"
    click RUN "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/run.py"
    click RB "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/run_bagging.sh"
    click RC "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/run_cyclic.sh"
    click SIM "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/sim.py"
    click RT1 "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/ray_tune_xgboost.py"
    click RT2 "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/ray_tune_xgboost_updated.py"
    click RU "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/update_ray_tune_xgboost.py"
    click TP "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/tuned_params.py"
    click RW "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/run_ray_tune.sh"
    click CML "https://github.com/moh-a-abde/fl-cml-pipeline/blob/main/.github/workflows/cml.yaml"
    click OUT "https://github.com/moh-a-abde/fl-cml-pipeline/tree/main/outputs/"
    click RES "https://github.com/moh-a-abde/fl-cml-pipeline/tree/main/results/"
    click TR "https://github.com/moh-a-abde/fl-cml-pipeline/tree/main/tune_results/"

    %% Styles
    classDef data fill:#cce5ff,stroke:#3399ff,color:#000;
    classDef fl fill:#d4edda,stroke:#28a745,color:#000;
    classDef orchestration fill:#ffe5b4,stroke:#ff8c00,color:#000;
    classDef tuning fill:#f0e68c,stroke:#b8860b,color:#000;
    classDef storage fill:#e2e3e5,stroke:#6c757d,color:#000;
```
