How moh-a-abde/fl-cml-pipeline works
Project type and purpose
• This is a full-stack machine-learning pipeline implementing privacy-preserving federated learning (FL) with the Flower framework.
• It ingests live/streamed network traffic, preprocesses it, partitions data among simulated clients, trains an XGBoost model in parallel (bagging) or sequentially (cyclic), and evaluates/exports results.
Top-level components (directory/filenames)
• Data Ingestion & Preprocessing
• data/livepreprocessing_socket.py (Kafka/Zeek stream reader)
• data/receiving_data.py (persist incoming records)
• dataset.py (batch loading, cleaning, train/test split, client partitioning)
• Federated Learning Core
• server.py + server_utils.py (FL server, strategy selection FedXgbBagging/FedXgbCyclic)
• client.py + client_utils.py (FL clients, local XGBoost training, model updates)
• Training Orchestration & Simulation
• run.py (end-to-end orchestration via Hydra)
• run_bagging.sh / run_cyclic.sh (shell wrappers)
• sim.py (local simulation harness)
• Hyperparameter Tuning
• ray_tune_xgboost*.py, tuned_params.py, tune_results/ (Ray Tune integration)
• Utilities & Reporting
• utils.py, visualization_utils.py (shared helpers, plots)
• CML workflow (.github/workflows/cml.yaml) for CI/CD and result reporting
• Outputs & Logs
• outputs/ (per-run model artifacts, predictions, .hydra config snapshots)
• results/ (aggregated metrics)
Key interactions & data flow
a. External Data Source (Kafka/Zeek stream) → Data Ingestion
→ data/… scripts persist raw network records
b. dataset.py reads raw CSVs → applies cleaning, feature engineering → partitions into client shards
c. FL Server (server.py) spins up on a designated port; defines federated strategy (bagging or cyclic)
d. Multiple FL Clients (client.py) connect to server via Flower RPC
• Each client loads its shard from dataset.py
• Performs local XGBoost training (client_utils.py)
• Returns weight updates or candidate trees to server
e. Server aggregates updates (FedXgbBagging or FedXgbCyclic) → updates global model → next round
f. At completion: server writes results.pkl, predictions_round_X.csv into outputs/
g. CML GitHub Action picks up outputs → generates dashboards/metrics in results/
Architectural patterns & principles
• Modular separation: Data layer vs ML core vs orchestration vs tuning
• Federated strategy pattern: interchangeable bagging/cyclic via Flower strategies
• Configuration-driven: Hydra for experiment parameters, .hydra snapshots under outputs
• Reproducibility: deterministic output folder structure (date/time) and CML integration