How eurekaer/fedvad works
Project Type and Purpose
• This is an open-source, research-oriented Federated Learning system for Video Anomaly Detection (FedVAD) that integrates GPT-driven semantic distillation.
• It orchestrates multiple local “clients” (each holding video feature subsets) to train a shared model without centralizing raw data, and refines local representations via a semantic distillation step powered by GPT/BLIP2/BERT utilities.
Major Components
a. Data & Feature Extraction
• Raw video datasets (UCF-Crime, XD-Violence, ShanghaiTech, UBnormal)
• Off-the-shelf I3D feature extractor (external repo)
b. Client Splitting Module (Client_Split/…)
• Strategies: Unsupervised, Weakly-Supervised (ST and XD variants)
• Outputs per-client feature lists and ground-truth arrays
c. Federated Learning Engine (Code/…)
• main.py – Coordinates training rounds, dispatches model to clients
• train.py – Local client training logic (forward/backward)
• test.py – Local evaluation on test splits
• distill.py – Semantic distillation using GPT embeddings
• fusion.py – Merges local model updates & distilled semantics
• cluster_utils.py – Helper functions for client clustering
• loss.py, layers.py, modules.py, model.py – Core model definitions and loss functions
• log.py – Experiment logging and metrics aggregation
d. External Semantic Services (other/…)
• bert_utils.ipynb, blip2_utils.ipynb, gpt4_utils.ipynb – Notebooks and helper scripts to query/fine-tune large language/vision models
e. Environment & Configuration
• environment.yml – Conda environment specification
• README.md – High-level description, requirements, dataset links
Relationships & Data Flow
1. Data Preparation: Raw Videos → Off-the-Shelf Feature Extractors → Saved Features
2. Split → Client_Split → Per-Client Feature Lists & GT Labels
3. Federated Loop (in main.py):
a. Server initializes global model
b. Server broadcasts model → Clients
c. Each Client runs train.py:
– Forward on local features
– Optionally call distill.py → query GPT/BLIP2 for semantic embeddings
– Compute local gradients (loss.py) & update local model
d. Clients return updates → fusion.py merges them (e.g., weighted averaging + distilled semantics)
e. log.py captures metrics each round
f. Repeat until convergence
4. Final Evaluation: Global model + per-client test.py → Performance metrics
Architectural Patterns & Design Principles
• Federated Client-Server Pattern (central aggregator, multiple learners)
• Modular separation: data splitting, model definition, training, distillation, logging
• Plugin-style semantic distillation: clear extensibility points for adding new LLM/VLM services
• Config-first: environment.yml centralizes dependency management