How prathameshatkare/final-year- works
1. Overall shape
- This is a research-oriented Python pipeline for cystic fibrosis diagnosis using federated learning.
- The repo is organized as a set of batch scripts that pass data through the filesystem, plus two inference surfaces: an edge CLI and a browser UI/API.
2. Core data model
- The model operates on a fixed tabular schema: age, sex, height, weight, fev1, BMI with target as the class label.
- Raw and intermediate datasets live in CSV form, including data/cystfibr.csv, data/nhanes_clean.csv, cleaned_cf_dataset.csv, augmented_cf_dataset.csv, and augmented_cf_dataset_3000cf.csv.
3. Preprocessing and statistical validation
- data_preprocessing_and_stat_validation.py is the first major stage.
- It appears to handle data cleaning plus clinical/statistical checks before any synthetic generation or model training.
- This stage is a gatekeeper: downstream augmentation and training assume the dataset has already been normalized into the core schema.
4. Minority augmentation layer
- Augmentation is handled by scripts such as data_augmentation_hybrid.py, data_augmentation_tvae.py, data_augmentation_smote_ctgan.py, and data_augmentation_guaan.py.
- The repo treats augmentation as a separate offline subsystem that expands the minority CF class before federated training.
- Outputs are persisted as augmented CSVs, which become the input to later partitioning.
5. Federated data boundary
- federated_data_partition.py converts the augmented dataset into per-client shards under federated_clients/.
- The tree shows five client CSVs, so the federated system is simulated locally rather than distributed across real remote devices.
- This is the main boundary between centralized preprocessing and decentralized training.
6. Federated training engine
- federated_pytorch.py is the core training entrypoint, with federated_training_cf.py and federated_simulation.py supporting the training workflow.
- The README indicates the aggregation strategy is FedProx + weighted FedAvg, so the architecture combines local client optimization with a weighted server-side aggregation step.
- Training produces model checkpoints such as best_federated_cf_model.pt and metrics JSON files.
7. Model selection and persistence
- Checkpoints and metrics are the primary interchange format between training and inference.
- The best model artifacts are stored at the repo root, while ablation variants are stored under ablation_runs/.
- This makes the repository reproducible: later stages read the saved .pt files rather than retraining in memory.
8. Evaluation, diagnostics, and quality gates
- Several scripts analyze model behavior after training: compute_confusion_matrix.py, model_comparison.py, threshold_calibration_results.csv, robustness_stress_test_results.csv, and realtime_metrics.py.
- clinical_strict_validation.py and model_leakage_diagnostics.csv suggest the project includes explicit checks for data leakage and clinical plausibility.
- These are offline verification steps, not part of the serving path.
9. Ablation and experimental analysis
- federated_ablation_runner.py generates feature-drop experiments, with outputs in ablation_runs/ and summary files like federated_ablation_results.csv and feature_ablation_results.csv.
- The repo stores multiple variants such as drop_age, drop_BMI, drop_fev1, etc., indicating feature importance is analyzed by retraining under ablated inputs.
- plot_federated_ablation.py turns these results into publication-ready figures.
10. Inference surfaces
- edge_device_app.py provides checkpoint-based local inference for an edge device use case.
- api_server.py, run_web_ui.py, and the web/ folder form a lightweight FastAPI-backed browser UI with static HTML/CSS/JS assets.
- Both inference surfaces depend on the trained checkpoint, so the training pipeline is the upstream producer and the UI/edge layers are downstream consumers.
11. Orchestration and execution style
- run_project_pipeline.py is the one-command orchestrator for the whole research workflow.
- The system is not a long-running microservice mesh; it is a pipeline of scripts and artifacts with a small serving layer at the end.
- That separation is important for an architecture diagram: offline research pipeline on one side, inference/serving on the other.
12. Documentation and publication artifacts
- The many .md files are not runtime components; they capture requirements, architecture notes, and paper drafts.
- Their presence signals that this repo is meant to support both software execution and academic reporting, with figures/tables generated from the same experiment outputs.
Main entry points: run_project_pipeline.py Pipeline runner · federated_pytorch.py Federated train · best_federated_cf_model.pt Best model · compute_confusion_matrix.py Evaluation
Components 19
Offline pipeline
Centralized data preparation, augmentation, partitioning, training, and evaluation scripts.
- Preprocess · Cleaning plus statistical/clinical validation before augmentation or training.
data_preprocessing_and_stat_validation.py - Augment data · Offline minority-class augmentation across hybrid, TVAE, SMOTE/CTGAN, and GUAAN variants.
data_augmentation_hybrid.py - Partition clients · Converts the augmented dataset into per-client shards under federated_clients/.
federated_data_partition.py - Federated train · Core federated learning engine and simulation entrypoint for checkpoint generation.
federated_pytorch.py - Simulation · Support layer for the local federated training loop.
federated_simulation.py - Evaluation · Confusion matrix, comparison, calibration, robustness, and leakage checks.
compute_confusion_matrix.py - Ablations · Runs feature ablation studies and stores variant checkpoints and summaries.
federated_ablation_runner.py - Pipeline runner · One-command orchestrator that ties preprocessing, augmentation, training, and evaluation together.
run_project_pipeline.py
Inference surfaces
Edge and browser-facing inference entrypoints that consume trained checkpoints.
- Edge app · Checkpoint-based inference for edge-device use cases.
edge_device_app.py - API server · Backend server for browser-based inference and UI requests.
api_server.py - Web UI · Minimal web interface and static assets for interactive inference.
web/index.html
Model artifacts
Persisted datasets, checkpoints, and experiment outputs used across stages.
- Raw data · Original and cleaned source datasets used to build the CF modeling table.
data/cystfibr.csv - Core dataset · Normalized CF dataset with the core clinical features and target label.
cleaned_cf_dataset.csv - Augmented set · Expanded training dataset used by partitioning and ablation runs.
augmented_cf_dataset_3000cf.csv - Client shards · Local simulated client datasets for federated training.
federated_clients/client_1.csv - Best model · Primary trained classifier artifact consumed by inference surfaces.
best_federated_cf_model.pt - Training metrics · Model quality outputs persisted alongside checkpoints.
best_federated_cf_metrics.json - Ablation runs · Persisted ablation checkpoints and metric files for feature importance analysis.
ablation_runs/all_features.pt
Docs
Requirements, paper drafts, and architecture notes.
- Project docs · Requirements, architecture notes, and paper drafts documenting the research workflow.
README.md
Connections 22
- Raw data → Preprocess: source data · Raw CSVs feed the preprocessing gate.
- Preprocess → Core dataset: normalized output · Preprocessing produces the canonical CF table.
- Core dataset → Augment data: clean input · Augmentation assumes the validated schema.
- Augment data → Augmented set: augmented CSV · Synthetic expansion is persisted as a larger training set.
- Augmented set → Partition clients: training set · The augmented dataset is split into local clients.
- Partition clients → Client shards: client shards · Partitioning writes per-client CSVs.
- Client shards → Federated train: local data · Federated training consumes client shards.
- Simulation → Federated train: execution support · Simulation coordinates the federated loop.
- Federated train → Best model: model · Training emits the best checkpoint.
- Federated train → Training metrics: metrics · Training persists summary metrics.
- Best model → Evaluation: evaluated by · Offline diagnostics use the trained model artifact.
- Training metrics → Evaluation: analysis input · Evaluation and comparison scripts read training outputs.
- Best model → Edge app: inference model · Edge app loads the saved checkpoint.
- Best model → API server: served model · API server uses the trained checkpoint for predictions.
- API server → Web UI: backend for · The browser UI talks to the API server.
- Ablations → Ablation runs: experiment outputs · Ablation runner stores variant checkpoints and metrics.
- Ablation runs → Evaluation: compared in · Ablation results feed analysis and plotting.
- Pipeline runner → Preprocess: orchestrates · Pipeline runner starts the preprocessing gate.
- Pipeline runner → Augment data: orchestrates · Pipeline runner triggers augmentation.
- Pipeline runner → Federated train: orchestrates · Pipeline runner launches federated training.
- Pipeline runner → Evaluation: orchestrates · Pipeline runner launches diagnostics.
- Project docs → Pipeline runner: documents · Repo docs describe the workflow and architecture.