prathameshatkare/new_cf

1. This repository is a Python-based machine learning pipeline for cystic fibrosis diagnosis, centered on tabular clinical data. The core schema is small and fixed: age, sex, height, weight, fev1, BMI, with a binary target label.

2. The data enters from raw CSV sources in data/ such as cystfibr.csv and nhanes_clean.csv. The first architectural boundary is the preprocessing/validation stage, which produces cleaned and statistically checked datasets like cleaned_cf_dataset.csv and related diagnostics.

3. A second major stage handles minority-class augmentation. The repo includes multiple augmentation strategies (data_augmentation_hybrid.py, data_augmentation_smote_ctgan.py, data_augmentation_tvae.py), with the README emphasizing TVAE-style tabular synthesis. This stage outputs augmented_cf_dataset.csv and is meant to improve rare-disease class balance before training.

4. After augmentation, the dataset is partitioned for federated learning. federated_data_partition.py creates per-client slices under federated_clients/, which form the privacy-preserving training boundary between local client data and the central aggregation process.

5. The federated training subsystem is implemented in scripts such as federated_pytorch.py, federated_training_cf.py, and federated_simulation.py. The README states the main algorithmic pattern is FedProx with weighted FedAvg, so the architecture is “client-local training + server-side aggregation” rather than a single centralized trainer.

6. The main model checkpoint and training metrics are persisted as artifacts, notably best_federated_cf_model.pt and best_federated_cf_metrics.json. These artifacts are reused downstream by edge inference, diagnostics, and web/API serving, so they are the handoff point between training and deployment.

7. There is a substantial evaluation and ablation layer around the trained model. federated_ablation_runner.py plus the ablation_runs/ directory capture feature-drop experiments (drop_age, drop_BMI, etc.), while CSV outputs like feature_ablation_results.csv and federated_ablation_results.csv support comparison across variants.

8. Model quality analysis is broader than accuracy alone. Scripts such as model_comparison.py, clinical_strict_validation.py, and the outputs model_leakage_diagnostics.csv, robustness_stress_test_results.csv, and threshold_calibration_results.csv show distinct diagnostic flows for leakage checks, clinical validity, stress testing, and decision-threshold tuning.

9. There is a dedicated edge inference path in edge_device_app.py. This is checkpoint-driven and loads the trained .pt model for local prediction, which makes it a separate deployment boundary from the federated training infrastructure.

10. The repository also includes a lightweight web delivery layer under web/ plus api_server.py, app_ui.py, and run_web_ui.py. The README indicates this is served through FastAPI/Uvicorn, with static frontend assets (index.html, styles.css, app.js) providing a browser-based interface over the model/API.

11. run_project_pipeline.py is the orchestration entry point for the whole workflow. It ties together preprocessing, augmentation, federated training, ablation, comparison, and plotting into a single repeatable run, while the manual command list in the README shows the same pipeline can be executed stepwise.

12. The outputs are publication-oriented as well as operational. The repo contains journal drafts, architecture diagrams, and plot artifacts such as federated_ablation_plot_paper_bw.pdf, which indicates the project supports both research reporting and executable ML system delivery.

13. In diagram terms, the key boundaries are: raw clinical data → cleaned/validated data → augmented data → federated client partitions → training/aggregation → model artifacts → diagnostics/plots → edge/API/web consumers. Python is the dominant runtime, PyTorch is the model/training framework, and FastAPI/Uvicorn is the serving layer.