# prathameshatkare/new_cf architecture

> Architecture diagram of the GitHub repository prathameshatkare/new_cf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/prathameshatkare/new_cf
- Repository: https://github.com/prathameshatkare/new_cf

## Overview

1. This repository is a Python-based machine learning pipeline for cystic fibrosis diagnosis, centered on tabular clinical data. The core schema is small and fixed: `age, sex, height, weight, fev1, BMI`, with a binary `target` label.

2. The data enters from raw CSV sources in `data/` such as `cystfibr.csv` and `nhanes_clean.csv`. The first architectural boundary is the preprocessing/validation stage, which produces cleaned and statistically checked datasets like `cleaned_cf_dataset.csv` and related diagnostics.

3. A second major stage handles minority-class augmentation. The repo includes multiple augmentation strategies (`data_augmentation_hybrid.py`, `data_augmentation_smote_ctgan.py`, `data_augmentation_tvae.py`), with the README emphasizing TVAE-style tabular synthesis. This stage outputs `augmented_cf_dataset.csv` and is meant to improve rare-disease class balance before training.

4. After augmentation, the dataset is partitioned for federated learning. `federated_data_partition.py` creates per-client slices under `federated_clients/`, which form the privacy-preserving training boundary between local client data and the central aggregation process.

5. The federated training subsystem is implemented in scripts such as `federated_pytorch.py`, `federated_training_cf.py`, and `federated_simulation.py`. The README states the main algorithmic pattern is FedProx with weighted FedAvg, so the architecture is “client-local training + server-side aggregation” rather than a single centralized trainer.

6. The main model checkpoint and training metrics are persisted as artifacts, notably `best_federated_cf_model.pt` and `best_federated_cf_metrics.json`. These artifacts are reused downstream by edge inference, diagnostics, and web/API serving, so they are the handoff point between training and deployment.

7. There is a substantial evaluation and ablation layer around the trained model. `federated_ablation_runner.py` plus the `ablation_runs/` directory capture feature-drop experiments (`drop_age`, `drop_BMI`, etc.), while CSV outputs like `feature_ablation_results.csv` and `federated_ablation_results.csv` support comparison across variants.

8. Model quality analysis is broader than accuracy alone. Scripts such as `model_comparison.py`, `clinical_strict_validation.py`, and the outputs `model_leakage_diagnostics.csv`, `robustness_stress_test_results.csv`, and `threshold_calibration_results.csv` show distinct diagnostic flows for leakage checks, clinical validity, stress testing, and decision-threshold tuning.

9. There is a dedicated edge inference path in `edge_device_app.py`. This is checkpoint-driven and loads the trained `.pt` model for local prediction, which makes it a separate deployment boundary from the federated training infrastructure.

10. The repository also includes a lightweight web delivery layer under `web/` plus `api_server.py`, `app_ui.py`, and `run_web_ui.py`. The README indicates this is served through FastAPI/Uvicorn, with static frontend assets (`index.html`, `styles.css`, `app.js`) providing a browser-based interface over the model/API.

11. `run_project_pipeline.py` is the orchestration entry point for the whole workflow. It ties together preprocessing, augmentation, federated training, ablation, comparison, and plotting into a single repeatable run, while the manual command list in the README shows the same pipeline can be executed stepwise.

12. The outputs are publication-oriented as well as operational. The repo contains journal drafts, architecture diagrams, and plot artifacts such as `federated_ablation_plot_paper_bw.pdf`, which indicates the project supports both research reporting and executable ML system delivery.

13. In diagram terms, the key boundaries are: raw clinical data → cleaned/validated data → augmented data → federated client partitions → training/aggregation → model artifacts → diagnostics/plots → edge/API/web consumers. Python is the dominant runtime, PyTorch is the model/training framework, and FastAPI/Uvicorn is the serving layer.

Main entry points: [`run_project_pipeline.py`](https://github.com/prathameshatkare/new_cf/blob/main/run_project_pipeline.py) (Project pipeline), [`best_federated_cf_model.pt`](https://github.com/prathameshatkare/new_cf/blob/main/best_federated_cf_model.pt) (Model artifact), [`data_preprocessing_and_stat_validation.py`](https://github.com/prathameshatkare/new_cf/blob/main/data_preprocessing_and_stat_validation.py) (Preprocess), [`data_augmentation_tvae.py`](https://github.com/prathameshatkare/new_cf/blob/main/data_augmentation_tvae.py) (Augment)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_data["Data pipeline"]
  node_raw_data[("Raw data<br/>clinical csv<br/>[cystfibr.csv]")]
  node_preprocess["Preprocess<br/>validation step"]
  node_clean_data["Cleaned set<br/>validated dataset"]
  node_augment["Augment<br/>tabular synthesis"]
  node_aug_data["Augmented set<br/>training dataset"]
  node_partition["Partition<br/>federated split"]
  node_clients[("Client data<br/>federated shards<br/>[client_1.csv]")]
end

subgraph group_group_training["Federated learning"]
  node_fed_train["Fed training<br/>federated optimizer"]
  node_fed_core["Fed core<br/>pytorch federation"]
  node_fed_sim["Simulation<br/>training driver"]
  node_checkpoint["Model artifact<br/>checkpoint"]
  node_pipeline["Project pipeline<br/>orchestrator"]
end

subgraph group_group_eval["Evaluation"]
  node_ablation["Ablations<br/>feature study"]
  node_diagnostics["Diagnostics<br/>model evaluation"]
  node_results["Results<br/>metrics outputs"]
end

subgraph group_group_deploy["Deployment"]
  node_edge_app["Edge app<br/>local inference<br/>[edge_device_app.py]"]
  node_api_ui["Web/API<br/>fastapi service<br/>[api_server.py]"]
  node_frontend["Frontend<br/>static ui<br/>[index.html]"]
end

subgraph group_group_docs["Research outputs"]
  node_papers["Papers<br/>research docs"]
end

node_raw_data -->|"clean"| node_preprocess
node_preprocess -->|"emit"| node_clean_data
node_clean_data -->|"augment"| node_augment
node_augment -->|"emit"| node_aug_data
node_aug_data -->|"split"| node_partition
node_partition -->|"write"| node_clients
node_clients -->|"train"| node_fed_train
node_fed_train -->|"uses"| node_fed_core
node_fed_core -->|"run"| node_fed_sim
node_fed_sim -->|"save"| node_checkpoint
node_checkpoint -->|"load"| node_edge_app
node_checkpoint -->|"serve"| node_api_ui
node_api_ui -->|"render"| node_frontend
node_checkpoint -->|"compare"| node_ablation
node_checkpoint -->|"analyze"| node_diagnostics
node_ablation -->|"record"| node_results
node_diagnostics -->|"record"| node_results
node_pipeline -->|"orchestrate"| node_preprocess
node_pipeline -->|"orchestrate"| node_augment
node_pipeline -->|"orchestrate"| node_fed_train
node_pipeline -->|"orchestrate"| node_ablation
node_pipeline -->|"orchestrate"| node_diagnostics
node_pipeline -->|"report"| node_papers

click node_raw_data "https://github.com/prathameshatkare/new_cf/blob/main/data/cystfibr.csv"
click node_preprocess "https://github.com/prathameshatkare/new_cf/blob/main/data_preprocessing_and_stat_validation.py"
click node_clean_data "https://github.com/prathameshatkare/new_cf/blob/main/cleaned_cf_dataset.csv"
click node_augment "https://github.com/prathameshatkare/new_cf/blob/main/data_augmentation_tvae.py"
click node_aug_data "https://github.com/prathameshatkare/new_cf/blob/main/augmented_cf_dataset.csv"
click node_partition "https://github.com/prathameshatkare/new_cf/blob/main/federated_data_partition.py"
click node_clients "https://github.com/prathameshatkare/new_cf/blob/main/federated_clients/client_1.csv"
click node_fed_train "https://github.com/prathameshatkare/new_cf/blob/main/federated_training_cf.py"
click node_fed_core "https://github.com/prathameshatkare/new_cf/blob/main/federated_pytorch.py"
click node_fed_sim "https://github.com/prathameshatkare/new_cf/blob/main/federated_simulation.py"
click node_checkpoint "https://github.com/prathameshatkare/new_cf/blob/main/best_federated_cf_model.pt"
click node_ablation "https://github.com/prathameshatkare/new_cf/blob/main/federated_ablation_runner.py"
click node_diagnostics "https://github.com/prathameshatkare/new_cf/blob/main/model_comparison.py"
click node_results "https://github.com/prathameshatkare/new_cf/blob/main/model_comparison_results_final.csv"
click node_edge_app "https://github.com/prathameshatkare/new_cf/blob/main/edge_device_app.py"
click node_api_ui "https://github.com/prathameshatkare/new_cf/blob/main/api_server.py"
click node_frontend "https://github.com/prathameshatkare/new_cf/blob/main/web/index.html"
click node_pipeline "https://github.com/prathameshatkare/new_cf/blob/main/run_project_pipeline.py"
click node_papers "https://github.com/prathameshatkare/new_cf/blob/main/JOURNAL_PAPER_IEEE.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_data,node_preprocess,node_clean_data,node_augment,node_aug_data,node_partition,node_clients toneBlue
class node_fed_train,node_fed_core,node_fed_sim,node_checkpoint,node_pipeline toneAmber
class node_ablation,node_diagnostics,node_results toneMint
class node_edge_app,node_api_ui,node_frontend toneRose
class node_papers toneIndigo
```

## Components

### Data pipeline

Raw clinical data, preprocessing, augmentation, and client partitioning.

- [Raw data](https://github.com/prathameshatkare/new_cf/blob/main/data/cystfibr.csv): clinical csv · Source tabular datasets for cystic fibrosis modeling. · `data/cystfibr.csv`
- [Preprocess](https://github.com/prathameshatkare/new_cf/blob/main/data_preprocessing_and_stat_validation.py): validation step · Cleans clinical records and performs statistical checks. · `data_preprocessing_and_stat_validation.py`
- [Cleaned set](https://github.com/prathameshatkare/new_cf/blob/main/cleaned_cf_dataset.csv): validated dataset · Preprocessed CF dataset ready for augmentation. · `cleaned_cf_dataset.csv`
- [Augment](https://github.com/prathameshatkare/new_cf/blob/main/data_augmentation_tvae.py): tabular synthesis · Minority-class augmentation using hybrid, SMOTE-CTGAN, and TVAE approaches. · `data_augmentation_tvae.py`
- [Augmented set](https://github.com/prathameshatkare/new_cf/blob/main/augmented_cf_dataset.csv): training dataset · Balanced dataset produced after augmentation. · `augmented_cf_dataset.csv`
- [Partition](https://github.com/prathameshatkare/new_cf/blob/main/federated_data_partition.py): federated split · Creates per-client slices for privacy-preserving training. · `federated_data_partition.py`
- [Client data](https://github.com/prathameshatkare/new_cf/blob/main/federated_clients/client_1.csv): federated shards · Local client CSV partitions used in federated simulation. · `federated_clients/client_1.csv`

### Federated learning

Client-side training, aggregation, and checkpoint artifacts.

- [Fed training](https://github.com/prathameshatkare/new_cf/blob/main/federated_training_cf.py): federated optimizer · Client-local training with server aggregation, including FedProx and weighted FedAvg. · `federated_training_cf.py`
- [Fed core](https://github.com/prathameshatkare/new_cf/blob/main/federated_pytorch.py): pytorch federation · Implementation backbone for local updates and aggregation logic. · `federated_pytorch.py`
- [Simulation](https://github.com/prathameshatkare/new_cf/blob/main/federated_simulation.py): training driver · Runs federated rounds in a controlled simulation environment. · `federated_simulation.py`
- [Model artifact](https://github.com/prathameshatkare/new_cf/blob/main/best_federated_cf_model.pt): checkpoint · Best federated model and metrics used by downstream consumers. · `best_federated_cf_model.pt`
- [Project pipeline](https://github.com/prathameshatkare/new_cf/blob/main/run_project_pipeline.py): orchestrator · Top-level workflow that ties preprocessing, augmentation, training, evaluation, and plotting together. · `run_project_pipeline.py`

### Evaluation

Ablations, comparisons, validation, and diagnostics.

- [Ablations](https://github.com/prathameshatkare/new_cf/blob/main/federated_ablation_runner.py): feature study · Feature-drop experiments and result capture across multiple variants. · `federated_ablation_runner.py`
- [Diagnostics](https://github.com/prathameshatkare/new_cf/blob/main/model_comparison.py): model evaluation · Leakage checks, strict validation, robustness, calibration, and comparisons. · `model_comparison.py`
- [Results](https://github.com/prathameshatkare/new_cf/blob/main/model_comparison_results_final.csv): metrics outputs · CSV and JSON outputs for comparison, ablation, calibration, and stress tests. · `model_comparison_results_final.csv`

### Deployment

Edge inference and web/API serving surfaces.

- [Edge app](https://github.com/prathameshatkare/new_cf/blob/main/edge_device_app.py): local inference · Checkpoint-driven inference on edge devices. · `edge_device_app.py`
- [Web/API](https://github.com/prathameshatkare/new_cf/blob/main/api_server.py): fastapi service · API server and browser UI for model access. · `api_server.py`
- [Frontend](https://github.com/prathameshatkare/new_cf/blob/main/web/index.html): static ui · Browser assets for the web interface. · `web/index.html`

### Research outputs

Paper drafts and architecture/reporting artifacts.

- [Papers](https://github.com/prathameshatkare/new_cf/blob/main/JOURNAL_PAPER_IEEE.md): research docs · Manuscripts and reporting materials for publication-oriented outputs. · `JOURNAL_PAPER_IEEE.md`

## Connections

- Raw data → Preprocess: clean · Raw clinical CSVs are validated and standardized.
- Preprocess → Cleaned set: emit · Preprocessing produces the cleaned dataset.
- Cleaned set → Augment: augment · Cleaned data feeds augmentation methods.
- Augment → Augmented set: emit · Augmentation produces the balanced training set.
- Augmented set → Partition: split · Augmented data is partitioned for federated clients.
- Partition → Client data: write · Client-specific slices are written to federated clients.
- Client data → Fed training: train · Local client datasets participate in federated rounds.
- Fed training → Fed core: uses · Training logic delegates to the federated PyTorch core.
- Fed core → Simulation: run · Simulation drives repeated training and aggregation rounds.
- Simulation → Model artifact: save · Best model and metrics are persisted as artifacts.
- Model artifact → Edge app: load · Edge inference loads the trained checkpoint.
- Model artifact → Web/API: serve · The web/API layer serves predictions from the trained model.
- Web/API → Frontend: render · Static browser assets provide the user interface.
- Model artifact → Ablations: compare · Ablation runs evaluate feature subsets against the main model.
- Model artifact → Diagnostics: analyze · Diagnostics inspect leakage, robustness, thresholds, and clinical validity.
- Ablations → Results: record · Ablation outputs are captured as CSV, JSON, and plots.
- Diagnostics → Results: record · Evaluation and validation outputs are persisted for comparison.
- Project pipeline → Preprocess: orchestrate · The top-level pipeline invokes preprocessing.
- Project pipeline → Augment: orchestrate · The top-level pipeline invokes augmentation.
- Project pipeline → Fed training: orchestrate · The top-level pipeline invokes federated training.
- Project pipeline → Ablations: orchestrate · The top-level pipeline invokes ablation studies.
- Project pipeline → Diagnostics: orchestrate · The top-level pipeline invokes evaluation and validation.
- Project pipeline → Papers: report · The workflow also supports publication-oriented outputs.