# prathameshatkare/final-year- architecture

> Architecture diagram of the GitHub repository prathameshatkare/final-year-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/prathameshatkare/final-year-
- Repository: https://github.com/prathameshatkare/final-year-

## Overview

1. **Overall shape**
   - This is a research-oriented Python pipeline for **cystic fibrosis diagnosis** using **federated learning**.
   - The repo is organized as a set of **batch scripts** that pass data through the filesystem, plus two inference surfaces: an **edge CLI** and a **browser UI/API**.

2. **Core data model**
   - The model operates on a fixed tabular schema: `age, sex, height, weight, fev1, BMI` with `target` as the class label.
   - Raw and intermediate datasets live in CSV form, including `data/cystfibr.csv`, `data/nhanes_clean.csv`, `cleaned_cf_dataset.csv`, `augmented_cf_dataset.csv`, and `augmented_cf_dataset_3000cf.csv`.

3. **Preprocessing and statistical validation**
   - `data_preprocessing_and_stat_validation.py` is the first major stage.
   - It appears to handle data cleaning plus clinical/statistical checks before any synthetic generation or model training.
   - This stage is a gatekeeper: downstream augmentation and training assume the dataset has already been normalized into the core schema.

4. **Minority augmentation layer**
   - Augmentation is handled by scripts such as `data_augmentation_hybrid.py`, `data_augmentation_tvae.py`, `data_augmentation_smote_ctgan.py`, and `data_augmentation_guaan.py`.
   - The repo treats augmentation as a separate offline subsystem that expands the minority CF class before federated training.
   - Outputs are persisted as augmented CSVs, which become the input to later partitioning.

5. **Federated data boundary**
   - `federated_data_partition.py` converts the augmented dataset into per-client shards under `federated_clients/`.
   - The tree shows five client CSVs, so the federated system is simulated locally rather than distributed across real remote devices.
   - This is the main boundary between centralized preprocessing and decentralized training.

6. **Federated training engine**
   - `federated_pytorch.py` is the core training entrypoint, with `federated_training_cf.py` and `federated_simulation.py` supporting the training workflow.
   - The README indicates the aggregation strategy is **FedProx + weighted FedAvg**, so the architecture combines local client optimization with a weighted server-side aggregation step.
   - Training produces model checkpoints such as `best_federated_cf_model.pt` and metrics JSON files.

7. **Model selection and persistence**
   - Checkpoints and metrics are the primary interchange format between training and inference.
   - The best model artifacts are stored at the repo root, while ablation variants are stored under `ablation_runs/`.
   - This makes the repository reproducible: later stages read the saved `.pt` files rather than retraining in memory.

8. **Evaluation, diagnostics, and quality gates**
   - Several scripts analyze model behavior after training: `compute_confusion_matrix.py`, `model_comparison.py`, `threshold_calibration_results.csv`, `robustness_stress_test_results.csv`, and `realtime_metrics.py`.
   - `clinical_strict_validation.py` and `model_leakage_diagnostics.csv` suggest the project includes explicit checks for data leakage and clinical plausibility.
   - These are offline verification steps, not part of the serving path.

9. **Ablation and experimental analysis**
   - `federated_ablation_runner.py` generates feature-drop experiments, with outputs in `ablation_runs/` and summary files like `federated_ablation_results.csv` and `feature_ablation_results.csv`.
   - The repo stores multiple variants such as `drop_age`, `drop_BMI`, `drop_fev1`, etc., indicating feature importance is analyzed by retraining under ablated inputs.
   - `plot_federated_ablation.py` turns these results into publication-ready figures.

10. **Inference surfaces**
   - `edge_device_app.py` provides **checkpoint-based local inference** for an edge device use case.
   - `api_server.py`, `run_web_ui.py`, and the `web/` folder form a lightweight **FastAPI-backed browser UI** with static HTML/CSS/JS assets.
   - Both inference surfaces depend on the trained checkpoint, so the training pipeline is the upstream producer and the UI/edge layers are downstream consumers.

11. **Orchestration and execution style**
   - `run_project_pipeline.py` is the one-command orchestrator for the whole research workflow.
   - The system is not a long-running microservice mesh; it is a **pipeline of scripts and artifacts** with a small serving layer at the end.
   - That separation is important for an architecture diagram: offline research pipeline on one side, inference/serving on the other.

12. **Documentation and publication artifacts**
   - The many `.md` files are not runtime components; they capture requirements, architecture notes, and paper drafts.
   - Their presence signals that this repo is meant to support both software execution and academic reporting, with figures/tables generated from the same experiment outputs.

Main entry points: [`run_project_pipeline.py`](https://github.com/prathameshatkare/final-year-/blob/main/run_project_pipeline.py) (Pipeline runner), [`federated_pytorch.py`](https://github.com/prathameshatkare/final-year-/blob/main/federated_pytorch.py) (Federated train), [`best_federated_cf_model.pt`](https://github.com/prathameshatkare/final-year-/blob/main/best_federated_cf_model.pt) (Best model), [`compute_confusion_matrix.py`](https://github.com/prathameshatkare/final-year-/blob/main/compute_confusion_matrix.py) (Evaluation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_pipeline["Offline pipeline"]
  node_node_preprocess["Preprocess<br/>validation gate"]
  node_node_augment["Augment data<br/>synthetic expansion"]
  node_node_partition["Partition clients<br/>federated split"]
  node_node_train["Federated train<br/>FedProx/FedAvg"]
  node_node_sim["Simulation<br/>federated workflow"]
  node_node_eval["Evaluation<br/>offline diagnostics"]
  node_node_ablation["Ablations<br/>feature-drop experiments"]
  node_node_orchestrator["Pipeline runner<br/>workflow script"]
end

subgraph group_group_serving["Inference surfaces"]
  node_node_edge["Edge app<br/>local inference<br/>[edge_device_app.py]"]
  node_node_api["API server<br/>FastAPI service<br/>[api_server.py]"]
  node_node_webui["Web UI<br/>browser frontend<br/>[index.html]"]
end

subgraph group_group_artifacts["Model artifacts"]
  node_node_raw_data["Raw data<br/>csv inputs<br/>[cystfibr.csv]"]
  node_node_clean_data["Core dataset<br/>tabular schema"]
  node_node_aug3000["Augmented set<br/>derived CSV"]
  node_node_clients["Client shards<br/>federated CSVs<br/>[client_1.csv]"]
  node_node_checkpoint{{"Best model<br/>PyTorch checkpoint"}}
  node_node_metrics["Training metrics<br/>JSON metrics"]
  node_node_ablation_artifacts{{"Ablation runs<br/>experiment outputs<br/>[all_features.pt]"}}
end

subgraph group_group_docs["Docs"]
  node_node_docs["Project docs<br/>requirements & drafts<br/>[README.md]"]
end

node_node_raw_data -->|"source data"| node_node_preprocess
node_node_preprocess -->|"normalized output"| node_node_clean_data
node_node_clean_data -->|"clean input"| node_node_augment
node_node_augment -->|"augmented CSV"| node_node_aug3000
node_node_aug3000 -->|"training set"| node_node_partition
node_node_partition -->|"client shards"| node_node_clients
node_node_clients -->|"local data"| node_node_train
node_node_sim -->|"execution support"| node_node_train
node_node_train -->|"model"| node_node_checkpoint
node_node_train -->|"metrics"| node_node_metrics
node_node_checkpoint -->|"evaluated by"| node_node_eval
node_node_metrics -->|"analysis input"| node_node_eval
node_node_checkpoint -->|"inference model"| node_node_edge
node_node_checkpoint -->|"served model"| node_node_api
node_node_api -->|"backend for"| node_node_webui
node_node_ablation -->|"experiment outputs"| node_node_ablation_artifacts
node_node_ablation_artifacts -->|"compared in"| node_node_eval
node_node_orchestrator -->|"orchestrates"| node_node_preprocess
node_node_orchestrator -->|"orchestrates"| node_node_augment
node_node_orchestrator -->|"orchestrates"| node_node_train
node_node_orchestrator -->|"orchestrates"| node_node_eval
node_node_docs -.->|"documents"| node_node_orchestrator

click node_node_raw_data "https://github.com/prathameshatkare/final-year-/blob/main/data/cystfibr.csv"
click node_node_clean_data "https://github.com/prathameshatkare/final-year-/blob/main/cleaned_cf_dataset.csv"
click node_node_preprocess "https://github.com/prathameshatkare/final-year-/blob/main/data_preprocessing_and_stat_validation.py"
click node_node_augment "https://github.com/prathameshatkare/final-year-/blob/main/data_augmentation_hybrid.py"
click node_node_aug3000 "https://github.com/prathameshatkare/final-year-/blob/main/augmented_cf_dataset_3000cf.csv"
click node_node_partition "https://github.com/prathameshatkare/final-year-/blob/main/federated_data_partition.py"
click node_node_clients "https://github.com/prathameshatkare/final-year-/blob/main/federated_clients/client_1.csv"
click node_node_train "https://github.com/prathameshatkare/final-year-/blob/main/federated_pytorch.py"
click node_node_sim "https://github.com/prathameshatkare/final-year-/blob/main/federated_simulation.py"
click node_node_checkpoint "https://github.com/prathameshatkare/final-year-/blob/main/best_federated_cf_model.pt"
click node_node_metrics "https://github.com/prathameshatkare/final-year-/blob/main/best_federated_cf_metrics.json"
click node_node_eval "https://github.com/prathameshatkare/final-year-/blob/main/compute_confusion_matrix.py"
click node_node_ablation "https://github.com/prathameshatkare/final-year-/blob/main/federated_ablation_runner.py"
click node_node_ablation_artifacts "https://github.com/prathameshatkare/final-year-/blob/main/ablation_runs/all_features.pt"
click node_node_edge "https://github.com/prathameshatkare/final-year-/blob/main/edge_device_app.py"
click node_node_api "https://github.com/prathameshatkare/final-year-/blob/main/api_server.py"
click node_node_webui "https://github.com/prathameshatkare/final-year-/blob/main/web/index.html"
click node_node_orchestrator "https://github.com/prathameshatkare/final-year-/blob/main/run_project_pipeline.py"
click node_node_docs "https://github.com/prathameshatkare/final-year-/blob/main/README.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_preprocess,node_node_augment,node_node_partition,node_node_train,node_node_sim,node_node_eval,node_node_ablation,node_node_orchestrator toneBlue
class node_node_edge,node_node_api,node_node_webui toneAmber
class node_node_raw_data,node_node_clean_data,node_node_aug3000,node_node_clients,node_node_checkpoint,node_node_metrics,node_node_ablation_artifacts toneMint
class node_node_docs toneRose
```

## Components

### Offline pipeline

Centralized data preparation, augmentation, partitioning, training, and evaluation scripts.

- [Preprocess](https://github.com/prathameshatkare/final-year-/blob/main/data_preprocessing_and_stat_validation.py): validation gate · Cleaning plus statistical/clinical validation before augmentation or training. · `data_preprocessing_and_stat_validation.py`
- [Augment data](https://github.com/prathameshatkare/final-year-/blob/main/data_augmentation_hybrid.py): synthetic expansion · Offline minority-class augmentation across hybrid, TVAE, SMOTE/CTGAN, and GUAAN variants. · `data_augmentation_hybrid.py`
- [Partition clients](https://github.com/prathameshatkare/final-year-/blob/main/federated_data_partition.py): federated split · Converts the augmented dataset into per-client shards under federated_clients/. · `federated_data_partition.py`
- [Federated train](https://github.com/prathameshatkare/final-year-/blob/main/federated_pytorch.py): FedProx/FedAvg · Core federated learning engine and simulation entrypoint for checkpoint generation. · `federated_pytorch.py`
- [Simulation](https://github.com/prathameshatkare/final-year-/blob/main/federated_simulation.py): federated workflow · Support layer for the local federated training loop. · `federated_simulation.py`
- [Evaluation](https://github.com/prathameshatkare/final-year-/blob/main/compute_confusion_matrix.py): offline diagnostics · Confusion matrix, comparison, calibration, robustness, and leakage checks. · `compute_confusion_matrix.py`
- [Ablations](https://github.com/prathameshatkare/final-year-/blob/main/federated_ablation_runner.py): feature-drop experiments · Runs feature ablation studies and stores variant checkpoints and summaries. · `federated_ablation_runner.py`
- [Pipeline runner](https://github.com/prathameshatkare/final-year-/blob/main/run_project_pipeline.py): workflow script · One-command orchestrator that ties preprocessing, augmentation, training, and evaluation together. · `run_project_pipeline.py`

### Inference surfaces

Edge and browser-facing inference entrypoints that consume trained checkpoints.

- [Edge app](https://github.com/prathameshatkare/final-year-/blob/main/edge_device_app.py): local inference · Checkpoint-based inference for edge-device use cases. · `edge_device_app.py`
- [API server](https://github.com/prathameshatkare/final-year-/blob/main/api_server.py): FastAPI service · Backend server for browser-based inference and UI requests. · `api_server.py`
- [Web UI](https://github.com/prathameshatkare/final-year-/blob/main/web/index.html): browser frontend · Minimal web interface and static assets for interactive inference. · `web/index.html`

### Model artifacts

Persisted datasets, checkpoints, and experiment outputs used across stages.

- [Raw data](https://github.com/prathameshatkare/final-year-/blob/main/data/cystfibr.csv): csv inputs · Original and cleaned source datasets used to build the CF modeling table. · `data/cystfibr.csv`
- [Core dataset](https://github.com/prathameshatkare/final-year-/blob/main/cleaned_cf_dataset.csv): tabular schema · Normalized CF dataset with the core clinical features and target label. · `cleaned_cf_dataset.csv`
- [Augmented set](https://github.com/prathameshatkare/final-year-/blob/main/augmented_cf_dataset_3000cf.csv): derived CSV · Expanded training dataset used by partitioning and ablation runs. · `augmented_cf_dataset_3000cf.csv`
- [Client shards](https://github.com/prathameshatkare/final-year-/blob/main/federated_clients/client_1.csv): federated CSVs · Local simulated client datasets for federated training. · `federated_clients/client_1.csv`
- [Best model](https://github.com/prathameshatkare/final-year-/blob/main/best_federated_cf_model.pt): PyTorch checkpoint · Primary trained classifier artifact consumed by inference surfaces. · `best_federated_cf_model.pt`
- [Training metrics](https://github.com/prathameshatkare/final-year-/blob/main/best_federated_cf_metrics.json): JSON metrics · Model quality outputs persisted alongside checkpoints. · `best_federated_cf_metrics.json`
- [Ablation runs](https://github.com/prathameshatkare/final-year-/blob/main/ablation_runs/all_features.pt): experiment outputs · Persisted ablation checkpoints and metric files for feature importance analysis. · `ablation_runs/all_features.pt`

### Docs

Requirements, paper drafts, and architecture notes.

- [Project docs](https://github.com/prathameshatkare/final-year-/blob/main/README.md): requirements & drafts · Requirements, architecture notes, and paper drafts documenting the research workflow. · `README.md`

## Connections

- Raw data → Preprocess: source data · Raw CSVs feed the preprocessing gate.
- Preprocess → Core dataset: normalized output · Preprocessing produces the canonical CF table.
- Core dataset → Augment data: clean input · Augmentation assumes the validated schema.
- Augment data → Augmented set: augmented CSV · Synthetic expansion is persisted as a larger training set.
- Augmented set → Partition clients: training set · The augmented dataset is split into local clients.
- Partition clients → Client shards: client shards · Partitioning writes per-client CSVs.
- Client shards → Federated train: local data · Federated training consumes client shards.
- Simulation → Federated train: execution support · Simulation coordinates the federated loop.
- Federated train → Best model: model · Training emits the best checkpoint.
- Federated train → Training metrics: metrics · Training persists summary metrics.
- Best model → Evaluation: evaluated by · Offline diagnostics use the trained model artifact.
- Training metrics → Evaluation: analysis input · Evaluation and comparison scripts read training outputs.
- Best model → Edge app: inference model · Edge app loads the saved checkpoint.
- Best model → API server: served model · API server uses the trained checkpoint for predictions.
- API server → Web UI: backend for · The browser UI talks to the API server.
- Ablations → Ablation runs: experiment outputs · Ablation runner stores variant checkpoints and metrics.
- Ablation runs → Evaluation: compared in · Ablation results feed analysis and plotting.
- Pipeline runner → Preprocess: orchestrates · Pipeline runner starts the preprocessing gate.
- Pipeline runner → Augment data: orchestrates · Pipeline runner triggers augmentation.
- Pipeline runner → Federated train: orchestrates · Pipeline runner launches federated training.
- Pipeline runner → Evaluation: orchestrates · Pipeline runner launches diagnostics.
- Project docs → Pipeline runner: documents · Repo docs describe the workflow and architecture.