# orcads/data-001-impact-of-synthetic-data-under-data-scarcity architecture

> Architecture diagram of the GitHub repository orcads/data-001-impact-of-synthetic-data-under-data-scarcity, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity
- Repository: https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity

## Overview

This repository implements a healthcare tabular synthetic-data scarcity benchmark. Its completed Gaussian Copula workflow compares synthetic augmentation with real-only training across scarcity levels, repeated splits, and classifiers, evaluating on held-out real data. Diagnostics cover fidelity, validity, reliability, and proximity; a separate CTGAN workflow is operationally paused, with verification components retained. The predictive orchestration and generator wiring are supported by sampled provenance sources; broader capability links follow the README and tree, whose unsampled implementations limit call-level detail.

Main entry points: [`results/provenance/gaussian_predictive_execution_v1_sources/src/experiments/augmentation.py`](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/results/provenance/gaussian_predictive_execution_v1_sources/src/experiments/augmentation.py) (Gaussian comparison), [`src/experiments/baseline.py`](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/experiments/baseline.py) (Real-only baseline), [`src/experiments/gaussian_diagnostics.py`](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/experiments/gaussian_diagnostics.py) (Gaussian diagnostics), [`src/experiments/ctgan_full_grid.py`](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/experiments/ctgan_full_grid.py) (CTGAN full grid)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data and scarcity"]
  node_loader["Dataset loading<br/>[loader.py]"]
  node_scarcity["Scarcity sampling<br/>[scarcity.py]"]
  node_validation["Domain validation<br/>[validation.py]"]
  node_preprocessing["Feature preprocessing<br/>[preprocessing.py]"]
end

subgraph group_generation["Synthetic generation"]
  node_gaussian["Gaussian Copula<br/>[gaussian_copula.py]"]
  node_ctgan["CTGAN generation<br/>[ctgan.py]"]
  node_tvae["TVAE generator<br/>[tvae.py]"]
end

subgraph group_experiments["Study workflows"]
  node_augmentation["Gaussian comparison<br/>[augmentation.py]"]
  node_baseline["Real-only baseline<br/>[baseline.py]"]
  node_diagnostics["Gaussian diagnostics"]
  node_ctgan_pilot["CTGAN pilot<br/>[ctgan_pilot.py]"]
  node_ctgan_grid["CTGAN full grid<br/>[ctgan_full_grid.py]"]
  node_logistic["Logistic regression"]
  node_forest["Random forest<br/>[random_forest.py]"]
end

subgraph group_assessment["Evaluation and reporting"]
  node_grid_verifier["Grid verification"]
  node_predictive["Predictive metrics<br/>[predictive.py]"]
  node_fidelity["Fidelity metrics<br/>[fidelity.py]"]
  node_privacy["Proximity metrics<br/>[privacy.py]"]
  node_statistics["Statistical summaries<br/>[statistics.py]"]
  node_plots["Result plots<br/>[plots.py]"]
  node_eda["Exploratory analysis<br/>[eda.py]"]
end

node_researcher(("Researcher"))
node_health_data["Healthcare datasets"]

node_researcher -->|"runs comparison"| node_augmentation
node_health_data -->|"provides records"| node_loader
node_augmentation -->|"samples training"| node_scarcity
node_augmentation -->|"generates augmentation"| node_gaussian
node_augmentation -->|"fits classifier"| node_logistic
node_augmentation -->|"fits classifier"| node_forest
node_augmentation -->|"scores predictions"| node_predictive
node_baseline -->|"samples training"| node_scarcity
node_baseline -->|"fits classifier"| node_logistic
node_baseline -->|"fits classifier"| node_forest
node_baseline -->|"evaluates predictions"| node_predictive
node_diagnostics -->|"generates samples"| node_gaussian
node_diagnostics -->|"checks domain"| node_validation
node_diagnostics -->|"measures fidelity"| node_fidelity
node_diagnostics -->|"measures proximity"| node_privacy
node_ctgan_pilot -->|"runs pilot"| node_ctgan
node_ctgan_grid -->|"dispatches conditions"| node_ctgan
node_ctgan_grid -->|"verifies conditions"| node_grid_verifier
node_augmentation -->|"summarizes metrics"| node_statistics
node_statistics -->|"supplies summaries"| node_plots
node_loader -->|"supports exploration"| node_eda

click node_loader "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/data/loader.py"
click node_scarcity "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/results/provenance/gaussian_predictive_execution_v1_sources/src/data/scarcity.py"
click node_validation "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/data/validation.py"
click node_preprocessing "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/data/preprocessing.py"
click node_gaussian "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/results/provenance/gaussian_predictive_execution_v1_sources/src/generators/gaussian_copula.py"
click node_ctgan "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/generators/ctgan.py"
click node_tvae "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/generators/tvae.py"
click node_augmentation "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/results/provenance/gaussian_predictive_execution_v1_sources/src/experiments/augmentation.py"
click node_baseline "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/experiments/baseline.py"
click node_diagnostics "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/experiments/gaussian_diagnostics.py"
click node_ctgan_pilot "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/experiments/ctgan_pilot.py"
click node_ctgan_grid "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/experiments/ctgan_full_grid.py"
click node_grid_verifier "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/evaluation/ctgan_full_grid_verifier.py"
click node_logistic "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/models/logistic_regression.py"
click node_forest "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/models/random_forest.py"
click node_predictive "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/evaluation/predictive.py"
click node_fidelity "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/evaluation/fidelity.py"
click node_privacy "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/evaluation/privacy.py"
click node_statistics "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/evaluation/statistics.py"
click node_plots "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/visualization/plots.py"
click node_eda "https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/visualization/eda.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_loader,node_scarcity,node_validation,node_preprocessing toneBlue
class node_gaussian,node_ctgan,node_tvae toneAmber
class node_augmentation,node_baseline,node_diagnostics,node_ctgan_pilot,node_ctgan_grid,node_logistic,node_forest toneMint
class node_grid_verifier,node_predictive,node_fidelity,node_privacy,node_statistics,node_plots,node_eda toneRose
class node_researcher,node_health_data toneIndigo
```

## Components

### Data and scarcity

- [Dataset loading](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/data/loader.py): `src/data/loader.py`
- [Scarcity sampling](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/results/provenance/gaussian_predictive_execution_v1_sources/src/data/scarcity.py): `results/provenance/gaussian_predictive_execution_v1_sources/src/data/scarcity.py`
- [Domain validation](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/data/validation.py): `src/data/validation.py`
- [Feature preprocessing](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/data/preprocessing.py): `src/data/preprocessing.py`

### Synthetic generation

- [Gaussian Copula](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/results/provenance/gaussian_predictive_execution_v1_sources/src/generators/gaussian_copula.py): `results/provenance/gaussian_predictive_execution_v1_sources/src/generators/gaussian_copula.py`
- [CTGAN generation](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/generators/ctgan.py): `src/generators/ctgan.py`
- [TVAE generator](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/generators/tvae.py): `src/generators/tvae.py`

### Study workflows

- [Gaussian comparison](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/results/provenance/gaussian_predictive_execution_v1_sources/src/experiments/augmentation.py): `results/provenance/gaussian_predictive_execution_v1_sources/src/experiments/augmentation.py`
- [Real-only baseline](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/experiments/baseline.py): `src/experiments/baseline.py`
- [Gaussian diagnostics](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/experiments/gaussian_diagnostics.py): `src/experiments/gaussian_diagnostics.py`
- [CTGAN pilot](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/experiments/ctgan_pilot.py): `src/experiments/ctgan_pilot.py`
- [CTGAN full grid](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/experiments/ctgan_full_grid.py): `src/experiments/ctgan_full_grid.py`
- [Logistic regression](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/models/logistic_regression.py): `src/models/logistic_regression.py`
- [Random forest](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/models/random_forest.py): `src/models/random_forest.py`

### Evaluation and reporting

- [Grid verification](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/evaluation/ctgan_full_grid_verifier.py): `src/evaluation/ctgan_full_grid_verifier.py`
- [Predictive metrics](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/evaluation/predictive.py): `src/evaluation/predictive.py`
- [Fidelity metrics](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/evaluation/fidelity.py): `src/evaluation/fidelity.py`
- [Proximity metrics](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/evaluation/privacy.py): `src/evaluation/privacy.py`
- [Statistical summaries](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/evaluation/statistics.py): `src/evaluation/statistics.py`
- [Result plots](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/visualization/plots.py): `src/visualization/plots.py`
- [Exploratory analysis](https://github.com/orcads/data-001-impact-of-synthetic-data-under-data-scarcity/blob/master/src/visualization/eda.py): `src/visualization/eda.py`

### Other components

- **Researcher**
- **Healthcare datasets**

## Connections

- Researcher → Gaussian comparison: runs comparison
- Healthcare datasets → Dataset loading: provides records
- Gaussian comparison → Scarcity sampling: samples training
- Gaussian comparison → Gaussian Copula: generates augmentation
- Gaussian comparison → Logistic regression: fits classifier
- Gaussian comparison → Random forest: fits classifier
- Gaussian comparison → Predictive metrics: scores predictions
- Real-only baseline → Scarcity sampling: samples training
- Real-only baseline → Logistic regression: fits classifier
- Real-only baseline → Random forest: fits classifier
- Real-only baseline → Predictive metrics: evaluates predictions
- Gaussian diagnostics → Gaussian Copula: generates samples
- Gaussian diagnostics → Domain validation: checks domain
- Gaussian diagnostics → Fidelity metrics: measures fidelity
- Gaussian diagnostics → Proximity metrics: measures proximity
- CTGAN pilot → CTGAN generation: runs pilot
- CTGAN full grid → CTGAN generation: dispatches conditions
- CTGAN full grid → Grid verification: verifies conditions
- Gaussian comparison → Statistical summaries: summarizes metrics
- Statistical summaries → Result plots: supplies summaries
- Dataset loading → Exploratory analysis: supports exploration