# ispirt/depa-training architecture

> Architecture diagram of the GitHub repository ispirt/depa-training, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ispirt/depa-training
- Repository: https://github.com/ispirt/depa-training

## Overview

DEPA for Training is a reference implementation of confidential clean rooms for privacy-preserving analytics and model training. A scenario supplies data preparation and pipeline configuration; the core framework supports dataset loading, joining, and differential-privacy-aware training, with evaluation and model outputs. Contracts, policy, and encrypted-data handling provide important governance and data-protection capabilities. The interactive demo discovers scenarios and exposes deployment and status controls. Edges are limited to README statements and relationships visible in the excerpts or source index; unsampled runtime wiring is left unasserted.

Main entry points: [`src/train/pytrain/pipeline_executor.py`](https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/pipeline_executor.py) (Pipeline Executor), [`src/train/pytrain/xgb_train.py`](https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/xgb_train.py) (XGBoost Training), [`gui-demo/app.py`](https://github.com/ispirt/depa-training/blob/main/gui-demo/app.py) (Demo Backend), [`src/train/pytrain/dl_train.py`](https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/dl_train.py) (Deep Learning)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_demo["Interactive Demo"]
  node_demo_ui["Demo UI<br/>[app.js]"]
  node_demo_api["Demo Backend<br/>[app.py]"]
end

subgraph group_scenario["Scenario Workflows"]
  node_scenario_catalog["Scenario Definitions"]
  node_scenario_config["Pipeline Config"]
  node_preprocessing["Data Preprocessing"]
  node_model_setup["Model Setup"]
  node_dataset_store[("Scenario Data")]
  node_model_artifacts["Trained Models"]
end

subgraph group_engine["Training Engine"]
  node_pipeline_cli["Pipeline Executor"]
  node_dataset_loader["Dataset Construction"]
  node_dataset_join["Dataset Joining<br/>[join.py]"]
  node_task_base["Task Base<br/>[task_base.py]"]
  node_deep_learning["Deep Learning<br/>[dl_train.py]"]
  node_xgb_training["XGBoost Training<br/>[xgb_train.py]"]
  node_model_builders["Model Construction"]
  node_loss_builders["Loss Construction"]
  node_evaluation["Evaluation<br/>[eval_tools.py]"]
end

subgraph group_privacy["Privacy And Governance"]
  node_encrypted_data["Encrypted Data Loader<br/>[encfs.sh]"]
  node_policy_rules["Policy Rules<br/>[policy.rego]"]
  node_contracts["Scenario Contracts"]
end

subgraph group_integrations["External Services"]
  node_contract_service["Contract Service"]
  node_azure["Azure Services"]
end

node_user(("Scenario Operator"))

node_user -->|"operates"| node_demo_ui
node_demo_ui -->|"requests scenarios"| node_demo_api
node_demo_api -->|"discovers"| node_scenario_catalog
node_demo_ui -->|"runs scripts"| node_demo_api
node_demo_api -->|"checks account"| node_azure
node_demo_api -->|"reads scenario"| node_scenario_config
node_scenario_config -.->|"configures"| node_pipeline_cli
node_pipeline_cli -.->|"loads datasets"| node_dataset_loader
node_pipeline_cli -.->|"dispatches join"| node_dataset_join
node_pipeline_cli -.->|"dispatches training"| node_deep_learning
node_pipeline_cli -.->|"dispatches training"| node_xgb_training
node_dataset_loader -.->|"reads data"| node_dataset_store
node_dataset_join -.->|"joins datasets"| node_dataset_store
node_xgb_training -->|"inherits from"| node_task_base
node_xgb_training -->|"imports"| node_dataset_loader
node_xgb_training -->|"imports"| node_evaluation
node_deep_learning -.->|"uses"| node_model_builders
node_deep_learning -.->|"uses"| node_loss_builders
node_evaluation -.->|"evaluates outputs"| node_model_artifacts
node_deep_learning -.->|"produces models"| node_model_artifacts
node_xgb_training -.->|"produces models"| node_model_artifacts
node_encrypted_data -.->|"loads encrypted data"| node_dataset_store
node_contracts -->|"works with"| node_contract_service
node_user -.->|"signs contract"| node_contract_service
node_scenario_config -.->|"sets policy"| node_policy_rules
node_azure -.->|"hosts clean room"| node_encrypted_data

click node_demo_ui "https://github.com/ispirt/depa-training/blob/main/gui-demo/static/js/app.js"
click node_demo_api "https://github.com/ispirt/depa-training/blob/main/gui-demo/app.py"
click node_scenario_catalog "https://github.com/ispirt/depa-training/tree/main/scenarios"
click node_scenario_config "https://github.com/ispirt/depa-training/tree/main/scenarios"
click node_preprocessing "https://github.com/ispirt/depa-training/tree/main/scenarios"
click node_model_setup "https://github.com/ispirt/depa-training/tree/main/scenarios"
click node_pipeline_cli "https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/pipeline_executor.py"
click node_dataset_loader "https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/utilities/dataset_constructor.py"
click node_dataset_join "https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/join.py"
click node_task_base "https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/task_base.py"
click node_deep_learning "https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/dl_train.py"
click node_xgb_training "https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/xgb_train.py"
click node_model_builders "https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/utilities/model_constructor.py"
click node_loss_builders "https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/utilities/loss_constructor.py"
click node_evaluation "https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/utilities/eval_tools.py"
click node_dataset_store "https://github.com/ispirt/depa-training/tree/main/scenarios"
click node_model_artifacts "https://github.com/ispirt/depa-training/tree/main/scenarios"
click node_encrypted_data "https://github.com/ispirt/depa-training/blob/main/src/encfs/encfs.sh"
click node_policy_rules "https://github.com/ispirt/depa-training/blob/main/src/policy/policy.rego"
click node_contracts "https://github.com/ispirt/depa-training/tree/main/scenarios"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_demo_ui,node_demo_api toneBlue
class node_scenario_catalog,node_scenario_config,node_preprocessing,node_model_setup,node_dataset_store,node_model_artifacts toneAmber
class node_pipeline_cli,node_dataset_loader,node_dataset_join,node_task_base,node_deep_learning,node_xgb_training,node_model_builders,node_loss_builders,node_evaluation toneMint
class node_encrypted_data,node_policy_rules,node_contracts toneRose
class node_contract_service,node_azure,node_user toneIndigo
```

## Components

### Interactive Demo

- [Demo UI](https://github.com/ispirt/depa-training/blob/main/gui-demo/static/js/app.js): `gui-demo/static/js/app.js`
- [Demo Backend](https://github.com/ispirt/depa-training/blob/main/gui-demo/app.py): `gui-demo/app.py`

### Scenario Workflows

- [Scenario Definitions](https://github.com/ispirt/depa-training/tree/main/scenarios): `scenarios`
- [Pipeline Config](https://github.com/ispirt/depa-training/tree/main/scenarios): `scenarios`
- [Data Preprocessing](https://github.com/ispirt/depa-training/tree/main/scenarios): `scenarios`
- [Model Setup](https://github.com/ispirt/depa-training/tree/main/scenarios): `scenarios`
- [Scenario Data](https://github.com/ispirt/depa-training/tree/main/scenarios): `scenarios`
- [Trained Models](https://github.com/ispirt/depa-training/tree/main/scenarios): `scenarios`

### Training Engine

- [Pipeline Executor](https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/pipeline_executor.py): `src/train/pytrain/pipeline_executor.py`
- [Dataset Construction](https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/utilities/dataset_constructor.py): `src/train/pytrain/utilities/dataset_constructor.py`
- [Dataset Joining](https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/join.py): `src/train/pytrain/join.py`
- [Task Base](https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/task_base.py): `src/train/pytrain/task_base.py`
- [Deep Learning](https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/dl_train.py): `src/train/pytrain/dl_train.py`
- [XGBoost Training](https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/xgb_train.py): `src/train/pytrain/xgb_train.py`
- [Model Construction](https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/utilities/model_constructor.py): `src/train/pytrain/utilities/model_constructor.py`
- [Loss Construction](https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/utilities/loss_constructor.py): `src/train/pytrain/utilities/loss_constructor.py`
- [Evaluation](https://github.com/ispirt/depa-training/blob/main/src/train/pytrain/utilities/eval_tools.py): `src/train/pytrain/utilities/eval_tools.py`

### Privacy And Governance

- [Encrypted Data Loader](https://github.com/ispirt/depa-training/blob/main/src/encfs/encfs.sh): `src/encfs/encfs.sh`
- [Policy Rules](https://github.com/ispirt/depa-training/blob/main/src/policy/policy.rego): `src/policy/policy.rego`
- [Scenario Contracts](https://github.com/ispirt/depa-training/tree/main/scenarios): `scenarios`

### External Services

- **Contract Service**
- **Azure Services**

### Other components

- **Scenario Operator**

## Connections

- Scenario Operator → Demo UI: operates (evidence: `README.md`)
- Demo UI → Demo Backend: requests scenarios (evidence: `gui-demo/static/js/app.js`)
- Demo Backend → Scenario Definitions: discovers (evidence: `gui-demo/app.py`)
- Demo UI → Demo Backend: runs scripts (evidence: `gui-demo/static/js/app.js`)
- Demo Backend → Azure Services: checks account (evidence: `gui-demo/app.py`)
- Demo Backend → Pipeline Config: reads scenario (evidence: `gui-demo/app.py`)
- Pipeline Config → Pipeline Executor: configures
- Pipeline Executor → Dataset Construction: loads datasets
- Pipeline Executor → Dataset Joining: dispatches join
- Pipeline Executor → Deep Learning: dispatches training
- Pipeline Executor → XGBoost Training: dispatches training
- Dataset Construction → Scenario Data: reads data
- Dataset Joining → Scenario Data: joins datasets
- XGBoost Training → Task Base: inherits from (evidence: `src/train/pytrain/xgb_train.py`)
- XGBoost Training → Dataset Construction: imports (evidence: `src/train/pytrain/xgb_train.py`)
- XGBoost Training → Evaluation: imports (evidence: `src/train/pytrain/xgb_train.py`)
- Deep Learning → Model Construction: uses
- Deep Learning → Loss Construction: uses
- Evaluation → Trained Models: evaluates outputs
- Deep Learning → Trained Models: produces models
- XGBoost Training → Trained Models: produces models
- Encrypted Data Loader → Scenario Data: loads encrypted data
- Scenario Contracts → Contract Service: works with (evidence: `README.md`)
- Scenario Operator → Contract Service: signs contract
- Pipeline Config → Policy Rules: sets policy
- Azure Services → Encrypted Data Loader: hosts clean room