# yangel20/localetl architecture

> Architecture diagram of the GitHub repository yangel20/localetl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yangel20/localetl
- Repository: https://github.com/yangel20/localetl
- Video tour (about a minute): https://gitdiagram.com/yangel20/localetl/video

## Overview

LocalETL is a local-first web app for turning uploaded CSVs into cleaned, validated PostgreSQL datasets. The user workflow runs from source upload and preview, through pipeline configuration and queued execution, to run monitoring and dataset exploration. The README documents the worker/ETL/storage flow; sampled code also shows pipeline API validation and frontend pipeline configuration. ETL internals and several API routes were not sampled, so their stage links rely on the README rather than inspected calls.

Main entry points: [`frontend/src/app/api/client.ts`](https://github.com/yangel20/localetl/blob/main/frontend/src/app/api/client.ts) (API client), [`backend/app/models/tables.py`](https://github.com/yangel20/localetl/blob/main/backend/app/models/tables.py) (PostgreSQL), [`frontend/src/app/App.tsx`](https://github.com/yangel20/localetl/blob/main/frontend/src/app/App.tsx) (Application shell), [`backend/app/services/runs/executor.py`](https://github.com/yangel20/localetl/blob/main/backend/app/services/runs/executor.py) (Run executor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Web application"]
  node_web["Application shell<br/>[App.tsx]"]
  node_source_ui["Sources and preview"]
  node_pipeline_ui["Pipeline editor<br/>[PipelineForm.tsx]"]
  node_run_ui["Run monitoring"]
  node_dataset_ui["Dataset explorer"]
  node_analytics_ui["Run analytics"]
  node_api_client["API client<br/>[client.ts]"]
end

subgraph group_api["API and jobs"]
  node_source_api["Source endpoints<br/>[sources.py]"]
  node_pipeline_api["Pipeline endpoints<br/>[pipelines.py]"]
  node_run_api["Run endpoints<br/>[runs.py]"]
  node_dataset_api["Dataset endpoints<br/>[datasets.py]"]
  node_metrics_api["Metrics endpoints<br/>[metrics.py]"]
  node_worker["Background worker<br/>[loop.py]"]
  node_executor["Run executor<br/>[executor.py]"]
end

subgraph group_etl["ETL processing"]
  node_extract["CSV extraction<br/>[extract.py]"]
  node_transform["Transform and cast<br/>[transform.py]"]
  node_validate["Record validation<br/>[validate.py]"]
  node_load["Dataset loading<br/>[load.py]"]
end

subgraph group_storage["Data and persistence"]
  node_csv_store["Local CSV storage<br/>[sources.py]"]
  node_db[("PostgreSQL<br/>[tables.py]")]
end

node_user(("User"))

node_user -->|"uses"| node_web
node_web -->|"routes to"| node_source_ui
node_web -->|"routes to"| node_pipeline_ui
node_web -->|"routes to"| node_run_ui
node_web -->|"routes to"| node_dataset_ui
node_source_ui -.->|"requests source APIs"| node_api_client
node_pipeline_ui -->|"creates and validates"| node_api_client
node_run_ui -.->|"requests run data"| node_api_client
node_dataset_ui -.->|"requests datasets"| node_api_client
node_analytics_ui -.->|"requests metrics"| node_api_client
node_api_client -.->|"HTTP requests"| node_source_api
node_api_client -.->|"HTTP requests"| node_pipeline_api
node_api_client -.->|"HTTP requests"| node_run_api
node_api_client -.->|"HTTP requests"| node_dataset_api
node_api_client -.->|"HTTP requests"| node_metrics_api
node_source_api -->|"stores uploads"| node_csv_store
node_source_api -->|"stores metadata"| node_db
node_pipeline_api -->|"persists config"| node_db
node_pipeline_api -->|"validates config"| node_validate
node_run_api -->|"queues run"| node_db
node_worker -->|"claims queued run"| node_db
node_worker -->|"executes run"| node_executor
node_executor -->|"starts ETL"| node_extract
node_executor -->|"applies transformations"| node_transform
node_executor -->|"validates records"| node_validate
node_executor -->|"loads results"| node_load
node_extract -->|"reads source file"| node_csv_store
node_load -->|"writes records"| node_db
node_dataset_api -->|"reads datasets"| node_db
node_metrics_api -->|"reads run metrics"| node_db
node_run_api -->|"reads run results"| node_db

click node_web "https://github.com/yangel20/localetl/blob/main/frontend/src/app/App.tsx"
click node_source_ui "https://github.com/yangel20/localetl/tree/main/frontend/src/pages/sources"
click node_pipeline_ui "https://github.com/yangel20/localetl/blob/main/frontend/src/features/pipelines/PipelineForm.tsx"
click node_run_ui "https://github.com/yangel20/localetl/tree/main/frontend/src/pages/runs"
click node_dataset_ui "https://github.com/yangel20/localetl/tree/main/frontend/src/features/explorer"
click node_analytics_ui "https://github.com/yangel20/localetl/tree/main/frontend/src/features/analytics"
click node_api_client "https://github.com/yangel20/localetl/blob/main/frontend/src/app/api/client.ts"
click node_source_api "https://github.com/yangel20/localetl/blob/main/backend/app/api/sources.py"
click node_pipeline_api "https://github.com/yangel20/localetl/blob/main/backend/app/api/pipelines.py"
click node_run_api "https://github.com/yangel20/localetl/blob/main/backend/app/api/runs.py"
click node_dataset_api "https://github.com/yangel20/localetl/blob/main/backend/app/api/datasets.py"
click node_metrics_api "https://github.com/yangel20/localetl/blob/main/backend/app/api/metrics.py"
click node_worker "https://github.com/yangel20/localetl/blob/main/backend/app/worker/loop.py"
click node_executor "https://github.com/yangel20/localetl/blob/main/backend/app/services/runs/executor.py"
click node_extract "https://github.com/yangel20/localetl/blob/main/backend/app/etl/extract.py"
click node_transform "https://github.com/yangel20/localetl/blob/main/backend/app/etl/transform.py"
click node_validate "https://github.com/yangel20/localetl/blob/main/backend/app/etl/validate.py"
click node_load "https://github.com/yangel20/localetl/blob/main/backend/app/etl/load.py"
click node_csv_store "https://github.com/yangel20/localetl/blob/main/backend/app/api/sources.py"
click node_db "https://github.com/yangel20/localetl/blob/main/backend/app/models/tables.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web,node_source_ui,node_pipeline_ui,node_run_ui,node_dataset_ui,node_analytics_ui,node_api_client,node_user toneBlue
class node_source_api,node_pipeline_api,node_run_api,node_dataset_api,node_metrics_api,node_worker,node_executor toneAmber
class node_extract,node_transform,node_validate,node_load toneMint
class node_csv_store,node_db toneRose
```

## Components

### Web application

- [Application shell](https://github.com/yangel20/localetl/blob/main/frontend/src/app/App.tsx): `frontend/src/app/App.tsx`
- [Sources and preview](https://github.com/yangel20/localetl/tree/main/frontend/src/pages/sources): `frontend/src/pages/sources`
- [Pipeline editor](https://github.com/yangel20/localetl/blob/main/frontend/src/features/pipelines/PipelineForm.tsx): `frontend/src/features/pipelines/PipelineForm.tsx`
- [Run monitoring](https://github.com/yangel20/localetl/tree/main/frontend/src/pages/runs): `frontend/src/pages/runs`
- [Dataset explorer](https://github.com/yangel20/localetl/tree/main/frontend/src/features/explorer): `frontend/src/features/explorer`
- [Run analytics](https://github.com/yangel20/localetl/tree/main/frontend/src/features/analytics): `frontend/src/features/analytics`
- [API client](https://github.com/yangel20/localetl/blob/main/frontend/src/app/api/client.ts): `frontend/src/app/api/client.ts`

### API and jobs

- [Source endpoints](https://github.com/yangel20/localetl/blob/main/backend/app/api/sources.py): `backend/app/api/sources.py`
- [Pipeline endpoints](https://github.com/yangel20/localetl/blob/main/backend/app/api/pipelines.py): `backend/app/api/pipelines.py`
- [Run endpoints](https://github.com/yangel20/localetl/blob/main/backend/app/api/runs.py): `backend/app/api/runs.py`
- [Dataset endpoints](https://github.com/yangel20/localetl/blob/main/backend/app/api/datasets.py): `backend/app/api/datasets.py`
- [Metrics endpoints](https://github.com/yangel20/localetl/blob/main/backend/app/api/metrics.py): `backend/app/api/metrics.py`
- [Background worker](https://github.com/yangel20/localetl/blob/main/backend/app/worker/loop.py): `backend/app/worker/loop.py`
- [Run executor](https://github.com/yangel20/localetl/blob/main/backend/app/services/runs/executor.py): `backend/app/services/runs/executor.py`

### ETL processing

- [CSV extraction](https://github.com/yangel20/localetl/blob/main/backend/app/etl/extract.py): `backend/app/etl/extract.py`
- [Transform and cast](https://github.com/yangel20/localetl/blob/main/backend/app/etl/transform.py): `backend/app/etl/transform.py`
- [Record validation](https://github.com/yangel20/localetl/blob/main/backend/app/etl/validate.py): `backend/app/etl/validate.py`
- [Dataset loading](https://github.com/yangel20/localetl/blob/main/backend/app/etl/load.py): `backend/app/etl/load.py`

### Data and persistence

- [Local CSV storage](https://github.com/yangel20/localetl/blob/main/backend/app/api/sources.py): `backend/app/api/sources.py`
- [PostgreSQL](https://github.com/yangel20/localetl/blob/main/backend/app/models/tables.py): `backend/app/models/tables.py`

### Other components

- **User**

## Connections

- User → Application shell: uses
- Application shell → Sources and preview: routes to
- Application shell → Pipeline editor: routes to
- Application shell → Run monitoring: routes to
- Application shell → Dataset explorer: routes to
- Sources and preview → API client: requests source APIs
- Pipeline editor → API client: creates and validates (evidence: `frontend/src/features/pipelines/PipelineForm.tsx`)
- Run monitoring → API client: requests run data
- Dataset explorer → API client: requests datasets
- Run analytics → API client: requests metrics
- API client → Source endpoints: HTTP requests
- API client → Pipeline endpoints: HTTP requests
- API client → Run endpoints: HTTP requests
- API client → Dataset endpoints: HTTP requests
- API client → Metrics endpoints: HTTP requests
- Source endpoints → Local CSV storage: stores uploads
- Source endpoints → PostgreSQL: stores metadata
- Pipeline endpoints → PostgreSQL: persists config (evidence: `backend/app/api/pipelines.py`)
- Pipeline endpoints → Record validation: validates config (evidence: `backend/app/api/pipelines.py`)
- Run endpoints → PostgreSQL: queues run
- Background worker → PostgreSQL: claims queued run
- Background worker → Run executor: executes run
- Run executor → CSV extraction: starts ETL
- Run executor → Transform and cast: applies transformations
- Run executor → Record validation: validates records
- Run executor → Dataset loading: loads results
- CSV extraction → Local CSV storage: reads source file
- Dataset loading → PostgreSQL: writes records
- Dataset endpoints → PostgreSQL: reads datasets
- Metrics endpoints → PostgreSQL: reads run metrics
- Run endpoints → PostgreSQL: reads run results