# lucasnovack/siphon architecture

> Architecture diagram of the GitHub repository lucasnovack/siphon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lucasnovack/siphon
- Repository: https://github.com/lucasnovack/siphon

## Overview

Siphon is a self-hosted Bronze-layer data pipeline platform, with a web UI and FastAPI service for managing connections, defining pipelines, previewing SQL, and inspecting runs. Its main runtime flow is pipeline scheduling or manual triggering, asynchronous Celery execution, incremental extraction with quality and schema checks, and Parquet output to S3-compatible storage. The graph includes auth, persistence, audit, and metrics as supporting layers. Some runtime wiring is documented in the README or SOURCE INDEX; optional or UI-to-API links without direct excerpts are left unverified.

Main entry points: [`src/siphon/main.py`](https://github.com/lucasnovack/siphon/blob/master/src/siphon/main.py) (FastAPI service), [`src/siphon/orm.py`](https://github.com/lucasnovack/siphon/blob/master/src/siphon/orm.py) (Run and pipeline models), [`frontend/src/App.tsx`](https://github.com/lucasnovack/siphon/blob/master/frontend/src/App.tsx) (Pipeline UI), [`src/siphon/pipelines/router.py`](https://github.com/lucasnovack/siphon/blob/master/src/siphon/pipelines/router.py) (Pipeline management)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["Web interface"]
  node_web_ui["Pipeline UI<br/>[App.tsx]"]
  node_auth_ui["UI authentication<br/>[AuthContext.tsx]"]
  node_pipeline_editor["Pipeline editor<br/>[PipelineWizard.tsx]"]
end

subgraph group_api["API and management"]
  node_api_service["FastAPI service<br/>[main.py]"]
  node_auth_api["Authentication<br/>[router.py]"]
  node_auth_guard["Principal checks<br/>[deps.py]"]
  node_connection_registry["Connection registry<br/>[router.py]"]
  node_pipeline_api["Pipeline management<br/>[router.py]"]
  node_preview_api["SQL preview<br/>[router.py]"]
  node_run_api["Run history<br/>[router.py]"]
  node_gdpr_api["Data purge and audit<br/>[router.py]"]
end

subgraph group_orchestration["Scheduling and execution"]
  node_scheduler["Schedule dispatch<br/>[scheduler.py]"]
  node_celery_tasks["Async task entry<br/>[tasks.py]"]
  node_worker["Extraction worker<br/>[worker.py]"]
end

subgraph group_extraction["Pipeline processing"]
  node_watermarks["Incremental watermarks<br/>[watermark.py]"]
  node_sql_source["SQL source<br/>[sql.py]"]
  node_source_plugins["Source plugins"]
  node_destination_plugins["S3 Parquet output<br/>[s3_parquet.py]"]
end

subgraph group_state["State and integrations"]
  node_run_state["Run and pipeline models<br/>[orm.py]"]
  node_database[("PostgreSQL")]
  node_task_broker["Celery broker"]
  node_object_storage[("S3-compatible storage")]
  node_metrics["Prometheus metrics<br/>[metrics.py]"]
end

node_user(("Operator"))

node_user -->|"uses"| node_web_ui
node_web_ui -.->|"authenticates"| node_auth_ui
node_web_ui -.->|"configures pipelines"| node_pipeline_editor
node_web_ui -.->|"calls API"| node_api_service
node_api_service -->|"routes to"| node_auth_api
node_api_service -->|"routes to"| node_connection_registry
node_api_service -->|"routes to"| node_pipeline_api
node_api_service -->|"routes to"| node_preview_api
node_api_service -->|"routes to"| node_run_api
node_api_service -->|"routes to"| node_gdpr_api
node_auth_api -->|"issues identity"| node_auth_guard
node_auth_guard -->|"loads user"| node_run_state
node_connection_registry -->|"stores connections"| node_run_state
node_connection_registry -.->|"tests connectivity"| node_object_storage
node_pipeline_api -->|"stores pipelines"| node_run_state
node_pipeline_api -->|"resyncs schedules"| node_scheduler
node_pipeline_api -->|"injects watermark"| node_watermarks
node_preview_api -->|"executes preview"| node_sql_source
node_scheduler -->|"dispatches task"| node_task_broker
node_api_service -->|"accepts job"| node_task_broker
node_task_broker -->|"delivers task"| node_celery_tasks
node_celery_tasks -->|"invokes worker"| node_worker
node_worker -->|"extracts batches"| node_source_plugins
node_source_plugins -->|"registers SQL"| node_sql_source
node_worker -->|"writes output"| node_destination_plugins
node_destination_plugins -->|"writes Parquet"| node_object_storage
node_worker -->|"persists run"| node_run_state
node_run_api -->|"reads run records"| node_run_state
node_gdpr_api -->|"purges files"| node_object_storage
node_gdpr_api -->|"records audit event"| node_run_state
node_api_service -->|"updates metrics"| node_metrics
node_run_state -.->|"persists state"| node_database

click node_web_ui "https://github.com/lucasnovack/siphon/blob/master/frontend/src/App.tsx"
click node_auth_ui "https://github.com/lucasnovack/siphon/blob/master/frontend/src/contexts/AuthContext.tsx"
click node_pipeline_editor "https://github.com/lucasnovack/siphon/blob/master/frontend/src/pages/pipelines/PipelineWizard.tsx"
click node_api_service "https://github.com/lucasnovack/siphon/blob/master/src/siphon/main.py"
click node_auth_api "https://github.com/lucasnovack/siphon/blob/master/src/siphon/auth/router.py"
click node_auth_guard "https://github.com/lucasnovack/siphon/blob/master/src/siphon/auth/deps.py"
click node_connection_registry "https://github.com/lucasnovack/siphon/blob/master/src/siphon/connections/router.py"
click node_pipeline_api "https://github.com/lucasnovack/siphon/blob/master/src/siphon/pipelines/router.py"
click node_preview_api "https://github.com/lucasnovack/siphon/blob/master/src/siphon/preview/router.py"
click node_run_api "https://github.com/lucasnovack/siphon/blob/master/src/siphon/runs/router.py"
click node_gdpr_api "https://github.com/lucasnovack/siphon/blob/master/src/siphon/gdpr/router.py"
click node_scheduler "https://github.com/lucasnovack/siphon/blob/master/src/siphon/scheduler.py"
click node_celery_tasks "https://github.com/lucasnovack/siphon/blob/master/src/siphon/tasks.py"
click node_worker "https://github.com/lucasnovack/siphon/blob/master/src/siphon/worker.py"
click node_watermarks "https://github.com/lucasnovack/siphon/blob/master/src/siphon/pipelines/watermark.py"
click node_sql_source "https://github.com/lucasnovack/siphon/blob/master/src/siphon/plugins/sources/sql.py"
click node_source_plugins "https://github.com/lucasnovack/siphon/tree/master/src/siphon/plugins/sources"
click node_destination_plugins "https://github.com/lucasnovack/siphon/blob/master/src/siphon/plugins/destinations/s3_parquet.py"
click node_run_state "https://github.com/lucasnovack/siphon/blob/master/src/siphon/orm.py"
click node_metrics "https://github.com/lucasnovack/siphon/blob/master/src/siphon/metrics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web_ui,node_auth_ui,node_pipeline_editor toneBlue
class node_api_service,node_auth_api,node_auth_guard,node_connection_registry,node_pipeline_api,node_preview_api,node_run_api,node_gdpr_api toneAmber
class node_scheduler,node_celery_tasks,node_worker toneMint
class node_watermarks,node_sql_source,node_source_plugins,node_destination_plugins toneRose
class node_run_state,node_database,node_task_broker,node_object_storage,node_metrics,node_user toneIndigo
```

## Components

### Web interface

- [Pipeline UI](https://github.com/lucasnovack/siphon/blob/master/frontend/src/App.tsx): `frontend/src/App.tsx`
- [UI authentication](https://github.com/lucasnovack/siphon/blob/master/frontend/src/contexts/AuthContext.tsx): `frontend/src/contexts/AuthContext.tsx`
- [Pipeline editor](https://github.com/lucasnovack/siphon/blob/master/frontend/src/pages/pipelines/PipelineWizard.tsx): `frontend/src/pages/pipelines/PipelineWizard.tsx`

### API and management

- [FastAPI service](https://github.com/lucasnovack/siphon/blob/master/src/siphon/main.py): `src/siphon/main.py`
- [Authentication](https://github.com/lucasnovack/siphon/blob/master/src/siphon/auth/router.py): `src/siphon/auth/router.py`
- [Principal checks](https://github.com/lucasnovack/siphon/blob/master/src/siphon/auth/deps.py): `src/siphon/auth/deps.py`
- [Connection registry](https://github.com/lucasnovack/siphon/blob/master/src/siphon/connections/router.py): `src/siphon/connections/router.py`
- [Pipeline management](https://github.com/lucasnovack/siphon/blob/master/src/siphon/pipelines/router.py): `src/siphon/pipelines/router.py`
- [SQL preview](https://github.com/lucasnovack/siphon/blob/master/src/siphon/preview/router.py): `src/siphon/preview/router.py`
- [Run history](https://github.com/lucasnovack/siphon/blob/master/src/siphon/runs/router.py): `src/siphon/runs/router.py`
- [Data purge and audit](https://github.com/lucasnovack/siphon/blob/master/src/siphon/gdpr/router.py): `src/siphon/gdpr/router.py`

### Scheduling and execution

- [Schedule dispatch](https://github.com/lucasnovack/siphon/blob/master/src/siphon/scheduler.py): `src/siphon/scheduler.py`
- [Async task entry](https://github.com/lucasnovack/siphon/blob/master/src/siphon/tasks.py): `src/siphon/tasks.py`
- [Extraction worker](https://github.com/lucasnovack/siphon/blob/master/src/siphon/worker.py): `src/siphon/worker.py`

### Pipeline processing

- [Incremental watermarks](https://github.com/lucasnovack/siphon/blob/master/src/siphon/pipelines/watermark.py): `src/siphon/pipelines/watermark.py`
- [SQL source](https://github.com/lucasnovack/siphon/blob/master/src/siphon/plugins/sources/sql.py): `src/siphon/plugins/sources/sql.py`
- [Source plugins](https://github.com/lucasnovack/siphon/tree/master/src/siphon/plugins/sources): `src/siphon/plugins/sources`
- [S3 Parquet output](https://github.com/lucasnovack/siphon/blob/master/src/siphon/plugins/destinations/s3_parquet.py): `src/siphon/plugins/destinations/s3_parquet.py`

### State and integrations

- [Run and pipeline models](https://github.com/lucasnovack/siphon/blob/master/src/siphon/orm.py): `src/siphon/orm.py`
- **PostgreSQL**
- **Celery broker**
- **S3-compatible storage**
- [Prometheus metrics](https://github.com/lucasnovack/siphon/blob/master/src/siphon/metrics.py): `src/siphon/metrics.py`

### Other components

- **Operator**

## Connections

- Operator → Pipeline UI: uses (evidence: `README.md`)
- Pipeline UI → UI authentication: authenticates
- Pipeline UI → Pipeline editor: configures pipelines
- Pipeline UI → FastAPI service: calls API
- FastAPI service → Authentication: routes to (evidence: `src/siphon/main.py`)
- FastAPI service → Connection registry: routes to (evidence: `src/siphon/main.py`)
- FastAPI service → Pipeline management: routes to (evidence: `src/siphon/main.py`)
- FastAPI service → SQL preview: routes to (evidence: `src/siphon/main.py`)
- FastAPI service → Run history: routes to (evidence: `src/siphon/main.py`)
- FastAPI service → Data purge and audit: routes to (evidence: `src/siphon/main.py`)
- Authentication → Principal checks: issues identity (evidence: `src/siphon/auth/router.py`)
- Principal checks → Run and pipeline models: loads user (evidence: `src/siphon/auth/deps.py`)
- Connection registry → Run and pipeline models: stores connections (evidence: `src/siphon/connections/router.py`)
- Connection registry → S3-compatible storage: tests connectivity
- Pipeline management → Run and pipeline models: stores pipelines (evidence: `src/siphon/pipelines/router.py`)
- Pipeline management → Schedule dispatch: resyncs schedules (evidence: `src/siphon/pipelines/router.py`)
- Pipeline management → Incremental watermarks: injects watermark (evidence: `src/siphon/pipelines/router.py`)
- SQL preview → SQL source: executes preview (evidence: `src/siphon/preview/router.py`)
- Schedule dispatch → Celery broker: dispatches task (evidence: `README.md`)
- FastAPI service → Celery broker: accepts job (evidence: `README.md`)
- Celery broker → Async task entry: delivers task (evidence: `README.md`)
- Async task entry → Extraction worker: invokes worker (evidence: `src/siphon/tasks.py`)
- Extraction worker → Source plugins: extracts batches (evidence: `src/siphon/worker.py`)
- Source plugins → SQL source: registers SQL (evidence: `src/siphon/plugins/sources/__init__.py`)
- Extraction worker → S3 Parquet output: writes output (evidence: `README.md`)
- S3 Parquet output → S3-compatible storage: writes Parquet (evidence: `README.md`)
- Extraction worker → Run and pipeline models: persists run (evidence: `README.md`)
- Run history → Run and pipeline models: reads run records (evidence: `src/siphon/runs/router.py`)
- Data purge and audit → S3-compatible storage: purges files (evidence: `src/siphon/gdpr/router.py`)
- Data purge and audit → Run and pipeline models: records audit event (evidence: `src/siphon/gdpr/router.py`)
- FastAPI service → Prometheus metrics: updates metrics (evidence: `src/siphon/main.py`)
- Run and pipeline models → PostgreSQL: persists state
