How lucasnovack/siphon works
Siphon is a self-hosted Bronze-layer data pipeline platform, with a web UI and FastAPI service for managing connections, defining pipelines, previewing SQL, and inspecting runs. Its main runtime flow is pipeline scheduling or manual triggering, asynchronous Celery execution, incremental extraction with quality and schema checks, and Parquet output to S3-compatible storage. The graph includes auth, persistence, audit, and metrics as supporting layers. Some runtime wiring is documented in the README or SOURCE INDEX; optional or UI-to-API links without direct excerpts are left unverified.
Main entry points: main.py FastAPI service · orm.py Run and pipeline models · App.tsx Pipeline UI · router.py Pipeline management
Components 24
Web interface
- Pipeline UI
frontend/src/App.tsx - UI authentication
frontend/src/contexts/AuthContext.tsx - Pipeline editor
frontend/src/pages/pipelines/PipelineWizard.tsx
API and management
- FastAPI service
src/siphon/main.py - Authentication
src/siphon/auth/router.py - Principal checks
src/siphon/auth/deps.py - Connection registry
src/siphon/connections/router.py - Pipeline management
src/siphon/pipelines/router.py - SQL preview
src/siphon/preview/router.py - Run history
src/siphon/runs/router.py - Data purge and audit
src/siphon/gdpr/router.py
Scheduling and execution
- Schedule dispatch
src/siphon/scheduler.py - Async task entry
src/siphon/tasks.py - Extraction worker
src/siphon/worker.py
Pipeline processing
- Incremental watermarks
src/siphon/pipelines/watermark.py - SQL source
src/siphon/plugins/sources/sql.py - Source plugins
src/siphon/plugins/sources - S3 Parquet output
src/siphon/plugins/destinations/s3_parquet.py
State and integrations
- Run and pipeline models
src/siphon/orm.py - PostgreSQL
- Celery broker
- S3-compatible storage
- Prometheus metrics
src/siphon/metrics.py
Other components
- Operator
Connections 32
- Operator → Pipeline UI: uses
README.md - Pipeline UI → UI authentication: authenticates
- Pipeline UI → Pipeline editor: configures pipelines
- Pipeline UI → FastAPI service: calls API
- FastAPI service → Authentication: routes to
src/siphon/main.py - FastAPI service → Connection registry: routes to
src/siphon/main.py - FastAPI service → Pipeline management: routes to
src/siphon/main.py - FastAPI service → SQL preview: routes to
src/siphon/main.py - FastAPI service → Run history: routes to
src/siphon/main.py - FastAPI service → Data purge and audit: routes to
src/siphon/main.py - Authentication → Principal checks: issues identity
src/siphon/auth/router.py - Principal checks → Run and pipeline models: loads user
src/siphon/auth/deps.py - Connection registry → Run and pipeline models: stores connections
src/siphon/connections/router.py - Connection registry → S3-compatible storage: tests connectivity
- Pipeline management → Run and pipeline models: stores pipelines
src/siphon/pipelines/router.py - Pipeline management → Schedule dispatch: resyncs schedules
src/siphon/pipelines/router.py - Pipeline management → Incremental watermarks: injects watermark
src/siphon/pipelines/router.py - SQL preview → SQL source: executes preview
src/siphon/preview/router.py - Schedule dispatch → Celery broker: dispatches task
README.md - FastAPI service → Celery broker: accepts job
README.md - Celery broker → Async task entry: delivers task
README.md - Async task entry → Extraction worker: invokes worker
src/siphon/tasks.py - Extraction worker → Source plugins: extracts batches
src/siphon/worker.py - Source plugins → SQL source: registers SQL
src/siphon/plugins/sources/__init__.py - Extraction worker → S3 Parquet output: writes output
README.md - S3 Parquet output → S3-compatible storage: writes Parquet
README.md - Extraction worker → Run and pipeline models: persists run
README.md - Run history → Run and pipeline models: reads run records
src/siphon/runs/router.py - Data purge and audit → S3-compatible storage: purges files
src/siphon/gdpr/router.py - Data purge and audit → Run and pipeline models: records audit event
src/siphon/gdpr/router.py - FastAPI service → Prometheus metrics: updates metrics
src/siphon/main.py - Run and pipeline models → PostgreSQL: persists state