How yangel20/localetl works
LocalETL is a local-first web app for turning uploaded CSVs into cleaned, validated PostgreSQL datasets. The user workflow runs from source upload and preview, through pipeline configuration and queued execution, to run monitoring and dataset exploration. The README documents the worker/ETL/storage flow; sampled code also shows pipeline API validation and frontend pipeline configuration. ETL internals and several API routes were not sampled, so their stage links rely on the README rather than inspected calls.
Main entry points: client.ts API client · tables.py PostgreSQL · App.tsx Application shell · executor.py Run executor
Components 21
Web application
- Application shell
frontend/src/app/App.tsx - Sources and preview
frontend/src/pages/sources - Pipeline editor
frontend/src/features/pipelines/PipelineForm.tsx - Run monitoring
frontend/src/pages/runs - Dataset explorer
frontend/src/features/explorer - Run analytics
frontend/src/features/analytics - API client
frontend/src/app/api/client.ts
API and jobs
- Source endpoints
backend/app/api/sources.py - Pipeline endpoints
backend/app/api/pipelines.py - Run endpoints
backend/app/api/runs.py - Dataset endpoints
backend/app/api/datasets.py - Metrics endpoints
backend/app/api/metrics.py - Background worker
backend/app/worker/loop.py - Run executor
backend/app/services/runs/executor.py
ETL processing
- CSV extraction
backend/app/etl/extract.py - Transform and cast
backend/app/etl/transform.py - Record validation
backend/app/etl/validate.py - Dataset loading
backend/app/etl/load.py
Data and persistence
- Local CSV storage
backend/app/api/sources.py - PostgreSQL
backend/app/models/tables.py
Other components
- User
Connections 31
- User → Application shell: uses
- Application shell → Sources and preview: routes to
- Application shell → Pipeline editor: routes to
- Application shell → Run monitoring: routes to
- Application shell → Dataset explorer: routes to
- Sources and preview → API client: requests source APIs
- Pipeline editor → API client: creates and validates
frontend/src/features/pipelines/PipelineForm.tsx - Run monitoring → API client: requests run data
- Dataset explorer → API client: requests datasets
- Run analytics → API client: requests metrics
- API client → Source endpoints: HTTP requests
- API client → Pipeline endpoints: HTTP requests
- API client → Run endpoints: HTTP requests
- API client → Dataset endpoints: HTTP requests
- API client → Metrics endpoints: HTTP requests
- Source endpoints → Local CSV storage: stores uploads
- Source endpoints → PostgreSQL: stores metadata
- Pipeline endpoints → PostgreSQL: persists config
backend/app/api/pipelines.py - Pipeline endpoints → Record validation: validates config
backend/app/api/pipelines.py - Run endpoints → PostgreSQL: queues run
- Background worker → PostgreSQL: claims queued run
- Background worker → Run executor: executes run
- Run executor → CSV extraction: starts ETL
- Run executor → Transform and cast: applies transformations
- Run executor → Record validation: validates records
- Run executor → Dataset loading: loads results
- CSV extraction → Local CSV storage: reads source file
- Dataset loading → PostgreSQL: writes records
- Dataset endpoints → PostgreSQL: reads datasets
- Metrics endpoints → PostgreSQL: reads run metrics
- Run endpoints → PostgreSQL: reads run results