How chandralekha77/enterprise-etl-data-quality-framework works
The product is a multi-source ETL and data-quality platform: Airflow initiates durable runs, source extracts are staged, transformations and quality gates govern promotion, and results feed read-only operational reporting. The README also describes dbt models, reconciliation, release gating, and dashboards; some implementations are not sampled, so those links are omitted or marked unverified rather than inferred. Edges cite only shown call/reference evidence or README statements.
Main entry points: airflow_tasks.py Airflow task services · main.py Reporting API · staging_loaders.py Replay-safe staging · dbt Warehouse transformations
Components 25
Source ingestion
- CRM and billing extracts
src/etl_framework/ingestion/database_sources.py - Customer status API
src/etl_framework/ingestion/customer_status_api.py - Operational CSV
src/etl_framework/ingestion/csv_source.py - Replay-safe staging
src/etl_framework/ingestion/staging_loaders.py
Run orchestration
- Airflow DAG
airflow/dags/enterprise_etl_pipeline.py - Airflow task services
src/etl_framework/orchestration/airflow_tasks.py - Checkpoints and leases
src/etl_framework/orchestration/pipeline.py - Run and batch tracking
src/etl_framework/control/run_tracking.py
Quality and release
- Warehouse transformations
dbt - Business rules
src/etl_framework/quality/business_rules.py - Validation results
src/etl_framework/quality/validation_results.py - Quality gate
src/etl_framework/quality/quality_gate.py - Source reconciliation
src/etl_framework/quality/reconciliation.py - Release gate
src/etl_framework/quality/release_gate.py
Operations reporting
- Reporting API
api/main.py - Report routes
api/routes/reports.py - Reporting repository
src/etl_framework/reporting/repository.py - Operations dashboard
api/dashboard.py - Run and replay controls
api/operator.py - Operational metrics
api/metrics.py
Database access
- Validated settings
src/etl_framework/config/settings.py - Connection pools
src/etl_framework/database/connections.py
Other components
- Source systems
- Warehouse
- Operators and viewers
Connections 30
- Airflow DAG → Airflow task services: invokes
airflow/dags/enterprise_etl_pipeline.py - Airflow task services → CRM and billing extracts: extracts
- Airflow task services → Customer status API: ingests
- Airflow task services → Operational CSV: loads
- Airflow task services → Replay-safe staging: loads staging
- Source systems → CRM and billing extracts: provides rows
README.md - Source systems → Customer status API: provides status
README.md - Source systems → Operational CSV: provides files
README.md - Airflow task services → Checkpoints and leases: manages checkpoints
src/etl_framework/orchestration/airflow_tasks.py - Airflow task services → Run and batch tracking: records runs
src/etl_framework/orchestration/airflow_tasks.py - Airflow task services → Warehouse transformations: runs models
src/etl_framework/orchestration/airflow_tasks.py - Replay-safe staging → Warehouse: writes staging
README.md - Warehouse transformations → Warehouse: transforms
README.md - Airflow task services → Business rules: validates
- Business rules → Validation results: stores failures
src/etl_framework/quality/business_rules.py - Airflow task services → Quality gate: gates quality
- Airflow task services → Source reconciliation: reconciles
- Airflow task services → Release gate: checks release
- Release gate → Warehouse: controls release
README.md - Reporting API → Report routes: routes requests
api/main.py - Reporting API → Operations dashboard: mounts
api/main.py - Reporting API → Run and replay controls: mounts
api/main.py - Reporting API → Operational metrics: mounts
api/main.py - Report routes → Reporting repository: queries
api/routes/reports.py - Reporting repository → Warehouse: reads reports
src/etl_framework/reporting/repository.py - Operators and viewers → Reporting API: requests reports
README.md - Operators and viewers → Operations dashboard: views dashboard
README.md - Operators and viewers → Run and replay controls: triggers runs
README.md - Validated settings → Connection pools: configures
src/etl_framework/database/connections.py - Airflow task services → Connection pools: uses engines
src/etl_framework/orchestration/airflow_tasks.py