# chandralekha77/enterprise-etl-data-quality-framework architecture

> Architecture diagram of the GitHub repository chandralekha77/enterprise-etl-data-quality-framework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chandralekha77/enterprise-etl-data-quality-framework
- Repository: https://github.com/chandralekha77/enterprise-etl-data-quality-framework

## Overview

The product is a multi-source ETL and data-quality platform: Airflow initiates durable runs, source extracts are staged, transformations and quality gates govern promotion, and results feed read-only operational reporting. The README also describes dbt models, reconciliation, release gating, and dashboards; some implementations are not sampled, so those links are omitted or marked unverified rather than inferred. Edges cite only shown call/reference evidence or README statements.

Main entry points: [`src/etl_framework/orchestration/airflow_tasks.py`](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/orchestration/airflow_tasks.py) (Airflow task services), [`api/main.py`](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/api/main.py) (Reporting API), [`src/etl_framework/ingestion/staging_loaders.py`](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/ingestion/staging_loaders.py) (Replay-safe staging), [`dbt`](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/tree/main/dbt) (Warehouse transformations)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Source ingestion"]
  node_crm["CRM and billing extracts"]
  node_status["Customer status API"]
  node_csv["Operational CSV<br/>[csv_source.py]"]
  node_staging[("Replay-safe staging<br/>[staging_loaders.py]")]
end

subgraph group_run["Run orchestration"]
  node_airflow["Airflow DAG"]
  node_tasks["Airflow task services<br/>[airflow_tasks.py]"]
  node_pipeline["Checkpoints and leases<br/>[pipeline.py]"]
  node_tracking[("Run and batch tracking<br/>[run_tracking.py]")]
end

subgraph group_quality["Quality and release"]
  node_dbt["Warehouse transformations"]
  node_rules["Business rules<br/>[business_rules.py]"]
  node_validation[("Validation results")]
  node_gate["Quality gate<br/>[quality_gate.py]"]
  node_reconcile["Source reconciliation<br/>[reconciliation.py]"]
  node_release["Release gate<br/>[release_gate.py]"]
end

subgraph group_report["Operations reporting"]
  node_api["Reporting API<br/>[main.py]"]
  node_reports["Report routes<br/>[reports.py]"]
  node_repo["Reporting repository<br/>[repository.py]"]
  node_dashboard["Operations dashboard<br/>[dashboard.py]"]
  node_operator["Run and replay controls<br/>[operator.py]"]
  node_metrics["Operational metrics<br/>[metrics.py]"]
end

subgraph group_infra["Database access"]
  node_settings["Validated settings<br/>[settings.py]"]
  node_connections["Connection pools<br/>[connections.py]"]
end

node_external_sources(("Source systems"))
node_warehouse[("Warehouse")]
node_clients(("Operators and viewers"))

node_airflow -->|"invokes"| node_tasks
node_tasks -->|"extracts"| node_crm
node_tasks -->|"ingests"| node_status
node_tasks -->|"loads"| node_csv
node_tasks -->|"loads staging"| node_staging
node_external_sources -.->|"provides rows"| node_crm
node_external_sources -.->|"provides status"| node_status
node_external_sources -.->|"provides files"| node_csv
node_tasks -.->|"manages checkpoints"| node_pipeline
node_tasks -->|"records runs"| node_tracking
node_tasks -->|"runs models"| node_dbt
node_staging -.->|"writes staging"| node_warehouse
node_dbt -.->|"transforms"| node_warehouse
node_tasks -->|"validates"| node_rules
node_rules -->|"stores failures"| node_validation
node_tasks -->|"gates quality"| node_gate
node_tasks -->|"reconciles"| node_reconcile
node_tasks -->|"checks release"| node_release
node_release -.->|"controls release"| node_warehouse
node_api -->|"routes requests"| node_reports
node_api -->|"mounts"| node_dashboard
node_api -->|"mounts"| node_operator
node_api -->|"mounts"| node_metrics
node_reports -->|"queries"| node_repo
node_repo -->|"reads reports"| node_warehouse
node_clients -.->|"requests reports"| node_api
node_clients -.->|"views dashboard"| node_dashboard
node_clients -.->|"triggers runs"| node_operator
node_settings -->|"configures"| node_connections
node_tasks -->|"uses engines"| node_connections

click node_airflow "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/airflow/dags/enterprise_etl_pipeline.py"
click node_tasks "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/orchestration/airflow_tasks.py"
click node_pipeline "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/orchestration/pipeline.py"
click node_tracking "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/control/run_tracking.py"
click node_dbt "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/tree/main/dbt"
click node_crm "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/ingestion/database_sources.py"
click node_status "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/ingestion/customer_status_api.py"
click node_csv "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/ingestion/csv_source.py"
click node_staging "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/ingestion/staging_loaders.py"
click node_rules "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/quality/business_rules.py"
click node_validation "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/quality/validation_results.py"
click node_gate "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/quality/quality_gate.py"
click node_reconcile "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/quality/reconciliation.py"
click node_release "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/quality/release_gate.py"
click node_api "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/api/main.py"
click node_reports "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/api/routes/reports.py"
click node_repo "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/reporting/repository.py"
click node_dashboard "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/api/dashboard.py"
click node_operator "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/api/operator.py"
click node_metrics "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/api/metrics.py"
click node_settings "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/config/settings.py"
click node_connections "https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/database/connections.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_crm,node_status,node_csv,node_staging,node_clients toneBlue
class node_airflow,node_tasks,node_pipeline,node_tracking,node_warehouse toneAmber
class node_dbt,node_rules,node_validation,node_gate,node_reconcile,node_release toneMint
class node_api,node_reports,node_repo,node_dashboard,node_operator,node_metrics toneRose
class node_settings,node_connections,node_external_sources toneIndigo
```

## Components

### Source ingestion

- [CRM and billing extracts](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/ingestion/database_sources.py): `src/etl_framework/ingestion/database_sources.py`
- [Customer status API](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/ingestion/customer_status_api.py): `src/etl_framework/ingestion/customer_status_api.py`
- [Operational CSV](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/ingestion/csv_source.py): `src/etl_framework/ingestion/csv_source.py`
- [Replay-safe staging](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/ingestion/staging_loaders.py): `src/etl_framework/ingestion/staging_loaders.py`

### Run orchestration

- [Airflow DAG](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/airflow/dags/enterprise_etl_pipeline.py): `airflow/dags/enterprise_etl_pipeline.py`
- [Airflow task services](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/orchestration/airflow_tasks.py): `src/etl_framework/orchestration/airflow_tasks.py`
- [Checkpoints and leases](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/orchestration/pipeline.py): `src/etl_framework/orchestration/pipeline.py`
- [Run and batch tracking](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/control/run_tracking.py): `src/etl_framework/control/run_tracking.py`

### Quality and release

- [Warehouse transformations](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/tree/main/dbt): `dbt`
- [Business rules](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/quality/business_rules.py): `src/etl_framework/quality/business_rules.py`
- [Validation results](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/quality/validation_results.py): `src/etl_framework/quality/validation_results.py`
- [Quality gate](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/quality/quality_gate.py): `src/etl_framework/quality/quality_gate.py`
- [Source reconciliation](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/quality/reconciliation.py): `src/etl_framework/quality/reconciliation.py`
- [Release gate](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/quality/release_gate.py): `src/etl_framework/quality/release_gate.py`

### Operations reporting

- [Reporting API](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/api/main.py): `api/main.py`
- [Report routes](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/api/routes/reports.py): `api/routes/reports.py`
- [Reporting repository](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/reporting/repository.py): `src/etl_framework/reporting/repository.py`
- [Operations dashboard](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/api/dashboard.py): `api/dashboard.py`
- [Run and replay controls](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/api/operator.py): `api/operator.py`
- [Operational metrics](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/api/metrics.py): `api/metrics.py`

### Database access

- [Validated settings](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/config/settings.py): `src/etl_framework/config/settings.py`
- [Connection pools](https://github.com/chandralekha77/enterprise-etl-data-quality-framework/blob/main/src/etl_framework/database/connections.py): `src/etl_framework/database/connections.py`

### Other components

- **Source systems**
- **Warehouse**
- **Operators and viewers**

## Connections

- Airflow DAG → Airflow task services: invokes (evidence: `airflow/dags/enterprise_etl_pipeline.py`)
- Airflow task services → CRM and billing extracts: extracts
- Airflow task services → Customer status API: ingests
- Airflow task services → Operational CSV: loads
- Airflow task services → Replay-safe staging: loads staging
- Source systems → CRM and billing extracts: provides rows (evidence: `README.md`)
- Source systems → Customer status API: provides status (evidence: `README.md`)
- Source systems → Operational CSV: provides files (evidence: `README.md`)
- Airflow task services → Checkpoints and leases: manages checkpoints (evidence: `src/etl_framework/orchestration/airflow_tasks.py`)
- Airflow task services → Run and batch tracking: records runs (evidence: `src/etl_framework/orchestration/airflow_tasks.py`)
- Airflow task services → Warehouse transformations: runs models (evidence: `src/etl_framework/orchestration/airflow_tasks.py`)
- Replay-safe staging → Warehouse: writes staging (evidence: `README.md`)
- Warehouse transformations → Warehouse: transforms (evidence: `README.md`)
- Airflow task services → Business rules: validates
- Business rules → Validation results: stores failures (evidence: `src/etl_framework/quality/business_rules.py`)
- Airflow task services → Quality gate: gates quality
- Airflow task services → Source reconciliation: reconciles
- Airflow task services → Release gate: checks release
- Release gate → Warehouse: controls release (evidence: `README.md`)
- Reporting API → Report routes: routes requests (evidence: `api/main.py`)
- Reporting API → Operations dashboard: mounts (evidence: `api/main.py`)
- Reporting API → Run and replay controls: mounts (evidence: `api/main.py`)
- Reporting API → Operational metrics: mounts (evidence: `api/main.py`)
- Report routes → Reporting repository: queries (evidence: `api/routes/reports.py`)
- Reporting repository → Warehouse: reads reports (evidence: `src/etl_framework/reporting/repository.py`)
- Operators and viewers → Reporting API: requests reports (evidence: `README.md`)
- Operators and viewers → Operations dashboard: views dashboard (evidence: `README.md`)
- Operators and viewers → Run and replay controls: triggers runs (evidence: `README.md`)
- Validated settings → Connection pools: configures (evidence: `src/etl_framework/database/connections.py`)
- Airflow task services → Connection pools: uses engines (evidence: `src/etl_framework/orchestration/airflow_tasks.py`)
