# trinathgundavaram/dq_complete_repo architecture

> Architecture diagram of the GitHub repository trinathgundavaram/dq_complete_repo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/trinathgundavaram/dq_complete_repo
- Repository: https://github.com/trinathgundavaram/dq_complete_repo

## Overview

GRE + Sampling provides two independent, configuration-driven data-quality workflows: rule evaluation records violations and verdicts, while stratified sampling selects cases for review. Both use shared source-connection infrastructure but keep their own configuration, run tracking, errors, and result tables. The repository also includes an Airflow bridge for rules and a metadata-table mirror utility. The main workflow links are supported by the README and sampled excerpts; some subsystem internals were not sampled.

Main entry points: [`rules_engine/runner.py`](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/rules_engine/runner.py) (Rule runner), [`sampling/sampling.py`](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/sampling/sampling.py) (Sampling workflow), [`rules_engine/executor.py`](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/rules_engine/executor.py) (Rule executor), [`run_by_process.py`](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/run_by_process.py) (Process CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_rules["Rules engine"]
  node_runner["Rule runner<br/>[runner.py]"]
  node_executor["Rule executor<br/>[executor.py]"]
  node_rule_config["Rule configuration<br/>[config.py]"]
  node_rule_db[("Rule metadata and results<br/>[schema.sql]")]
end

subgraph group_sampling["Sampling engine"]
  node_sampling_engine["Sampling workflow<br/>[sampling.py]"]
  node_sampling_config["Sampling configuration<br/>[config.py]"]
  node_sampling_db[("Sampling metadata and selections<br/>[schema.sql]")]
end

subgraph group_connections["Source connections"]
  node_factory["Connection factory"]
  node_sources[("Source databases and files")]
end

subgraph group_ops["Scheduling and sync"]
  node_cli["Process CLI<br/>[run_by_process.py]"]
  node_airflow_dag["Airflow DAG<br/>[gre_rules_dag.py]"]
  node_airflow_bridge["Airflow task bridge<br/>[gre_rules_task.py]"]
  node_sync["Metadata sync"]
  node_postgres[("Postgres mirror")]
  node_teradata[("Teradata metadata")]
end

node_caller(("Engine caller"))

node_caller -->|"runs group"| node_runner
node_caller -->|"runs config"| node_sampling_engine
node_cli -->|"dispatches rules"| node_runner
node_cli -->|"dispatches sampling"| node_sampling_engine
node_runner -->|"loads settings"| node_rule_config
node_runner -->|"reads rules; tracks run"| node_rule_db
node_runner -->|"executes rules"| node_executor
node_executor -->|"queries source"| node_sources
node_executor -->|"writes exceptions and verdicts"| node_rule_db
node_sampling_engine -->|"loads settings"| node_sampling_config
node_sampling_engine -->|"queries candidates"| node_sources
node_sampling_engine -->|"writes run and selections"| node_sampling_db
node_factory -->|"opens adapters"| node_sources
node_airflow_dag -->|"calls task"| node_airflow_bridge
node_airflow_bridge -->|"invokes rules"| node_runner
node_sync -->|"reads tables"| node_teradata
node_sync -->|"mirrors tables"| node_postgres

click node_cli "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/run_by_process.py"
click node_runner "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/rules_engine/runner.py"
click node_executor "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/rules_engine/executor.py"
click node_rule_config "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/rules_engine/config.py"
click node_rule_db "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/rules_engine/schema.sql"
click node_sampling_engine "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/sampling/sampling.py"
click node_sampling_config "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/sampling/config.py"
click node_sampling_db "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/sampling/schema.sql"
click node_factory "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/db/connection_factory.py"
click node_airflow_dag "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/airflow/gre_rules_dag.py"
click node_airflow_bridge "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/airflow/rules_engine/gre_rules_task.py"
click node_sync "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/metadata_sync/sync_from_teradata.py"
click node_postgres "https://github.com/trinathgundavaram/dq_complete_repo/blob/main/metadata_sync/create_postgres_tables.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_executor,node_rule_config,node_rule_db toneBlue
class node_sampling_engine,node_sampling_config,node_sampling_db toneAmber
class node_factory,node_sources toneMint
class node_cli,node_airflow_dag,node_airflow_bridge,node_sync,node_postgres,node_teradata toneRose
class node_caller toneIndigo
```

## Components

### Rules engine

- [Rule runner](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/rules_engine/runner.py): `rules_engine/runner.py`
- [Rule executor](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/rules_engine/executor.py): `rules_engine/executor.py`
- [Rule configuration](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/rules_engine/config.py): `rules_engine/config.py`
- [Rule metadata and results](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/rules_engine/schema.sql): `rules_engine/schema.sql`

### Sampling engine

- [Sampling workflow](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/sampling/sampling.py): `sampling/sampling.py`
- [Sampling configuration](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/sampling/config.py): `sampling/config.py`
- [Sampling metadata and selections](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/sampling/schema.sql): `sampling/schema.sql`

### Source connections

- [Connection factory](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/db/connection_factory.py): `db/connection_factory.py`
- **Source databases and files**

### Scheduling and sync

- [Process CLI](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/run_by_process.py): `run_by_process.py`
- [Airflow DAG](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/airflow/gre_rules_dag.py): `airflow/gre_rules_dag.py`
- [Airflow task bridge](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/airflow/rules_engine/gre_rules_task.py): `airflow/rules_engine/gre_rules_task.py`
- [Metadata sync](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/metadata_sync/sync_from_teradata.py): `metadata_sync/sync_from_teradata.py`
- [Postgres mirror](https://github.com/trinathgundavaram/dq_complete_repo/blob/main/metadata_sync/create_postgres_tables.py): `metadata_sync/create_postgres_tables.py`
- **Teradata metadata**

### Other components

- **Engine caller**

## Connections

- Engine caller → Rule runner: runs group
- Engine caller → Sampling workflow: runs config
- Process CLI → Rule runner: dispatches rules
- Process CLI → Sampling workflow: dispatches sampling
- Rule runner → Rule configuration: loads settings
- Rule runner → Rule metadata and results: reads rules; tracks run
- Rule runner → Rule executor: executes rules
- Rule executor → Source databases and files: queries source
- Rule executor → Rule metadata and results: writes exceptions and verdicts
- Sampling workflow → Sampling configuration: loads settings
- Sampling workflow → Source databases and files: queries candidates
- Sampling workflow → Sampling metadata and selections: writes run and selections
- Connection factory → Source databases and files: opens adapters
- Airflow DAG → Airflow task bridge: calls task
- Airflow task bridge → Rule runner: invokes rules
- Metadata sync → Teradata metadata: reads tables
- Metadata sync → Postgres mirror: mirrors tables