How trinathgundavaram/dq_complete_repo works
GRE + Sampling provides two independent, configuration-driven data-quality workflows: rule evaluation records violations and verdicts, while stratified sampling selects cases for review. Both use shared source-connection infrastructure but keep their own configuration, run tracking, errors, and result tables. The repository also includes an Airflow bridge for rules and a metadata-table mirror utility. The main workflow links are supported by the README and sampled excerpts; some subsystem internals were not sampled.
Main entry points: runner.py Rule runner · sampling.py Sampling workflow · executor.py Rule executor · run_by_process.py Process CLI
Components 16
Rules engine
- Rule runner
rules_engine/runner.py - Rule executor
rules_engine/executor.py - Rule configuration
rules_engine/config.py - Rule metadata and results
rules_engine/schema.sql
Sampling engine
- Sampling workflow
sampling/sampling.py - Sampling configuration
sampling/config.py - Sampling metadata and selections
sampling/schema.sql
Source connections
- Connection factory
db/connection_factory.py - Source databases and files
Scheduling and sync
- Process CLI
run_by_process.py - Airflow DAG
airflow/gre_rules_dag.py - Airflow task bridge
airflow/rules_engine/gre_rules_task.py - Metadata sync
metadata_sync/sync_from_teradata.py - Postgres mirror
metadata_sync/create_postgres_tables.py - Teradata metadata
Other components
- Engine caller
Connections 17
- Engine caller → Rule runner: runs group
- Engine caller → Sampling workflow: runs config
- Process CLI → Rule runner: dispatches rules
- Process CLI → Sampling workflow: dispatches sampling
- Rule runner → Rule configuration: loads settings
- Rule runner → Rule metadata and results: reads rules; tracks run
- Rule runner → Rule executor: executes rules
- Rule executor → Source databases and files: queries source
- Rule executor → Rule metadata and results: writes exceptions and verdicts
- Sampling workflow → Sampling configuration: loads settings
- Sampling workflow → Source databases and files: queries candidates
- Sampling workflow → Sampling metadata and selections: writes run and selections
- Connection factory → Source databases and files: opens adapters
- Airflow DAG → Airflow task bridge: calls task
- Airflow task bridge → Rule runner: invokes rules
- Metadata sync → Teradata metadata: reads tables
- Metadata sync → Postgres mirror: mirrors tables