# chandrakondur/arch-diagrams architecture

> Architecture diagram of the GitHub repository chandrakondur/arch-diagrams, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chandrakondur/arch-diagrams
- Repository: https://github.com/chandrakondur/arch-diagrams

## Overview

This repository implements Snowflake-oriented metadata pipelines. Codebook deliveries are scanned, checked, loaded, and dispositioned; separate pipelines validate data conformance, load enrichment data, and export registry metadata as DCAT JSON. Runner-to-processor links are supported by the source index and the ingestion runner excerpt; the README documents the product areas. Some processor internals and storage interactions were not sampled, so those relationships are omitted rather than inferred.

Main entry points: [`src/pipelines/ingestion/ingestion_runner.py`](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/ingestion_runner.py) (Ingestion runner), [`src/pipelines/dcat/dcat_runner.py`](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/dcat_runner.py) (DCAT runner), [`src/pipelines/data_quality/data_quality_runner.py`](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/data_quality/data_quality_runner.py) (Quality runner), [`src/pipelines/utils/proc_log.py`](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/proc_log.py) (Process logging)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Codebook ingestion"]
  node_ingest_runner["Ingestion runner"]
  node_ingest_config["Ingestion config<br/>[config.py]"]
  node_preflight["Delivery preflight"]
  node_contract["Contract validation"]
  node_loader["Codebook loader<br/>[load_processor.py]"]
  node_disposition["File disposition"]
end

subgraph group_quality["Data quality"]
  node_quality_runner["Quality runner"]
  node_quality_config["Quality config<br/>[config.py]"]
  node_conformance["Conformance checks"]
  node_work_claim["Work claiming"]
  node_work_queue["Work queue"]
end

subgraph group_enrichment["Enrichment"]
  node_enrichment_runner["Enrichment runner"]
  node_enrichment_load["Enrichment loader"]
  node_enrichment_config["Enrichment config<br/>[config.py]"]
end

subgraph group_dcat["DCAT export"]
  node_dcat_runner["DCAT runner<br/>[dcat_runner.py]"]
  node_dcat_config["DCAT config<br/>[config.py]"]
  node_series_stage["Series staging"]
  node_dataset_stage["Dataset staging"]
  node_dcat_json["DCAT generation"]
  node_dictionary_json["Dictionary export"]
  node_json_output["JSON output<br/>[json_output.py]"]
end

subgraph group_shared["Pipeline foundation"]
  node_etl["Snowpark ETL session<br/>[etl.py]"]
  node_deployment["Deployment namespace<br/>[deployment.py]"]
  node_databases["Database constants<br/>[databases.py]"]
  node_settings["Shared configuration<br/>[config.py]"]
  node_logger["Logging<br/>[logger.py]"]
  node_proc_log["Process logging<br/>[proc_log.py]"]
  node_decorators["Pipeline decorators<br/>[decorators.py]"]
  node_normalize["Name normalization<br/>[normalize.py]"]
end

node_source[("Codebook stage")]
node_snowflake[("Snowflake")]
node_export_stage[("Export stage")]

node_source -.->|"supplies deliveries"| node_ingest_runner
node_ingest_runner -->|"runs preflight"| node_preflight
node_ingest_runner -->|"runs contract checks"| node_contract
node_ingest_runner -->|"runs loader"| node_loader
node_ingest_runner -->|"runs disposition"| node_disposition
node_ingest_runner -->|"uses session"| node_etl
node_quality_runner -->|"runs checks"| node_conformance
node_quality_runner -->|"claims work"| node_work_claim
node_quality_runner -->|"reads queue"| node_work_queue
node_quality_runner -->|"uses session"| node_etl
node_conformance -->|"normalizes names"| node_normalize
node_enrichment_runner -->|"runs loader"| node_enrichment_load
node_enrichment_runner -->|"uses session"| node_etl
node_dcat_runner -->|"stages series"| node_series_stage
node_dcat_runner -->|"stages datasets"| node_dataset_stage
node_dcat_runner -->|"generates catalogue"| node_dcat_json
node_dcat_runner -->|"generates dictionary"| node_dictionary_json
node_dcat_json -->|"writes JSON"| node_json_output
node_dictionary_json -->|"writes JSON"| node_json_output
node_dcat_runner -.->|"exports artifacts"| node_export_stage
node_databases -->|"gets namespace"| node_deployment
node_quality_config -->|"uses names"| node_databases
node_ingest_config -->|"uses names"| node_databases
node_dcat_config -->|"uses names"| node_databases
node_enrichment_config -->|"uses names"| node_databases
node_etl -->|"opens session"| node_snowflake
node_proc_log -->|"gets log table"| node_settings
node_proc_log -->|"echoes messages"| node_logger
node_quality_runner -->|"uses process log"| node_proc_log
node_ingest_runner -->|"uses process log"| node_proc_log
node_dcat_runner -->|"uses process log"| node_proc_log
node_enrichment_runner -->|"uses decorators"| node_decorators

click node_ingest_runner "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/ingestion_runner.py"
click node_ingest_config "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/config.py"
click node_preflight "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/processors/preflight_processor.py"
click node_contract "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/processors/contract_processor.py"
click node_loader "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/processors/load_processor.py"
click node_disposition "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/processors/disposition_processor.py"
click node_quality_runner "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/data_quality/data_quality_runner.py"
click node_quality_config "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/data_quality/config.py"
click node_conformance "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/data_quality/processors/conformance_processor.py"
click node_work_claim "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/data_quality/processors/work_claim_processor.py"
click node_work_queue "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/data_quality/processors/work_queue_processor.py"
click node_enrichment_runner "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/enrichment/enrichment_runner.py"
click node_enrichment_load "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/enrichment/processors/enrichment_load_processor.py"
click node_enrichment_config "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/enrichment/config.py"
click node_dcat_runner "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/dcat_runner.py"
click node_dcat_config "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/config.py"
click node_series_stage "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/processors/series_stg_processor.py"
click node_dataset_stage "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/processors/dataset_stg_processor.py"
click node_dcat_json "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/processors/dcat_json_processor.py"
click node_dictionary_json "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/processors/dictionary_json_processor.py"
click node_json_output "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/json_output.py"
click node_etl "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/etl.py"
click node_deployment "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/deployment.py"
click node_databases "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/databases.py"
click node_settings "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/config.py"
click node_logger "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/logger.py"
click node_proc_log "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/proc_log.py"
click node_decorators "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/decorators.py"
click node_normalize "https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/normalize.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest_runner,node_ingest_config,node_preflight,node_contract,node_loader,node_disposition toneBlue
class node_quality_runner,node_quality_config,node_conformance,node_work_claim,node_work_queue,node_source,node_snowflake,node_export_stage toneAmber
class node_enrichment_runner,node_enrichment_load,node_enrichment_config toneMint
class node_dcat_runner,node_dcat_config,node_series_stage,node_dataset_stage,node_dcat_json,node_dictionary_json,node_json_output toneRose
class node_etl,node_deployment,node_databases,node_settings,node_logger,node_proc_log,node_decorators,node_normalize toneIndigo
```

## Components

### Codebook ingestion

- [Ingestion runner](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/ingestion_runner.py): `src/pipelines/ingestion/ingestion_runner.py`
- [Ingestion config](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/config.py): `src/pipelines/ingestion/config.py`
- [Delivery preflight](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/processors/preflight_processor.py): `src/pipelines/ingestion/processors/preflight_processor.py`
- [Contract validation](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/processors/contract_processor.py): `src/pipelines/ingestion/processors/contract_processor.py`
- [Codebook loader](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/processors/load_processor.py): `src/pipelines/ingestion/processors/load_processor.py`
- [File disposition](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/ingestion/processors/disposition_processor.py): `src/pipelines/ingestion/processors/disposition_processor.py`

### Data quality

- [Quality runner](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/data_quality/data_quality_runner.py): `src/pipelines/data_quality/data_quality_runner.py`
- [Quality config](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/data_quality/config.py): `src/pipelines/data_quality/config.py`
- [Conformance checks](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/data_quality/processors/conformance_processor.py): `src/pipelines/data_quality/processors/conformance_processor.py`
- [Work claiming](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/data_quality/processors/work_claim_processor.py): `src/pipelines/data_quality/processors/work_claim_processor.py`
- [Work queue](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/data_quality/processors/work_queue_processor.py): `src/pipelines/data_quality/processors/work_queue_processor.py`

### Enrichment

- [Enrichment runner](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/enrichment/enrichment_runner.py): `src/pipelines/enrichment/enrichment_runner.py`
- [Enrichment loader](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/enrichment/processors/enrichment_load_processor.py): `src/pipelines/enrichment/processors/enrichment_load_processor.py`
- [Enrichment config](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/enrichment/config.py): `src/pipelines/enrichment/config.py`

### DCAT export

- [DCAT runner](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/dcat_runner.py): `src/pipelines/dcat/dcat_runner.py`
- [DCAT config](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/config.py): `src/pipelines/dcat/config.py`
- [Series staging](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/processors/series_stg_processor.py): `src/pipelines/dcat/processors/series_stg_processor.py`
- [Dataset staging](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/processors/dataset_stg_processor.py): `src/pipelines/dcat/processors/dataset_stg_processor.py`
- [DCAT generation](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/processors/dcat_json_processor.py): `src/pipelines/dcat/processors/dcat_json_processor.py`
- [Dictionary export](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/processors/dictionary_json_processor.py): `src/pipelines/dcat/processors/dictionary_json_processor.py`
- [JSON output](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/dcat/json_output.py): `src/pipelines/dcat/json_output.py`

### Pipeline foundation

- [Snowpark ETL session](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/etl.py): `src/pipelines/utils/etl.py`
- [Deployment namespace](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/deployment.py): `src/pipelines/utils/deployment.py`
- [Database constants](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/databases.py): `src/pipelines/utils/databases.py`
- [Shared configuration](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/config.py): `src/pipelines/utils/config.py`
- [Logging](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/logger.py): `src/pipelines/utils/logger.py`
- [Process logging](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/proc_log.py): `src/pipelines/utils/proc_log.py`
- [Pipeline decorators](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/decorators.py): `src/pipelines/utils/decorators.py`
- [Name normalization](https://github.com/chandrakondur/arch-diagrams/blob/main/src/pipelines/utils/normalize.py): `src/pipelines/utils/normalize.py`

### Other components

- **Codebook stage**
- **Snowflake**
- **Export stage**

## Connections

- Codebook stage → Ingestion runner: supplies deliveries
- Ingestion runner → Delivery preflight: runs preflight (evidence: `src/pipelines/ingestion/ingestion_runner.py`)
- Ingestion runner → Contract validation: runs contract checks (evidence: `src/pipelines/ingestion/ingestion_runner.py`)
- Ingestion runner → Codebook loader: runs loader (evidence: `src/pipelines/ingestion/ingestion_runner.py`)
- Ingestion runner → File disposition: runs disposition (evidence: `src/pipelines/ingestion/ingestion_runner.py`)
- Ingestion runner → Snowpark ETL session: uses session (evidence: `src/pipelines/ingestion/ingestion_runner.py`)
- Quality runner → Conformance checks: runs checks (evidence: `src/pipelines/data_quality/data_quality_runner.py`)
- Quality runner → Work claiming: claims work (evidence: `src/pipelines/data_quality/data_quality_runner.py`)
- Quality runner → Work queue: reads queue (evidence: `src/pipelines/data_quality/data_quality_runner.py`)
- Quality runner → Snowpark ETL session: uses session (evidence: `src/pipelines/data_quality/data_quality_runner.py`)
- Conformance checks → Name normalization: normalizes names (evidence: `src/pipelines/data_quality/processors/conformance_processor.py`)
- Enrichment runner → Enrichment loader: runs loader (evidence: `src/pipelines/enrichment/enrichment_runner.py`)
- Enrichment runner → Snowpark ETL session: uses session (evidence: `src/pipelines/enrichment/enrichment_runner.py`)
- DCAT runner → Series staging: stages series (evidence: `src/pipelines/dcat/dcat_runner.py`)
- DCAT runner → Dataset staging: stages datasets (evidence: `src/pipelines/dcat/dcat_runner.py`)
- DCAT runner → DCAT generation: generates catalogue (evidence: `src/pipelines/dcat/dcat_runner.py`)
- DCAT runner → Dictionary export: generates dictionary (evidence: `src/pipelines/dcat/dcat_runner.py`)
- DCAT generation → JSON output: writes JSON (evidence: `src/pipelines/dcat/processors/dcat_json_processor.py`)
- Dictionary export → JSON output: writes JSON (evidence: `src/pipelines/dcat/processors/dictionary_json_processor.py`)
- DCAT runner → Export stage: exports artifacts
- Database constants → Deployment namespace: gets namespace (evidence: `src/pipelines/utils/databases.py`)
- Quality config → Database constants: uses names (evidence: `src/pipelines/data_quality/config.py`)
- Ingestion config → Database constants: uses names (evidence: `src/pipelines/ingestion/config.py`)
- DCAT config → Database constants: uses names (evidence: `src/pipelines/dcat/config.py`)
- Enrichment config → Database constants: uses names (evidence: `src/pipelines/enrichment/config.py`)
- Snowpark ETL session → Snowflake: opens session (evidence: `src/pipelines/utils/etl.py`)
- Process logging → Shared configuration: gets log table (evidence: `src/pipelines/utils/proc_log.py`)
- Process logging → Logging: echoes messages (evidence: `src/pipelines/utils/proc_log.py`)
- Quality runner → Process logging: uses process log (evidence: `src/pipelines/data_quality/data_quality_runner.py`)
- Ingestion runner → Process logging: uses process log (evidence: `src/pipelines/ingestion/ingestion_runner.py`)
- DCAT runner → Process logging: uses process log (evidence: `src/pipelines/dcat/dcat_runner.py`)
- Enrichment runner → Pipeline decorators: uses decorators (evidence: `src/pipelines/enrichment/enrichment_runner.py`)
