How chandrakondur/arch-diagrams works
This repository implements Snowflake-oriented metadata pipelines. Codebook deliveries are scanned, checked, loaded, and dispositioned; separate pipelines validate data conformance, load enrichment data, and export registry metadata as DCAT JSON. Runner-to-processor links are supported by the source index and the ingestion runner excerpt; the README documents the product areas. Some processor internals and storage interactions were not sampled, so those relationships are omitted rather than inferred.
Main entry points: ingestion_runner.py Ingestion runner · dcat_runner.py DCAT runner · data_quality_runner.py Quality runner · proc_log.py Process logging
Components 32
Codebook ingestion
- Ingestion runner
src/pipelines/ingestion/ingestion_runner.py - Ingestion config
src/pipelines/ingestion/config.py - Delivery preflight
src/pipelines/ingestion/processors/preflight_processor.py - Contract validation
src/pipelines/ingestion/processors/contract_processor.py - Codebook loader
src/pipelines/ingestion/processors/load_processor.py - File disposition
src/pipelines/ingestion/processors/disposition_processor.py
Data quality
- Quality runner
src/pipelines/data_quality/data_quality_runner.py - Quality config
src/pipelines/data_quality/config.py - Conformance checks
src/pipelines/data_quality/processors/conformance_processor.py - Work claiming
src/pipelines/data_quality/processors/work_claim_processor.py - Work queue
src/pipelines/data_quality/processors/work_queue_processor.py
Enrichment
- Enrichment runner
src/pipelines/enrichment/enrichment_runner.py - Enrichment loader
src/pipelines/enrichment/processors/enrichment_load_processor.py - Enrichment config
src/pipelines/enrichment/config.py
DCAT export
- DCAT runner
src/pipelines/dcat/dcat_runner.py - DCAT config
src/pipelines/dcat/config.py - Series staging
src/pipelines/dcat/processors/series_stg_processor.py - Dataset staging
src/pipelines/dcat/processors/dataset_stg_processor.py - DCAT generation
src/pipelines/dcat/processors/dcat_json_processor.py - Dictionary export
src/pipelines/dcat/processors/dictionary_json_processor.py - JSON output
src/pipelines/dcat/json_output.py
Pipeline foundation
- Snowpark ETL session
src/pipelines/utils/etl.py - Deployment namespace
src/pipelines/utils/deployment.py - Database constants
src/pipelines/utils/databases.py - Shared configuration
src/pipelines/utils/config.py - Logging
src/pipelines/utils/logger.py - Process logging
src/pipelines/utils/proc_log.py - Pipeline decorators
src/pipelines/utils/decorators.py - Name normalization
src/pipelines/utils/normalize.py
Other components
- Codebook stage
- Snowflake
- Export stage
Connections 32
- Codebook stage → Ingestion runner: supplies deliveries
- Ingestion runner → Delivery preflight: runs preflight
src/pipelines/ingestion/ingestion_runner.py - Ingestion runner → Contract validation: runs contract checks
src/pipelines/ingestion/ingestion_runner.py - Ingestion runner → Codebook loader: runs loader
src/pipelines/ingestion/ingestion_runner.py - Ingestion runner → File disposition: runs disposition
src/pipelines/ingestion/ingestion_runner.py - Ingestion runner → Snowpark ETL session: uses session
src/pipelines/ingestion/ingestion_runner.py - Quality runner → Conformance checks: runs checks
src/pipelines/data_quality/data_quality_runner.py - Quality runner → Work claiming: claims work
src/pipelines/data_quality/data_quality_runner.py - Quality runner → Work queue: reads queue
src/pipelines/data_quality/data_quality_runner.py - Quality runner → Snowpark ETL session: uses session
src/pipelines/data_quality/data_quality_runner.py - Conformance checks → Name normalization: normalizes names
src/pipelines/data_quality/processors/conformance_processor.py - Enrichment runner → Enrichment loader: runs loader
src/pipelines/enrichment/enrichment_runner.py - Enrichment runner → Snowpark ETL session: uses session
src/pipelines/enrichment/enrichment_runner.py - DCAT runner → Series staging: stages series
src/pipelines/dcat/dcat_runner.py - DCAT runner → Dataset staging: stages datasets
src/pipelines/dcat/dcat_runner.py - DCAT runner → DCAT generation: generates catalogue
src/pipelines/dcat/dcat_runner.py - DCAT runner → Dictionary export: generates dictionary
src/pipelines/dcat/dcat_runner.py - DCAT generation → JSON output: writes JSON
src/pipelines/dcat/processors/dcat_json_processor.py - Dictionary export → JSON output: writes JSON
src/pipelines/dcat/processors/dictionary_json_processor.py - DCAT runner → Export stage: exports artifacts
- Database constants → Deployment namespace: gets namespace
src/pipelines/utils/databases.py - Quality config → Database constants: uses names
src/pipelines/data_quality/config.py - Ingestion config → Database constants: uses names
src/pipelines/ingestion/config.py - DCAT config → Database constants: uses names
src/pipelines/dcat/config.py - Enrichment config → Database constants: uses names
src/pipelines/enrichment/config.py - Snowpark ETL session → Snowflake: opens session
src/pipelines/utils/etl.py - Process logging → Shared configuration: gets log table
src/pipelines/utils/proc_log.py - Process logging → Logging: echoes messages
src/pipelines/utils/proc_log.py - Quality runner → Process logging: uses process log
src/pipelines/data_quality/data_quality_runner.py - Ingestion runner → Process logging: uses process log
src/pipelines/ingestion/ingestion_runner.py - DCAT runner → Process logging: uses process log
src/pipelines/dcat/dcat_runner.py - Enrichment runner → Pipeline decorators: uses decorators
src/pipelines/enrichment/enrichment_runner.py