How trinathgundavaram/framework works
This repository provides a manually invoked AWS Glue job for loading one CSV into one existing Postgres table per run. The main path is job parameters → S3 CSV read → column/key and mode handling → Postgres insert/upsert. Credentials come from Secrets Manager; the connection helper owns that lookup and database connection. Seed files represent distinct compliance metadata domains. The README also documents Glue failure alerting and design material on orchestration, reuse, late arrival, and schema; those areas are shown conservatively because their implementation details were not sampled.
Main entry points: seed_data Metadata seed CSVs · glue_job_metadata_load.py Glue Python Shell · rds_conn.py RdsClient · glue-job.tf Failure alerting
Components 22
Metadata loading
- Glue Python Shell
aws/glue/code/glue_job_metadata_load.py - Run parameters
aws/glue/code/glue_job_metadata_load.py - Input CSV in S3
- CSV reader
aws/glue/code/glue_job_metadata_load.py - Load and conflict handling
aws/glue/code/glue_job_metadata_load.py - RdsClient
aws/glue/code/rds_conn.py - Postgres credentials
- AWS Secrets Manager
Compliance metadata
- Existing Postgres tables
- Metadata seed CSVs
aws/glue/code/seed_data - Source systems
aws/glue/code/seed_data/compliance_source_system.csv - Dataset source crosswalk
aws/glue/code/seed_data/compliance_dataset_source_xwalk.csv - Request intake
aws/glue/code/seed_data/compliance_request_intake.csv - Run types
aws/glue/code/seed_data/compliance_run_type.csv - Compliance exceptions audit
aws/glue/code/seed_data/cms_compliance_exceptions_audit.csv
Operations and design
- Failure alerting
aws/glue/glue-job.tf - SNS and CloudWatch
- Orchestration design
docs/design/orchestration-flow.md - Reuse and carry-forward
docs/design/reuse-and-late-arrival.md - Schema design
docs/design/schema-design.md - Framework design
docs/design/framework-master.md
Other components
- Job operator
Connections 18
- Job operator → Glue Python Shell: starts run
- Job operator → Run parameters: supplies
- Run parameters → Glue Python Shell: configure run
- Glue Python Shell → CSV reader: calls reader
- CSV reader → Input CSV in S3: reads object
- CSV reader → Load and conflict handling: returns rows
- Glue Python Shell → RdsClient: connects through
- RdsClient → AWS Secrets Manager: requests credentials
- AWS Secrets Manager → Postgres credentials: returns secret
- RdsClient → Existing Postgres tables: opens connection
- Load and conflict handling → Existing Postgres tables: inserts or upserts
- Metadata seed CSVs → Source systems: includes
- Metadata seed CSVs → Dataset source crosswalk: includes
- Metadata seed CSVs → Request intake: includes
- Metadata seed CSVs → Run types: includes
- Metadata seed CSVs → Compliance exceptions audit: includes
- Failure alerting → SNS and CloudWatch: configures alerts
- Glue Python Shell → Failure alerting: failure events