# trinathgundavaram/framework architecture

> Architecture diagram of the GitHub repository trinathgundavaram/framework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/trinathgundavaram/framework
- Repository: https://github.com/trinathgundavaram/framework
- Video tour (about a minute): https://gitdiagram.com/trinathgundavaram/framework/video

## Overview

This repository provides a manually invoked AWS Glue job for loading one CSV into one existing Postgres table per run. The main path is job parameters → S3 CSV read → column/key and mode handling → Postgres insert/upsert. Credentials come from Secrets Manager; the connection helper owns that lookup and database connection. Seed files represent distinct compliance metadata domains. The README also documents Glue failure alerting and design material on orchestration, reuse, late arrival, and schema; those areas are shown conservatively because their implementation details were not sampled.

Main entry points: [`aws/glue/code/seed_data`](https://github.com/trinathgundavaram/framework/tree/main/aws/glue/code/seed_data) (Metadata seed CSVs), [`aws/glue/code/glue_job_metadata_load.py`](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/glue_job_metadata_load.py) (Glue Python Shell), [`aws/glue/code/rds_conn.py`](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/rds_conn.py) (RdsClient), [`aws/glue/glue-job.tf`](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/glue-job.tf) (Failure alerting)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Metadata loading"]
  node_glue["Glue Python Shell"]
  node_parameters["Run parameters"]
  node_s3[("Input CSV in S3")]
  node_csv["CSV reader"]
  node_load["Load and conflict handling"]
  node_connector["RdsClient<br/>[rds_conn.py]"]
  node_secret[("Postgres credentials")]
  node_secrets["AWS Secrets Manager"]
end

subgraph group_data["Compliance metadata"]
  node_postgres[("Existing Postgres tables")]
  node_seed["Metadata seed CSVs"]
  node_source["Source systems"]
  node_xref["Dataset source crosswalk"]
  node_intake["Request intake"]
  node_runtypes["Run types"]
  node_exceptions["Compliance exceptions audit"]
end

subgraph group_operations["Operations and design"]
  node_alerts["Failure alerting<br/>[glue-job.tf]"]
  node_alerting["SNS and CloudWatch"]
  node_orchestration["Orchestration design"]
  node_reuse["Reuse and carry-forward"]
  node_schema["Schema design<br/>[schema-design.md]"]
  node_framework["Framework design"]
end

node_operator(("Job operator"))

node_operator -->|"starts run"| node_glue
node_operator -->|"supplies"| node_parameters
node_parameters -->|"configure run"| node_glue
node_glue -->|"calls reader"| node_csv
node_csv -->|"reads object"| node_s3
node_csv -->|"returns rows"| node_load
node_glue -->|"connects through"| node_connector
node_connector -->|"requests credentials"| node_secrets
node_secrets -->|"returns secret"| node_secret
node_connector -->|"opens connection"| node_postgres
node_load -->|"inserts or upserts"| node_postgres
node_seed -->|"includes"| node_source
node_seed -->|"includes"| node_xref
node_seed -->|"includes"| node_intake
node_seed -->|"includes"| node_runtypes
node_seed -->|"includes"| node_exceptions
node_alerts -->|"configures alerts"| node_alerting
node_glue -.->|"failure events"| node_alerts

click node_glue "https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/glue_job_metadata_load.py"
click node_parameters "https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/glue_job_metadata_load.py"
click node_csv "https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/glue_job_metadata_load.py"
click node_load "https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/glue_job_metadata_load.py"
click node_connector "https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/rds_conn.py"
click node_seed "https://github.com/trinathgundavaram/framework/tree/main/aws/glue/code/seed_data"
click node_source "https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/seed_data/compliance_source_system.csv"
click node_xref "https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/seed_data/compliance_dataset_source_xwalk.csv"
click node_intake "https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/seed_data/compliance_request_intake.csv"
click node_runtypes "https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/seed_data/compliance_run_type.csv"
click node_exceptions "https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/seed_data/cms_compliance_exceptions_audit.csv"
click node_alerts "https://github.com/trinathgundavaram/framework/blob/main/aws/glue/glue-job.tf"
click node_orchestration "https://github.com/trinathgundavaram/framework/blob/main/docs/design/orchestration-flow.md"
click node_reuse "https://github.com/trinathgundavaram/framework/blob/main/docs/design/reuse-and-late-arrival.md"
click node_schema "https://github.com/trinathgundavaram/framework/blob/main/docs/design/schema-design.md"
click node_framework "https://github.com/trinathgundavaram/framework/blob/main/docs/design/framework-master.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_glue,node_parameters,node_s3,node_csv,node_load,node_connector,node_secret,node_secrets toneBlue
class node_postgres,node_seed,node_source,node_xref,node_intake,node_runtypes,node_exceptions toneAmber
class node_alerts,node_alerting,node_orchestration,node_reuse,node_schema,node_framework toneMint
class node_operator toneIndigo
```

## Components

### Metadata loading

- [Glue Python Shell](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/glue_job_metadata_load.py): `aws/glue/code/glue_job_metadata_load.py`
- [Run parameters](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/glue_job_metadata_load.py): `aws/glue/code/glue_job_metadata_load.py`
- **Input CSV in S3**
- [CSV reader](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/glue_job_metadata_load.py): `aws/glue/code/glue_job_metadata_load.py`
- [Load and conflict handling](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/glue_job_metadata_load.py): `aws/glue/code/glue_job_metadata_load.py`
- [RdsClient](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/rds_conn.py): `aws/glue/code/rds_conn.py`
- **Postgres credentials**
- **AWS Secrets Manager**

### Compliance metadata

- **Existing Postgres tables**
- [Metadata seed CSVs](https://github.com/trinathgundavaram/framework/tree/main/aws/glue/code/seed_data): `aws/glue/code/seed_data`
- [Source systems](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/seed_data/compliance_source_system.csv): `aws/glue/code/seed_data/compliance_source_system.csv`
- [Dataset source crosswalk](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/seed_data/compliance_dataset_source_xwalk.csv): `aws/glue/code/seed_data/compliance_dataset_source_xwalk.csv`
- [Request intake](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/seed_data/compliance_request_intake.csv): `aws/glue/code/seed_data/compliance_request_intake.csv`
- [Run types](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/seed_data/compliance_run_type.csv): `aws/glue/code/seed_data/compliance_run_type.csv`
- [Compliance exceptions audit](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/code/seed_data/cms_compliance_exceptions_audit.csv): `aws/glue/code/seed_data/cms_compliance_exceptions_audit.csv`

### Operations and design

- [Failure alerting](https://github.com/trinathgundavaram/framework/blob/main/aws/glue/glue-job.tf): `aws/glue/glue-job.tf`
- **SNS and CloudWatch**
- [Orchestration design](https://github.com/trinathgundavaram/framework/blob/main/docs/design/orchestration-flow.md): `docs/design/orchestration-flow.md`
- [Reuse and carry-forward](https://github.com/trinathgundavaram/framework/blob/main/docs/design/reuse-and-late-arrival.md): `docs/design/reuse-and-late-arrival.md`
- [Schema design](https://github.com/trinathgundavaram/framework/blob/main/docs/design/schema-design.md): `docs/design/schema-design.md`
- [Framework design](https://github.com/trinathgundavaram/framework/blob/main/docs/design/framework-master.md): `docs/design/framework-master.md`

### Other components

- **Job operator**

## Connections

- Job operator → Glue Python Shell: starts run
- Job operator → Run parameters: supplies
- Run parameters → Glue Python Shell: configure run
- Glue Python Shell → CSV reader: calls reader
- CSV reader → Input CSV in S3: reads object
- CSV reader → Load and conflict handling: returns rows
- Glue Python Shell → RdsClient: connects through
- RdsClient → AWS Secrets Manager: requests credentials
- AWS Secrets Manager → Postgres credentials: returns secret
- RdsClient → Existing Postgres tables: opens connection
- Load and conflict handling → Existing Postgres tables: inserts or upserts
- Metadata seed CSVs → Source systems: includes
- Metadata seed CSVs → Dataset source crosswalk: includes
- Metadata seed CSVs → Request intake: includes
- Metadata seed CSVs → Run types: includes
- Metadata seed CSVs → Compliance exceptions audit: includes
- Failure alerting → SNS and CloudWatch: configures alerts
- Glue Python Shell → Failure alerting: failure events