# datawithrocky/open_code_test architecture

> Architecture diagram of the GitHub repository datawithrocky/open_code_test, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datawithrocky/open_code_test
- Repository: https://github.com/datawithrocky/open_code_test

## Overview

This repository contains Python data-migration and ingestion workflows: Spark jobs selectively read source data and append it to PostgreSQL, while pandas utilities map file-derived tables into target schemas, including relational merges. The README is empty, so the architecture is inferred from the sampled source. The mapping scripts import `data_std`, which is not present in the tree; their relationship to the sampled `data_std_ident.py` and `data_std_new.py` implementations is therefore unclear.

Main entry points: [`upload_df_spark.py`](https://github.com/datawithrocky/open_code_test/blob/main/upload_df_spark.py) (Generic ingestion), [`ingestion.py`](https://github.com/datawithrocky/open_code_test/blob/main/ingestion.py) (Selective ingestion), [`file_map_new.py`](https://github.com/datawithrocky/open_code_test/blob/main/file_map_new.py) (File mapping), [`failmapping_ident.py`](https://github.com/datawithrocky/open_code_test/blob/main/failmapping_ident.py) (IDENT file mapping)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Spark ingestion"]
  node_ingestion_job["Selective ingestion<br/>[ingestion.py]"]
  node_generic_job["Generic ingestion<br/>[upload_df_spark.py]"]
end

subgraph group_mapping["Table mapping"]
  node_mapping_job["File mapping<br/>[file_map_new.py]"]
  node_mapping_job_ident["IDENT file mapping"]
  node_relational_merge["Relational merge<br/>[data_std_new.py]"]
  node_relational_merge_ident["IDENT relational merge<br/>[data_std_ident.py]"]
end

subgraph group_settings["Runtime settings"]
  node_ingestion_rules["Ingestion rules<br/>[ingestion_rules.py]"]
  node_selective_config["Selective config<br/>[config.py]"]
  node_database_config["Database config<br/>[db_config.py]"]
end

node_source_db[("Source databases")]
node_landing_postgres[("Landing PostgreSQL")]
node_mapping_postgres[("Mapping PostgreSQL")]
node_merge_database[("Migration database")]

node_ingestion_job -->|"loads settings"| node_selective_config
node_ingestion_job -->|"reads tables"| node_source_db
node_ingestion_job -->|"appends rows"| node_landing_postgres
node_generic_job -->|"loads settings"| node_database_config
node_generic_job -->|"loads rules"| node_ingestion_rules
node_generic_job -->|"reads tables"| node_source_db
node_generic_job -->|"appends rows"| node_landing_postgres
node_mapping_job -->|"reads tables"| node_source_db
node_mapping_job -->|"reads target"| node_mapping_postgres
node_mapping_job -->|"replaces table"| node_mapping_postgres
node_mapping_job_ident -->|"reads tables"| node_source_db
node_mapping_job_ident -->|"reads target"| node_mapping_postgres
node_mapping_job_ident -->|"replaces table"| node_mapping_postgres
node_relational_merge -->|"reads source"| node_merge_database
node_relational_merge_ident -->|"reads source"| node_merge_database

click node_ingestion_job "https://github.com/datawithrocky/open_code_test/blob/main/ingestion.py"
click node_generic_job "https://github.com/datawithrocky/open_code_test/blob/main/upload_df_spark.py"
click node_ingestion_rules "https://github.com/datawithrocky/open_code_test/blob/main/ingestion_rules.py"
click node_selective_config "https://github.com/datawithrocky/open_code_test/blob/main/config.py"
click node_database_config "https://github.com/datawithrocky/open_code_test/blob/main/db_config.py"
click node_mapping_job "https://github.com/datawithrocky/open_code_test/blob/main/file_map_new.py"
click node_mapping_job_ident "https://github.com/datawithrocky/open_code_test/blob/main/failmapping_ident.py"
click node_relational_merge "https://github.com/datawithrocky/open_code_test/blob/main/data_std_new.py"
click node_relational_merge_ident "https://github.com/datawithrocky/open_code_test/blob/main/data_std_ident.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion_job,node_generic_job toneBlue
class node_mapping_job,node_mapping_job_ident,node_relational_merge,node_relational_merge_ident,node_source_db,node_landing_postgres,node_mapping_postgres,node_merge_database toneAmber
class node_ingestion_rules,node_selective_config,node_database_config toneMint
```

## Components

### Spark ingestion

- [Selective ingestion](https://github.com/datawithrocky/open_code_test/blob/main/ingestion.py): `ingestion.py`
- [Generic ingestion](https://github.com/datawithrocky/open_code_test/blob/main/upload_df_spark.py): `upload_df_spark.py`

### Table mapping

- [File mapping](https://github.com/datawithrocky/open_code_test/blob/main/file_map_new.py): `file_map_new.py`
- [IDENT file mapping](https://github.com/datawithrocky/open_code_test/blob/main/failmapping_ident.py): `failmapping_ident.py`
- [Relational merge](https://github.com/datawithrocky/open_code_test/blob/main/data_std_new.py): `data_std_new.py`
- [IDENT relational merge](https://github.com/datawithrocky/open_code_test/blob/main/data_std_ident.py): `data_std_ident.py`

### Runtime settings

- [Ingestion rules](https://github.com/datawithrocky/open_code_test/blob/main/ingestion_rules.py): `ingestion_rules.py`
- [Selective config](https://github.com/datawithrocky/open_code_test/blob/main/config.py): `config.py`
- [Database config](https://github.com/datawithrocky/open_code_test/blob/main/db_config.py): `db_config.py`

### Other components

- **Source databases**
- **Landing PostgreSQL**
- **Mapping PostgreSQL**
- **Migration database**

## Connections

- Selective ingestion → Selective config: loads settings (evidence: `ingestion.py`)
- Selective ingestion → Source databases: reads tables (evidence: `ingestion.py`)
- Selective ingestion → Landing PostgreSQL: appends rows (evidence: `ingestion.py`)
- Generic ingestion → Database config: loads settings (evidence: `upload_df_spark.py`)
- Generic ingestion → Ingestion rules: loads rules (evidence: `upload_df_spark.py`)
- Generic ingestion → Source databases: reads tables (evidence: `upload_df_spark.py`)
- Generic ingestion → Landing PostgreSQL: appends rows (evidence: `upload_df_spark.py`)
- File mapping → Source databases: reads tables (evidence: `file_map_new.py`)
- File mapping → Mapping PostgreSQL: reads target (evidence: `file_map_new.py`)
- File mapping → Mapping PostgreSQL: replaces table (evidence: `file_map_new.py`)
- IDENT file mapping → Source databases: reads tables (evidence: `failmapping_ident.py`)
- IDENT file mapping → Mapping PostgreSQL: reads target (evidence: `failmapping_ident.py`)
- IDENT file mapping → Mapping PostgreSQL: replaces table (evidence: `failmapping_ident.py`)
- Relational merge → Migration database: reads source (evidence: `data_std_new.py`)
- IDENT relational merge → Migration database: reads source (evidence: `data_std_ident.py`)
