How datawithrocky/open_code_test works
This repository contains Python data-migration and ingestion workflows: Spark jobs selectively read source data and append it to PostgreSQL, while pandas utilities map file-derived tables into target schemas, including relational merges. The README is empty, so the architecture is inferred from the sampled source. The mapping scripts import data_std, which is not present in the tree; their relationship to the sampled data_std_ident.py and data_std_new.py implementations is therefore unclear.
Main entry points: upload_df_spark.py Generic ingestion · ingestion.py Selective ingestion · file_map_new.py File mapping · failmapping_ident.py IDENT file mapping
Components 13
Spark ingestion
- Selective ingestion
ingestion.py - Generic ingestion
upload_df_spark.py
Table mapping
- File mapping
file_map_new.py - IDENT file mapping
failmapping_ident.py - Relational merge
data_std_new.py - IDENT relational merge
data_std_ident.py
Runtime settings
- Ingestion rules
ingestion_rules.py - Selective config
config.py - Database config
db_config.py
Other components
- Source databases
- Landing PostgreSQL
- Mapping PostgreSQL
- Migration database
Connections 15
- Selective ingestion → Selective config: loads settings
ingestion.py - Selective ingestion → Source databases: reads tables
ingestion.py - Selective ingestion → Landing PostgreSQL: appends rows
ingestion.py - Generic ingestion → Database config: loads settings
upload_df_spark.py - Generic ingestion → Ingestion rules: loads rules
upload_df_spark.py - Generic ingestion → Source databases: reads tables
upload_df_spark.py - Generic ingestion → Landing PostgreSQL: appends rows
upload_df_spark.py - File mapping → Source databases: reads tables
file_map_new.py - File mapping → Mapping PostgreSQL: reads target
file_map_new.py - File mapping → Mapping PostgreSQL: replaces table
file_map_new.py - IDENT file mapping → Source databases: reads tables
failmapping_ident.py - IDENT file mapping → Mapping PostgreSQL: reads target
failmapping_ident.py - IDENT file mapping → Mapping PostgreSQL: replaces table
failmapping_ident.py - Relational merge → Migration database: reads source
data_std_new.py - IDENT relational merge → Migration database: reads source
data_std_ident.py