How roshoutlook/databricks_datareplication works
This repository appears to support metadata-driven data replication through Databricks. The clearest runtime path is the Docker Python entry point submitting a Databricks job for the modular replication notebook and fetching its output. The tree also names full/incremental replication, post-replication testing, data upload, and source-definition files; their internal wiring is not shown in the sampled source, so those relationships are left unspecified.
Main entry points: Docker_ADB_Call.py Docker job caller · Modular_Replication_Code.ipynb Modular replication
Components 9
Job invocation
- Docker job caller
Metadatamanagement/Docker/Docker_ADB_Call.py - Python API caller
Metadatamanagement/Python/ADB_Py_API.py - Databricks jobs
Replication workflows
- Modular replication
Metadatamanagement/Python/Modular_Replication_Code.ipynb - Full and incremental replication
Metadatamanagement/Python/Full and Incremental Data Replication.ipynb - PySpark data upload
Metadatamanagement/Python/Pyspark_upload_data_to_DB_Script.ipynb - Post-replication test
Metadatamanagement/Python/Post_Replication_Test.ipynb
Source metadata
- Source definitions
Metadatamanagement/SourceDefinitionFiles
Other components
- Operator
Connections 4
- Operator → Docker job caller: invokes
- Docker job caller → Databricks jobs: submits job
- Databricks jobs → Modular replication: runs task
- Docker job caller → Databricks jobs: fetches output