# roshoutlook/databricks_datareplication architecture

> Architecture diagram of the GitHub repository roshoutlook/databricks_datareplication, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/roshoutlook/databricks_datareplication
- Repository: https://github.com/roshoutlook/databricks_datareplication

## Overview

This repository appears to support metadata-driven data replication through Databricks. The clearest runtime path is the Docker Python entry point submitting a Databricks job for the modular replication notebook and fetching its output. The tree also names full/incremental replication, post-replication testing, data upload, and source-definition files; their internal wiring is not shown in the sampled source, so those relationships are left unspecified.

Main entry points: [`Metadatamanagement/Docker/Docker_ADB_Call.py`](https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Docker/Docker_ADB_Call.py) (Docker job caller), [`Metadatamanagement/Python/Modular_Replication_Code.ipynb`](https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Python/Modular_Replication_Code.ipynb) (Modular replication)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_invocation["Job invocation"]
  node_docker_call["Docker job caller<br/>[Docker_ADB_Call.py]"]
  node_python_api["Python API caller<br/>[ADB_Py_API.py]"]
  node_databricks["Databricks jobs"]
end

subgraph group_replication["Replication workflows"]
  node_modular["Modular replication"]
  node_full_incremental["Full and incremental replication"]
  node_upload["PySpark data upload"]
  node_post_test["Post-replication test"]
end

subgraph group_metadata["Source metadata"]
  node_source_specs["Source definitions"]
end

node_user(("Operator"))

node_user -->|"invokes"| node_docker_call
node_docker_call -->|"submits job"| node_databricks
node_databricks -->|"runs task"| node_modular
node_docker_call -->|"fetches output"| node_databricks

click node_docker_call "https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Docker/Docker_ADB_Call.py"
click node_python_api "https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Python/ADB_Py_API.py"
click node_modular "https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Python/Modular_Replication_Code.ipynb"
click node_full_incremental "https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Python/Full%20and%20Incremental%20Data%20Replication.ipynb"
click node_upload "https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Python/Pyspark_upload_data_to_DB_Script.ipynb"
click node_post_test "https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Python/Post_Replication_Test.ipynb"
click node_source_specs "https://github.com/roshoutlook/databricks_datareplication/tree/main/Metadatamanagement/SourceDefinitionFiles"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_docker_call,node_python_api,node_databricks toneBlue
class node_modular,node_full_incremental,node_upload,node_post_test toneAmber
class node_source_specs toneMint
class node_user toneIndigo
```

## Components

### Job invocation

- [Docker job caller](https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Docker/Docker_ADB_Call.py): `Metadatamanagement/Docker/Docker_ADB_Call.py`
- [Python API caller](https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Python/ADB_Py_API.py): `Metadatamanagement/Python/ADB_Py_API.py`
- **Databricks jobs**

### Replication workflows

- [Modular replication](https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Python/Modular_Replication_Code.ipynb): `Metadatamanagement/Python/Modular_Replication_Code.ipynb`
- [Full and incremental replication](https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Python/Full%20and%20Incremental%20Data%20Replication.ipynb): `Metadatamanagement/Python/Full and Incremental Data Replication.ipynb`
- [PySpark data upload](https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Python/Pyspark_upload_data_to_DB_Script.ipynb): `Metadatamanagement/Python/Pyspark_upload_data_to_DB_Script.ipynb`
- [Post-replication test](https://github.com/roshoutlook/databricks_datareplication/blob/main/Metadatamanagement/Python/Post_Replication_Test.ipynb): `Metadatamanagement/Python/Post_Replication_Test.ipynb`

### Source metadata

- [Source definitions](https://github.com/roshoutlook/databricks_datareplication/tree/main/Metadatamanagement/SourceDefinitionFiles): `Metadatamanagement/SourceDefinitionFiles`

### Other components

- **Operator**

## Connections

- Operator → Docker job caller: invokes
- Docker job caller → Databricks jobs: submits job
- Databricks jobs → Modular replication: runs task
- Docker job caller → Databricks jobs: fetches output