# amritpsingh/elt-data-engineering-project architecture

> Architecture diagram of the GitHub repository amritpsingh/elt-data-engineering-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amritpsingh/elt-data-engineering-project
- Repository: https://github.com/amritpsingh/elt-data-engineering-project

## Overview

This repository is an orchestrated PostgreSQL ELT demo. Its Airflow DAG triggers an Airbyte sync and then runs dbt; dbt’s SQL models provide the transformation stage, with results stored in the destination database. A separate Python script implements a pg_dump/psql transfer path, but the DAG’s direct invocation of that script is commented out. The README also documents source-database initialization and data-quality checks; the latter’s implementation is not present in the sampled source. Some integration wiring is therefore left unverified.

Main entry points: [`airflow/dags/elt_dag.py`](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/airflow/dags/elt_dag.py) (dbt runner), [`elt/elt_script.py`](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/elt/elt_script.py) (pg_dump / psql), [`custom_postgres/models/example/film_ratings.sql`](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/custom_postgres/models/example/film_ratings.sql) (Ratings model), [`source_db_init/init.sql`](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/source_db_init/init.sql) (Source initialization)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_airflow["Airflow DAG<br/>[elt_dag.py]"]
end

subgraph group_transfer["Data Transfer"]
  node_airbyte["Airbyte sync"]
  node_elt_script["pg_dump / psql<br/>[elt_script.py]"]
end

subgraph group_databases["PostgreSQL"]
  node_source_init["Source initialization<br/>[init.sql]"]
  node_source_db[("Source PostgreSQL")]
  node_destination_db[("Destination PostgreSQL")]
end

subgraph group_transform["dbt Transformations"]
  node_dbt_runner["dbt runner<br/>[elt_dag.py]"]
  node_film_models["Film models<br/>[films.sql]"]
  node_actor_models["Actor models<br/>[actors.sql]"]
  node_film_actor_model["Film-actor model<br/>[film_actors.sql]"]
  node_ratings_model["Ratings model<br/>[film_ratings.sql]"]
  node_ratings_macro["Ratings macro"]
end

node_user(("Pipeline operator"))

node_user -->|"triggers DAG"| node_airflow
node_airflow -->|"triggers sync"| node_airbyte
node_airflow -->|"runs after sync"| node_dbt_runner
node_source_init -->|"initializes"| node_source_db
node_airbyte -.->|"syncs from"| node_source_db
node_airbyte -.->|"syncs to"| node_destination_db
node_elt_script -->|"dumps from"| node_source_db
node_elt_script -->|"loads into"| node_destination_db
node_dbt_runner -->|"transforms in"| node_destination_db
node_dbt_runner -->|"runs models"| node_film_models
node_dbt_runner -->|"runs models"| node_actor_models
node_dbt_runner -->|"runs models"| node_film_actor_model
node_dbt_runner -->|"runs models"| node_ratings_model
node_ratings_model -.->|"uses macro"| node_ratings_macro

click node_airflow "https://github.com/amritpsingh/elt-data-engineering-project/blob/main/airflow/dags/elt_dag.py"
click node_elt_script "https://github.com/amritpsingh/elt-data-engineering-project/blob/main/elt/elt_script.py"
click node_source_init "https://github.com/amritpsingh/elt-data-engineering-project/blob/main/source_db_init/init.sql"
click node_dbt_runner "https://github.com/amritpsingh/elt-data-engineering-project/blob/main/airflow/dags/elt_dag.py"
click node_film_models "https://github.com/amritpsingh/elt-data-engineering-project/blob/main/custom_postgres/models/example/films.sql"
click node_actor_models "https://github.com/amritpsingh/elt-data-engineering-project/blob/main/custom_postgres/models/example/actors.sql"
click node_film_actor_model "https://github.com/amritpsingh/elt-data-engineering-project/blob/main/custom_postgres/models/example/film_actors.sql"
click node_ratings_model "https://github.com/amritpsingh/elt-data-engineering-project/blob/main/custom_postgres/models/example/film_ratings.sql"
click node_ratings_macro "https://github.com/amritpsingh/elt-data-engineering-project/blob/main/custom_postgres/macros/film_ratings_macro.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow toneBlue
class node_airbyte,node_elt_script toneAmber
class node_source_init,node_source_db,node_destination_db toneMint
class node_dbt_runner,node_film_models,node_actor_models,node_film_actor_model,node_ratings_model,node_ratings_macro toneRose
class node_user toneIndigo
```

## Components

### Orchestration

- [Airflow DAG](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/airflow/dags/elt_dag.py): `airflow/dags/elt_dag.py`

### Data Transfer

- **Airbyte sync**
- [pg_dump / psql](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/elt/elt_script.py): `elt/elt_script.py`

### PostgreSQL

- [Source initialization](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/source_db_init/init.sql): `source_db_init/init.sql`
- **Source PostgreSQL**
- **Destination PostgreSQL**

### dbt Transformations

- [dbt runner](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/airflow/dags/elt_dag.py): `airflow/dags/elt_dag.py`
- [Film models](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/custom_postgres/models/example/films.sql): `custom_postgres/models/example/films.sql`
- [Actor models](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/custom_postgres/models/example/actors.sql): `custom_postgres/models/example/actors.sql`
- [Film-actor model](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/custom_postgres/models/example/film_actors.sql): `custom_postgres/models/example/film_actors.sql`
- [Ratings model](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/custom_postgres/models/example/film_ratings.sql): `custom_postgres/models/example/film_ratings.sql`
- [Ratings macro](https://github.com/amritpsingh/elt-data-engineering-project/blob/main/custom_postgres/macros/film_ratings_macro.sql): `custom_postgres/macros/film_ratings_macro.sql`

### Other components

- **Pipeline operator**

## Connections

- Pipeline operator → Airflow DAG: triggers DAG (evidence: `README.md`)
- Airflow DAG → Airbyte sync: triggers sync (evidence: `airflow/dags/elt_dag.py`)
- Airflow DAG → dbt runner: runs after sync (evidence: `airflow/dags/elt_dag.py`)
- Source initialization → Source PostgreSQL: initializes (evidence: `README.md`)
- Airbyte sync → Source PostgreSQL: syncs from
- Airbyte sync → Destination PostgreSQL: syncs to
- pg_dump / psql → Source PostgreSQL: dumps from (evidence: `elt/elt_script.py`)
- pg_dump / psql → Destination PostgreSQL: loads into (evidence: `elt/elt_script.py`)
- dbt runner → Destination PostgreSQL: transforms in (evidence: `README.md`)
- dbt runner → Film models: runs models (evidence: `README.md`)
- dbt runner → Actor models: runs models (evidence: `README.md`)
- dbt runner → Film-actor model: runs models (evidence: `README.md`)
- dbt runner → Ratings model: runs models (evidence: `README.md`)
- Ratings model → Ratings macro: uses macro
