How amritpsingh/elt-data-engineering-project works
This repository is an orchestrated PostgreSQL ELT demo. Its Airflow DAG triggers an Airbyte sync and then runs dbt; dbt’s SQL models provide the transformation stage, with results stored in the destination database. A separate Python script implements a pg_dump/psql transfer path, but the DAG’s direct invocation of that script is commented out. The README also documents source-database initialization and data-quality checks; the latter’s implementation is not present in the sampled source. Some integration wiring is therefore left unverified.
Main entry points: elt_dag.py dbt runner · elt_script.py pg_dump / psql · film_ratings.sql Ratings model · init.sql Source initialization
Components 13
Orchestration
- Airflow DAG
airflow/dags/elt_dag.py
Data Transfer
- Airbyte sync
- pg_dump / psql
elt/elt_script.py
PostgreSQL
- Source initialization
source_db_init/init.sql - Source PostgreSQL
- Destination PostgreSQL
dbt Transformations
- dbt runner
airflow/dags/elt_dag.py - Film models
custom_postgres/models/example/films.sql - Actor models
custom_postgres/models/example/actors.sql - Film-actor model
custom_postgres/models/example/film_actors.sql - Ratings model
custom_postgres/models/example/film_ratings.sql - Ratings macro
custom_postgres/macros/film_ratings_macro.sql
Other components
- Pipeline operator
Connections 14
- Pipeline operator → Airflow DAG: triggers DAG
README.md - Airflow DAG → Airbyte sync: triggers sync
airflow/dags/elt_dag.py - Airflow DAG → dbt runner: runs after sync
airflow/dags/elt_dag.py - Source initialization → Source PostgreSQL: initializes
README.md - Airbyte sync → Source PostgreSQL: syncs from
- Airbyte sync → Destination PostgreSQL: syncs to
- pg_dump / psql → Source PostgreSQL: dumps from
elt/elt_script.py - pg_dump / psql → Destination PostgreSQL: loads into
elt/elt_script.py - dbt runner → Destination PostgreSQL: transforms in
README.md - dbt runner → Film models: runs models
README.md - dbt runner → Actor models: runs models
README.md - dbt runner → Film-actor model: runs models
README.md - dbt runner → Ratings model: runs models
README.md - Ratings model → Ratings macro: uses macro