# ahmednbail/transport-analysis-data-pipeline architecture

> Architecture diagram of the GitHub repository ahmednbail/transport-analysis-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ahmednbail/transport-analysis-data-pipeline
- Repository: https://github.com/ahmednbail/transport-analysis-data-pipeline

## Overview

1. **What this repo is**
   - This is a batch data pipeline for transport/taxi analytics.
   - The core flow is: **source CSVs → GCS data lake → BigQuery warehouse → dbt transformations → analytical marts**.
   - **Apache Airflow** orchestrates the pipeline; **dbt** owns the transformation layer.

2. **Orchestration layer**
   - The Airflow code lives under `dags/`.
   - There are three main DAGs:
     - `Extract&Transfer-DAG.py` for ingestion/loading work.
     - `DBT-DAG.py` for running dbt models.
     - `Master_DAG.py` for coordinating the overall end-to-end sequence.
   - `dags/upload.py` is likely a helper used by the ingestion DAG to move files into the target storage/warehouse.

3. **Ingestion and landing zone**
   - The repo README explicitly positions **Google Cloud Storage (GCS)** as the data lake.
   - The ingestion side appears to be responsible for bringing raw transport CSVs into the pipeline and making them available for warehouse loading.
   - This is the first hard boundary: raw or semi-raw data is kept separate from modeled warehouse tables.

4. **Warehouse layer**
   - **BigQuery** is the central warehouse.
   - dbt models are written to build warehouse tables rather than serving files directly from the lake.
   - The architecture implies that the DAGs perform load/transfer steps into BigQuery, then dbt reads from BigQuery to create modeled datasets.

5. **dbt project structure**
   - `dbt_project.yml` defines the dbt project.
   - The `models/` directory is split into standard analytics layers:
     - `models/staging/`
     - `models/intermediat/` (intermediate transformations)
     - `models/marts/` (final business-facing tables)
   - This is a classic layered transformation boundary: raw warehouse tables are normalized in staging, reshaped in intermediate models, and published as marts.

6. **Concrete modeled outputs**
   - Staging: `models/staging/stg_taxi.sql` is the first clean abstraction over source data.
   - Intermediate: `models/intermediat/int_trips.sql` combines or enriches trip-level logic.
   - Marts:
     - `models/marts/fact_trip.sql`
     - `models/marts/dim_location.sql`
     - `models/marts/dim_vendors.sql`
   - These names show a star-schema-style output: a central trip fact table with supporting dimensions.

7. **Reusable SQL logic**
   - `macros/` contains shared dbt logic:
     - `safe_cast.sql` for defensive type conversion.
     - `get_trip_duration_minutes.sql` for derived trip metrics.
     - `get_vendor_names.sql` for vendor lookup/normalization.
   - These macros reduce duplicated SQL and centralize domain logic used across models.

8. **Reference data and lookups**
   - `seeds/` contains CSV lookup tables:
     - `payment_type_lookup.csv`
     - `taxi_zone_lookup.csv`
   - In dbt, seeds typically become small reference tables used to enrich fact/dimension models.
   - This means some business context is managed as versioned static data inside the repo, not as an external dependency.

9. **Data quality and metadata**
   - `models/source.yaml`, `macros/macros_properties.yml`, and `seeds/seeds_properties.yml` indicate dbt documentation/tests/schema definitions are part of the project.
   - `tests/dags/test_dag_example.py` and `.astro/test_dag_integrity_default.py` show DAG integrity checks are used to prevent broken Airflow deployments.
   - `snapshots/` exists, so the project is prepared for slowly changing dimension / historical change tracking, even if the directory is currently empty.

10. **Deployment/runtime packaging**
   - `Dockerfile`, `requirements.txt`, `packages.txt`, and `packages.yml` show the project is meant to run in a containerized, reproducible environment.
   - The `.astro/` directory indicates an **Astronomer-managed Airflow** setup, which is important for how the DAGs are packaged and validated.
   - `airflow_settings.yaml` suggests environment-specific Airflow configuration and connection management.

11. **Analysis and presentation artifacts**
   - `analyses/` is a dbt workspace for ad hoc SQL exploration.
   - `assets/` contains non-code documentation/media, including the logo used in the README.
   - `Power BI` is mentioned in the README as the final visualization layer, but there is no BI code in the repo; it is a downstream consumer of the marts.

12. **How to draw the architecture**
   - Draw five main boxes in sequence:
     - **Source data / transport CSVs**
     - **GCS landing zone**
     - **Airflow orchestration**
     - **BigQuery warehouse**
     - **dbt layered models**
     - then **Power BI** on top of the marts.
   - Inside dbt, show the path **staging → intermediate → marts**.
   - Around the pipeline, show shared assets for **macros**, **seeds/lookups**, **tests**, and **containerized deployment**.

Main entry points: [`.astro/config.yaml`](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/.astro/config.yaml) (Airflow), [`dags/Extract&Transfer-DAG.py`](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/dags/Extract&Transfer-DAG.py) (Extract DAG), [`models/marts/fact_trip.sql`](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/models/marts/fact_trip.sql) (Marts), [`dags/Master_DAG.py`](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/dags/Master_DAG.py) (Master DAG)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sources["Sources & landing"]
  node_source_csvs["Source CSVs<br/>raw input"]
  node_gcs_landing[("GCS Lake<br/>data lake")]
end

subgraph group_orchestration["Airflow orchestration"]
  node_airflow(("Airflow<br/>orchestrator<br/>[config.yaml]"))
  node_extract_dag["Extract DAG<br/>ingest DAG"]
  node_master_dag["Master DAG<br/>control DAG<br/>[Master_DAG.py]"]
  node_dbt_dag["dbt DAG<br/>transform DAG<br/>[DBT-DAG.py]"]
  node_upload_helper["Upload Helper<br/>ingest helper<br/>[upload.py]"]
end

subgraph group_warehouse["Warehouse & dbt"]
  node_bq_warehouse[("BigQuery<br/>warehouse")]
  node_dbt_project["dbt Project<br/>modeling project<br/>[dbt_project.yml]"]
  node_staging_models["Staging<br/>dbt layer<br/>[stg_taxi.sql]"]
  node_intermediate_models["Intermediate<br/>dbt layer<br/>[int_trips.sql]"]
  node_mart_models["Marts<br/>dbt layer<br/>[fact_trip.sql]"]
  node_macros["Macros<br/>shared SQL<br/>[safe_cast.sql]"]
  node_seeds["Seeds<br/>lookup data"]
end

subgraph group_platform["Platform & quality"]
  node_tests["Tests<br/>validation"]
  node_runtime["Container Runtime<br/>deployment"]
  node_snapshots["Snapshots<br/>history tracking<br/>[.gitkeep]"]
  node_power_bi(("Power BI<br/>bi consumer"))
end

node_source_csvs -->|"land raw data"| node_gcs_landing
node_gcs_landing -->|"ingest"| node_extract_dag
node_airflow -->|"runs"| node_extract_dag
node_airflow -->|"runs"| node_master_dag
node_airflow -->|"runs"| node_dbt_dag
node_extract_dag -->|"uses"| node_upload_helper
node_extract_dag -->|"load"| node_bq_warehouse
node_master_dag -->|"sequence"| node_extract_dag
node_master_dag -->|"sequence"| node_dbt_dag
node_bq_warehouse -->|"source"| node_dbt_project
node_dbt_project -->|"defines"| node_staging_models
node_staging_models -->|"refine"| node_intermediate_models
node_intermediate_models -->|"publish"| node_mart_models
node_staging_models -->|"reuse"| node_macros
node_intermediate_models -->|"reuse"| node_macros
node_mart_models -->|"enrich"| node_seeds
node_dbt_dag -->|"executes"| node_dbt_project
node_tests -.->|"guard"| node_airflow
node_runtime -->|"packages"| node_airflow
node_snapshots -.->|"extend"| node_mart_models
node_mart_models -->|"serve"| node_power_bi

click node_airflow "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/.astro/config.yaml"
click node_extract_dag "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/dags/Extract&Transfer-DAG.py"
click node_master_dag "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/dags/Master_DAG.py"
click node_dbt_dag "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/dags/DBT-DAG.py"
click node_upload_helper "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/dags/upload.py"
click node_dbt_project "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/dbt_project.yml"
click node_staging_models "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/models/staging/stg_taxi.sql"
click node_intermediate_models "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/models/intermediat/int_trips.sql"
click node_mart_models "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/models/marts/fact_trip.sql"
click node_macros "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/macros/safe_cast.sql"
click node_seeds "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/seeds/payment_type_lookup.csv"
click node_tests "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/tests/dags/test_dag_example.py"
click node_runtime "https://github.com/ahmednbail/transport-analysis-data-pipeline/tree/main/Dockerfile"
click node_snapshots "https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/snapshots/.gitkeep"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_source_csvs,node_gcs_landing toneBlue
class node_airflow,node_extract_dag,node_master_dag,node_dbt_dag,node_upload_helper toneAmber
class node_bq_warehouse,node_dbt_project,node_staging_models,node_intermediate_models,node_mart_models,node_macros,node_seeds toneMint
class node_tests,node_runtime,node_snapshots,node_power_bi toneRose
```

## Components

### Sources & landing

Raw transport data and the landing zone it first enters

- **Source CSVs**: raw input · Transport/taxi source files entering the pipeline
- **GCS Lake**: data lake · Google Cloud Storage landing zone for raw and semi-raw files

### Airflow orchestration

DAGs and helpers that coordinate ingestion and dbt runs

- [Airflow](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/.astro/config.yaml): orchestrator · Apache Airflow runtime coordinating the batch pipeline · `.astro/config.yaml`
- [Extract DAG](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/dags/Extract&Transfer-DAG.py): ingest DAG · Loads and transfers incoming data · `dags/Extract&Transfer-DAG.py`
- [Master DAG](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/dags/Master_DAG.py): control DAG · Coordinates end-to-end execution sequence · `dags/Master_DAG.py`
- [dbt DAG](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/dags/DBT-DAG.py): transform DAG · Triggers dbt transformation runs · `dags/DBT-DAG.py`
- [Upload Helper](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/dags/upload.py): ingest helper · Shared file-moving logic used by ingestion tasks · `dags/upload.py`

### Warehouse & dbt

BigQuery-backed transformation layers and analytical outputs

- **BigQuery**: warehouse · Central warehouse for loaded and modeled tables
- [dbt Project](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/dbt_project.yml): modeling project · dbt configuration and model definitions · `dbt_project.yml`
- [Staging](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/models/staging/stg_taxi.sql): dbt layer · Clean abstractions over source tables · `models/staging/stg_taxi.sql`
- [Intermediate](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/models/intermediat/int_trips.sql): dbt layer · Enrichment and trip-level reshaping logic · `models/intermediat/int_trips.sql`
- [Marts](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/models/marts/fact_trip.sql): dbt layer · Final fact and dimension outputs · `models/marts/fact_trip.sql`
- [Macros](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/macros/safe_cast.sql): shared SQL · Reusable dbt logic for casting, duration, and vendor lookups · `macros/safe_cast.sql`
- [Seeds](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/seeds/payment_type_lookup.csv): lookup data · Versioned reference tables for payments and taxi zones · `seeds/payment_type_lookup.csv`

### Platform & quality

Packaging, metadata, tests, and supporting assets

- [Tests](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/tests/dags/test_dag_example.py): validation · DAG integrity and project validation checks · `tests/dags/test_dag_example.py`
- [Container Runtime](https://github.com/ahmednbail/transport-analysis-data-pipeline/tree/main/Dockerfile): deployment · Containerized environment and dependency packaging · `Dockerfile`
- [Snapshots](https://github.com/ahmednbail/transport-analysis-data-pipeline/blob/main/snapshots/.gitkeep): history tracking · Prepared area for slowly changing or historical capture · `snapshots/.gitkeep`
- **Power BI**: bi consumer · Downstream analytics consumer of the marts

## Connections

- Source CSVs → GCS Lake: land raw data · Raw transport CSVs are stored in the lake first
- GCS Lake → Extract DAG: ingest · Airflow ingestion tasks pick up landed files
- Airflow → Extract DAG: runs · Airflow executes the ingestion DAG
- Airflow → Master DAG: runs · Airflow executes the coordinating DAG
- Airflow → dbt DAG: runs · Airflow executes dbt transformation runs
- Extract DAG → Upload Helper: uses · The ingestion DAG relies on shared upload logic
- Extract DAG → BigQuery: load · Ingested data is loaded into the warehouse
- Master DAG → Extract DAG: sequence · Master orchestration starts ingestion first
- Master DAG → dbt DAG: sequence · Master orchestration triggers transformation after load
- BigQuery → dbt Project: source · dbt models read from warehouse tables
- dbt Project → Staging: defines · Project configuration includes staging models
- Staging → Intermediate: refine · Cleaned data flows into intermediate transformations
- Intermediate → Marts: publish · Intermediate logic feeds final marts
- Staging → Macros: reuse · Staging and downstream models share SQL macros
- Intermediate → Macros: reuse · Transformation logic reuses macros
- Marts → Seeds: enrich · Marts are enriched with lookup/reference data
- dbt DAG → dbt Project: executes · The dbt DAG runs the project models
- Tests → Airflow: guard · Integrity tests help protect DAG deployment
- Container Runtime → Airflow: packages · Docker and dependency files package the Airflow runtime
- Snapshots → Marts: extend · Snapshots provide a path for historical modeling
- Marts → Power BI: serve · Analytical marts feed downstream BI reporting