# dhai02/big-data-nyc architecture

> Architecture diagram of the GitHub repository dhai02/big-data-nyc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dhai02/big-data-nyc
- Repository: https://github.com/dhai02/big-data-nyc

## Overview

1. **What this repo is**
   - This is a batch data pipeline for NYC taxi data built around **Apache Spark + HDFS**.
   - The architecture follows a **Bronze → Silver → Gold** pattern: raw data lands in HDFS, is cleaned/enriched, then aggregated into analysis-ready datasets.

2. **Execution environment**
   - The primary runtime is **Docker Compose**, which brings up the Hadoop/HDFS and Spark stack plus a Jupyter environment.
   - The repo also includes a manual **WSL2/Hadoop/Spark** install path, but that is clearly secondary to the containerized flow.
   - Key infrastructure files are `docker-compose.yml`, `docker/Dockerfile`, and the Hadoop config/env files under `docker/`.

3. **Ingestion boundary: external data → Bronze**
   - `src/ingestion.py` is the entry point for pulling NYC taxi source data into the pipeline.
   - Its role is to place **raw, untouched parquet** into HDFS under `/nyc_taxi/bronze/raw/`.
   - This is the first hard boundary: data becomes part of the managed pipeline only after landing in HDFS.

4. **Cleaning/enrichment boundary: Bronze → Silver**
   - `src/02_data_cleaning.py` transforms Bronze data into the Silver layer.
   - `src/cleaning_utils.py` holds the reusable cleaning logic that the tests target directly.
   - The Silver output is not just cleaned; it is also **enriched** with derived fields such as trip duration, pickup hour/date/day-of-week, tip percentage, and speed.

5. **Gold layer: analytical aggregates**
   - `src/03_aggregation_time.py` produces time-based summaries in `/nyc_taxi/gold/aggregated_by_time/`.
   - `src/04_aggregation_vendor.py` produces vendor/business summaries in `/nyc_taxi/gold/aggregated_by_vendor/`.
   - The README shows these Gold outputs are partitioned into analysis-friendly datasets like by hour, by date, by day-of-week, by hour/day-of-week, by vendor, by payment method, top pickup locations, and top routes.

6. **Verification stage**
   - `src/05_verify_results.py` is a dedicated quality/check step after aggregation.
   - This suggests the pipeline treats correctness as an explicit stage, not just an afterthought.
   - The presence of verification scripts and tests indicates a clear separation between transformation logic and validation.

7. **Exploration and presentation layer**
   - `src/01_eda.py` and `src/06_visualization.py` support exploratory analysis and charting, not core ETL.
   - The notebooks (`notebooks/01_eda.ipynb`, `02_cleaning_demo.ipynb`, `03_aggregation_viz.ipynb`) are the demo/analysis boundary for human inspection and assignment deliverables.
   - In architecture terms, these sit downstream of the pipeline and consume Silver/Gold outputs rather than raw data.

8. **Data organization in HDFS**
   - The HDFS root is `/nyc_taxi/` with clear layer separation:
     - `bronze/raw/`
     - `silver/cleaned/`
     - `gold/aggregated_by_time/`
     - `gold/aggregated_by_vendor/`
   - This layout is central to the repo’s architecture because each script writes to one layer and the next step reads from it.

9. **Local artifacts and sample outputs**
   - `src/data/` contains parquet files that look like saved outputs or reference datasets for local use/documentation.
   - The README also describes copying data from HDFS to `/app/data`, which is mounted back to the host.
   - This creates a bridge between cluster storage and local inspection without changing the pipeline itself.

10. **Testing boundary**
   - `tests/test_cleaning.py`, `tests/test_cleaning_utils.py`, and `tests/test_aggregation.py` provide coverage for transformation logic.
   - The tests focus on business logic correctness around cleaning and aggregation rather than deployment.
   - This implies the core architecture is transformation-heavy, with tests anchored around deterministic Spark/DataFrame behavior.

11. **Supporting scripts and docs**
   - `run_pipeline.sh` and the scripts under `setup/` support local/manual execution and environment verification.
   - `docs/` contains assignment-specific documentation, which appears to support reporting rather than runtime behavior.
   - These are operational aids around the pipeline, not part of the main data path.

12. **High-level data flow**
   - The main flow is: **source data ingestion → HDFS Bronze parquet → Spark cleaning/enrichment → HDFS Silver parquet → Spark aggregations → HDFS Gold parquet → verification/EDA/visualization**.
   - The key architectural constraint is that Spark jobs read and write through HDFS, so storage and compute are decoupled.
   - Jupyter and notebooks consume the curated Silver/Gold outputs for analysis and presentation, not raw ingestion.

Main entry points: [`src/02_data_cleaning.py`](https://github.com/dhai02/big-data-nyc/blob/main/src/02_data_cleaning.py) (Cleaning), [`src/03_aggregation_time.py`](https://github.com/dhai02/big-data-nyc/blob/main/src/03_aggregation_time.py) (Time agg), [`src/04_aggregation_vendor.py`](https://github.com/dhai02/big-data-nyc/blob/main/src/04_aggregation_vendor.py) (Vendor agg), [`docker-compose.yml`](https://github.com/dhai02/big-data-nyc/blob/main/docker-compose.yml) (Compose)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_infra["Runtime"]
  node_docker_compose["Compose<br/>orchestrator<br/>[docker-compose.yml]"]
  node_dockerfile["Image<br/>container build"]
  node_hadoop_conf["Hadoop env<br/>cluster config<br/>[hadoop-client.xml]"]
  node_hadoop_env["Cluster env<br/>[hadoop.env]"]
end

subgraph group_pipeline["Pipeline"]
  node_ingestion["Ingestion<br/>spark job<br/>[ingestion.py]"]
  node_cleaning["Cleaning<br/>spark job"]
  node_cleaning_utils["Rules<br/>transform utils<br/>[cleaning_utils.py]"]
  node_agg_time["Time agg<br/>spark job"]
  node_agg_vendor["Vendor agg<br/>spark job"]
end

subgraph group_analytics["Exploration"]
  node_eda["EDA<br/>analysis script<br/>[01_eda.py]"]
  node_viz["Charts<br/>visualization"]
  node_notebooks["Notebooks<br/>notebook layer<br/>[01_eda.ipynb]"]
  node_sample_data[("Samples<br/>parquet outputs<br/>[readme.md]")]
end

subgraph group_quality["Verification"]
  node_verify["Verify<br/>quality check"]
  node_tests_clean["Cleaning tests<br/>test suite<br/>[test_cleaning.py]"]
  node_tests_agg["Agg tests<br/>test suite"]
end

subgraph group_support["Support"]
  node_setup_script["Setup<br/>install scripts"]
  node_verify_env["Env check<br/>setup check<br/>[verify_env.sh]"]
  node_run_pipeline["Run pipeline<br/>runner script<br/>[run_pipeline.sh]"]
  node_readme["README<br/>docs<br/>[README.md]"]
end

node_docker_compose -->|"builds"| node_dockerfile
node_docker_compose -->|"loads"| node_hadoop_conf
node_docker_compose -->|"configures"| node_hadoop_env
node_ingestion -->|"Bronze→Silver"| node_cleaning
node_cleaning -->|"uses"| node_cleaning_utils
node_cleaning -->|"Silver→Gold"| node_agg_time
node_cleaning -->|"Silver→Gold"| node_agg_vendor
node_agg_time -->|"checks"| node_verify
node_agg_vendor -->|"checks"| node_verify
node_cleaning -.->|"covered by"| node_tests_clean
node_cleaning_utils -.->|"covered by"| node_tests_clean
node_agg_time -.->|"covered by"| node_tests_agg
node_agg_vendor -.->|"covered by"| node_tests_agg
node_cleaning -->|"feeds"| node_eda
node_agg_time -->|"feeds"| node_viz
node_agg_vendor -->|"feeds"| node_viz
node_cleaning -->|"explored in"| node_notebooks
node_agg_time -->|"explored in"| node_notebooks
node_agg_vendor -->|"explored in"| node_notebooks
node_agg_time -->|"exports"| node_sample_data
node_agg_vendor -->|"exports"| node_sample_data
node_run_pipeline -->|"runs"| node_ingestion
node_setup_script -->|"supports"| node_verify_env
node_readme -.->|"documents"| node_docker_compose
node_readme -.->|"documents"| node_run_pipeline

click node_docker_compose "https://github.com/dhai02/big-data-nyc/blob/main/docker-compose.yml"
click node_dockerfile "https://github.com/dhai02/big-data-nyc/tree/main/docker/Dockerfile"
click node_hadoop_conf "https://github.com/dhai02/big-data-nyc/blob/main/docker/hadoop-client.xml"
click node_hadoop_env "https://github.com/dhai02/big-data-nyc/blob/main/docker/hadoop.env"
click node_ingestion "https://github.com/dhai02/big-data-nyc/blob/main/src/ingestion.py"
click node_cleaning "https://github.com/dhai02/big-data-nyc/blob/main/src/02_data_cleaning.py"
click node_cleaning_utils "https://github.com/dhai02/big-data-nyc/blob/main/src/cleaning_utils.py"
click node_agg_time "https://github.com/dhai02/big-data-nyc/blob/main/src/03_aggregation_time.py"
click node_agg_vendor "https://github.com/dhai02/big-data-nyc/blob/main/src/04_aggregation_vendor.py"
click node_verify "https://github.com/dhai02/big-data-nyc/blob/main/src/05_verify_results.py"
click node_eda "https://github.com/dhai02/big-data-nyc/blob/main/src/01_eda.py"
click node_viz "https://github.com/dhai02/big-data-nyc/blob/main/src/06_visualization.py"
click node_notebooks "https://github.com/dhai02/big-data-nyc/blob/main/notebooks/01_eda.ipynb"
click node_sample_data "https://github.com/dhai02/big-data-nyc/blob/main/src/data/readme.md"
click node_tests_clean "https://github.com/dhai02/big-data-nyc/blob/main/tests/test_cleaning.py"
click node_tests_agg "https://github.com/dhai02/big-data-nyc/blob/main/tests/test_aggregation.py"
click node_setup_script "https://github.com/dhai02/big-data-nyc/blob/main/setup/install_hadoop_spark.sh"
click node_verify_env "https://github.com/dhai02/big-data-nyc/blob/main/setup/verify_env.sh"
click node_run_pipeline "https://github.com/dhai02/big-data-nyc/blob/main/run_pipeline.sh"
click node_readme "https://github.com/dhai02/big-data-nyc/blob/main/README.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_docker_compose,node_dockerfile,node_hadoop_conf,node_hadoop_env toneBlue
class node_ingestion,node_cleaning,node_cleaning_utils,node_agg_time,node_agg_vendor toneAmber
class node_eda,node_viz,node_notebooks,node_sample_data toneMint
class node_verify,node_tests_clean,node_tests_agg toneRose
class node_setup_script,node_verify_env,node_run_pipeline,node_readme toneIndigo
```

## Components

### Runtime

Containerized Hadoop/Spark execution and environment setup.

- [Compose](https://github.com/dhai02/big-data-nyc/blob/main/docker-compose.yml): orchestrator · Primary container runtime for Hadoop, Spark, and notebook services. · `docker-compose.yml`
- [Image](https://github.com/dhai02/big-data-nyc/tree/main/docker/Dockerfile): container build · Custom Docker image for the Spark/Hadoop environment. · `docker/Dockerfile`
- [Hadoop env](https://github.com/dhai02/big-data-nyc/blob/main/docker/hadoop-client.xml): cluster config · Hadoop client and environment configuration used by the runtime. · `docker/hadoop-client.xml`
- [Cluster env](https://github.com/dhai02/big-data-nyc/blob/main/docker/hadoop.env): cluster env · Environment variables for the Hadoop/Spark stack. · `docker/hadoop.env`

### Pipeline

Bronze to Silver to Gold Spark data flow in HDFS.

- [Ingestion](https://github.com/dhai02/big-data-nyc/blob/main/src/ingestion.py): spark job · Entry point that lands raw taxi data into Bronze in HDFS. · `src/ingestion.py`
- [Cleaning](https://github.com/dhai02/big-data-nyc/blob/main/src/02_data_cleaning.py): spark job · Transforms Bronze data into cleaned and enriched Silver datasets. · `src/02_data_cleaning.py`
- [Rules](https://github.com/dhai02/big-data-nyc/blob/main/src/cleaning_utils.py): transform utils · Reusable cleaning and enrichment logic shared by the cleaning stage and tests. · `src/cleaning_utils.py`
- [Time agg](https://github.com/dhai02/big-data-nyc/blob/main/src/03_aggregation_time.py): spark job · Builds Gold summaries grouped by time dimensions. · `src/03_aggregation_time.py`
- [Vendor agg](https://github.com/dhai02/big-data-nyc/blob/main/src/04_aggregation_vendor.py): spark job · Builds Gold summaries grouped by vendor and business dimensions. · `src/04_aggregation_vendor.py`

### Exploration

EDA, visualization, notebooks, and sample outputs.

- [EDA](https://github.com/dhai02/big-data-nyc/blob/main/src/01_eda.py): analysis script · Exploratory analysis over curated pipeline outputs. · `src/01_eda.py`
- [Charts](https://github.com/dhai02/big-data-nyc/blob/main/src/06_visualization.py): visualization · Visualization entry point for Gold and demo outputs. · `src/06_visualization.py`
- [Notebooks](https://github.com/dhai02/big-data-nyc/blob/main/notebooks/01_eda.ipynb): notebook layer · Interactive analysis and demonstration notebooks. · `notebooks/01_eda.ipynb`
- [Samples](https://github.com/dhai02/big-data-nyc/blob/main/src/data/readme.md): parquet outputs · Local parquet artifacts and example analysis datasets. · `src/data/readme.md`

### Verification

Tests and result checks for transformation logic.

- [Verify](https://github.com/dhai02/big-data-nyc/blob/main/src/05_verify_results.py): quality check · Post-aggregation validation and result checking stage. · `src/05_verify_results.py`
- [Cleaning tests](https://github.com/dhai02/big-data-nyc/blob/main/tests/test_cleaning.py): test suite · Tests for cleaning transformations and utility rules. · `tests/test_cleaning.py`
- [Agg tests](https://github.com/dhai02/big-data-nyc/blob/main/tests/test_aggregation.py): test suite · Tests for aggregation outputs and business logic. · `tests/test_aggregation.py`

### Support

Docs and manual setup helpers.

- [Setup](https://github.com/dhai02/big-data-nyc/blob/main/setup/install_hadoop_spark.sh): install scripts · Manual Hadoop/Spark install and environment verification helpers. · `setup/install_hadoop_spark.sh`
- [Env check](https://github.com/dhai02/big-data-nyc/blob/main/setup/verify_env.sh): setup check · Validates local environment readiness for the stack. · `setup/verify_env.sh`
- [Run pipeline](https://github.com/dhai02/big-data-nyc/blob/main/run_pipeline.sh): runner script · Convenience script for executing the pipeline end to end. · `run_pipeline.sh`
- [README](https://github.com/dhai02/big-data-nyc/blob/main/README.md): docs · Primary project documentation and usage guide. · `README.md`

## Connections

- Compose → Image: builds · Compose uses the custom image for the stack.
- Compose → Hadoop env: loads · Runtime mounts Hadoop client configuration.
- Compose → Cluster env: configures · Runtime environment variables are injected through Compose.
- Ingestion → Cleaning: Bronze→Silver · Cleaned data is produced from raw Bronze parquet.
- Cleaning → Rules: uses · Cleaning job delegates reusable transformation logic.
- Cleaning → Time agg: Silver→Gold · Time aggregations consume cleaned Silver data.
- Cleaning → Vendor agg: Silver→Gold · Vendor aggregations consume cleaned Silver data.
- Time agg → Verify: checks · Verification validates time-based aggregates.
- Vendor agg → Verify: checks · Verification validates vendor-based aggregates.
- Cleaning → Cleaning tests: covered by · Cleaning behavior is validated by targeted tests.
- Rules → Cleaning tests: covered by · Utility rules are tested directly.
- Time agg → Agg tests: covered by · Aggregation outputs are tested.
- Vendor agg → Agg tests: covered by · Aggregation outputs are tested.
- Cleaning → EDA: feeds · EDA consumes curated cleaned data.
- Time agg → Charts: feeds · Visualization consumes time-based Gold outputs.
- Vendor agg → Charts: feeds · Visualization consumes vendor-based Gold outputs.
- Cleaning → Notebooks: explored in · Notebooks demonstrate cleaning and data inspection.
- Time agg → Notebooks: explored in · Notebooks use aggregated outputs for presentation.
- Vendor agg → Notebooks: explored in · Notebooks use aggregated outputs for presentation.
- Time agg → Samples: exports · Representative parquet outputs are stored for local use.
- Vendor agg → Samples: exports · Representative parquet outputs are stored for local use.
- Run pipeline → Ingestion: runs · Pipeline script orchestrates the ETL stages.
- Setup → Env check: supports · Install and environment check scripts are paired helpers.
- README → Compose: documents · Documentation describes the containerized execution path.
- README → Run pipeline: documents · Documentation describes local execution of the pipeline.
