How dhai02/big-data-nyc works
1. What this repo is
- This is a batch data pipeline for NYC taxi data built around Apache Spark + HDFS.
- The architecture follows a Bronze → Silver → Gold pattern: raw data lands in HDFS, is cleaned/enriched, then aggregated into analysis-ready datasets.
2. Execution environment
- The primary runtime is Docker Compose, which brings up the Hadoop/HDFS and Spark stack plus a Jupyter environment.
- The repo also includes a manual WSL2/Hadoop/Spark install path, but that is clearly secondary to the containerized flow.
- Key infrastructure files are docker-compose.yml, docker/Dockerfile, and the Hadoop config/env files under docker/.
3. Ingestion boundary: external data → Bronze
- src/ingestion.py is the entry point for pulling NYC taxi source data into the pipeline.
- Its role is to place raw, untouched parquet into HDFS under /nyc_taxi/bronze/raw/.
- This is the first hard boundary: data becomes part of the managed pipeline only after landing in HDFS.
4. Cleaning/enrichment boundary: Bronze → Silver
- src/02_data_cleaning.py transforms Bronze data into the Silver layer.
- src/cleaning_utils.py holds the reusable cleaning logic that the tests target directly.
- The Silver output is not just cleaned; it is also enriched with derived fields such as trip duration, pickup hour/date/day-of-week, tip percentage, and speed.
5. Gold layer: analytical aggregates
- src/03_aggregation_time.py produces time-based summaries in /nyc_taxi/gold/aggregated_by_time/.
- src/04_aggregation_vendor.py produces vendor/business summaries in /nyc_taxi/gold/aggregated_by_vendor/.
- The README shows these Gold outputs are partitioned into analysis-friendly datasets like by hour, by date, by day-of-week, by hour/day-of-week, by vendor, by payment method, top pickup locations, and top routes.
6. Verification stage
- src/05_verify_results.py is a dedicated quality/check step after aggregation.
- This suggests the pipeline treats correctness as an explicit stage, not just an afterthought.
- The presence of verification scripts and tests indicates a clear separation between transformation logic and validation.
7. Exploration and presentation layer
- src/01_eda.py and src/06_visualization.py support exploratory analysis and charting, not core ETL.
- The notebooks (notebooks/01_eda.ipynb, 02_cleaning_demo.ipynb, 03_aggregation_viz.ipynb) are the demo/analysis boundary for human inspection and assignment deliverables.
- In architecture terms, these sit downstream of the pipeline and consume Silver/Gold outputs rather than raw data.
8. Data organization in HDFS
- The HDFS root is /nyc_taxi/ with clear layer separation:
- bronze/raw/
- silver/cleaned/
- gold/aggregated_by_time/
- gold/aggregated_by_vendor/
- This layout is central to the repo’s architecture because each script writes to one layer and the next step reads from it.
9. Local artifacts and sample outputs
- src/data/ contains parquet files that look like saved outputs or reference datasets for local use/documentation.
- The README also describes copying data from HDFS to /app/data, which is mounted back to the host.
- This creates a bridge between cluster storage and local inspection without changing the pipeline itself.
10. Testing boundary
- tests/test_cleaning.py, tests/test_cleaning_utils.py, and tests/test_aggregation.py provide coverage for transformation logic.
- The tests focus on business logic correctness around cleaning and aggregation rather than deployment.
- This implies the core architecture is transformation-heavy, with tests anchored around deterministic Spark/DataFrame behavior.
11. Supporting scripts and docs
- run_pipeline.sh and the scripts under setup/ support local/manual execution and environment verification.
- docs/ contains assignment-specific documentation, which appears to support reporting rather than runtime behavior.
- These are operational aids around the pipeline, not part of the main data path.
12. High-level data flow
- The main flow is: source data ingestion → HDFS Bronze parquet → Spark cleaning/enrichment → HDFS Silver parquet → Spark aggregations → HDFS Gold parquet → verification/EDA/visualization.
- The key architectural constraint is that Spark jobs read and write through HDFS, so storage and compute are decoupled.
- Jupyter and notebooks consume the curated Silver/Gold outputs for analysis and presentation, not raw ingestion.
Main entry points: 02_data_cleaning.py Cleaning · 03_aggregation_time.py Time agg · 04_aggregation_vendor.py Vendor agg · docker-compose.yml Compose
Components 20
Runtime
Containerized Hadoop/Spark execution and environment setup.
- Compose · Primary container runtime for Hadoop, Spark, and notebook services.
docker-compose.yml - Image · Custom Docker image for the Spark/Hadoop environment.
docker/Dockerfile - Hadoop env · Hadoop client and environment configuration used by the runtime.
docker/hadoop-client.xml - Cluster env · Environment variables for the Hadoop/Spark stack.
docker/hadoop.env
Pipeline
Bronze to Silver to Gold Spark data flow in HDFS.
- Ingestion · Entry point that lands raw taxi data into Bronze in HDFS.
src/ingestion.py - Cleaning · Transforms Bronze data into cleaned and enriched Silver datasets.
src/02_data_cleaning.py - Rules · Reusable cleaning and enrichment logic shared by the cleaning stage and tests.
src/cleaning_utils.py - Time agg · Builds Gold summaries grouped by time dimensions.
src/03_aggregation_time.py - Vendor agg · Builds Gold summaries grouped by vendor and business dimensions.
src/04_aggregation_vendor.py
Exploration
EDA, visualization, notebooks, and sample outputs.
- EDA · Exploratory analysis over curated pipeline outputs.
src/01_eda.py - Charts · Visualization entry point for Gold and demo outputs.
src/06_visualization.py - Notebooks · Interactive analysis and demonstration notebooks.
notebooks/01_eda.ipynb - Samples · Local parquet artifacts and example analysis datasets.
src/data/readme.md
Verification
Tests and result checks for transformation logic.
- Verify · Post-aggregation validation and result checking stage.
src/05_verify_results.py - Cleaning tests · Tests for cleaning transformations and utility rules.
tests/test_cleaning.py - Agg tests · Tests for aggregation outputs and business logic.
tests/test_aggregation.py
Support
Docs and manual setup helpers.
- Setup · Manual Hadoop/Spark install and environment verification helpers.
setup/install_hadoop_spark.sh - Env check · Validates local environment readiness for the stack.
setup/verify_env.sh - Run pipeline · Convenience script for executing the pipeline end to end.
run_pipeline.sh - README · Primary project documentation and usage guide.
README.md
Connections 25
- Compose → Image: builds · Compose uses the custom image for the stack.
- Compose → Hadoop env: loads · Runtime mounts Hadoop client configuration.
- Compose → Cluster env: configures · Runtime environment variables are injected through Compose.
- Ingestion → Cleaning: Bronze→Silver · Cleaned data is produced from raw Bronze parquet.
- Cleaning → Rules: uses · Cleaning job delegates reusable transformation logic.
- Cleaning → Time agg: Silver→Gold · Time aggregations consume cleaned Silver data.
- Cleaning → Vendor agg: Silver→Gold · Vendor aggregations consume cleaned Silver data.
- Time agg → Verify: checks · Verification validates time-based aggregates.
- Vendor agg → Verify: checks · Verification validates vendor-based aggregates.
- Cleaning → Cleaning tests: covered by · Cleaning behavior is validated by targeted tests.
- Rules → Cleaning tests: covered by · Utility rules are tested directly.
- Time agg → Agg tests: covered by · Aggregation outputs are tested.
- Vendor agg → Agg tests: covered by · Aggregation outputs are tested.
- Cleaning → EDA: feeds · EDA consumes curated cleaned data.
- Time agg → Charts: feeds · Visualization consumes time-based Gold outputs.
- Vendor agg → Charts: feeds · Visualization consumes vendor-based Gold outputs.
- Cleaning → Notebooks: explored in · Notebooks demonstrate cleaning and data inspection.
- Time agg → Notebooks: explored in · Notebooks use aggregated outputs for presentation.
- Vendor agg → Notebooks: explored in · Notebooks use aggregated outputs for presentation.
- Time agg → Samples: exports · Representative parquet outputs are stored for local use.
- Vendor agg → Samples: exports · Representative parquet outputs are stored for local use.
- Run pipeline → Ingestion: runs · Pipeline script orchestrates the ETL stages.
- Setup → Env check: supports · Install and environment check scripts are paired helpers.
- README → Compose: documents · Documentation describes the containerized execution path.
- README → Run pipeline: documents · Documentation describes local execution of the pipeline.