How nalexx6/diploma works
This repository implements a NYC taxi trip analytics pipeline: trips can be simulated into Kafka, consumed and normalized by Spark streaming into Parquet, then aggregated by a Spark batch job and written to PostgreSQL. An EMR step manager launches and monitors the Spark jobs. A separate pandas application performs local trip analyses. The README is empty, so capabilities and wiring here are based on the sampled source and file tree; configuration details and unsampled infrastructure behavior are not inferred.
Main entry points: batch.py Spark batch job · streaming.py Spark streaming job · simulation.py Trip simulator · __init__.py Spark utilities
Components 16
Job orchestration
- EMR step manager
emr_steps_execution/steps_manager.py
Trip ingestion
- Spark streaming job
spark_apps/streaming.py - Trip simulator
spark_apps/simulation.py - Kafka topics
Trip analytics
- Spark batch job
spark_apps/batch.py - Spark utilities
spark_apps/utils/__init__.py - Trip metrics
spark_apps/metrics.py - Spark configuration
spark_apps/config.yaml - Taxi metadata
nyc_taxi_metadata - Local analysis app
python_apps/main.py - Pandas analyses
python_apps/functions.py
Data outputs
- Trip Parquet data
- PostgreSQL metrics
- Local Parquet files
Other components
- Operator
- Amazon EMR
Connections 20
- Operator → EMR step manager: starts jobs
- EMR step manager → Amazon EMR: submits and monitors
- Amazon EMR → Spark batch job: runs submitted job
- Amazon EMR → Spark streaming job: runs submitted job
- Trip simulator → Spark utilities: loads config and session
- Trip simulator → Trip Parquet data: reads trips
- Trip simulator → Kafka topics: publishes trips
- Spark streaming job → Spark utilities: loads config and session
- Spark streaming job → Kafka topics: reads trip events
- Spark streaming job → Trip Parquet data: appends trip data
- Spark batch job → Spark utilities: loads data and saves metrics
- Spark batch job → Trip metrics: calls aggregations
- Spark batch job → Trip Parquet data: reads trip data
- Spark batch job → Taxi metadata: reads licenses and zones
- Local analysis app → Local Parquet files: reads trip files
- Local analysis app → Pandas analyses: calls analyses
- Spark utilities → PostgreSQL metrics: writes metric tables
- Spark batch job → Spark configuration: loads configuration
- Spark streaming job → Spark configuration: loads configuration
- Trip simulator → Spark configuration: loads configuration