How ram200509/weather-mapreduce works
This project generates synthetic daily weather records, computes city/year temperatures through a local mapper–sort–reducer job, and computes rainfall, summer-temperature, and heat analytics. The Streamlit dashboard reads the dataset and generated CSVs, displays city-focused trends and metrics, and fits a per-city linear regression for 2027. The README documents these capabilities; source excerpts show the Python MapReduce and extra-analytics implementations, while Spark is described as optional/unused. Edges reflect only relationships visible in the excerpts or README.
Main entry points: app.py Streamlit dashboard · run_jobs.py Job runner · spark_jobs.py Spark analytics · weather_data.csv Weather records
Components 15
Weather data
- Synthetic generator
data/generate_weather_data.py - Weather records
data/weather_data.csv
Analytics processing
- Job runner
mapreduce/run_jobs.py - City-year mapper
mapreduce/mapper.py - Shuffle and sort
- Temperature reducer
mapreduce/reducer.py - Extra analytics
mapreduce/extra_jobs.py - Spark analytics
mapreduce/spark_jobs.py
Analytics results
- Yearly averages
output/avg_temp_city_year.csv - Monthly rainfall
output/rainfall_city_month.csv - Summer temperatures
output/summer_temp_city.csv - Extreme heat counts
output/extreme_heat_days.csv
Dashboard
- Streamlit dashboard
dashboard/app.py - 2027 regression forecast
dashboard/app.py
Other components
- Dashboard user
Connections 25
- Synthetic generator → Weather records: writes records
data/generate_weather_data.py - Job runner → City-year mapper: invokes
mapreduce/run_jobs.py - Job runner → Shuffle and sort: pipes through
mapreduce/run_jobs.py - Job runner → Temperature reducer: invokes
mapreduce/run_jobs.py - Job runner → Weather records: reads input
mapreduce/run_jobs.py - Job runner → Yearly averages: writes averages
mapreduce/run_jobs.py - City-year mapper → Shuffle and sort: emits keyed rows
README.md - Shuffle and sort → Temperature reducer: groups keys
README.md - Temperature reducer → Yearly averages: produces averages
README.md - Extra analytics → Weather records: reads records
mapreduce/extra_jobs.py - Extra analytics → Monthly rainfall: writes totals
mapreduce/extra_jobs.py - Extra analytics → Summer temperatures: writes averages
mapreduce/extra_jobs.py - Extra analytics → Extreme heat counts: writes counts
mapreduce/extra_jobs.py - Spark analytics → Weather records: reads records
mapreduce/spark_jobs.py - Spark analytics → Yearly averages: writes averages
mapreduce/spark_jobs.py - Spark analytics → Monthly rainfall: writes totals
mapreduce/spark_jobs.py - Spark analytics → Summer temperatures: writes averages
mapreduce/spark_jobs.py - Spark analytics → Extreme heat counts: writes counts
mapreduce/spark_jobs.py - Dashboard user → Streamlit dashboard: views
README.md - Streamlit dashboard → Weather records: reads records
dashboard/app.py - Streamlit dashboard → Yearly averages: reads averages
dashboard/app.py - Streamlit dashboard → Monthly rainfall: reads rainfall
dashboard/app.py - Streamlit dashboard → Summer temperatures: reads summer data
dashboard/app.py - Streamlit dashboard → Extreme heat counts: reads heat counts
dashboard/app.py - Streamlit dashboard → 2027 regression forecast: fits model
dashboard/app.py