How nguyenlehoaitam/topic120-distributeddatabase works
This is a simulated three-stage weather-event streaming and alert pipeline, with batch runners for 30- and 60-second windows and an interactive dashboard. The primary workflow reads generated sensor CSV events, cleans and transforms them in separate processes, then applies event-time lateness handling and county-window aggregation before reporting results. Checkpointing and a crash demonstration are supporting capabilities. The runner and node wiring is documented and partly sampled; dashboard and failure-demo details rely on the README and partial excerpts.
Main entry points: main_30s.py 30s pipeline runner · main_60s.py 60s pipeline runner · failure_sim.py Crash simulation · node_aggregate.py County aggregation
Components 18
Data input
- Dataset generator
src/generator.py - Weather events CSV
data/weather_sensors.csv
Stream processing
- 30s pipeline runner
src/main_30s.py - 60s pipeline runner
src/main_60s.py - Clean and deduplicate
src/node_clean.py - Unit transform
src/node_transform.py - County aggregation
src/node_aggregate.py - Tumbling windows
src/windowing.py - Late-event policy
src/watermark.py
State and recovery
- Checkpoint manager
src/state_manager.py - Node checkpoints
- Crash simulation
src/failure_sim.py
Reporting and dashboard
- 30s report
results/report_30s.txt - 60s report
results/report_60s.txt - Interactive dashboard
dashboard.py - Latency metrics
src/metrics.py - Failure log
results/failure_log.txt
Other components
- Operator
Connections 29
- Operator → Dataset generator: starts generation
- Dataset generator → Weather events CSV: writes events
- Operator → 30s pipeline runner: starts run
- Operator → 60s pipeline runner: starts run
- 30s pipeline runner → Weather events CSV: reads events
- 60s pipeline runner → Weather events CSV: reads events
- 30s pipeline runner → Clean and deduplicate: dispatches events
- 60s pipeline runner → Clean and deduplicate: dispatches events
- Clean and deduplicate → Unit transform: queues clean events
- Unit transform → County aggregation: queues transformed events
- County aggregation → Tumbling windows: adds events
- County aggregation → Late-event policy: checks lateness
- Clean and deduplicate → Checkpoint manager: saves and loads
- Unit transform → Checkpoint manager: saves and loads
- County aggregation → Checkpoint manager: saves and loads
- Checkpoint manager → Node checkpoints: reads and writes
- 30s pipeline runner → 30s report: writes report
- 60s pipeline runner → 60s report: writes report
- Operator → Interactive dashboard: sets parameters
- Interactive dashboard → Weather events CSV: reads events
- Interactive dashboard → Tumbling windows: aggregates events
- Interactive dashboard → Late-event policy: applies lateness
- 30s pipeline runner → Latency metrics: records timings
- 60s pipeline runner → Latency metrics: records timings
- Operator → Crash simulation: starts demo
- Crash simulation → Clean and deduplicate: starts node
- Crash simulation → Unit transform: simulates crash
- Crash simulation → County aggregation: starts resilient node
- Crash simulation → Failure log: writes log