How ahmed-bs11/cycling-stations-monitoring works
The product collects live Velib station data, streams it through Kafka and Spark, and exposes station status through Elasticsearch/Kibana, Hive, and empty-station reporting. The graph separates collection, stream processing, and analytics/monitoring. Source excerpts support the producer, Spark stream, sinks, and monitor; Kibana’s visualization role is documented in the README, but its data connection is not shown in source.
Main entry points: pyspark_consumer.py Spark consumer · Stations_producer.py Station producer · monitor_stations.py Station monitor
Components 13
Data collection
- Station producer
Stations_producer.py - Kafka topic
Stream processing
- Spark consumer
pyspark_consumer.py - JSON parsing
pyspark_consumer.py - Empty station filter
pyspark_consumer.py - Console alerts
pyspark_consumer.py - Batch indexing callback
pyspark_consumer.py
Analytics and monitoring
- Elasticsearch index
- Hive station table
- Hive record check
pyspark_consumer.py - Kibana dashboards
- Station monitor
monitor_stations.py
Other components
- JCDecaux API
Connections 13
- Station producer → JCDecaux API: fetches stations
Stations_producer.py - Station producer → Kafka topic: publishes records
Stations_producer.py - Spark consumer → Kafka topic: reads topic
pyspark_consumer.py - Spark consumer → JSON parsing: configures stream
pyspark_consumer.py - JSON parsing → Empty station filter: filters empty
pyspark_consumer.py - Empty station filter → Console alerts: writes alerts
pyspark_consumer.py - JSON parsing → Batch indexing callback: writes batches
pyspark_consumer.py - Batch indexing callback → Elasticsearch index: indexes records
pyspark_consumer.py - Batch indexing callback → Hive station table: appends rows
pyspark_consumer.py - Spark consumer → Hive record check: queries table
pyspark_consumer.py - Hive station table → Hive record check: provides records
pyspark_consumer.py - Station monitor → Kafka topic: consumes topic
monitor_stations.py - Elasticsearch index → Kibana dashboards: visualizes data
README.md