# ahmed-bs11/cycling-stations-monitoring architecture

> Architecture diagram of the GitHub repository ahmed-bs11/cycling-stations-monitoring, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ahmed-bs11/cycling-stations-monitoring
- Repository: https://github.com/ahmed-bs11/cycling-stations-monitoring

## Overview

The product collects live Velib station data, streams it through Kafka and Spark, and exposes station status through Elasticsearch/Kibana, Hive, and empty-station reporting. The graph separates collection, stream processing, and analytics/monitoring. Source excerpts support the producer, Spark stream, sinks, and monitor; Kibana’s visualization role is documented in the README, but its data connection is not shown in source.

Main entry points: [`pyspark_consumer.py`](https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py) (Spark consumer), [`Stations_producer.py`](https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/Stations_producer.py) (Station producer), [`monitor_stations.py`](https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/monitor_stations.py) (Station monitor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Data collection"]
  node_producer["Station producer"]
  node_kafka["Kafka topic"]
end

subgraph group_streaming["Stream processing"]
  node_spark["Spark consumer"]
  node_parse["JSON parsing"]
  node_empty["Empty station filter"]
  node_console["Console alerts"]
  node_batch["Batch indexing callback"]
end

subgraph group_analytics["Analytics and monitoring"]
  node_elastic[("Elasticsearch index")]
  node_hive[("Hive station table")]
  node_hivecheck["Hive record check"]
  node_kibana["Kibana dashboards"]
  node_monitor["Station monitor"]
end

node_jcdecaux(("JCDecaux API"))

node_producer -->|"fetches stations"| node_jcdecaux
node_producer -->|"publishes records"| node_kafka
node_spark -->|"reads topic"| node_kafka
node_spark -->|"configures stream"| node_parse
node_parse -->|"filters empty"| node_empty
node_empty -->|"writes alerts"| node_console
node_parse -->|"writes batches"| node_batch
node_batch -->|"indexes records"| node_elastic
node_batch -->|"appends rows"| node_hive
node_spark -->|"queries table"| node_hivecheck
node_hive -->|"provides records"| node_hivecheck
node_monitor -->|"consumes topic"| node_kafka
node_elastic -.->|"visualizes data"| node_kibana

click node_producer "https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/Stations_producer.py"
click node_spark "https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py"
click node_parse "https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py"
click node_empty "https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py"
click node_console "https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py"
click node_batch "https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py"
click node_hivecheck "https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py"
click node_monitor "https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/monitor_stations.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer,node_kafka toneBlue
class node_spark,node_parse,node_empty,node_console,node_batch toneAmber
class node_elastic,node_hive,node_hivecheck,node_kibana,node_monitor,node_jcdecaux toneMint
```

## Components

### Data collection

- [Station producer](https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/Stations_producer.py): `Stations_producer.py`
- **Kafka topic**

### Stream processing

- [Spark consumer](https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py): `pyspark_consumer.py`
- [JSON parsing](https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py): `pyspark_consumer.py`
- [Empty station filter](https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py): `pyspark_consumer.py`
- [Console alerts](https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py): `pyspark_consumer.py`
- [Batch indexing callback](https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py): `pyspark_consumer.py`

### Analytics and monitoring

- **Elasticsearch index**
- **Hive station table**
- [Hive record check](https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/pyspark_consumer.py): `pyspark_consumer.py`
- **Kibana dashboards**
- [Station monitor](https://github.com/ahmed-bs11/cycling-stations-monitoring/blob/main/monitor_stations.py): `monitor_stations.py`

### Other components

- **JCDecaux API**

## Connections

- Station producer → JCDecaux API: fetches stations (evidence: `Stations_producer.py`)
- Station producer → Kafka topic: publishes records (evidence: `Stations_producer.py`)
- Spark consumer → Kafka topic: reads topic (evidence: `pyspark_consumer.py`)
- Spark consumer → JSON parsing: configures stream (evidence: `pyspark_consumer.py`)
- JSON parsing → Empty station filter: filters empty (evidence: `pyspark_consumer.py`)
- Empty station filter → Console alerts: writes alerts (evidence: `pyspark_consumer.py`)
- JSON parsing → Batch indexing callback: writes batches (evidence: `pyspark_consumer.py`)
- Batch indexing callback → Elasticsearch index: indexes records (evidence: `pyspark_consumer.py`)
- Batch indexing callback → Hive station table: appends rows (evidence: `pyspark_consumer.py`)
- Spark consumer → Hive record check: queries table (evidence: `pyspark_consumer.py`)
- Hive station table → Hive record check: provides records (evidence: `pyspark_consumer.py`)
- Station monitor → Kafka topic: consumes topic (evidence: `monitor_stations.py`)
- Elasticsearch index → Kibana dashboards: visualizes data (evidence: `README.md`)