# ciskander2/citi-bike-kafka-pipeline architecture

> Architecture diagram of the GitHub repository ciskander2/citi-bike-kafka-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ciskander2/citi-bike-kafka-pipeline
- Repository: https://github.com/ciskander2/citi-bike-kafka-pipeline

## Overview

The repository implements a Citi Bike streaming analytics pipeline: GBFS station updates are published to Kafka, consumed in batches into S3 and PostgreSQL, optionally transformed into Parquet with Spark, modeled with dbt, and queried by a Streamlit dashboard. The main streaming and dashboard paths are supported by excerpts; dbt behavior and several pipeline links are documented in the README, while model internals were not sampled.

Main entry points: [`dashboard/app.py`](https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/dashboard/app.py) (Streamlit dashboard), [`citibike_dbt/models/staging`](https://github.com/ciskander2/citi-bike-kafka-pipeline/tree/main/citibike_dbt/models/staging) (Staging models), [`src/simple_consumer.py`](https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/src/simple_consumer.py) (Batched consumer), [`src/citibike_producer.py`](https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/src/citibike_producer.py) (Kafka producer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Event ingestion"]
  node_producer["Kafka producer"]
  node_kafka["Station status topic"]
  node_consumer["Batched consumer<br/>[simple_consumer.py]"]
end

subgraph group_storage["Data storage"]
  node_s3raw[("Raw JSONL lake")]
  node_postgres[("Postgres warehouse")]
  node_s3parquet[("Parquet outputs")]
end

subgraph group_transform["Analytics transforms"]
  node_spark["Spark transform"]
  node_dbt["Staging models"]
  node_top20["Top stations"]
  node_no_docking["No docking analysis"]
  node_capacity["Capacity analysis"]
  node_hourly["Hourly status"]
  node_utilization["Utilization ratio"]
end

subgraph group_presentation["Analytics dashboard"]
  node_dashboard["Streamlit dashboard<br/>[app.py]"]
end

node_gbfs(("GBFS feed"))

node_gbfs -->|"provides updates"| node_producer
node_producer -->|"publishes events"| node_kafka
node_kafka -->|"delivers events"| node_consumer
node_consumer -->|"writes batches"| node_s3raw
node_consumer -->|"inserts batches"| node_postgres
node_s3raw -->|"reads JSONL"| node_spark
node_spark -->|"writes Parquet"| node_s3parquet
node_postgres -->|"models warehouse data"| node_dbt
node_dbt -.->|"includes"| node_top20
node_dbt -.->|"includes"| node_no_docking
node_dbt -.->|"includes"| node_capacity
node_dbt -.->|"includes"| node_hourly
node_dbt -.->|"includes"| node_utilization
node_dashboard -->|"queries metrics"| node_postgres
node_dashboard -->|"queries model"| node_top20
node_dashboard -->|"queries model"| node_no_docking
node_dashboard -->|"queries model"| node_capacity
node_dashboard -->|"queries model"| node_hourly
node_dashboard -->|"queries model"| node_utilization

click node_producer "https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/src/citibike_producer.py"
click node_consumer "https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/src/simple_consumer.py"
click node_spark "https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/src/spark_transformation.py"
click node_dbt "https://github.com/ciskander2/citi-bike-kafka-pipeline/tree/main/citibike_dbt/models/staging"
click node_dashboard "https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/dashboard/app.py"
click node_top20 "https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/citibike_dbt/models/staging/top_20_stations.sql"
click node_no_docking "https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/citibike_dbt/models/staging/no_docking_but_availability.sql"
click node_capacity "https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/citibike_dbt/models/staging/capacity_zero_availability.sql"
click node_hourly "https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/citibike_dbt/models/staging/hourly_bike_status.sql"
click node_utilization "https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/citibike_dbt/models/staging/utlilization_ratio.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer,node_kafka,node_consumer toneBlue
class node_s3raw,node_postgres,node_s3parquet toneAmber
class node_spark,node_dbt,node_top20,node_no_docking,node_capacity,node_hourly,node_utilization toneMint
class node_dashboard toneRose
class node_gbfs toneIndigo
```

## Components

### Event ingestion

- [Kafka producer](https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/src/citibike_producer.py): `src/citibike_producer.py`
- **Station status topic**
- [Batched consumer](https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/src/simple_consumer.py): `src/simple_consumer.py`

### Data storage

- **Raw JSONL lake**
- **Postgres warehouse**
- **Parquet outputs**

### Analytics transforms

- [Spark transform](https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/src/spark_transformation.py): `src/spark_transformation.py`
- [Staging models](https://github.com/ciskander2/citi-bike-kafka-pipeline/tree/main/citibike_dbt/models/staging): `citibike_dbt/models/staging`
- [Top stations](https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/citibike_dbt/models/staging/top_20_stations.sql): `citibike_dbt/models/staging/top_20_stations.sql`
- [No docking analysis](https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/citibike_dbt/models/staging/no_docking_but_availability.sql): `citibike_dbt/models/staging/no_docking_but_availability.sql`
- [Capacity analysis](https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/citibike_dbt/models/staging/capacity_zero_availability.sql): `citibike_dbt/models/staging/capacity_zero_availability.sql`
- [Hourly status](https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/citibike_dbt/models/staging/hourly_bike_status.sql): `citibike_dbt/models/staging/hourly_bike_status.sql`
- [Utilization ratio](https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/citibike_dbt/models/staging/utlilization_ratio.sql): `citibike_dbt/models/staging/utlilization_ratio.sql`

### Analytics dashboard

- [Streamlit dashboard](https://github.com/ciskander2/citi-bike-kafka-pipeline/blob/main/dashboard/app.py): `dashboard/app.py`

### Other components

- **GBFS feed**

## Connections

- GBFS feed → Kafka producer: provides updates (evidence: `README.md`)
- Kafka producer → Station status topic: publishes events (evidence: `src/citibike_producer.py`)
- Station status topic → Batched consumer: delivers events (evidence: `README.md`)
- Batched consumer → Raw JSONL lake: writes batches (evidence: `src/simple_consumer.py`)
- Batched consumer → Postgres warehouse: inserts batches (evidence: `src/simple_consumer.py`)
- Raw JSONL lake → Spark transform: reads JSONL (evidence: `src/spark_transformation.py`)
- Spark transform → Parquet outputs: writes Parquet (evidence: `src/spark_transformation.py`)
- Postgres warehouse → Staging models: models warehouse data (evidence: `README.md`)
- Staging models → Top stations: includes
- Staging models → No docking analysis: includes
- Staging models → Capacity analysis: includes
- Staging models → Hourly status: includes
- Staging models → Utilization ratio: includes
- Streamlit dashboard → Postgres warehouse: queries metrics (evidence: `dashboard/app.py`)
- Streamlit dashboard → Top stations: queries model (evidence: `dashboard/app.py`)
- Streamlit dashboard → No docking analysis: queries model (evidence: `dashboard/app.py`)
- Streamlit dashboard → Capacity analysis: queries model (evidence: `dashboard/app.py`)
- Streamlit dashboard → Hourly status: queries model (evidence: `dashboard/app.py`)
- Streamlit dashboard → Utilization ratio: queries model (evidence: `dashboard/app.py`)
