# san089/optimizing-public-transportation architecture

> Architecture diagram of the GitHub repository san089/optimizing-public-transportation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/san089/optimizing-public-transportation
- Repository: https://github.com/san089/optimizing-public-transportation

## Overview

##### Main Components
- Producers: Create events related to train arrivals, turnstile usage, and weather.
- In the producers/ directory, note files like simulation.py, connector.py, and the models (including schemas) which indicate how messages are structured.
- That producers ingest data from multiple sources (CSV files in producers/data and startup) and publish them to Kafka topics.
- Consumers: Process and consume the events from Kafka.
- In the consumers/ directory, there are multiple consumers:
◦ consumer.py – a core consumer handling events,
◦ faust_stream – using Faust for stream processing,
◦ ksql.py – a script likely using ksql for querying stream data.
- The server.py and associated templates (status.html) indicate a UI component that displays real-time status.
- Data Sources/Initiation: The startup/ directory includes initialization scripts and CSV datasets that seed the simulation with additional data.
- External Systems and Infrastructure:
- Apache Kafka (and its ecosystem such as REST Proxy and Kafka Connect) acts as the backbone.
- Postgres SQL is mentioned in the README for station data extraction via Kafka Connect.
- Docker and Docker Compose are used to orchestrate the services as indicated in the README.

##### Relationships and Data Flow
- A Kafka component in the center. Label it clearly to indicate it is the event pipeline.
- Kafka to the Consumers, representing how different consumer types (Faust stream processing, KSQL, and the UI server) read from Kafka topics.
- External data flows where Postgres (accessed via Kafka Connect) is providing additional station information to Kafka.
- Optionally, indicate that Docker Compose coordinates the startup and interaction of these services.

##### Architectural Patterns / Design Principles
- Streaming architecture – using publish/subscribe messaging patterns between producers and consumers.
- That the design is modular – splitting producers and consumers into separate directories implies loosely coupled components.
- If applicable, mention that Faust is used for stream processing, while KSQL offers a SQL-like interface to Kafka for querying.
- The separation of simulation logic and configuration (such as logging.ini files) also suggests attention to decoupling concerns.

##### Relevant Technologies
- Apache Kafka (with related tools like REST Proxy and Kafka Connect)
- Python (the language chosen, with separate virtual environments for producers and consumers)
- Docker & Docker Compose (for service orchestration)
- Postgres (for station data integration)
- Faust (for stream processing) and KSQL (for stream querying)

##### Additional Details
- That every component from the project (including supporting folders like startup and docs) is represented if relevant. For instance, the startup/ folder might be summarized as an initialization or seeding process feeding into the simulation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Docker Orchestrator
    DC("Docker Compose Orchestrator"):::docker

    %% Data Sources / Initialization Subgraph
    subgraph "Data Sources / Initialization"
        DS1("Startup Stream (stream.py)"):::datasource
        DS2("Startup Clicks (clicks.py)"):::datasource
        DS3("CTA Stations CSV"):::datasource
    end

    %% Producers Subgraph
    subgraph "Producers"
        P1("Simulation (simulation.py)"):::producers
        P2("Connector (connector.py)"):::producers
        P3("Models (schemas)"):::producers
        P4("Logging (producers)"):::producers
    end

    %% Kafka Cluster
    K("Apache Kafka (Cluster)"):::kafka

    %% Consumers Subgraph
    subgraph "Consumers"
        C1("Consumer (consumer.py)"):::consumers
        C2("Faust Stream (faust_stream.py)"):::consumers
        C3("KSQL Query (ksql.py)"):::consumers
        C4("Models (Consumers)"):::consumers
        C5("UI Server (server.py)"):::consumers
        C6("Status UI (status.html)"):::consumers
        C7("Logging (consumers)"):::consumers
    end

    %% External Postgres
    PG("Postgres Database"):::external

    %% Connections from Docker Compose
    DC --> DS1
    DC --> DS2
    DC --> DS3
    DC --> P1
    DC --> P2
    DC --> P3
    DC --> P4
    DC --> K
    DC --> PG
    DC --> C1
    DC --> C2
    DC --> C3
    DC --> C4
    DC --> C5
    DC --> C6
    DC --> C7

    %% Data flow from Data Sources to Producers
    DS1 -->|"seed-data"| P1
    DS2 -->|"seed-data"| P1
    DS3 -->|"seed-data"| P1

    %% Producers publish events to Kafka Cluster
    P1 -->|"publishes"| K
    P2 -->|"publishes"| K
    P3 -->|"schemas"| K

    %% External Postgres feeds data via Kafka Connect into Kafka Cluster
    PG -->|"KafkaConnect"| K

    %% Kafka Cluster forwards messages to Consumers
    K -->|"consumed-by"| C1
    K -->|"consumed-by"| C2
    K -->|"consumed-by"| C3
    K -->|"updates"| C5

    %% UI components interaction
    C5 -->|"renders"| C6

    %% Styling
    classDef producers fill:#aaddff,stroke:#333,stroke-width:2px;
    classDef consumers fill:#aaffaa,stroke:#333,stroke-width:2px;
    classDef kafka fill:#ffcc00,stroke:#333,stroke-width:2px;
    classDef datasource fill:#dda0dd,stroke:#333,stroke-width:2px;
    classDef docker fill:#ff9933,stroke:#333,stroke-width:2px;
    classDef external fill:#cccccc,stroke:#333,stroke-width:2px;

    %% Click Events for Producers
    click P1 "https://github.com/san089/optimizing-public-transportation/blob/master/producers/simulation.py"
    click P2 "https://github.com/san089/optimizing-public-transportation/blob/master/producers/connector.py"
    click P3 "https://github.com/san089/optimizing-public-transportation/tree/master/producers/models"
    click P4 "https://github.com/san089/optimizing-public-transportation/blob/master/producers/logging.ini"

    %% Click Events for Consumers
    click C1 "https://github.com/san089/optimizing-public-transportation/blob/master/consumers/consumer.py"
    click C2 "https://github.com/san089/optimizing-public-transportation/blob/master/consumers/faust_stream.py"
    click C3 "https://github.com/san089/optimizing-public-transportation/blob/master/consumers/ksql.py"
    click C4 "https://github.com/san089/optimizing-public-transportation/tree/master/consumers/models"
    click C5 "https://github.com/san089/optimizing-public-transportation/blob/master/consumers/server.py"
    click C6 "https://github.com/san089/optimizing-public-transportation/blob/master/consumers/templates/status.html"
    click C7 "https://github.com/san089/optimizing-public-transportation/blob/master/consumers/logging.ini"

    %% Click Events for Data Sources
    click DS1 "https://github.com/san089/optimizing-public-transportation/blob/master/startup/stream.py"
    click DS2 "https://github.com/san089/optimizing-public-transportation/blob/master/startup/clicks.py"
    click DS3 "https://github.com/san089/optimizing-public-transportation/blob/master/startup/cta_stations.csv"
```
