How san089/optimizing-public-transportation works
Main Components
• Producers: Create events related to train arrivals, turnstile usage, and weather.
• In the producers/ directory, note files like simulation.py, connector.py, and the models (including schemas) which indicate how messages are structured.
• That producers ingest data from multiple sources (CSV files in producers/data and startup) and publish them to Kafka topics.
• Consumers: Process and consume the events from Kafka.
• In the consumers/ directory, there are multiple consumers:
◦ consumer.py – a core consumer handling events,
◦ faust_stream – using Faust for stream processing,
◦ ksql.py – a script likely using ksql for querying stream data.
• The server.py and associated templates (status.html) indicate a UI component that displays real-time status.
• Data Sources/Initiation: The startup/ directory includes initialization scripts and CSV datasets that seed the simulation with additional data.
• External Systems and Infrastructure:
• Apache Kafka (and its ecosystem such as REST Proxy and Kafka Connect) acts as the backbone.
• Postgres SQL is mentioned in the README for station data extraction via Kafka Connect.
• Docker and Docker Compose are used to orchestrate the services as indicated in the README.
Relationships and Data Flow
• A Kafka component in the center. Label it clearly to indicate it is the event pipeline.
• Kafka to the Consumers, representing how different consumer types (Faust stream processing, KSQL, and the UI server) read from Kafka topics.
• External data flows where Postgres (accessed via Kafka Connect) is providing additional station information to Kafka.
• Optionally, indicate that Docker Compose coordinates the startup and interaction of these services.
Architectural Patterns / Design Principles
• Streaming architecture – using publish/subscribe messaging patterns between producers and consumers.
• That the design is modular – splitting producers and consumers into separate directories implies loosely coupled components.
• If applicable, mention that Faust is used for stream processing, while KSQL offers a SQL-like interface to Kafka for querying.
• The separation of simulation logic and configuration (such as logging.ini files) also suggests attention to decoupling concerns.
Relevant Technologies
• Apache Kafka (with related tools like REST Proxy and Kafka Connect)
• Python (the language chosen, with separate virtual environments for producers and consumers)
• Docker & Docker Compose (for service orchestration)
• Postgres (for station data integration)
• Faust (for stream processing) and KSQL (for stream querying)
Additional Details
• That every component from the project (including supporting folders like startup and docs) is represented if relevant. For instance, the startup/ folder might be summarized as an initialization or seeding process feeding into the simulation.