How divya-a10/weverse-scale-streaming-pipeline works
This is a local prototype of a high-throughput live-stream telemetry pipeline. Simulated clients POST events to a FastAPI gateway, which forwards them over TCP to a background listener; a queue feeds a worker that aggregates events into 10-second stream windows and persists batches to PostgreSQL. A Streamlit dashboard reads stored metrics for live charts and records. The README also describes cloud-service equivalents, but the sampled runtime implements local TCP and PostgreSQL rather than Pub/Sub or Dataflow. The graph reflects only relationships shown in the excerpts or README.
Main entry points: locustfile.py Locust Simulator · ingestion_gateway.py FastAPI Gateway · stream_processor.py TCP Listener · dashboard.py Live Dashboard
Components 9
Event Ingestion
- Locust Simulator
locustfile.py - FastAPI Gateway
ingestion_gateway.py - TCP Forwarder
ingestion_gateway.py
Stream Processing
- TCP Listener
stream_processor.py - Event Queue
stream_processor.py - Window Aggregator
stream_processor.py
Metrics Storage
- PostgreSQL Metrics
Operations Monitoring
- Live Dashboard
dashboard.py
Other components
- Event Client
Connections 8
- Event Client → Locust Simulator: generates traffic
README.md - Locust Simulator → FastAPI Gateway: POSTs events
locustfile.py - FastAPI Gateway → TCP Forwarder: validates and forwards
ingestion_gateway.py - TCP Forwarder → TCP Listener: sends TCP payload
ingestion_gateway.py - TCP Listener → Event Queue: enqueues events
stream_processor.py - Event Queue → Window Aggregator: consumes events
stream_processor.py - Window Aggregator → PostgreSQL Metrics: batch upserts
stream_processor.py - Live Dashboard → PostgreSQL Metrics: queries metrics
dashboard.py