# divya-a10/weverse-scale-streaming-pipeline architecture

> Architecture diagram of the GitHub repository divya-a10/weverse-scale-streaming-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/divya-a10/weverse-scale-streaming-pipeline
- Repository: https://github.com/divya-a10/weverse-scale-streaming-pipeline

## Overview

This is a local prototype of a high-throughput live-stream telemetry pipeline. Simulated clients POST events to a FastAPI gateway, which forwards them over TCP to a background listener; a queue feeds a worker that aggregates events into 10-second stream windows and persists batches to PostgreSQL. A Streamlit dashboard reads stored metrics for live charts and records. The README also describes cloud-service equivalents, but the sampled runtime implements local TCP and PostgreSQL rather than Pub/Sub or Dataflow. The graph reflects only relationships shown in the excerpts or README.

Main entry points: [`locustfile.py`](https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/locustfile.py) (Locust Simulator), [`ingestion_gateway.py`](https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/ingestion_gateway.py) (FastAPI Gateway), [`stream_processor.py`](https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/stream_processor.py) (TCP Listener), [`dashboard.py`](https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/dashboard.py) (Live Dashboard)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Event Ingestion"]
  node_load_simulator["Locust Simulator<br/>[locustfile.py]"]
  node_gateway["FastAPI Gateway"]
  node_tcp_sender["TCP Forwarder"]
end

subgraph group_processing["Stream Processing"]
  node_tcp_listener["TCP Listener"]
  node_event_queue["Event Queue"]
  node_window_worker["Window Aggregator"]
end

subgraph group_storage["Metrics Storage"]
  node_postgres[("PostgreSQL Metrics")]
end

subgraph group_monitoring["Operations Monitoring"]
  node_dashboard["Live Dashboard<br/>[dashboard.py]"]
end

node_client(("Event Client"))

node_client -->|"generates traffic"| node_load_simulator
node_load_simulator -->|"POSTs events"| node_gateway
node_gateway -->|"validates and forwards"| node_tcp_sender
node_tcp_sender -->|"sends TCP payload"| node_tcp_listener
node_tcp_listener -->|"enqueues events"| node_event_queue
node_event_queue -->|"consumes events"| node_window_worker
node_window_worker -->|"batch upserts"| node_postgres
node_dashboard -->|"queries metrics"| node_postgres

click node_load_simulator "https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/locustfile.py"
click node_gateway "https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/ingestion_gateway.py"
click node_tcp_sender "https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/ingestion_gateway.py"
click node_tcp_listener "https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/stream_processor.py"
click node_event_queue "https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/stream_processor.py"
click node_window_worker "https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/stream_processor.py"
click node_dashboard "https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/dashboard.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_load_simulator,node_gateway,node_tcp_sender,node_client toneBlue
class node_tcp_listener,node_event_queue,node_window_worker toneAmber
class node_postgres toneMint
class node_dashboard toneRose
```

## Components

### Event Ingestion

- [Locust Simulator](https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/locustfile.py): `locustfile.py`
- [FastAPI Gateway](https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/ingestion_gateway.py): `ingestion_gateway.py`
- [TCP Forwarder](https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/ingestion_gateway.py): `ingestion_gateway.py`

### Stream Processing

- [TCP Listener](https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/stream_processor.py): `stream_processor.py`
- [Event Queue](https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/stream_processor.py): `stream_processor.py`
- [Window Aggregator](https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/stream_processor.py): `stream_processor.py`

### Metrics Storage

- **PostgreSQL Metrics**

### Operations Monitoring

- [Live Dashboard](https://github.com/divya-a10/weverse-scale-streaming-pipeline/blob/main/dashboard.py): `dashboard.py`

### Other components

- **Event Client**

## Connections

- Event Client → Locust Simulator: generates traffic (evidence: `README.md`)
- Locust Simulator → FastAPI Gateway: POSTs events (evidence: `locustfile.py`)
- FastAPI Gateway → TCP Forwarder: validates and forwards (evidence: `ingestion_gateway.py`)
- TCP Forwarder → TCP Listener: sends TCP payload (evidence: `ingestion_gateway.py`)
- TCP Listener → Event Queue: enqueues events (evidence: `stream_processor.py`)
- Event Queue → Window Aggregator: consumes events (evidence: `stream_processor.py`)
- Window Aggregator → PostgreSQL Metrics: batch upserts (evidence: `stream_processor.py`)
- Live Dashboard → PostgreSQL Metrics: queries metrics (evidence: `dashboard.py`)