# suu00k1459/ost-synstream architecture

> Architecture diagram of the GitHub repository suu00k1459/ost-synstream, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/suu00k1459/ost-synstream
- Repository: https://github.com/suu00k1459/ost-synstream

## Overview

OST-SynStream is a containerized platform for replaying sensor telemetry, processing streaming windows, generating synthetic time series, and monitoring/evaluating their quality. The main workflow replays dataset records to Kafka, processes them into normalized windows and features, then supports anomaly/drift monitoring and real-versus-synthetic comparison. Batch utilities prepare splits and benchmark downstream utility; storage and visualization provide analytics and monitoring capabilities described by the repository structure. Some service implementations were not sampled, so their connections are left unspecified.

Main entry points: [`ingestion_flink/replay_producer.py`](https://github.com/suu00k1459/ost-synstream/blob/main/ingestion_flink/replay_producer.py) (Telemetry Replay), [`stream_analytics/metrics.py`](https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/metrics.py) (Similarity and Privacy Metrics), [`ingestion_flink/flink_window_processor.py`](https://github.com/suu00k1459/ost-synstream/blob/main/ingestion_flink/flink_window_processor.py) (Window Processor), [`ingestion_flink/prepare_batch_splits.py`](https://github.com/suu00k1459/ost-synstream/blob/main/ingestion_flink/prepare_batch_splits.py) (Batch Split Preparation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Replay and Ingestion"]
  node_dataset["Dataset Loader<br/>[datasets.py]"]
  node_config["Stream Configuration<br/>[config.py]"]
  node_replay["Telemetry Replay<br/>[replay_producer.py]"]
  node_windowing["Window Processor"]
  node_splits["Batch Split Preparation"]
end

subgraph group_generation["Synthetic Generation"]
  node_generator["Online Generator"]
  node_timegan["TimeGAN Training<br/>[train_timegan.py]"]
  node_timevae["TimeVAE Training<br/>[train_timevae.py]"]
end

subgraph group_analytics["Analytics and Evaluation"]
  node_river["Anomaly and Drift Monitor"]
  node_comparator["Window Comparator"]
  node_metrics["Similarity and Privacy Metrics<br/>[metrics.py]"]
  node_loading["Analytics Data Loading<br/>[data_loading.py]"]
  node_benchmark["Downstream Utility Benchmark"]
  node_synthetic_check["Synthetic Data Checks<br/>[check_synthetic.py]"]
  node_retrain["Retraining Evaluation<br/>[retrain_effect.py]"]
  node_batch_eval["Batch Generation Evaluation<br/>[evaluate_batch.py]"]
end

subgraph group_persistence["Storage"]
  node_storage_consumer["Kafka Storage Sink<br/>[kafka_to_influx.py]"]
  node_window_handlers["Window Analytics Handlers"]
  node_analytics_handlers["Analytics Handlers"]
  node_query["Influx Query Helper<br/>[query_helper.py]"]
end

subgraph group_ui["Visualization"]
  node_dashboard["Monitoring Dashboard<br/>[app.py]"]
  node_viz_sink["Visualization Sink"]
end

node_operator(("Operator"))
node_kafka["Kafka Topics"]
node_influx[("InfluxDB")]

node_replay -->|"loads records"| node_dataset
node_replay -->|"publishes raw records"| node_kafka
node_windowing -->|"reads and publishes windows"| node_kafka
node_windowing -->|"uses topic settings"| node_config
node_splits -->|"loads dataset"| node_dataset
node_splits -->|"uses window settings"| node_config
node_river -->|"publishes scores and alerts"| node_kafka
node_comparator -->|"computes comparisons"| node_metrics
node_comparator -->|"publishes comparison results"| node_kafka
node_benchmark -->|"computes metrics"| node_metrics
node_benchmark -->|"loads benchmark data"| node_loading
node_river -->|"uses data helpers"| node_loading
node_synthetic_check -->|"checks similarity and privacy"| node_metrics
node_operator -.->|"starts replay"| node_replay
node_operator -.->|"views monitoring"| node_dashboard

click node_dataset "https://github.com/suu00k1459/ost-synstream/blob/main/common/datasets.py"
click node_config "https://github.com/suu00k1459/ost-synstream/blob/main/common/config.py"
click node_replay "https://github.com/suu00k1459/ost-synstream/blob/main/ingestion_flink/replay_producer.py"
click node_windowing "https://github.com/suu00k1459/ost-synstream/blob/main/ingestion_flink/flink_window_processor.py"
click node_splits "https://github.com/suu00k1459/ost-synstream/blob/main/ingestion_flink/prepare_batch_splits.py"
click node_generator "https://github.com/suu00k1459/ost-synstream/blob/main/synthetic_gen/online_generator_service.py"
click node_timegan "https://github.com/suu00k1459/ost-synstream/blob/main/synthetic_gen/train_timegan.py"
click node_timevae "https://github.com/suu00k1459/ost-synstream/blob/main/synthetic_gen/train_timevae.py"
click node_river "https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/river_anomaly_drift.py"
click node_comparator "https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/window_comparator.py"
click node_metrics "https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/metrics.py"
click node_loading "https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/data_loading.py"
click node_benchmark "https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/batch_tstr_benchmark.py"
click node_synthetic_check "https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/check_synthetic.py"
click node_retrain "https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/retrain_effect.py"
click node_batch_eval "https://github.com/suu00k1459/ost-synstream/blob/main/synthetic_gen/evaluate_batch.py"
click node_storage_consumer "https://github.com/suu00k1459/ost-synstream/blob/main/storage_influx/kafka_to_influx.py"
click node_window_handlers "https://github.com/suu00k1459/ost-synstream/blob/main/storage_influx/handlers_windows.py"
click node_analytics_handlers "https://github.com/suu00k1459/ost-synstream/blob/main/storage_influx/handlers_analytics.py"
click node_query "https://github.com/suu00k1459/ost-synstream/blob/main/storage_influx/query_helper.py"
click node_dashboard "https://github.com/suu00k1459/ost-synstream/blob/main/visualization/streamlit_app/app.py"
click node_viz_sink "https://github.com/suu00k1459/ost-synstream/blob/main/visualization/influx_sink_service.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset,node_config,node_replay,node_windowing,node_splits toneBlue
class node_generator,node_timegan,node_timevae,node_influx toneAmber
class node_river,node_comparator,node_metrics,node_loading,node_benchmark,node_synthetic_check,node_retrain,node_batch_eval toneMint
class node_storage_consumer,node_window_handlers,node_analytics_handlers,node_query toneRose
class node_dashboard,node_viz_sink,node_operator,node_kafka toneIndigo
```

## Components

### Replay and Ingestion

- [Dataset Loader](https://github.com/suu00k1459/ost-synstream/blob/main/common/datasets.py): `common/datasets.py`
- [Stream Configuration](https://github.com/suu00k1459/ost-synstream/blob/main/common/config.py): `common/config.py`
- [Telemetry Replay](https://github.com/suu00k1459/ost-synstream/blob/main/ingestion_flink/replay_producer.py): `ingestion_flink/replay_producer.py`
- [Window Processor](https://github.com/suu00k1459/ost-synstream/blob/main/ingestion_flink/flink_window_processor.py): `ingestion_flink/flink_window_processor.py`
- [Batch Split Preparation](https://github.com/suu00k1459/ost-synstream/blob/main/ingestion_flink/prepare_batch_splits.py): `ingestion_flink/prepare_batch_splits.py`

### Synthetic Generation

- [Online Generator](https://github.com/suu00k1459/ost-synstream/blob/main/synthetic_gen/online_generator_service.py): `synthetic_gen/online_generator_service.py`
- [TimeGAN Training](https://github.com/suu00k1459/ost-synstream/blob/main/synthetic_gen/train_timegan.py): `synthetic_gen/train_timegan.py`
- [TimeVAE Training](https://github.com/suu00k1459/ost-synstream/blob/main/synthetic_gen/train_timevae.py): `synthetic_gen/train_timevae.py`

### Analytics and Evaluation

- [Anomaly and Drift Monitor](https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/river_anomaly_drift.py): `stream_analytics/river_anomaly_drift.py`
- [Window Comparator](https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/window_comparator.py): `stream_analytics/window_comparator.py`
- [Similarity and Privacy Metrics](https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/metrics.py): `stream_analytics/metrics.py`
- [Analytics Data Loading](https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/data_loading.py): `stream_analytics/data_loading.py`
- [Downstream Utility Benchmark](https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/batch_tstr_benchmark.py): `stream_analytics/batch_tstr_benchmark.py`
- [Synthetic Data Checks](https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/check_synthetic.py): `stream_analytics/check_synthetic.py`
- [Retraining Evaluation](https://github.com/suu00k1459/ost-synstream/blob/main/stream_analytics/retrain_effect.py): `stream_analytics/retrain_effect.py`
- [Batch Generation Evaluation](https://github.com/suu00k1459/ost-synstream/blob/main/synthetic_gen/evaluate_batch.py): `synthetic_gen/evaluate_batch.py`

### Storage

- [Kafka Storage Sink](https://github.com/suu00k1459/ost-synstream/blob/main/storage_influx/kafka_to_influx.py): `storage_influx/kafka_to_influx.py`
- [Window Analytics Handlers](https://github.com/suu00k1459/ost-synstream/blob/main/storage_influx/handlers_windows.py): `storage_influx/handlers_windows.py`
- [Analytics Handlers](https://github.com/suu00k1459/ost-synstream/blob/main/storage_influx/handlers_analytics.py): `storage_influx/handlers_analytics.py`
- [Influx Query Helper](https://github.com/suu00k1459/ost-synstream/blob/main/storage_influx/query_helper.py): `storage_influx/query_helper.py`

### Visualization

- [Monitoring Dashboard](https://github.com/suu00k1459/ost-synstream/blob/main/visualization/streamlit_app/app.py): `visualization/streamlit_app/app.py`
- [Visualization Sink](https://github.com/suu00k1459/ost-synstream/blob/main/visualization/influx_sink_service.py): `visualization/influx_sink_service.py`

### Other components

- **Operator**
- **Kafka Topics**
- **InfluxDB**

## Connections

- Telemetry Replay → Dataset Loader: loads records (evidence: `ingestion_flink/replay_producer.py`)
- Telemetry Replay → Kafka Topics: publishes raw records (evidence: `ingestion_flink/replay_producer.py`)
- Window Processor → Kafka Topics: reads and publishes windows (evidence: `ingestion_flink/flink_window_processor.py`)
- Window Processor → Stream Configuration: uses topic settings (evidence: `ingestion_flink/flink_window_processor.py`)
- Batch Split Preparation → Dataset Loader: loads dataset (evidence: `ingestion_flink/prepare_batch_splits.py`)
- Batch Split Preparation → Stream Configuration: uses window settings (evidence: `ingestion_flink/prepare_batch_splits.py`)
- Anomaly and Drift Monitor → Kafka Topics: publishes scores and alerts (evidence: `stream_analytics/river_anomaly_drift.py`)
- Window Comparator → Similarity and Privacy Metrics: computes comparisons (evidence: `stream_analytics/window_comparator.py`)
- Window Comparator → Kafka Topics: publishes comparison results (evidence: `stream_analytics/window_comparator.py`)
- Downstream Utility Benchmark → Similarity and Privacy Metrics: computes metrics (evidence: `stream_analytics/batch_tstr_benchmark.py`)
- Downstream Utility Benchmark → Analytics Data Loading: loads benchmark data (evidence: `stream_analytics/batch_tstr_benchmark.py`)
- Anomaly and Drift Monitor → Analytics Data Loading: uses data helpers (evidence: `stream_analytics/river_anomaly_drift.py`)
- Synthetic Data Checks → Similarity and Privacy Metrics: checks similarity and privacy (evidence: `stream_analytics/check_synthetic.py`)
- Operator → Telemetry Replay: starts replay
- Operator → Monitoring Dashboard: views monitoring
