How suu00k1459/ost-synstream works
OST-SynStream is a containerized platform for replaying sensor telemetry, processing streaming windows, generating synthetic time series, and monitoring/evaluating their quality. The main workflow replays dataset records to Kafka, processes them into normalized windows and features, then supports anomaly/drift monitoring and real-versus-synthetic comparison. Batch utilities prepare splits and benchmark downstream utility; storage and visualization provide analytics and monitoring capabilities described by the repository structure. Some service implementations were not sampled, so their connections are left unspecified.
Main entry points: replay_producer.py Telemetry Replay · metrics.py Similarity and Privacy Metrics · flink_window_processor.py Window Processor · prepare_batch_splits.py Batch Split Preparation
Components 25
Replay and Ingestion
- Dataset Loader
common/datasets.py - Stream Configuration
common/config.py - Telemetry Replay
ingestion_flink/replay_producer.py - Window Processor
ingestion_flink/flink_window_processor.py - Batch Split Preparation
ingestion_flink/prepare_batch_splits.py
Synthetic Generation
- Online Generator
synthetic_gen/online_generator_service.py - TimeGAN Training
synthetic_gen/train_timegan.py - TimeVAE Training
synthetic_gen/train_timevae.py
Analytics and Evaluation
- Anomaly and Drift Monitor
stream_analytics/river_anomaly_drift.py - Window Comparator
stream_analytics/window_comparator.py - Similarity and Privacy Metrics
stream_analytics/metrics.py - Analytics Data Loading
stream_analytics/data_loading.py - Downstream Utility Benchmark
stream_analytics/batch_tstr_benchmark.py - Synthetic Data Checks
stream_analytics/check_synthetic.py - Retraining Evaluation
stream_analytics/retrain_effect.py - Batch Generation Evaluation
synthetic_gen/evaluate_batch.py
Storage
- Kafka Storage Sink
storage_influx/kafka_to_influx.py - Window Analytics Handlers
storage_influx/handlers_windows.py - Analytics Handlers
storage_influx/handlers_analytics.py - Influx Query Helper
storage_influx/query_helper.py
Visualization
- Monitoring Dashboard
visualization/streamlit_app/app.py - Visualization Sink
visualization/influx_sink_service.py
Other components
- Operator
- Kafka Topics
- InfluxDB
Connections 15
- Telemetry Replay → Dataset Loader: loads records
ingestion_flink/replay_producer.py - Telemetry Replay → Kafka Topics: publishes raw records
ingestion_flink/replay_producer.py - Window Processor → Kafka Topics: reads and publishes windows
ingestion_flink/flink_window_processor.py - Window Processor → Stream Configuration: uses topic settings
ingestion_flink/flink_window_processor.py - Batch Split Preparation → Dataset Loader: loads dataset
ingestion_flink/prepare_batch_splits.py - Batch Split Preparation → Stream Configuration: uses window settings
ingestion_flink/prepare_batch_splits.py - Anomaly and Drift Monitor → Kafka Topics: publishes scores and alerts
stream_analytics/river_anomaly_drift.py - Window Comparator → Similarity and Privacy Metrics: computes comparisons
stream_analytics/window_comparator.py - Window Comparator → Kafka Topics: publishes comparison results
stream_analytics/window_comparator.py - Downstream Utility Benchmark → Similarity and Privacy Metrics: computes metrics
stream_analytics/batch_tstr_benchmark.py - Downstream Utility Benchmark → Analytics Data Loading: loads benchmark data
stream_analytics/batch_tstr_benchmark.py - Anomaly and Drift Monitor → Analytics Data Loading: uses data helpers
stream_analytics/river_anomaly_drift.py - Synthetic Data Checks → Similarity and Privacy Metrics: checks similarity and privacy
stream_analytics/check_synthetic.py - Operator → Telemetry Replay: starts replay
- Operator → Monitoring Dashboard: views monitoring