How faresalawneh/llm-graduation-project works
INFERSCOPE replays steady or burst BurstGPT traces against vLLM, streams inference metrics through OpenTelemetry, and combines them with GPU telemetry in Prometheus for Grafana dashboards and a JWT-protected REST API. Completed runs can also be recorded in MLflow with metrics, artifacts, and container state. The main flow is documented and supported by sampled replay, exporter, API, and logging code; stack wiring and dashboard details are based on the README and repository paths.
Main entry points: aiperf_trace_replay.py AIPerf replay · app.py Observability API · mlflow_log_run.py MLflow logger · otel-collector-config.yml OTel collector
Components 16
Trace replay
- Steady trace
monitoring/scripts/steady_window.csv - Burst trace
monitoring/scripts/burst_window.csv - AIPerf replay
monitoring/scripts/aiperf_trace_replay.py - vLLM server
Telemetry pipeline
- OTel collector
monitoring/otel/otel-collector-config.yml - NVML exporter
monitoring/exporters/nvml_exporter.py - Scrape config
monitoring/prometheus/prometheus.yml - Prometheus
Metrics access
- Grafana dashboards
monitoring/grafana/provisioning/dashboards - Observability API
api/app.py - JWT and roles
api/auth.py
Run tracking
- Run artifacts
monitoring/scripts/aiperf_trace_replay.py - MLflow logger
monitoring/scripts/mlflow_log_run.py - MLflow store
Other components
- Operator
- API user
Connections 19
- Steady trace → AIPerf replay: replay input
- Burst trace → AIPerf replay: replay input
- AIPerf replay → vLLM server: HTTP requests
- AIPerf replay → OTel collector: streams OTLP
- AIPerf replay → Run artifacts: writes results
- vLLM server → Prometheus: scraped metrics
- OTel collector → Prometheus: scraped metrics
- NVML exporter → Prometheus: scraped gauges
- Scrape config → Prometheus: defines scrapes
- Operator → AIPerf replay: starts replay
- Prometheus → Grafana dashboards: serves queries
- Operator → Grafana dashboards: views dashboards
- API user → Observability API: requests metrics
- Observability API → JWT and roles: checks access
- Observability API → Prometheus: queries metrics
- Operator → MLflow logger: logs completed run
- MLflow logger → Run artifacts: reads artifacts
- MLflow logger → MLflow store: logs run
- AIPerf replay → Prometheus: optional GPU queries