# aicoe/prometheus-data-science architecture

> Architecture diagram of the GitHub repository aicoe/prometheus-data-science, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aicoe/prometheus-data-science
- Repository: https://github.com/aicoe/prometheus-data-science

## Overview

This repository is a data-science toolkit for analyzing Prometheus metrics, not a continuously running service. Its main workflow reformats metric JSON packets into per-series Pandas data, then applies forecasting and anomaly-detection techniques. Separate metadata tools visualize label behavior and reduce metadata dimensions; histogram and summary analysis is also a documented capability. The formatter and several analysis scripts are excerpted; links between preprocessing and forecasting follow the README, while unsampled implementations are represented only where their paths are known.

Main entry points: [`format_to_pandas.py`](https://github.com/aicoe/prometheus-data-science/blob/master/format_to_pandas.py) (Packet formatter), [`metadata_analysis/graph_metadata.py`](https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/graph_metadata.py) (Metadata timelines), [`metadata_analysis/graph_specific_ts.py`](https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/graph_specific_ts.py) (Specific series plots), [`metadata_analysis/t_sne_for_metadata.py`](https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/t_sne_for_metadata.py) (Metadata t-SNE)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_preprocess["Preprocessing"]
  node_formatter["Packet formatter"]
  node_series_pickle["Series pickle"]
end

subgraph group_metadata["Metadata Analysis"]
  node_metadata_graph["Metadata timelines<br/>[graph_metadata.py]"]
  node_metadata_specific["Specific series plots"]
  node_metadata_tsne["Metadata t-SNE"]
  node_metadata_labels["Label histograms"]
  node_single_series["Single-series extraction"]
end

subgraph group_forecast["Forecasting"]
  node_arima["ARIMA training<br/>[arima_train.py]"]
  node_smoothing["Exponential smoothing"]
  node_holt_winters["Holt-Winters training"]
  node_fourier["Fourier training<br/>[fourier_train.py]"]
  node_prophet["Prophet training<br/>[prophet_train.py]"]
  node_model_compare["Fourier-Prophet comparison"]
end

subgraph group_anomaly["Anomaly Analysis"]
  node_time_series_graphs["Series graphing<br/>[graphing_ts.py]"]
  node_detector["Anomaly decision rules"]
end

node_prometheus["Metric packets"]

node_prometheus -->|"provides packets"| node_formatter
node_formatter -->|"writes series"| node_series_pickle
node_series_pickle -->|"supplies series"| node_arima
node_series_pickle -->|"supplies series"| node_smoothing
node_series_pickle -->|"supplies series"| node_holt_winters
node_series_pickle -->|"supplies series"| node_fourier
node_series_pickle -->|"supplies series"| node_prophet
node_series_pickle -->|"supplies series"| node_model_compare
node_prometheus -->|"reads packets"| node_metadata_graph
node_prometheus -->|"reads packets"| node_metadata_specific
node_prometheus -->|"reads packets"| node_metadata_tsne
node_prometheus -->|"reads packets"| node_metadata_labels
node_prometheus -->|"reads packets"| node_single_series
node_series_pickle -->|"loads series"| node_time_series_graphs

click node_formatter "https://github.com/aicoe/prometheus-data-science/blob/master/format_to_pandas.py"
click node_metadata_graph "https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/graph_metadata.py"
click node_metadata_specific "https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/graph_specific_ts.py"
click node_metadata_tsne "https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/t_sne_for_metadata.py"
click node_metadata_labels "https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/plot_metadata_labels.py"
click node_single_series "https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/get_single_ts_all.py"
click node_arima "https://github.com/aicoe/prometheus-data-science/blob/master/arima_train.py"
click node_smoothing "https://github.com/aicoe/prometheus-data-science/blob/master/exp_smoothing_train.py"
click node_holt_winters "https://github.com/aicoe/prometheus-data-science/blob/master/holt_winters_train.py"
click node_fourier "https://github.com/aicoe/prometheus-data-science/blob/master/fourier_train.py"
click node_prophet "https://github.com/aicoe/prometheus-data-science/blob/master/prophet_train.py"
click node_model_compare "https://github.com/aicoe/prometheus-data-science/blob/master/compare_fourier_prophet.py"
click node_time_series_graphs "https://github.com/aicoe/prometheus-data-science/blob/master/graphing_ts.py"
click node_detector "https://github.com/aicoe/prometheus-data-science/blob/master/anomaly_detector.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_formatter,node_series_pickle toneBlue
class node_metadata_graph,node_metadata_specific,node_metadata_tsne,node_metadata_labels,node_single_series toneAmber
class node_arima,node_smoothing,node_holt_winters,node_fourier,node_prophet,node_model_compare toneMint
class node_time_series_graphs,node_detector toneRose
class node_prometheus toneIndigo
```

## Components

### Preprocessing

- [Packet formatter](https://github.com/aicoe/prometheus-data-science/blob/master/format_to_pandas.py): `format_to_pandas.py`
- **Series pickle**

### Metadata Analysis

- [Metadata timelines](https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/graph_metadata.py): `metadata_analysis/graph_metadata.py`
- [Specific series plots](https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/graph_specific_ts.py): `metadata_analysis/graph_specific_ts.py`
- [Metadata t-SNE](https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/t_sne_for_metadata.py): `metadata_analysis/t_sne_for_metadata.py`
- [Label histograms](https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/plot_metadata_labels.py): `metadata_analysis/plot_metadata_labels.py`
- [Single-series extraction](https://github.com/aicoe/prometheus-data-science/blob/master/metadata_analysis/get_single_ts_all.py): `metadata_analysis/get_single_ts_all.py`

### Forecasting

- [ARIMA training](https://github.com/aicoe/prometheus-data-science/blob/master/arima_train.py): `arima_train.py`
- [Exponential smoothing](https://github.com/aicoe/prometheus-data-science/blob/master/exp_smoothing_train.py): `exp_smoothing_train.py`
- [Holt-Winters training](https://github.com/aicoe/prometheus-data-science/blob/master/holt_winters_train.py): `holt_winters_train.py`
- [Fourier training](https://github.com/aicoe/prometheus-data-science/blob/master/fourier_train.py): `fourier_train.py`
- [Prophet training](https://github.com/aicoe/prometheus-data-science/blob/master/prophet_train.py): `prophet_train.py`
- [Fourier-Prophet comparison](https://github.com/aicoe/prometheus-data-science/blob/master/compare_fourier_prophet.py): `compare_fourier_prophet.py`

### Anomaly Analysis

- [Series graphing](https://github.com/aicoe/prometheus-data-science/blob/master/graphing_ts.py): `graphing_ts.py`
- [Anomaly decision rules](https://github.com/aicoe/prometheus-data-science/blob/master/anomaly_detector.py): `anomaly_detector.py`

### Other components

- **Metric packets**

## Connections

- Metric packets → Packet formatter: provides packets (evidence: `README.md`)
- Packet formatter → Series pickle: writes series (evidence: `README.md`)
- Series pickle → ARIMA training: supplies series (evidence: `README.md`)
- Series pickle → Exponential smoothing: supplies series (evidence: `README.md`)
- Series pickle → Holt-Winters training: supplies series (evidence: `README.md`)
- Series pickle → Fourier training: supplies series (evidence: `README.md`)
- Series pickle → Prophet training: supplies series (evidence: `README.md`)
- Series pickle → Fourier-Prophet comparison: supplies series (evidence: `README.md`)
- Metric packets → Metadata timelines: reads packets (evidence: `metadata_analysis/graph_metadata.py`)
- Metric packets → Specific series plots: reads packets (evidence: `metadata_analysis/graph_specific_ts.py`)
- Metric packets → Metadata t-SNE: reads packets (evidence: `metadata_analysis/t_sne_for_metadata.py`)
- Metric packets → Label histograms: reads packets (evidence: `metadata_analysis/plot_metadata_labels.py`)
- Metric packets → Single-series extraction: reads packets (evidence: `metadata_analysis/get_single_ts_all.py`)
- Series pickle → Series graphing: loads series (evidence: `graphing_ts.py`)
