# mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs architecture
> Architecture diagram of the GitHub repository mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-24). It is an AI-made overview: check details against the source.
- Interactive diagram: https://gitdiagram.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs
- Repository: https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs
## Overview
1. **What the repo is**
- This repository is a local Kubernetes AIOps lab, not a general-purpose app. It demonstrates two monitoring patterns on the same sample service: real-time CPU anomaly detection and predictive analysis.
- The whole system is meant to run on a laptop using a local Kind cluster.
2. **Execution boundary**
- The architecture splits into two main environments: the **local machine** and the **Kind Kubernetes cluster**.
- The local machine runs setup scripts, traffic generators, and port-forwards. The cluster runs the Product Store app plus the monitoring stack.
3. **Cluster topology**
- The Kind setup uses a dedicated cluster named **`aiops-lab`** with **1 control-plane node and 2 worker nodes**.
- This matters architecturally because the labs are designed to exercise multi-node Kubernetes scheduling and monitoring behavior, while still staying fully local.
4. **Application subsystem**
- The sample workload is **Product Store**, implemented in `apps/product-store/server.py`.
- It exposes three endpoints: `/health`, `/products`, and `/load`.
- The app is intentionally simple: normal requests are low-cost, while `/load` burns CPU to simulate an incident.
5. **How the app is deployed**
- In Lab 1, the app is not built as a container image. Instead, the Python source is stored in a **ConfigMap** and mounted into the deployment.
- The app runs as **two pods** in the `aiops-lab` namespace, behind an internal Kubernetes Service.
- CPU and memory requests are explicitly set, and the CPU request becomes the baseline used by the alert logic.
6. **Monitoring stack**
- The monitoring layer is installed in the **`monitoring` namespace** using the **`kube-prometheus-stack`** Helm chart.
- This gives the lab its three core observability components:
- **Prometheus** for scraping metrics and evaluating rules
- **Alertmanager** for routing alerts
- **Grafana** for dashboarding
- The Slack webhook is injected through generated Helm values, keeping secrets out of git.
7. **Lab 1 data flow: anomaly detection**
- Traffic reaches Product Store, CPU usage rises, and **Prometheus** scrapes the pod metrics.
- The Lab 1 rule compares **actual CPU usage / requested CPU**.
- If the ratio stays above **0.75 for 2 minutes**, Prometheus marks the alert as firing.
- **Alertmanager** receives that firing alert and sends the notification to **Slack** through an incoming webhook.
8. **Lab 1 visualization path**
- **Grafana** reads from Prometheus and shows the same CPU behavior on a dashboard.
- The dashboard is part of the repo’s Kubernetes manifests, so it is deployed alongside the app and alert rule.
- This gives a second validation path: the alert is not only sent, it is also visible as a metric trend.
9. **Lab 2 data flow: predictive analysis**
- Lab 2 keeps the same app and monitoring stack, but adds a second Prometheus rule.
- That rule uses **`predict_linear`** on recent CPU ratio samples to forecast whether the workload will exceed the same threshold soon.
- This is a forward-looking warning, not ML: it is Prometheus time-series forecasting over the last 10 minutes, predicting 5 minutes ahead.
10. **Alerting boundary**
- Alert routing is intentionally separated from alert detection.
- **Prometheus decides whether a condition is true**, and **Alertmanager decides where it goes**.
- Slack is the external notification sink; it is not part of the cluster and is configured entirely through the webhook.
11. **Traffic generation**
- The repo includes shell scripts that generate two traffic patterns:
- normal baseline requests to `/health` and `/products`
- CPU-heavy requests to `/load`
- This traffic is the input that drives the monitoring story; there is no synthetic telemetry generator separate from the app traffic itself.
12. **Operational scripts**
- Top-level scripts handle cluster lifecycle and validation:
- create/delete the Kind cluster
- check required tools
- verify that the lab deployed correctly
- Lab-specific scripts handle deployment, cleanup, traffic, and port-forwarding.
- Architecturally, these scripts are the control plane for the lab, while Kubernetes is the runtime plane.
13. **Access pattern**
- The services are internal by default.
- Local access is provided through **`kubectl port-forward`**, exposing Grafana and the Product Store API on localhost without changing cluster networking or service type.
14. **Repository structure as system boundaries**
- `clusters/kind/` defines the local cluster substrate.
- `apps/product-store/` contains the workload source.
- `labs/lab-01-anomaly-detection/` contains the baseline monitoring app manifests, rule, dashboard, scripts, and traffic.
- `labs/lab-02-predictive-analysis/` adds the predictive rule on top of the same base environment.
- `diagrams/` documents the intended end-to-end flows, especially the alert path to Slack.
15. **What to show in an architecture diagram**
- The key nodes are: local machine, Kind cluster, `aiops-lab` namespace, `monitoring` namespace, Product Store pods, Prometheus, Alertmanager, Grafana, and Slack.
- The main arrows are:
- local scripts → Kind cluster
- traffic → Product Store Service → Product Store pods
- Product Store pods → Prometheus
- Prometheus → Alertmanager → Slack
- Prometheus → Grafana
- Lab 2 adds one extra rule path from Prometheus evaluation to a predictive alert, but reuses the same runtime and monitoring stack.
Main entry points: [`labs/lab-01-anomaly-detection/monitoring/values-template.yaml`](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/monitoring/values-template.yaml) (Helm values), [`labs/lab-01-anomaly-detection/k8s/02-product-store-deployment.yaml`](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/02-product-store-deployment.yaml) (Product pods), [`labs/lab-01-anomaly-detection/k8s/03-product-store-service.yaml`](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/03-product-store-service.yaml) (Store service), [`labs/lab-01-anomaly-detection/k8s/01-product-store-configmap.yaml`](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/01-product-store-configmap.yaml) (App ConfigMap)
## Diagram (Mermaid)
```mermaid
flowchart TD
subgraph group_local["Local machine"]
node_scripts["Lab scripts
shell control"]
node_kindcfg["Kind config
cluster config"]
node_lab1deploy["Deploy lab
deploy script
[deploy-lab.sh]"]
node_traffic["Traffic gen
traffic scripts
[normal-traffic.sh]"]
node_portfwd["Port-forward
access script
[port-forward.sh]"]
end
subgraph group_cluster["Kind cluster"]
node_kindcluster[("Kind cluster
kubernetes runtime")]
node_aionamespace[("aiops-lab
namespace
[00-namespace.yaml]")]
node_monnamespace[("monitoring
namespace")]
end
subgraph group_appns["aiops-lab namespace"]
node_appsource["Product Store
python app
[server.py]"]
node_configmap["App ConfigMap
k8s config"]
node_podworkload[("Product pods
k8s workload")]
node_service["Store service
cluster service"]
node_cpualert["CPU alert
prometheus rule"]
node_predictive["Predictive rule
forecast rule"]
node_dashboard["Grafana dashboard
dashboard manifest"]
end
subgraph group_monns["monitoring namespace"]
node_monitoringvals["Helm values"]
node_prometheus[("Prometheus
metrics engine")]
node_alertmanager["Alertmanager
alert router"]
node_grafana["Grafana
dashboarding"]
end
node_slack(("Slack
external sink"))
node_scripts -->|"create/manage"| node_kindcluster
node_kindcfg -->|"configure"| node_kindcluster
node_lab1deploy -->|"deploy"| node_kindcluster
node_traffic -->|"generate requests"| node_service
node_portfwd -->|"expose"| node_service
node_kindcluster -->|"hosts"| node_aionamespace
node_kindcluster -->|"hosts"| node_monnamespace
node_appsource -->|"stored as"| node_configmap
node_configmap -->|"mounted into"| node_podworkload
node_podworkload -->|"exposed by"| node_service
node_cpualert -->|"evaluated by"| node_prometheus
node_predictive -->|"evaluated by"| node_prometheus
node_podworkload -->|"scraped by"| node_prometheus
node_prometheus -->|"fires alerts"| node_alertmanager
node_alertmanager -->|"notifies"| node_slack
node_prometheus -->|"queried by"| node_grafana
node_dashboard -->|"loads into"| node_grafana
node_monitoringvals -->|"configures"| node_prometheus
node_monitoringvals -->|"configures"| node_alertmanager
node_monitoringvals -->|"configures"| node_grafana
click node_scripts "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/tree/main/scripts"
click node_kindcfg "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/clusters/kind/aiops-lab-kind-config.yaml"
click node_lab1deploy "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/scripts/deploy-lab.sh"
click node_traffic "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/traffic/normal-traffic.sh"
click node_portfwd "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/scripts/port-forward.sh"
click node_aionamespace "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/00-namespace.yaml"
click node_appsource "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/apps/product-store/server.py"
click node_configmap "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/01-product-store-configmap.yaml"
click node_podworkload "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/02-product-store-deployment.yaml"
click node_service "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/03-product-store-service.yaml"
click node_cpualert "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/04-product-store-cpu-alert.yaml"
click node_predictive "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-02-predictive-analysis/product-store-predictive-rule.yaml"
click node_dashboard "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/05-grafana-dashboard.yaml"
click node_monitoringvals "https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/monitoring/values-template.yaml"
classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scripts,node_kindcfg,node_lab1deploy,node_traffic,node_portfwd toneBlue
class node_kindcluster,node_aionamespace,node_monnamespace toneAmber
class node_appsource,node_configmap,node_podworkload,node_service,node_cpualert,node_predictive,node_dashboard toneMint
class node_monitoringvals,node_prometheus,node_alertmanager,node_grafana toneRose
class node_slack toneNeutral
```
## Components
### Local machine
Scripts and traffic generators that control the lab from the laptop.
- [Lab scripts](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/tree/main/scripts): shell control · Top-level cluster lifecycle and validation scripts. · `scripts`
- [Kind config](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/clusters/kind/aiops-lab-kind-config.yaml): cluster config · Defines the local aiops-lab Kind topology. · `clusters/kind/aiops-lab-kind-config.yaml`
- [Deploy lab](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/scripts/deploy-lab.sh): deploy script · Installs the baseline anomaly-detection lab. · `labs/lab-01-anomaly-detection/scripts/deploy-lab.sh`
- [Traffic gen](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/traffic/normal-traffic.sh): traffic scripts · Baseline and CPU-burn request generators. · `labs/lab-01-anomaly-detection/traffic/normal-traffic.sh`
- [Port-forward](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/scripts/port-forward.sh): access script · Exposes cluster services on localhost. · `labs/lab-01-anomaly-detection/scripts/port-forward.sh`
### Kind cluster
Local Kubernetes runtime created as aiops-lab.
- **Kind cluster**: kubernetes runtime · Local multi-node cluster running the lab.
- [aiops-lab](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/00-namespace.yaml): namespace · Namespace for the sample workload. · `labs/lab-01-anomaly-detection/k8s/00-namespace.yaml`
- **monitoring**: namespace · Namespace for the observability stack.
### aiops-lab namespace
Product Store workload and lab-specific manifests.
- [Product Store](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/apps/product-store/server.py): python app · Simple service with health, product, and load endpoints. · `apps/product-store/server.py`
- [App ConfigMap](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/01-product-store-configmap.yaml): k8s config · Mounts the Python source into the workload. · `labs/lab-01-anomaly-detection/k8s/01-product-store-configmap.yaml`
- [Product pods](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/02-product-store-deployment.yaml): k8s workload · Two-pod Product Store deployment with CPU baseline requests. · `labs/lab-01-anomaly-detection/k8s/02-product-store-deployment.yaml`
- [Store service](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/03-product-store-service.yaml): cluster service · Internal Service fronting the Product Store pods. · `labs/lab-01-anomaly-detection/k8s/03-product-store-service.yaml`
- [CPU alert](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/04-product-store-cpu-alert.yaml): prometheus rule · Firing alert when CPU usage exceeds request ratio. · `labs/lab-01-anomaly-detection/k8s/04-product-store-cpu-alert.yaml`
- [Predictive rule](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-02-predictive-analysis/product-store-predictive-rule.yaml): forecast rule · Lab 2 forecast using predict_linear over recent CPU ratio. · `labs/lab-02-predictive-analysis/product-store-predictive-rule.yaml`
- [Grafana dashboard](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/k8s/05-grafana-dashboard.yaml): dashboard manifest · Dashboard for validating the same CPU behavior visually. · `labs/lab-01-anomaly-detection/k8s/05-grafana-dashboard.yaml`
### monitoring namespace
Prometheus stack and dashboarding components.
- [Helm values](https://github.com/mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs/blob/main/labs/lab-01-anomaly-detection/monitoring/values-template.yaml): helm values · Generated values for kube-prometheus-stack and Slack webhook wiring. · `labs/lab-01-anomaly-detection/monitoring/values-template.yaml`
- **Prometheus**: metrics engine · Scrapes metrics and evaluates recording and alert rules.
- **Alertmanager**: alert router · Routes firing alerts to the configured notification sink.
- **Grafana**: dashboarding · Reads from Prometheus and visualizes workload behavior.
### Other components
- **Slack**: external sink · Incoming webhook target for alert notifications.
## Connections
- Lab scripts → Kind cluster: create/manage · Creates, deletes, and validates the local cluster.
- Kind config → Kind cluster: configure · Defines the multi-node Kind topology.
- Deploy lab → Kind cluster: deploy · Installs the lab workload and monitoring manifests.
- Traffic gen → Store service: generate requests · Drives normal and CPU-heavy app traffic.
- Port-forward → Store service: expose · Makes the internal service reachable on localhost.
- Kind cluster → aiops-lab: hosts · Runs the application namespace inside the cluster.
- Kind cluster → monitoring: hosts · Runs the monitoring namespace inside the cluster.
- Product Store → App ConfigMap: stored as · Application source is mounted from a ConfigMap in lab 1.
- App ConfigMap → Product pods: mounted into · Supplies the runtime code to the deployment.
- Product pods → Store service: exposed by · Service fronts the two Product Store pods.
- CPU alert → Prometheus: evaluated by · Prometheus evaluates the CPU ratio alert rule.
- Predictive rule → Prometheus: evaluated by · Prometheus evaluates the predictive forecast rule.
- Product pods → Prometheus: scraped by · Prometheus scrapes pod metrics from the workload.
- Prometheus → Alertmanager: fires alerts · Firing alerts are handed off for routing.
- Alertmanager → Slack: notifies · Slack receives alerts via incoming webhook.
- Prometheus → Grafana: queried by · Grafana uses Prometheus as its data source.
- Grafana dashboard → Grafana: loads into · Dashboard manifest is deployed with the lab.
- Helm values → Prometheus: configures · Helm values configure the Prometheus stack.
- Helm values → Alertmanager: configures · Helm values wire notification routing.
- Helm values → Grafana: configures · Helm values enable Grafana in the stack.