mlops-devops-04-26/aiops-anomaly-detection-predictive-analysis-labs

1. What the repo is

- This repository is a local Kubernetes AIOps lab, not a general-purpose app. It demonstrates two monitoring patterns on the same sample service: real-time CPU anomaly detection and predictive analysis.

- The whole system is meant to run on a laptop using a local Kind cluster.

2. Execution boundary

- The architecture splits into two main environments: the local machine and the Kind Kubernetes cluster.

- The local machine runs setup scripts, traffic generators, and port-forwards. The cluster runs the Product Store app plus the monitoring stack.

3. Cluster topology

- The Kind setup uses a dedicated cluster named aiops-lab with 1 control-plane node and 2 worker nodes.

- This matters architecturally because the labs are designed to exercise multi-node Kubernetes scheduling and monitoring behavior, while still staying fully local.

4. Application subsystem

- The sample workload is Product Store, implemented in apps/product-store/server.py.

- It exposes three endpoints: /health, /products, and /load.

- The app is intentionally simple: normal requests are low-cost, while /load burns CPU to simulate an incident.

5. How the app is deployed

- In Lab 1, the app is not built as a container image. Instead, the Python source is stored in a ConfigMap and mounted into the deployment.

- The app runs as two pods in the aiops-lab namespace, behind an internal Kubernetes Service.

- CPU and memory requests are explicitly set, and the CPU request becomes the baseline used by the alert logic.

6. Monitoring stack

- The monitoring layer is installed in the monitoring namespace using the kube-prometheus-stack Helm chart.

- This gives the lab its three core observability components:

- Prometheus for scraping metrics and evaluating rules

- Alertmanager for routing alerts

- Grafana for dashboarding

- The Slack webhook is injected through generated Helm values, keeping secrets out of git.

7. Lab 1 data flow: anomaly detection

- Traffic reaches Product Store, CPU usage rises, and Prometheus scrapes the pod metrics.

- The Lab 1 rule compares actual CPU usage / requested CPU.

- If the ratio stays above 0.75 for 2 minutes, Prometheus marks the alert as firing.

- Alertmanager receives that firing alert and sends the notification to Slack through an incoming webhook.

8. Lab 1 visualization path

- Grafana reads from Prometheus and shows the same CPU behavior on a dashboard.

- The dashboard is part of the repo’s Kubernetes manifests, so it is deployed alongside the app and alert rule.

- This gives a second validation path: the alert is not only sent, it is also visible as a metric trend.

9. Lab 2 data flow: predictive analysis

- Lab 2 keeps the same app and monitoring stack, but adds a second Prometheus rule.

- That rule uses predict_linear on recent CPU ratio samples to forecast whether the workload will exceed the same threshold soon.

- This is a forward-looking warning, not ML: it is Prometheus time-series forecasting over the last 10 minutes, predicting 5 minutes ahead.

10. Alerting boundary

- Alert routing is intentionally separated from alert detection.

- Prometheus decides whether a condition is true, and Alertmanager decides where it goes.

- Slack is the external notification sink; it is not part of the cluster and is configured entirely through the webhook.

11. Traffic generation

- The repo includes shell scripts that generate two traffic patterns:

- normal baseline requests to /health and /products

- CPU-heavy requests to /load

- This traffic is the input that drives the monitoring story; there is no synthetic telemetry generator separate from the app traffic itself.

12. Operational scripts

- Top-level scripts handle cluster lifecycle and validation:

- create/delete the Kind cluster

- check required tools

- verify that the lab deployed correctly

- Lab-specific scripts handle deployment, cleanup, traffic, and port-forwarding.

- Architecturally, these scripts are the control plane for the lab, while Kubernetes is the runtime plane.

13. Access pattern

- The services are internal by default.

- Local access is provided through kubectl port-forward, exposing Grafana and the Product Store API on localhost without changing cluster networking or service type.

14. Repository structure as system boundaries

- clusters/kind/ defines the local cluster substrate.

- apps/product-store/ contains the workload source.

- labs/lab-01-anomaly-detection/ contains the baseline monitoring app manifests, rule, dashboard, scripts, and traffic.

- labs/lab-02-predictive-analysis/ adds the predictive rule on top of the same base environment.

- diagrams/ documents the intended end-to-end flows, especially the alert path to Slack.

15. What to show in an architecture diagram

- The key nodes are: local machine, Kind cluster, aiops-lab namespace, monitoring namespace, Product Store pods, Prometheus, Alertmanager, Grafana, and Slack.

- The main arrows are:

- local scripts → Kind cluster

- traffic → Product Store Service → Product Store pods

- Product Store pods → Prometheus

- Prometheus → Alertmanager → Slack

- Prometheus → Grafana

- Lab 2 adds one extra rule path from Prometheus evaluation to a predictive alert, but reuses the same runtime and monitoring stack.