How mlops-devops-04-26/kubernetes-ai-log-analysis-rca-slack-bot works
1. Repository purpose
- This is a Kubernetes-based demo system for AIOps: a Python Slack bot analyzes cluster logs/telemetry with an LLM, while a sample microservice and load generator create behavior to observe.
- The repo is organized around three runtime domains: aiops-bot, microservices, and monitoring.
2. Top-level deployment model
- Everything is designed to run inside a local KIND cluster.
- The kind-config.yaml and README show several NodePort mappings so the cluster can be accessed from the host for Grafana, Prometheus, Alertmanager, and the sample service.
- The setup.sh and update-deployments.sh scripts indicate the repo expects repeatable local environment setup and image reference updates.
3. AIOps bot subsystem
- aiops-bot/ contains a Python service packaged as a Docker image and deployed to Kubernetes with its own deployment.yaml and rbac.yaml.
- Its boundary is important: it is not a general app server, but an operator-facing bot that needs Kubernetes read permissions and secret access.
- It depends on external integrations configured via environment variables/secrets: Slack Bot Token, Slack App Token, Slack webhook/channel, and HuggingFace API/model.
4. Bot data flow
- The bot receives Slack commands/events, queries cluster information/logs/metrics, sends prompts to the HuggingFace model API, and returns analysis/RCA back to Slack.
- The repo structure implies this is the “intelligence” layer that sits between Kubernetes telemetry and Slack notifications.
- RBAC is a key boundary here: the bot can only do what rbac.yaml grants.
5. Sample workload / microservice subsystem
- microservices/service1/ is the target application workload.
- It has its own Dockerfile, Python app, requirements, and Kubernetes deployment manifest, meaning it is meant to be deployed independently as a simple observable service.
- This service provides the system under test for metrics, errors, and log patterns that the bot and monitoring stack can react to.
6. Load generation subsystem
- load-generator/ is a separate Python containerized workload that likely creates traffic or failure pressure against the sample service.
- Its role is to stimulate the microservice so the monitoring and RCA pipeline has events to detect.
- Architecturally it is an upstream producer of load, not a user-facing service.
7. Observability stack
- monitoring/ contains the Prometheus, Grafana, Alertmanager, and alert-rule configuration.
- The repository uses a mix of Helm values (prometheus-values.yaml, grafana-values.yaml, loki-values.yaml) and raw Kubernetes manifests (grafana-deployment.yaml, prometheus-service.yaml, alertmanager-config.yaml, ingress.yaml, etc.).
- This stack is the telemetry backend that collects metrics, evaluates rules, and visualizes system state.
8. Alerting and dashboard path
- monitoring/prometheus-rules.yaml and monitoring/microservices-alerts.yaml define the alerting conditions for the system.
- Prometheus evaluates these rules, Alertmanager receives firing alerts, and Grafana provides visualization/dashboard access.
- The repo also includes a Grafana dashboard manifest, implying a prebuilt view into the service and cluster behavior.
9. Logging/metrics boundary
- The presence of loki-values.yaml indicates logs may also be centralized through Loki in addition to Prometheus metrics.
- This matters because the RCA bot is described as a log-analysis bot; the architecture likely combines metrics from Prometheus with logs from Loki or Kubernetes logs for diagnosis.
- The monitoring layer is therefore the evidence source for the bot’s analysis pipeline.
10. Infrastructure and runtime assumptions
- All application components are Python-based and containerized with Docker.
- Kubernetes is the runtime, with namespaces separating concerns: aiops-bot, microservices, and monitoring.
- External services are limited but important: Slack for interaction and HuggingFace Router/API for LLM inference.
11. Operational workflow
- The README shows a workflow of: build images, push to Docker Hub, patch deployment image references, create a KIND cluster, install monitoring, deploy workloads, then deploy the bot.
- That means the repo is designed for a local reproducible demo environment, not just a single application.
- The update-deployments.sh script is the main coupling point between image publishing and Kubernetes manifests.
12. CI/CD and automation
- The .github/workflows/main.yml file indicates at least one GitHub Actions pipeline exists for automated checks/builds/deployment-related tasks.
- Even without its exact contents, it is part of the repo’s delivery path and likely validates or packages the containerized components.
13. How to draw the architecture
- A good diagram should show:
- Host machine running Docker, kubectl, Helm, and KIND.
- Three Kubernetes namespaces or logical zones.
- The sample service and load generator interacting inside microservices.
- Monitoring components collecting cluster/app telemetry.
- The bot consuming telemetry/cluster data, calling HuggingFace, and returning results to Slack.
- The main directional flow is: load generator → service1 → telemetry/alerts/logs → bot analysis → Slack response.
Main entry points: prometheus-values.yaml Prometheus · aiops-bot/app.py Bot app · monitoring Monitoring · service1/app.py Service 1
Components 17
Host machine
Local development machine running the demo tooling and cluster lifecycle scripts
- Setup scripts · Build, push, and patch deployment references for repeatable local setup
setup.sh - Image patcher · Updates Kubernetes manifests with published image tags
update-deployments.sh - KIND config · Local cluster topology and NodePort exposure settings
kind-config.yaml - CI workflow · Repository automation for validation/build/deploy tasks
.github/workflows/main.yml
KIND cluster
Local Kubernetes cluster hosting the bot, workloads, and observability stack
- AIOps bot · Operator-facing Slack bot that analyzes telemetry with an LLM
aiops-bot - Bot app · Receives Slack events, gathers cluster evidence, and produces RCA responses
aiops-bot/app.py - Bot RBAC · Kubernetes permissions boundary for reading cluster data and secrets
aiops-bot/rbac.yaml - Microservices · Sample workload domain used to generate observable behavior
microservices - Service 1 · Observable sample service targeted by load and monitored for failures
microservices/service1/app.py - Load generator · Produces traffic or pressure against the sample service
load-generator/app.py - Monitoring · Telemetry and alerting stack for metrics, logs, and dashboards
monitoring - Prometheus · Scrapes metrics and evaluates alert rules
monitoring/prometheus-values.yaml - Alertmanager · Receives firing alerts and routes notifications
monitoring/alertmanager-config.yaml - Grafana · Visualizes cluster and workload state through dashboards
monitoring/grafana-deployment.yaml - Loki · Central log store used for log-driven diagnosis
monitoring/loki-values.yaml
External services
Outside integrations used for chat and model inference
- Slack · User-facing interface for bot commands and analysis responses
- HuggingFace API · External model endpoint used for prompt-based analysis
Connections 22
- Setup scripts → KIND config: create cluster · Uses the KIND config to stand up the local environment
- Setup scripts → Image patcher: patch images · Coordinates image reference updates before deployment
- CI workflow → Setup scripts: automate · Runs repository automation around build and deploy steps
- AIOps bot → Bot app: runs · Namespace hosts the bot container
- AIOps bot → Bot RBAC: binds · Attaches permissions required for cluster and secret access
- Bot app → Slack: commands/responses · Receives requests and posts analysis back to Slack
- Bot app → HuggingFace API: prompts · Sends evidence to the LLM for diagnosis and summarization
- Bot app → Prometheus: queries metrics · Reads cluster and workload telemetry for analysis
- Bot app → Loki: reads logs · Consumes logs when available for RCA context
- Bot RBAC → Prometheus: permits access · Allows the bot to read cluster resources and telemetry sources
- Microservices → Service 1: runs · Namespace hosts the sample application
- Microservices → Load generator: runs · Namespace hosts the traffic generator workload
- Load generator → Service 1: triggers traffic · Generates requests or pressure against the service
- Service 1 → Prometheus: exposes metrics · Service emits signals scraped by the monitoring stack
- Service 1 → Loki: emits logs · Application logs feed centralized log analysis
- Monitoring → Prometheus: hosts · Monitoring namespace contains the metrics backend
- Monitoring → Alertmanager: hosts · Monitoring namespace contains alert routing
- Monitoring → Grafana: hosts · Monitoring namespace contains dashboards
- Monitoring → Loki: hosts · Monitoring namespace contains log storage
- Prometheus → Alertmanager: fires alerts · Alert rules generate notifications when conditions match
- Prometheus → Grafana: visualizes · Grafana dashboards are typically backed by Prometheus data
- Grafana → Slack: operator view · Provides human visibility into system behavior