How camilb/prometheus-kubernetes works
Project type and purpose
• This is an open‑source Kubernetes monitoring “stack” for AWS/GCP/Azure clusters, delivered as pure manifests, scripts and assets (no Helm).
• It provides a turnkey Prometheus Operator‑based solution, comprising Prometheus v2, Alertmanager, Grafana, and exporters (node‑exporter, kube‑state‑metrics, cAdvisor) all wired together by Kubernetes ServiceMonitors/CRDs.
Key directories and what they imply
• assets/…: user‑modifiable configuration templates (Alertmanager YAML, Grafana dashboards, Prometheus rules)
• manifests/…: Kubernetes YAML definitions organized by component (alertmanager, grafana, prometheus, exporters, self‑hosted core‑k8s)
• scripts/: helper scripts to assemble ConfigMaps or Secrets from assets
• tools/: shell wrappers for ingress, port‑forwarding and custom‑configuration workflows
• deploy & teardown: entry‑point scripts that apply or delete the whole monitoring stack
Main components
a. Kubernetes Cluster
• API Server & Control Plane (Scheduler, Controller Manager) – these are monitored via ServiceMonitors in manifests/k8s/self‑hosted
• Worker Nodes running kubelet, kube‑proxy, Docker/Container runtime
b. Prometheus Operator (Deployment + CRDs)
• Watches ServiceMonitor, Prometheus, Alertmanager CRs
• Dynamically instantiates Prometheus StatefulSet, Alertmanager StatefulSet, Service objects, RBAC
c. Prometheus StatefulSet
• Scrapes metrics from:
– kube‑api, kubelet (10250/TCP)
– kube‑dns (10054/10055)
– node‑exporter DaemonSet on each node (9100/TCP)
– kube‑state‑metrics Deployment
– Prometheus Operator metrics, Kubernetes components
• Persists TSDB in PVC
d. Alertmanager StatefulSet
• Receives alerts from Prometheus
• Routes notifications via email (SMTP) or Slack (token)
e. Grafana Deployment
• Reads dashboards from ConfigMaps (generated by scripts)
• Queries Prometheus via HTTP Service
f. Exporters & Metrics Providers
• node‑exporter (DaemonSet + Service) – host metrics
• kube‑state‑metrics (Deployment + Service) – cluster object metrics
• (Optional) cAdvisor on host port 4194 for container metrics on K8s ≥1.12
g. Ingress / Proxies (optional)
• NGINX Ingress Controller + BasicAuth Secret
• tools/*_proxy.sh for local port‑forwarding
Architectural patterns & design principles
• Operator pattern: declarative CRDs drive dynamic creation of StatefulSets, RBAC, Services
• Separation of concerns:
• Stateful workloads (Prometheus, Alertmanager)
• Stateless dashboards (Grafana)
• DaemonSets for per‑node exporters
• Dedicated namespace (e.g. “monitoring”)
• Immutability: YAML manifests under version control, generated ConfigMaps via scripts
• Extensibility: assets/ and tools/custom‑configuration allow injection of custom dashboards, alert rules, Prometheus operator flags