How microsoft/aiopslab works
1. AIOpsLab is an evaluation framework, not a single app. Its architecture centers on an orchestration loop that connects an agent, a Kubernetes-hosted application, fault injection, workload generation, telemetry collection, and scoring.
2. There are two main execution shapes: local interactive use through cli.py, and remote service use through service.py. In the remote mode, the repo exposes an HTTP service with endpoints like health, problems, agents, and simulate so agents can be driven from outside the cluster host.
3. The orchestrator is the control plane of the system. aiopslab/orchestrator/orchestrator.py coordinates problem setup, agent registration, action parsing, session tracking, and evaluation. It is backed by task-specific abstractions for detection, localization, analysis, and mitigation, plus an action model for each task type.
4. Problems are the key composition unit. Each problem binds together five pieces: an application, a task, a fault injector, a workload generator, and an evaluator. The aiopslab/orchestrator/problems/ tree contains many concrete benchmark scenarios, and registry.py is the catalog used to expose them.
5. Applications live under aiopslab/service/apps/ and represent deployable cloud/microservice environments such as social network, hotel reservation, train ticket, flower, and TiDB operator setups. Application metadata in aiopslab/service/metadata/ describes Helm release names, chart paths, and namespaces so the orchestrator can deploy them into Kubernetes.
6. Fault injection is split by target layer. aiopslab/generators/fault/ includes application-level, OS/kernel, hardware, operator, observability, and virtualized injectors. Several scenarios also have prebuilt Chaos YAML manifests and shell scripts for Kubernetes-specific disruptions such as pod kill, pod failure, network delay, and loss.
7. Workload generation is separate from faults. aiopslab/generators/workload/wrk.py wraps wrk2-based load generation, and individual problems choose how to drive traffic into the deployed app. This keeps load creation isolated from problem logic and makes each benchmark scenario reproducible.
8. The service layer is the cluster-facing adapter. aiopslab/service/helm.py, kubectl.py, and shell.py abstract deployment and command execution against Kubernetes. This is the boundary between the Python orchestration code and the actual cluster runtime.
9. Observability is handled as its own subsystem under aiopslab/observer/. Prometheus, Filebeat, and Logstash are packaged there via Helm charts, while log_api.py, metric_api.py, and trace_api.py provide local persistence APIs for logs, metrics, and traces. The architecture therefore supports both in-cluster telemetry stacks and file-backed collection for experiments.
10. Agents are external plug-ins, not hard-coded into the orchestrator. The clients/ directory contains LLM-backed agent adapters for OpenAI-style APIs, Azure identity-based auth, DeepSeek, Qwen, Llama, vLLM, and a React-style agent wrapper. The orchestrator interacts with agents through a simple async get_action contract, then evaluates traces and submitted solutions with qualitative and quantitative scorers in aiopslab/orchestrator/evaluators/.
11. The repo also includes deployment and environment orchestration around the core framework. kind/ supports local Kubernetes clusters, while scripts/ansible/ and scripts/terraform/ provision remote clusters and configure the lab end-to-end. These are infrastructure boundaries rather than application logic, but they materially affect how the system is run.
12. Data flows through the system in a predictable loop: deploy app → start workload → inject fault → collect telemetry → agent observes state and returns an action → orchestrator applies the action through service adapters → session records the trace → evaluator scores the outcome. This makes the benchmark reproducible and gives a clear diagrammatic separation between environment, control, observation, and assessment.
Main entry points: orchestrator.py Orchestrator · base.py Applications · __init__.py Problem specs · observe.py Observer
Components 23
Core orchestration
Control plane that runs benchmark sessions and evaluates outcomes.
- CLI · Local interactive entrypoint for running AIOpsLab sessions.
cli.py - Orchestrator · Coordinates problem setup, agent registration, action parsing, sessions, and evaluation.
aiopslab/orchestrator/orchestrator.py - Session · Tracks a benchmark run, its actions, and result artifacts.
aiopslab/session.py - Action parser · Converts agent output into executable actions for the orchestrator.
aiopslab/orchestrator/parser.py - Task model · Task-specific abstractions for detection, localization, analysis, and mitigation.
aiopslab/orchestrator/tasks/base.py - Evaluators · Qualitative and quantitative scorers for benchmark outcomes.
aiopslab/orchestrator/evaluators/qualitative.py - Problem registry · Catalog of benchmark scenarios that bind app, task, fault, workload, and evaluator.
aiopslab/orchestrator/problems/registry.py - Problem specs · Concrete benchmark scenario implementations.
aiopslab/orchestrator/problems/__init__.py
Cluster service layer
Deployment and execution adapters that talk to Kubernetes and the lab cluster.
- Service API · Remote HTTP service exposing health, problems, agents, and simulation endpoints.
service.py - Applications · Cloud and microservice environments deployed into Kubernetes.
aiopslab/service/apps/base.py - App metadata · Helm release names, chart paths, and namespaces for app deployment.
aiopslab/service/metadata/__init__.py - Helm adapter · Helm-based deployment and release management.
aiopslab/service/helm.py - Kubectl adapter · Runs cluster commands against Kubernetes resources.
aiopslab/service/kubectl.py - Shell adapter · General shell execution boundary used by the service layer.
aiopslab/service/shell.py
Observability
Telemetry stack and local persistence APIs for logs, metrics, and traces.
- Observer · Packages in-cluster telemetry and file-backed collection APIs.
aiopslab/observer/observe.py - Telemetry APIs · Local APIs for logs, metrics, traces, and extraction utilities.
aiopslab/observer/log_api.py - Prometheus stack · Prometheus-based metrics collection and exporters.
aiopslab/observer/prometheus/prometheus/Chart.yaml - Logging stack · Filebeat and Logstash components for log collection and processing.
aiopslab/observer/filebeat/Chart.yaml
Signal generation
Fault and workload generators used by problems to create reproducible scenarios.
- Fault injectors · Layered injectors for app, OS, hardware, operator, observability, and virtualized faults.
aiopslab/generators/fault/base.py - Workload generator · Wrk2-based traffic generator used to drive target applications.
aiopslab/generators/workload/wrk.py
Extensions
External agents and infrastructure used to run the lab in different environments.
- Agent clients · LLM-backed agent adapters accessed through a simple async action contract.
clients/client.py - Kind cluster · Local Kubernetes cluster setup for interactive use.
kind/kind-config-x86.yaml - Provisioning · Ansible and Terraform scripts for remote cluster setup and lab provisioning.
scripts/terraform/deploy.py
Connections 26
- CLI → Orchestrator: start session · CLI drives the control plane for local interactive runs.
- Service API → Orchestrator: remote control · HTTP service exposes orchestrator functions to external callers.
- Orchestrator → Problem registry: select problem · The orchestrator loads a benchmark scenario from the registry.
- Problem registry → Problem specs: instantiate · Registry resolves to concrete problem definitions.
- Problem specs → Applications: bind app · Each problem anchors a specific application environment.
- Problem specs → Fault injectors: bind fault · Each problem chooses an appropriate injector or disruption script.
- Problem specs → Workload generator: bind load · Each problem specifies how traffic is generated.
- Problem specs → Evaluators: bind scorer · Each problem attaches evaluation logic.
- Orchestrator → Action parser: parse action · Agent output is converted into executable actions.
- Orchestrator → Task model: task flow · Task abstractions shape detection, localization, analysis, and mitigation flows.
- Orchestrator → Session: record run · Session state stores traces and run artifacts.
- Orchestrator → Evaluators: score outcome · Completed sessions are evaluated against qualitative and quantitative criteria.
- Applications → App metadata: deploy spec · Application definitions rely on metadata for Helm and namespace details.
- Helm adapter → Applications: deploy app · Helm adapter installs or updates application releases.
- Kubectl adapter → Applications: operate cluster · Kubectl adapter performs cluster-side operations on deployed apps.
- Shell adapter → Kubectl adapter: exec commands · Shell layer underpins command execution used by cluster adapters.
- Fault injectors → Applications: target env · Fault injectors act on application and cluster components.
- Fault injectors → Observer: stress telemetry · Fault scenarios may target observability infrastructure as part of a benchmark.
- Workload generator → Applications: drive traffic · Load generator sends requests into deployed applications.
- Observer → Telemetry APIs: persist signals · Observer includes local APIs for storing and retrieving telemetry.
- Observer → Prometheus stack: metrics stack · Prometheus-based collection is packaged as a separate subsystem.
- Observer → Logging stack: logs stack · Filebeat and Logstash provide log collection and processing.
- Agent clients → Orchestrator: get_action · External agents supply actions through the orchestrator contract.
- Kind cluster → Service API: local runtime · Kind provides the local Kubernetes environment used by the service layer.
- Provisioning → Kind cluster: provision cluster · Automation prepares local or remote cluster environments.
- Provisioning → Service API: setup lab · Infrastructure scripts configure the lab runtime around the service boundary.