# microsoft/aiopslab architecture

> Architecture diagram of the GitHub repository microsoft/aiopslab, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/aiopslab
- Repository: https://github.com/microsoft/aiopslab

## Overview

1. AIOpsLab is an evaluation framework, not a single app. Its architecture centers on an orchestration loop that connects an agent, a Kubernetes-hosted application, fault injection, workload generation, telemetry collection, and scoring.

2. There are two main execution shapes: local interactive use through `cli.py`, and remote service use through `service.py`. In the remote mode, the repo exposes an HTTP service with endpoints like health, problems, agents, and simulate so agents can be driven from outside the cluster host.

3. The orchestrator is the control plane of the system. `aiopslab/orchestrator/orchestrator.py` coordinates problem setup, agent registration, action parsing, session tracking, and evaluation. It is backed by task-specific abstractions for detection, localization, analysis, and mitigation, plus an action model for each task type.

4. Problems are the key composition unit. Each problem binds together five pieces: an application, a task, a fault injector, a workload generator, and an evaluator. The `aiopslab/orchestrator/problems/` tree contains many concrete benchmark scenarios, and `registry.py` is the catalog used to expose them.

5. Applications live under `aiopslab/service/apps/` and represent deployable cloud/microservice environments such as social network, hotel reservation, train ticket, flower, and TiDB operator setups. Application metadata in `aiopslab/service/metadata/` describes Helm release names, chart paths, and namespaces so the orchestrator can deploy them into Kubernetes.

6. Fault injection is split by target layer. `aiopslab/generators/fault/` includes application-level, OS/kernel, hardware, operator, observability, and virtualized injectors. Several scenarios also have prebuilt Chaos YAML manifests and shell scripts for Kubernetes-specific disruptions such as pod kill, pod failure, network delay, and loss.

7. Workload generation is separate from faults. `aiopslab/generators/workload/wrk.py` wraps wrk2-based load generation, and individual problems choose how to drive traffic into the deployed app. This keeps load creation isolated from problem logic and makes each benchmark scenario reproducible.

8. The service layer is the cluster-facing adapter. `aiopslab/service/helm.py`, `kubectl.py`, and `shell.py` abstract deployment and command execution against Kubernetes. This is the boundary between the Python orchestration code and the actual cluster runtime.

9. Observability is handled as its own subsystem under `aiopslab/observer/`. Prometheus, Filebeat, and Logstash are packaged there via Helm charts, while `log_api.py`, `metric_api.py`, and `trace_api.py` provide local persistence APIs for logs, metrics, and traces. The architecture therefore supports both in-cluster telemetry stacks and file-backed collection for experiments.

10. Agents are external plug-ins, not hard-coded into the orchestrator. The `clients/` directory contains LLM-backed agent adapters for OpenAI-style APIs, Azure identity-based auth, DeepSeek, Qwen, Llama, vLLM, and a React-style agent wrapper. The orchestrator interacts with agents through a simple async `get_action` contract, then evaluates traces and submitted solutions with qualitative and quantitative scorers in `aiopslab/orchestrator/evaluators/`.

11. The repo also includes deployment and environment orchestration around the core framework. `kind/` supports local Kubernetes clusters, while `scripts/ansible/` and `scripts/terraform/` provision remote clusters and configure the lab end-to-end. These are infrastructure boundaries rather than application logic, but they materially affect how the system is run.

12. Data flows through the system in a predictable loop: deploy app → start workload → inject fault → collect telemetry → agent observes state and returns an action → orchestrator applies the action through service adapters → session records the trace → evaluator scores the outcome. This makes the benchmark reproducible and gives a clear diagrammatic separation between environment, control, observation, and assessment.

Main entry points: [`aiopslab/orchestrator/orchestrator.py`](https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/orchestrator.py) (Orchestrator), [`aiopslab/service/apps/base.py`](https://github.com/microsoft/aiopslab/blob/main/aiopslab/service/apps/base.py) (Applications), [`aiopslab/orchestrator/problems/__init__.py`](https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/problems/__init__.py) (Problem specs), [`aiopslab/observer/observe.py`](https://github.com/microsoft/aiopslab/blob/main/aiopslab/observer/observe.py) (Observer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core orchestration"]
  node_cli["CLI<br/>entrypoint<br/>[cli.py]"]
  node_orchestrator["Orchestrator<br/>control plane<br/>[orchestrator.py]"]
  node_session["Session<br/>trace state<br/>[session.py]"]
  node_parser["Action parser<br/>command parsing<br/>[parser.py]"]
  node_tasks["Task model<br/>task abstractions<br/>[base.py]"]
  node_evals["Evaluators<br/>scoring<br/>[qualitative.py]"]
  node_problems["Problem registry<br/>benchmark catalog<br/>[registry.py]"]
  node_problem_impl["Problem specs<br/>scenarios<br/>[__init__.py]"]
end

subgraph group_service["Cluster service layer"]
  node_service_py["Service API<br/>http entrypoint<br/>[service.py]"]
  node_apps["Applications<br/>deployable apps<br/>[base.py]"]
  node_metadata["App metadata<br/>deployment metadata<br/>[__init__.py]"]
  node_helm["Helm adapter<br/>deploy helper<br/>[helm.py]"]
  node_kubectl["Kubectl adapter<br/>cluster exec<br/>[kubectl.py]"]
  node_shell["Shell adapter<br/>process exec<br/>[shell.py]"]
end

subgraph group_observability["Observability"]
  node_observer["Observer<br/>telemetry subsystem<br/>[observe.py]"]
  node_telemetry_api["Telemetry APIs<br/>local persistence<br/>[log_api.py]"]
  node_prom_stack[("Prometheus stack<br/>helm observability<br/>[Chart.yaml]")]
  node_logging_stack[("Logging stack<br/>helm observability<br/>[Chart.yaml]")]
end

subgraph group_generation["Signal generation"]
  node_faults["Fault injectors<br/>fault generation<br/>[base.py]"]
  node_workload["Workload generator<br/>load generation<br/>[wrk.py]"]
end

subgraph group_extensions["Extensions"]
  node_clients["Agent clients<br/>external agents<br/>[client.py]"]
  node_kind["Kind cluster<br/>local k8s"]
  node_provisioning["Provisioning<br/>infra automation<br/>[deploy.py]"]
end

node_cli -->|"start session"| node_orchestrator
node_service_py -->|"remote control"| node_orchestrator
node_orchestrator -->|"select problem"| node_problems
node_problems -->|"instantiate"| node_problem_impl
node_problem_impl -->|"bind app"| node_apps
node_problem_impl -->|"bind fault"| node_faults
node_problem_impl -->|"bind load"| node_workload
node_problem_impl -->|"bind scorer"| node_evals
node_orchestrator -->|"parse action"| node_parser
node_orchestrator -->|"task flow"| node_tasks
node_orchestrator -->|"record run"| node_session
node_orchestrator -->|"score outcome"| node_evals
node_apps -->|"deploy spec"| node_metadata
node_helm -->|"deploy app"| node_apps
node_kubectl -->|"operate cluster"| node_apps
node_shell -->|"exec commands"| node_kubectl
node_faults -->|"target env"| node_apps
node_faults -->|"stress telemetry"| node_observer
node_workload -->|"drive traffic"| node_apps
node_observer -->|"persist signals"| node_telemetry_api
node_observer -->|"metrics stack"| node_prom_stack
node_observer -->|"logs stack"| node_logging_stack
node_clients -->|"get_action"| node_orchestrator
node_kind -->|"local runtime"| node_service_py
node_provisioning -->|"provision cluster"| node_kind
node_provisioning -->|"setup lab"| node_service_py

click node_cli "https://github.com/microsoft/aiopslab/blob/main/cli.py"
click node_service_py "https://github.com/microsoft/aiopslab/blob/main/service.py"
click node_orchestrator "https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/orchestrator.py"
click node_session "https://github.com/microsoft/aiopslab/blob/main/aiopslab/session.py"
click node_parser "https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/parser.py"
click node_tasks "https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/tasks/base.py"
click node_evals "https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/evaluators/qualitative.py"
click node_problems "https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/problems/registry.py"
click node_problem_impl "https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/problems/__init__.py"
click node_apps "https://github.com/microsoft/aiopslab/blob/main/aiopslab/service/apps/base.py"
click node_metadata "https://github.com/microsoft/aiopslab/blob/main/aiopslab/service/metadata/__init__.py"
click node_helm "https://github.com/microsoft/aiopslab/blob/main/aiopslab/service/helm.py"
click node_kubectl "https://github.com/microsoft/aiopslab/blob/main/aiopslab/service/kubectl.py"
click node_shell "https://github.com/microsoft/aiopslab/blob/main/aiopslab/service/shell.py"
click node_faults "https://github.com/microsoft/aiopslab/blob/main/aiopslab/generators/fault/base.py"
click node_workload "https://github.com/microsoft/aiopslab/blob/main/aiopslab/generators/workload/wrk.py"
click node_observer "https://github.com/microsoft/aiopslab/blob/main/aiopslab/observer/observe.py"
click node_telemetry_api "https://github.com/microsoft/aiopslab/blob/main/aiopslab/observer/log_api.py"
click node_prom_stack "https://github.com/microsoft/aiopslab/blob/main/aiopslab/observer/prometheus/prometheus/Chart.yaml"
click node_logging_stack "https://github.com/microsoft/aiopslab/blob/main/aiopslab/observer/filebeat/Chart.yaml"
click node_clients "https://github.com/microsoft/aiopslab/blob/main/clients/client.py"
click node_kind "https://github.com/microsoft/aiopslab/blob/main/kind/kind-config-x86.yaml"
click node_provisioning "https://github.com/microsoft/aiopslab/blob/main/scripts/terraform/deploy.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_orchestrator,node_session,node_parser,node_tasks,node_evals,node_problems,node_problem_impl toneBlue
class node_service_py,node_apps,node_metadata,node_helm,node_kubectl,node_shell toneAmber
class node_observer,node_telemetry_api,node_prom_stack,node_logging_stack toneMint
class node_faults,node_workload toneRose
class node_clients,node_kind,node_provisioning toneIndigo
```

## Components

### Core orchestration

Control plane that runs benchmark sessions and evaluates outcomes.

- [CLI](https://github.com/microsoft/aiopslab/blob/main/cli.py): entrypoint · Local interactive entrypoint for running AIOpsLab sessions. · `cli.py`
- [Orchestrator](https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/orchestrator.py): control plane · Coordinates problem setup, agent registration, action parsing, sessions, and evaluation. · `aiopslab/orchestrator/orchestrator.py`
- [Session](https://github.com/microsoft/aiopslab/blob/main/aiopslab/session.py): trace state · Tracks a benchmark run, its actions, and result artifacts. · `aiopslab/session.py`
- [Action parser](https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/parser.py): command parsing · Converts agent output into executable actions for the orchestrator. · `aiopslab/orchestrator/parser.py`
- [Task model](https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/tasks/base.py): task abstractions · Task-specific abstractions for detection, localization, analysis, and mitigation. · `aiopslab/orchestrator/tasks/base.py`
- [Evaluators](https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/evaluators/qualitative.py): scoring · Qualitative and quantitative scorers for benchmark outcomes. · `aiopslab/orchestrator/evaluators/qualitative.py`
- [Problem registry](https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/problems/registry.py): benchmark catalog · Catalog of benchmark scenarios that bind app, task, fault, workload, and evaluator. · `aiopslab/orchestrator/problems/registry.py`
- [Problem specs](https://github.com/microsoft/aiopslab/blob/main/aiopslab/orchestrator/problems/__init__.py): scenarios · Concrete benchmark scenario implementations. · `aiopslab/orchestrator/problems/__init__.py`

### Cluster service layer

Deployment and execution adapters that talk to Kubernetes and the lab cluster.

- [Service API](https://github.com/microsoft/aiopslab/blob/main/service.py): http entrypoint · Remote HTTP service exposing health, problems, agents, and simulation endpoints. · `service.py`
- [Applications](https://github.com/microsoft/aiopslab/blob/main/aiopslab/service/apps/base.py): deployable apps · Cloud and microservice environments deployed into Kubernetes. · `aiopslab/service/apps/base.py`
- [App metadata](https://github.com/microsoft/aiopslab/blob/main/aiopslab/service/metadata/__init__.py): deployment metadata · Helm release names, chart paths, and namespaces for app deployment. · `aiopslab/service/metadata/__init__.py`
- [Helm adapter](https://github.com/microsoft/aiopslab/blob/main/aiopslab/service/helm.py): deploy helper · Helm-based deployment and release management. · `aiopslab/service/helm.py`
- [Kubectl adapter](https://github.com/microsoft/aiopslab/blob/main/aiopslab/service/kubectl.py): cluster exec · Runs cluster commands against Kubernetes resources. · `aiopslab/service/kubectl.py`
- [Shell adapter](https://github.com/microsoft/aiopslab/blob/main/aiopslab/service/shell.py): process exec · General shell execution boundary used by the service layer. · `aiopslab/service/shell.py`

### Observability

Telemetry stack and local persistence APIs for logs, metrics, and traces.

- [Observer](https://github.com/microsoft/aiopslab/blob/main/aiopslab/observer/observe.py): telemetry subsystem · Packages in-cluster telemetry and file-backed collection APIs. · `aiopslab/observer/observe.py`
- [Telemetry APIs](https://github.com/microsoft/aiopslab/blob/main/aiopslab/observer/log_api.py): local persistence · Local APIs for logs, metrics, traces, and extraction utilities. · `aiopslab/observer/log_api.py`
- [Prometheus stack](https://github.com/microsoft/aiopslab/blob/main/aiopslab/observer/prometheus/prometheus/Chart.yaml): helm observability · Prometheus-based metrics collection and exporters. · `aiopslab/observer/prometheus/prometheus/Chart.yaml`
- [Logging stack](https://github.com/microsoft/aiopslab/blob/main/aiopslab/observer/filebeat/Chart.yaml): helm observability · Filebeat and Logstash components for log collection and processing. · `aiopslab/observer/filebeat/Chart.yaml`

### Signal generation

Fault and workload generators used by problems to create reproducible scenarios.

- [Fault injectors](https://github.com/microsoft/aiopslab/blob/main/aiopslab/generators/fault/base.py): fault generation · Layered injectors for app, OS, hardware, operator, observability, and virtualized faults. · `aiopslab/generators/fault/base.py`
- [Workload generator](https://github.com/microsoft/aiopslab/blob/main/aiopslab/generators/workload/wrk.py): load generation · Wrk2-based traffic generator used to drive target applications. · `aiopslab/generators/workload/wrk.py`

### Extensions

External agents and infrastructure used to run the lab in different environments.

- [Agent clients](https://github.com/microsoft/aiopslab/blob/main/clients/client.py): external agents · LLM-backed agent adapters accessed through a simple async action contract. · `clients/client.py`
- [Kind cluster](https://github.com/microsoft/aiopslab/blob/main/kind/kind-config-x86.yaml): local k8s · Local Kubernetes cluster setup for interactive use. · `kind/kind-config-x86.yaml`
- [Provisioning](https://github.com/microsoft/aiopslab/blob/main/scripts/terraform/deploy.py): infra automation · Ansible and Terraform scripts for remote cluster setup and lab provisioning. · `scripts/terraform/deploy.py`

## Connections

- CLI → Orchestrator: start session · CLI drives the control plane for local interactive runs.
- Service API → Orchestrator: remote control · HTTP service exposes orchestrator functions to external callers.
- Orchestrator → Problem registry: select problem · The orchestrator loads a benchmark scenario from the registry.
- Problem registry → Problem specs: instantiate · Registry resolves to concrete problem definitions.
- Problem specs → Applications: bind app · Each problem anchors a specific application environment.
- Problem specs → Fault injectors: bind fault · Each problem chooses an appropriate injector or disruption script.
- Problem specs → Workload generator: bind load · Each problem specifies how traffic is generated.
- Problem specs → Evaluators: bind scorer · Each problem attaches evaluation logic.
- Orchestrator → Action parser: parse action · Agent output is converted into executable actions.
- Orchestrator → Task model: task flow · Task abstractions shape detection, localization, analysis, and mitigation flows.
- Orchestrator → Session: record run · Session state stores traces and run artifacts.
- Orchestrator → Evaluators: score outcome · Completed sessions are evaluated against qualitative and quantitative criteria.
- Applications → App metadata: deploy spec · Application definitions rely on metadata for Helm and namespace details.
- Helm adapter → Applications: deploy app · Helm adapter installs or updates application releases.
- Kubectl adapter → Applications: operate cluster · Kubectl adapter performs cluster-side operations on deployed apps.
- Shell adapter → Kubectl adapter: exec commands · Shell layer underpins command execution used by cluster adapters.
- Fault injectors → Applications: target env · Fault injectors act on application and cluster components.
- Fault injectors → Observer: stress telemetry · Fault scenarios may target observability infrastructure as part of a benchmark.
- Workload generator → Applications: drive traffic · Load generator sends requests into deployed applications.
- Observer → Telemetry APIs: persist signals · Observer includes local APIs for storing and retrieving telemetry.
- Observer → Prometheus stack: metrics stack · Prometheus-based collection is packaged as a separate subsystem.
- Observer → Logging stack: logs stack · Filebeat and Logstash provide log collection and processing.
- Agent clients → Orchestrator: get_action · External agents supply actions through the orchestrator contract.
- Kind cluster → Service API: local runtime · Kind provides the local Kubernetes environment used by the service layer.
- Provisioning → Kind cluster: provision cluster · Automation prepares local or remote cluster environments.
- Provisioning → Service API: setup lab · Infrastructure scripts configure the lab runtime around the service boundary.
