# nvidia/ai-factory-operations-agent architecture

> Architecture diagram of the GitHub repository nvidia/ai-factory-operations-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/ai-factory-operations-agent
- Repository: https://github.com/nvidia/ai-factory-operations-agent

## Overview

AI Factory Operations Agent deploys a governed, extensible operations agent on Kubernetes. Operators use the browser UI, while scripts and MCP clients can use headless interfaces; the agent gathers evidence through specialized operational tools and returns root-cause summaries or recommendations. Kubernetes, observability, Slurm, and optional cluster-management integrations are distinct capability branches. Sampled extensions show approval, policy, and mutation-audit layers around potentially mutating actions. The README documents the deployment and interface architecture; unsampled module internals and some wiring are represented only where documented.

Main entry points: [`helm/mosaic-stack/templates/openclaw.yaml`](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/templates/openclaw.yaml) (Agent runtime), [`helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/index.ts`](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/index.ts) (Remote SSH tools), [`helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/index.ts`](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/index.ts) (Kubernetes tools), [`helm/mosaic-stack/files/openclaw-seed/extensions/observability/index.ts`](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/observability/index.ts) (Metrics and logs)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User Interfaces"]
  node_ui["Browser UI<br/>[mosaic-ui.yaml]"]
  node_headless["HTTP headless API"]
  node_mcpbridge["MCP bridge<br/>[mosaic-mcp.mjs]"]
end

subgraph group_agent["Agent Governance"]
  node_openclaw["Agent runtime<br/>[openclaw.yaml]"]
  node_guard["Automation guard<br/>[index.ts]"]
  node_approval["Slack approval<br/>[slack-approval.ts]"]
  node_access["Access context"]
  node_diagnosticpolicy["Diagnostic policy"]
  node_ledger[("Mutation ledger<br/>[mutation-ledger.ts]")]
end

subgraph group_operations["Operations Tools"]
  node_kubernetes["Kubernetes tools<br/>[index.ts]"]
  node_kubepolicy["Kubernetes policy<br/>[policy.ts]"]
  node_kuberunner["Kubernetes runner<br/>[runner.ts]"]
  node_observability["Metrics and logs<br/>[index.ts]"]
  node_grafana["Grafana tools<br/>[grafana.ts]"]
  node_loki["Loki tools<br/>[loki.ts]"]
  node_slurm["Slurm tools<br/>[index.ts]"]
  node_bcm["BCM tools<br/>[index.ts]"]
  node_dcgm["DCGM tools<br/>[index.ts]"]
  node_iraop["Cluster management<br/>[index.ts]"]
  node_ssh["Remote SSH tools<br/>[index.ts]"]
  node_sshpolicy["SSH policy<br/>[policy.ts]"]
  node_sshrunner["SSH runner<br/>[runner.ts]"]
end

subgraph group_deployment["Deployment"]
  node_inference["Optional vLLM<br/>[vllm.yaml]"]
end

node_operator(("Cluster operator"))
node_automation(("Scripts and MCP clients"))
node_external["Configured cluster services"]

node_operator -->|"uses"| node_ui
node_automation -->|"sends prompt"| node_headless
node_automation -->|"uses"| node_mcpbridge
node_mcpbridge -->|"bridges requests"| node_headless
node_ui -->|"starts agent workflow"| node_openclaw
node_headless -->|"starts agent workflow"| node_openclaw
node_openclaw -.->|"checks access"| node_guard
node_guard -->|"configures access"| node_access
node_guard -->|"installs broker"| node_approval
node_approval -->|"records decisions"| node_ledger
node_access -->|"checks commands"| node_diagnosticpolicy
node_openclaw -.->|"dispatches tools"| node_kubernetes
node_openclaw -.->|"dispatches tools"| node_observability
node_openclaw -.->|"dispatches tools"| node_slurm
node_openclaw -.->|"dispatches tools"| node_bcm
node_openclaw -.->|"dispatches tools"| node_dcgm
node_openclaw -.->|"dispatches tools"| node_iraop
node_openclaw -.->|"dispatches tools"| node_ssh
node_kubernetes -->|"resolves policy"| node_kubepolicy
node_kubernetes -->|"runs commands"| node_kuberunner
node_kubernetes -->|"records mutations"| node_ledger
node_observability -->|"registers tools"| node_grafana
node_observability -->|"registers tools"| node_loki
node_observability -->|"queries services"| node_external
node_slurm -->|"inspects scheduler evidence"| node_external
node_bcm -->|"records mutations"| node_ledger
node_bcm -->|"requests approval"| node_access
node_ssh -->|"classifies requests"| node_sshpolicy
node_ssh -->|"runs commands"| node_sshrunner
node_ssh -->|"records mutations"| node_ledger
node_ssh -->|"requests approval"| node_access
node_ssh -->|"checks diagnostics"| node_diagnosticpolicy
node_sshrunner -->|"uses host type"| node_sshpolicy
node_openclaw -.->|"uses when configured"| node_inference
node_openclaw -->|"accesses configured modules"| node_external

click node_ui "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/templates/mosaic-ui.yaml"
click node_headless "https://github.com/nvidia/ai-factory-operations-agent/tree/main/helm/mosaic-stack"
click node_mcpbridge "https://github.com/nvidia/ai-factory-operations-agent/blob/main/utils/mosaic-mcp.mjs"
click node_openclaw "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/templates/openclaw.yaml"
click node_guard "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/automation-guard/index.ts"
click node_approval "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/automation-guard/slack-approval.ts"
click node_access "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/automation-context.ts"
click node_diagnosticpolicy "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/diagnostic-policy.ts"
click node_ledger "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/mutation-ledger.ts"
click node_kubernetes "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/index.ts"
click node_kubepolicy "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/policy.ts"
click node_kuberunner "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/runner.ts"
click node_observability "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/observability/index.ts"
click node_grafana "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/observability/grafana.ts"
click node_loki "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/observability/loki.ts"
click node_slurm "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/slurm/index.ts"
click node_bcm "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/bcm/index.ts"
click node_dcgm "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/dcgm/index.ts"
click node_iraop "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/iraop/index.ts"
click node_ssh "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/index.ts"
click node_sshpolicy "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/policy.ts"
click node_sshrunner "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/runner.ts"
click node_inference "https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/templates/vllm.yaml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_headless,node_mcpbridge,node_automation toneBlue
class node_openclaw,node_guard,node_approval,node_access,node_diagnosticpolicy,node_ledger toneAmber
class node_kubernetes,node_kubepolicy,node_kuberunner,node_observability,node_grafana,node_loki,node_slurm,node_bcm,node_dcgm,node_iraop,node_ssh,node_sshpolicy,node_sshrunner toneMint
class node_inference toneRose
class node_operator,node_external toneIndigo
```

## Components

### User Interfaces

- [Browser UI](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/templates/mosaic-ui.yaml): `helm/mosaic-stack/templates/mosaic-ui.yaml`
- [HTTP headless API](https://github.com/nvidia/ai-factory-operations-agent/tree/main/helm/mosaic-stack): `helm/mosaic-stack`
- [MCP bridge](https://github.com/nvidia/ai-factory-operations-agent/blob/main/utils/mosaic-mcp.mjs): `utils/mosaic-mcp.mjs`

### Agent Governance

- [Agent runtime](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/templates/openclaw.yaml): `helm/mosaic-stack/templates/openclaw.yaml`
- [Automation guard](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/automation-guard/index.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/automation-guard/index.ts`
- [Slack approval](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/automation-guard/slack-approval.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/automation-guard/slack-approval.ts`
- [Access context](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/automation-context.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/automation-context.ts`
- [Diagnostic policy](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/diagnostic-policy.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/diagnostic-policy.ts`
- [Mutation ledger](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/mutation-ledger.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/mutation-ledger.ts`

### Operations Tools

- [Kubernetes tools](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/index.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/index.ts`
- [Kubernetes policy](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/policy.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/policy.ts`
- [Kubernetes runner](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/runner.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/runner.ts`
- [Metrics and logs](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/observability/index.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/observability/index.ts`
- [Grafana tools](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/observability/grafana.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/observability/grafana.ts`
- [Loki tools](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/observability/loki.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/observability/loki.ts`
- [Slurm tools](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/slurm/index.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/slurm/index.ts`
- [BCM tools](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/bcm/index.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/bcm/index.ts`
- [DCGM tools](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/dcgm/index.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/dcgm/index.ts`
- [Cluster management](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/iraop/index.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/iraop/index.ts`
- [Remote SSH tools](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/index.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/index.ts`
- [SSH policy](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/policy.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/policy.ts`
- [SSH runner](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/runner.ts): `helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/runner.ts`

### Deployment

- [Optional vLLM](https://github.com/nvidia/ai-factory-operations-agent/blob/main/helm/mosaic-stack/templates/vllm.yaml): `helm/mosaic-stack/templates/vllm.yaml`

### Other components

- **Cluster operator**
- **Scripts and MCP clients**
- **Configured cluster services**

## Connections

- Cluster operator → Browser UI: uses (evidence: `README.md`)
- Scripts and MCP clients → HTTP headless API: sends prompt (evidence: `README.md`)
- Scripts and MCP clients → MCP bridge: uses (evidence: `README.md`)
- MCP bridge → HTTP headless API: bridges requests (evidence: `README.md`)
- Browser UI → Agent runtime: starts agent workflow (evidence: `README.md`)
- HTTP headless API → Agent runtime: starts agent workflow (evidence: `README.md`)
- Agent runtime → Automation guard: checks access
- Automation guard → Access context: configures access (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/automation-guard/index.ts`)
- Automation guard → Slack approval: installs broker (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/automation-guard/index.ts`)
- Slack approval → Mutation ledger: records decisions (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/automation-guard/slack-approval.ts`)
- Access context → Diagnostic policy: checks commands (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/automation-context.ts`)
- Agent runtime → Kubernetes tools: dispatches tools
- Agent runtime → Metrics and logs: dispatches tools
- Agent runtime → Slurm tools: dispatches tools
- Agent runtime → BCM tools: dispatches tools
- Agent runtime → DCGM tools: dispatches tools
- Agent runtime → Cluster management: dispatches tools
- Agent runtime → Remote SSH tools: dispatches tools
- Kubernetes tools → Kubernetes policy: resolves policy (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/index.ts`)
- Kubernetes tools → Kubernetes runner: runs commands (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/index.ts`)
- Kubernetes tools → Mutation ledger: records mutations (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/kubernetes/index.ts`)
- Metrics and logs → Grafana tools: registers tools (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/observability/index.ts`)
- Metrics and logs → Loki tools: registers tools (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/observability/index.ts`)
- Metrics and logs → Configured cluster services: queries services (evidence: `README.md`)
- Slurm tools → Configured cluster services: inspects scheduler evidence (evidence: `README.md`)
- BCM tools → Mutation ledger: records mutations (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/bcm/index.ts`)
- BCM tools → Access context: requests approval (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/bcm/index.ts`)
- Remote SSH tools → SSH policy: classifies requests (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/index.ts`)
- Remote SSH tools → SSH runner: runs commands (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/index.ts`)
- Remote SSH tools → Mutation ledger: records mutations (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/index.ts`)
- Remote SSH tools → Access context: requests approval (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/index.ts`)
- Remote SSH tools → Diagnostic policy: checks diagnostics (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/index.ts`)
- SSH runner → SSH policy: uses host type (evidence: `helm/mosaic-stack/files/openclaw-seed/extensions/remote-ssh/runner.ts`)
- Agent runtime → Optional vLLM: uses when configured (evidence: `README.md`)
- Agent runtime → Configured cluster services: accesses configured modules (evidence: `README.md`)
