# web-infra-dev/midscene architecture

> Architecture diagram of the GitHub repository web-infra-dev/midscene, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/web-infra-dev/midscene
- Repository: https://github.com/web-infra-dev/midscene

## Overview

Midscene is a vision-driven GUI automation agent and E2E testing toolkit for web, mobile, and desktop. The main workflow is a user’s natural-language task entering an Agent, which plans and executes visual actions through a platform integration, checks the visible result, and produces observable execution data for reports. Distinct supporting branches cover declarative YAML tests and custom Nodes, interactive reports, and the Playground. Several links use README-level documentation because the sampled excerpts do not expose their runtime wiring.

Main entry points: [`packages/core/src/agent/agent.ts`](https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/agent/agent.ts) (GUI Agent), [`packages/core/src/agent/tasks.ts`](https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/agent/tasks.ts) (Task execution), [`packages/playground-app/src/controller/usePlaygroundController.ts`](https://github.com/web-infra-dev/midscene/blob/main/packages/playground-app/src/controller/usePlaygroundController.ts) (Playground controller), [`packages/core/src/yaml/execution-session.ts`](https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/yaml/execution-session.ts) (YAML execution)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_agent["Agent execution"]
  node_agent["GUI Agent<br/>[agent.ts]"]
  node_tasks["Task execution<br/>[tasks.ts]"]
  node_task_build["Task building<br/>[task-builder.ts]"]
  node_ui_utils["UI utilities<br/>[ui-utils.ts]"]
  node_model["Model adapters"]
end

subgraph group_platforms["Platform integrations"]
  node_web["Web integration"]
  node_android["Android integration"]
  node_ios["iOS integration"]
  node_desktop["Desktop integration"]
end

subgraph group_testing["Test framework"]
  node_yaml["YAML execution"]
  node_cli["Test CLI<br/>[index.ts]"]
end

subgraph group_observability["Reports and observability"]
  node_report["HTML reports<br/>[App.tsx]"]
  node_report_export["Markdown export<br/>[markdown-export.ts]"]
  node_recorder["Browser recorder"]
  node_recorder_store[("Recorder storage<br/>[indexedDB.ts]")]
end

subgraph group_playground["Playground"]
  node_visualizer["Playground visualizer"]
  node_playground_controller["Playground controller"]
  node_playground_server["Playground server<br/>[server.ts]"]
end

node_user(("Test author"))

node_user -->|"describes task"| node_agent
node_agent -->|"executes tasks"| node_tasks
node_tasks -->|"uses builder"| node_task_build
node_tasks -->|"uses utilities"| node_ui_utils
node_agent -->|"requests visual reasoning"| node_model
node_agent -->|"drives interface"| node_web
node_agent -->|"drives interface"| node_android
node_agent -->|"drives interface"| node_ios
node_agent -->|"drives interface"| node_desktop
node_cli -->|"runs cases"| node_yaml
node_yaml -->|"expresses test intent"| node_agent
node_agent -->|"records execution"| node_report
node_report -->|"exports Markdown"| node_report_export
node_user -->|"tries instructions"| node_playground_controller
node_playground_controller -->|"controls session"| node_playground_server
node_playground_controller -->|"configures visualizer"| node_visualizer
node_user -->|"records interactions"| node_recorder
node_recorder -->|"persists sessions"| node_recorder_store

click node_agent "https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/agent/agent.ts"
click node_tasks "https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/agent/tasks.ts"
click node_task_build "https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/agent/task-builder.ts"
click node_ui_utils "https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/agent/ui-utils.ts"
click node_model "https://github.com/web-infra-dev/midscene/tree/main/packages/core/src/ai-model/model-adapter"
click node_web "https://github.com/web-infra-dev/midscene/tree/main/packages/web-integration/src"
click node_android "https://github.com/web-infra-dev/midscene/tree/main/packages/android/src"
click node_ios "https://github.com/web-infra-dev/midscene/blob/main/packages/ios/src/ios-webdriver-client.ts"
click node_desktop "https://github.com/web-infra-dev/midscene/tree/main/packages/computer/src"
click node_yaml "https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/yaml/execution-session.ts"
click node_cli "https://github.com/web-infra-dev/midscene/blob/main/packages/cli/src/index.ts"
click node_report "https://github.com/web-infra-dev/midscene/blob/main/apps/report/src/App.tsx"
click node_report_export "https://github.com/web-infra-dev/midscene/blob/main/apps/report/src/utils/markdown-export.ts"
click node_visualizer "https://github.com/web-infra-dev/midscene/blob/main/packages/visualizer/src/component/universal-playground/providers/storage-provider.ts"
click node_playground_controller "https://github.com/web-infra-dev/midscene/blob/main/packages/playground-app/src/controller/usePlaygroundController.ts"
click node_playground_server "https://github.com/web-infra-dev/midscene/blob/main/packages/playground/src/server.ts"
click node_recorder "https://github.com/web-infra-dev/midscene/blob/main/apps/chrome-extension/src/extension/recorder/hooks/useRecordingSession.ts"
click node_recorder_store "https://github.com/web-infra-dev/midscene/blob/main/apps/chrome-extension/src/utils/indexedDB.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_agent,node_tasks,node_task_build,node_ui_utils,node_model toneBlue
class node_web,node_android,node_ios,node_desktop toneAmber
class node_yaml,node_cli toneMint
class node_report,node_report_export,node_recorder,node_recorder_store toneRose
class node_visualizer,node_playground_controller,node_playground_server,node_user toneIndigo
```

## Components

### Agent execution

- [GUI Agent](https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/agent/agent.ts): `packages/core/src/agent/agent.ts`
- [Task execution](https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/agent/tasks.ts): `packages/core/src/agent/tasks.ts`
- [Task building](https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/agent/task-builder.ts): `packages/core/src/agent/task-builder.ts`
- [UI utilities](https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/agent/ui-utils.ts): `packages/core/src/agent/ui-utils.ts`
- [Model adapters](https://github.com/web-infra-dev/midscene/tree/main/packages/core/src/ai-model/model-adapter): `packages/core/src/ai-model/model-adapter`

### Platform integrations

- [Web integration](https://github.com/web-infra-dev/midscene/tree/main/packages/web-integration/src): `packages/web-integration/src`
- [Android integration](https://github.com/web-infra-dev/midscene/tree/main/packages/android/src): `packages/android/src`
- [iOS integration](https://github.com/web-infra-dev/midscene/blob/main/packages/ios/src/ios-webdriver-client.ts): `packages/ios/src/ios-webdriver-client.ts`
- [Desktop integration](https://github.com/web-infra-dev/midscene/tree/main/packages/computer/src): `packages/computer/src`

### Test framework

- [YAML execution](https://github.com/web-infra-dev/midscene/blob/main/packages/core/src/yaml/execution-session.ts): `packages/core/src/yaml/execution-session.ts`
- [Test CLI](https://github.com/web-infra-dev/midscene/blob/main/packages/cli/src/index.ts): `packages/cli/src/index.ts`

### Reports and observability

- [HTML reports](https://github.com/web-infra-dev/midscene/blob/main/apps/report/src/App.tsx): `apps/report/src/App.tsx`
- [Markdown export](https://github.com/web-infra-dev/midscene/blob/main/apps/report/src/utils/markdown-export.ts): `apps/report/src/utils/markdown-export.ts`
- [Browser recorder](https://github.com/web-infra-dev/midscene/blob/main/apps/chrome-extension/src/extension/recorder/hooks/useRecordingSession.ts): `apps/chrome-extension/src/extension/recorder/hooks/useRecordingSession.ts`
- [Recorder storage](https://github.com/web-infra-dev/midscene/blob/main/apps/chrome-extension/src/utils/indexedDB.ts): `apps/chrome-extension/src/utils/indexedDB.ts`

### Playground

- [Playground visualizer](https://github.com/web-infra-dev/midscene/blob/main/packages/visualizer/src/component/universal-playground/providers/storage-provider.ts): `packages/visualizer/src/component/universal-playground/providers/storage-provider.ts`
- [Playground controller](https://github.com/web-infra-dev/midscene/blob/main/packages/playground-app/src/controller/usePlaygroundController.ts): `packages/playground-app/src/controller/usePlaygroundController.ts`
- [Playground server](https://github.com/web-infra-dev/midscene/blob/main/packages/playground/src/server.ts): `packages/playground/src/server.ts`

### Other components

- **Test author**

## Connections

- Test author → GUI Agent: describes task (evidence: `README.md`)
- GUI Agent → Task execution: executes tasks (evidence: `README.md`)
- Task execution → Task building: uses builder (evidence: `packages/core/src/agent/tasks.ts`)
- Task execution → UI utilities: uses utilities (evidence: `packages/core/src/agent/tasks.ts`)
- GUI Agent → Model adapters: requests visual reasoning (evidence: `README.md`)
- GUI Agent → Web integration: drives interface (evidence: `README.md`)
- GUI Agent → Android integration: drives interface (evidence: `README.md`)
- GUI Agent → iOS integration: drives interface (evidence: `README.md`)
- GUI Agent → Desktop integration: drives interface (evidence: `README.md`)
- Test CLI → YAML execution: runs cases (evidence: `README.md`)
- YAML execution → GUI Agent: expresses test intent (evidence: `README.md`)
- GUI Agent → HTML reports: records execution (evidence: `README.md`)
- HTML reports → Markdown export: exports Markdown (evidence: `apps/report/src/App.tsx`)
- Test author → Playground controller: tries instructions (evidence: `README.md`)
- Playground controller → Playground server: controls session (evidence: `packages/playground-app/src/controller/usePlaygroundController.ts`)
- Playground controller → Playground visualizer: configures visualizer (evidence: `packages/playground-app/src/controller/usePlaygroundController.ts`)
- Test author → Browser recorder: records interactions (evidence: `README.md`)
- Browser recorder → Recorder storage: persists sessions (evidence: `apps/chrome-extension/src/extension/recorder/hooks/useRecordingSession.ts`)
