How web-infra-dev/midscene works
Midscene is a vision-driven GUI automation agent and E2E testing toolkit for web, mobile, and desktop. The main workflow is a user’s natural-language task entering an Agent, which plans and executes visual actions through a platform integration, checks the visible result, and produces observable execution data for reports. Distinct supporting branches cover declarative YAML tests and custom Nodes, interactive reports, and the Playground. Several links use README-level documentation because the sampled excerpts do not expose their runtime wiring.
Main entry points: agent.ts GUI Agent · tasks.ts Task execution · usePlaygroundController.ts Playground controller · execution-session.ts YAML execution
Components 19
Agent execution
- GUI Agent
packages/core/src/agent/agent.ts - Task execution
packages/core/src/agent/tasks.ts - Task building
packages/core/src/agent/task-builder.ts - UI utilities
packages/core/src/agent/ui-utils.ts - Model adapters
packages/core/src/ai-model/model-adapter
Platform integrations
- Web integration
packages/web-integration/src - Android integration
packages/android/src - iOS integration
packages/ios/src/ios-webdriver-client.ts - Desktop integration
packages/computer/src
Test framework
- YAML execution
packages/core/src/yaml/execution-session.ts - Test CLI
packages/cli/src/index.ts
Reports and observability
- HTML reports
apps/report/src/App.tsx - Markdown export
apps/report/src/utils/markdown-export.ts - Browser recorder
apps/chrome-extension/src/extension/recorder/hooks/useRecordingSession.ts - Recorder storage
apps/chrome-extension/src/utils/indexedDB.ts
Playground
- Playground visualizer
packages/visualizer/src/component/universal-playground/providers/storage-provider.ts - Playground controller
packages/playground-app/src/controller/usePlaygroundController.ts - Playground server
packages/playground/src/server.ts
Other components
- Test author
Connections 18
- Test author → GUI Agent: describes task
README.md - GUI Agent → Task execution: executes tasks
README.md - Task execution → Task building: uses builder
packages/core/src/agent/tasks.ts - Task execution → UI utilities: uses utilities
packages/core/src/agent/tasks.ts - GUI Agent → Model adapters: requests visual reasoning
README.md - GUI Agent → Web integration: drives interface
README.md - GUI Agent → Android integration: drives interface
README.md - GUI Agent → iOS integration: drives interface
README.md - GUI Agent → Desktop integration: drives interface
README.md - Test CLI → YAML execution: runs cases
README.md - YAML execution → GUI Agent: expresses test intent
README.md - GUI Agent → HTML reports: records execution
README.md - HTML reports → Markdown export: exports Markdown
apps/report/src/App.tsx - Test author → Playground controller: tries instructions
README.md - Playground controller → Playground server: controls session
packages/playground-app/src/controller/usePlaygroundController.ts - Playground controller → Playground visualizer: configures visualizer
packages/playground-app/src/controller/usePlaygroundController.ts - Test author → Browser recorder: records interactions
README.md - Browser recorder → Recorder storage: persists sessions
apps/chrome-extension/src/extension/recorder/hooks/useRecordingSession.ts