# basitaliyousafzai/omniagent architecture

> Architecture diagram of the GitHub repository basitaliyousafzai/omniagent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/basitaliyousafzai/omniagent
- Repository: https://github.com/basitaliyousafzai/omniagent

## Overview

OmniAgent OS combines a multimodal desktop-control agent with a YouTube video production pipeline. A user interacts through the desktop interface; the agent reasons over goals and observations, dispatches tools, and gates risky actions for approval. The separate video factory coordinates script generation, narration, media retrieval, audio mastering, composition, and upload, with a database supporting idempotency and telemetry. Edges reflect excerpted calls, indexed references, or README statements; unsampled implementation details and unshown wiring are omitted.

Main entry points: [`backend/agent/orchestrator.py`](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/agent/orchestrator.py) (ReAct Agent), [`backend/app.py`](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/app.py) (API and Event Bus), [`frontend/src/App.tsx`](https://github.com/basitaliyousafzai/omniagent/blob/main/frontend/src/App.tsx) (React Workspace), [`modules/youtube_automator/orchestrator.py`](https://github.com/basitaliyousafzai/omniagent/blob/main/modules/youtube_automator/orchestrator.py) (Pipeline Orchestrator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["Desktop Interface"]
  node_desktop["Desktop Shell<br/>[desktop_app.py]"]
  node_frontend["React Workspace<br/>[App.tsx]"]
  node_chat["Agent Chat<br/>[AgentChat.tsx]"]
  node_socket["Agent Socket<br/>[useAgentSocket.ts]"]
  node_youtube_ui["YouTube Dashboard"]
end

subgraph group_agent["Agent Runtime"]
  node_api["API and Event Bus<br/>[app.py]"]
  node_orchestrator["ReAct Agent<br/>[orchestrator.py]"]
  node_tools["Tool Registry<br/>[tool_registry.py]"]
  node_gateway["Provider Gateway"]
  node_fallback["Failover Router<br/>[fallback_router.py]"]
  node_context["Context Trimmer<br/>[context_trimmer.py]"]
  node_vault[("Settings and Secrets<br/>[vault.py]")]
end

subgraph group_control["Control and Safety"]
  node_approval["Approval Bridge<br/>[approval_bridge.py]"]
  node_guardrails["Safety Guardrails<br/>[guardrails.py]"]
  node_audit[("Audit Journal<br/>[audit_logger.py]")]
  node_kill["Emergency Stop<br/>[kill_switch.py]"]
  node_gui["GUI Controller<br/>[gui_controller.py]"]
  node_windows["Window Manager<br/>[window_manager.py]"]
  node_automation["UI Automation<br/>[ui_automation.py]"]
  node_files["Workspace Files<br/>[file_tools.py]"]
  node_shell["Shell Runner<br/>[shell_runner.py]"]
  node_apps["App Recipes"]
  node_screen["Screen Capture<br/>[screen_capture.py]"]
  node_coordinates["Coordinate Mapper"]
  node_speech["Speech I/O<br/>[stt.py]"]
  node_voice["Voice Activity<br/>[vad_listener.py]"]
  node_tts["Text to Speech<br/>[tts.py]"]
end

subgraph group_video["Video Factory"]
  node_yt_router["YouTube API Router<br/>[youtube_router.py]"]
  node_pipeline["Pipeline Orchestrator<br/>[orchestrator.py]"]
  node_scripts["Script Engine<br/>[script_engine.py]"]
  node_voicegen["Voice Synthesizer"]
  node_media["Media Retriever<br/>[media_retriever.py]"]
  node_video_db[("Pipeline Database<br/>[db_manager.py]")]
end

node_user(("Desktop User"))

node_user -->|"uses"| node_frontend
node_desktop -.->|"hosts"| node_api
node_frontend -->|"renders"| node_chat
node_frontend -->|"uses"| node_socket
node_frontend -->|"renders"| node_youtube_ui
node_frontend -->|"presents"| node_user
node_api -->|"dispatches"| node_orchestrator
node_api -->|"reads settings"| node_vault
node_api -->|"captures"| node_screen
node_api -->|"uses"| node_speech
node_api -->|"uses"| node_tts
node_api -->|"routes"| node_yt_router
node_orchestrator -->|"requests completion"| node_fallback
node_orchestrator -->|"trims context"| node_context
node_orchestrator -->|"dispatches tools"| node_tools
node_orchestrator -->|"requests approval"| node_approval
node_orchestrator -->|"checks risk"| node_guardrails
node_orchestrator -->|"executes actions"| node_gui
node_orchestrator -->|"finds windows"| node_windows
node_orchestrator -->|"observes screen"| node_screen
node_fallback -->|"routes requests"| node_gateway
node_fallback -->|"reads keys"| node_vault
node_fallback -->|"logs decisions"| node_audit
node_gateway -->|"reads credentials"| node_vault
node_gui -->|"maps coordinates"| node_coordinates
node_gui -->|"checks abort"| node_kill
node_gui -->|"ensures desktop"| node_windows
node_files -->|"checks writes"| node_guardrails
node_kill -->|"records trigger"| node_audit
node_approval -->|"records decisions"| node_audit
node_yt_router -->|"invokes pipeline"| node_pipeline
node_pipeline -->|"generates script"| node_scripts
node_pipeline -->|"synthesizes voice"| node_voicegen
node_pipeline -->|"retrieves media"| node_media
node_pipeline -->|"tracks runs"| node_video_db

click node_desktop "https://github.com/basitaliyousafzai/omniagent/blob/main/desktop_app.py"
click node_frontend "https://github.com/basitaliyousafzai/omniagent/blob/main/frontend/src/App.tsx"
click node_chat "https://github.com/basitaliyousafzai/omniagent/blob/main/frontend/src/components/AgentChat.tsx"
click node_socket "https://github.com/basitaliyousafzai/omniagent/blob/main/frontend/src/hooks/useAgentSocket.ts"
click node_youtube_ui "https://github.com/basitaliyousafzai/omniagent/blob/main/frontend/src/components/YouTubeDashboard.tsx"
click node_api "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/app.py"
click node_orchestrator "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/agent/orchestrator.py"
click node_tools "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/agent/tool_registry.py"
click node_gateway "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/gateway/provider_gateway.py"
click node_fallback "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/gateway/fallback_router.py"
click node_context "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/gateway/context_trimmer.py"
click node_vault "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/gateway/vault.py"
click node_approval "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/safety/approval_bridge.py"
click node_guardrails "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/safety/guardrails.py"
click node_audit "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/safety/audit_logger.py"
click node_kill "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/safety/kill_switch.py"
click node_gui "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/os_control/gui_controller.py"
click node_windows "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/os_control/window_manager.py"
click node_automation "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/os_control/ui_automation.py"
click node_files "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/os_control/file_tools.py"
click node_shell "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/os_control/shell_runner.py"
click node_apps "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/os_control/app_orchestrator.py"
click node_screen "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/multimodal/screen_capture.py"
click node_coordinates "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/multimodal/coordinate_mapper.py"
click node_speech "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/multimodal/stt.py"
click node_voice "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/multimodal/vad_listener.py"
click node_tts "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/multimodal/tts.py"
click node_yt_router "https://github.com/basitaliyousafzai/omniagent/blob/main/backend/routers/youtube_router.py"
click node_pipeline "https://github.com/basitaliyousafzai/omniagent/blob/main/modules/youtube_automator/orchestrator.py"
click node_scripts "https://github.com/basitaliyousafzai/omniagent/blob/main/modules/youtube_automator/core/script_engine.py"
click node_voicegen "https://github.com/basitaliyousafzai/omniagent/blob/main/modules/youtube_automator/core/voice_synthesizer.py"
click node_media "https://github.com/basitaliyousafzai/omniagent/blob/main/modules/youtube_automator/core/media_retriever.py"
click node_video_db "https://github.com/basitaliyousafzai/omniagent/blob/main/modules/youtube_automator/core/db_manager.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_desktop,node_frontend,node_chat,node_socket,node_youtube_ui,node_user toneBlue
class node_api,node_orchestrator,node_tools,node_gateway,node_fallback,node_context,node_vault toneAmber
class node_approval,node_guardrails,node_audit,node_kill,node_gui,node_windows,node_automation,node_files,node_shell,node_apps,node_screen,node_coordinates,node_speech,node_voice,node_tts toneMint
class node_yt_router,node_pipeline,node_scripts,node_voicegen,node_media,node_video_db toneRose
```

## Components

### Desktop Interface

- [Desktop Shell](https://github.com/basitaliyousafzai/omniagent/blob/main/desktop_app.py): `desktop_app.py`
- [React Workspace](https://github.com/basitaliyousafzai/omniagent/blob/main/frontend/src/App.tsx): `frontend/src/App.tsx`
- [Agent Chat](https://github.com/basitaliyousafzai/omniagent/blob/main/frontend/src/components/AgentChat.tsx): `frontend/src/components/AgentChat.tsx`
- [Agent Socket](https://github.com/basitaliyousafzai/omniagent/blob/main/frontend/src/hooks/useAgentSocket.ts): `frontend/src/hooks/useAgentSocket.ts`
- [YouTube Dashboard](https://github.com/basitaliyousafzai/omniagent/blob/main/frontend/src/components/YouTubeDashboard.tsx): `frontend/src/components/YouTubeDashboard.tsx`

### Agent Runtime

- [API and Event Bus](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/app.py): `backend/app.py`
- [ReAct Agent](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/agent/orchestrator.py): `backend/agent/orchestrator.py`
- [Tool Registry](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/agent/tool_registry.py): `backend/agent/tool_registry.py`
- [Provider Gateway](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/gateway/provider_gateway.py): `backend/gateway/provider_gateway.py`
- [Failover Router](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/gateway/fallback_router.py): `backend/gateway/fallback_router.py`
- [Context Trimmer](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/gateway/context_trimmer.py): `backend/gateway/context_trimmer.py`
- [Settings and Secrets](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/gateway/vault.py): `backend/gateway/vault.py`

### Control and Safety

- [Approval Bridge](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/safety/approval_bridge.py): `backend/safety/approval_bridge.py`
- [Safety Guardrails](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/safety/guardrails.py): `backend/safety/guardrails.py`
- [Audit Journal](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/safety/audit_logger.py): `backend/safety/audit_logger.py`
- [Emergency Stop](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/safety/kill_switch.py): `backend/safety/kill_switch.py`
- [GUI Controller](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/os_control/gui_controller.py): `backend/os_control/gui_controller.py`
- [Window Manager](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/os_control/window_manager.py): `backend/os_control/window_manager.py`
- [UI Automation](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/os_control/ui_automation.py): `backend/os_control/ui_automation.py`
- [Workspace Files](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/os_control/file_tools.py): `backend/os_control/file_tools.py`
- [Shell Runner](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/os_control/shell_runner.py): `backend/os_control/shell_runner.py`
- [App Recipes](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/os_control/app_orchestrator.py): `backend/os_control/app_orchestrator.py`
- [Screen Capture](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/multimodal/screen_capture.py): `backend/multimodal/screen_capture.py`
- [Coordinate Mapper](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/multimodal/coordinate_mapper.py): `backend/multimodal/coordinate_mapper.py`
- [Speech I/O](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/multimodal/stt.py): `backend/multimodal/stt.py`
- [Voice Activity](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/multimodal/vad_listener.py): `backend/multimodal/vad_listener.py`
- [Text to Speech](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/multimodal/tts.py): `backend/multimodal/tts.py`

### Video Factory

- [YouTube API Router](https://github.com/basitaliyousafzai/omniagent/blob/main/backend/routers/youtube_router.py): `backend/routers/youtube_router.py`
- [Pipeline Orchestrator](https://github.com/basitaliyousafzai/omniagent/blob/main/modules/youtube_automator/orchestrator.py): `modules/youtube_automator/orchestrator.py`
- [Script Engine](https://github.com/basitaliyousafzai/omniagent/blob/main/modules/youtube_automator/core/script_engine.py): `modules/youtube_automator/core/script_engine.py`
- [Voice Synthesizer](https://github.com/basitaliyousafzai/omniagent/blob/main/modules/youtube_automator/core/voice_synthesizer.py): `modules/youtube_automator/core/voice_synthesizer.py`
- [Media Retriever](https://github.com/basitaliyousafzai/omniagent/blob/main/modules/youtube_automator/core/media_retriever.py): `modules/youtube_automator/core/media_retriever.py`
- [Pipeline Database](https://github.com/basitaliyousafzai/omniagent/blob/main/modules/youtube_automator/core/db_manager.py): Supports pipeline idempotency and telemetry. · `modules/youtube_automator/core/db_manager.py`

### Other components

- **Desktop User**

## Connections

- Desktop User → React Workspace: uses (evidence: `README.md`)
- Desktop Shell → API and Event Bus: hosts
- React Workspace → Agent Chat: renders (evidence: `frontend/src/App.tsx`)
- React Workspace → Agent Socket: uses (evidence: `frontend/src/App.tsx`)
- React Workspace → YouTube Dashboard: renders (evidence: `frontend/src/App.tsx`)
- React Workspace → Desktop User: presents (evidence: `README.md`)
- API and Event Bus → ReAct Agent: dispatches (evidence: `backend/app.py`)
- API and Event Bus → Settings and Secrets: reads settings (evidence: `backend/app.py`)
- API and Event Bus → Screen Capture: captures (evidence: `backend/app.py`)
- API and Event Bus → Speech I/O: uses (evidence: `backend/app.py`)
- API and Event Bus → Text to Speech: uses (evidence: `backend/app.py`)
- API and Event Bus → YouTube API Router: routes (evidence: `backend/app.py`)
- ReAct Agent → Failover Router: requests completion (evidence: `backend/agent/orchestrator.py`)
- ReAct Agent → Context Trimmer: trims context (evidence: `backend/agent/orchestrator.py`)
- ReAct Agent → Tool Registry: dispatches tools (evidence: `backend/agent/orchestrator.py`)
- ReAct Agent → Approval Bridge: requests approval (evidence: `backend/agent/orchestrator.py`)
- ReAct Agent → Safety Guardrails: checks risk (evidence: `backend/agent/orchestrator.py`)
- ReAct Agent → GUI Controller: executes actions (evidence: `backend/agent/orchestrator.py`)
- ReAct Agent → Window Manager: finds windows (evidence: `backend/agent/orchestrator.py`)
- ReAct Agent → Screen Capture: observes screen (evidence: `backend/agent/orchestrator.py`)
- Failover Router → Provider Gateway: routes requests (evidence: `backend/gateway/fallback_router.py`)
- Failover Router → Settings and Secrets: reads keys (evidence: `backend/gateway/fallback_router.py`)
- Failover Router → Audit Journal: logs decisions (evidence: `backend/gateway/fallback_router.py`)
- Provider Gateway → Settings and Secrets: reads credentials (evidence: `backend/gateway/provider_gateway.py`)
- GUI Controller → Coordinate Mapper: maps coordinates (evidence: `backend/os_control/gui_controller.py`)
- GUI Controller → Emergency Stop: checks abort (evidence: `backend/os_control/gui_controller.py`)
- GUI Controller → Window Manager: ensures desktop (evidence: `backend/os_control/gui_controller.py`)
- Workspace Files → Safety Guardrails: checks writes (evidence: `backend/os_control/file_tools.py`)
- Emergency Stop → Audit Journal: records trigger (evidence: `backend/safety/kill_switch.py`)
- Approval Bridge → Audit Journal: records decisions (evidence: `backend/safety/approval_bridge.py`)
- YouTube API Router → Pipeline Orchestrator: invokes pipeline (evidence: `backend/routers/youtube_router.py`)
- Pipeline Orchestrator → Script Engine: generates script (evidence: `modules/youtube_automator/orchestrator.py`)
- Pipeline Orchestrator → Voice Synthesizer: synthesizes voice (evidence: `modules/youtube_automator/orchestrator.py`)
- Pipeline Orchestrator → Media Retriever: retrieves media (evidence: `modules/youtube_automator/orchestrator.py`)
- Pipeline Orchestrator → Pipeline Database: tracks runs (evidence: `modules/youtube_automator/orchestrator.py`)
