How basitaliyousafzai/omniagent works
OmniAgent OS combines a multimodal desktop-control agent with a YouTube video production pipeline. A user interacts through the desktop interface; the agent reasons over goals and observations, dispatches tools, and gates risky actions for approval. The separate video factory coordinates script generation, narration, media retrieval, audio mastering, composition, and upload, with a database supporting idempotency and telemetry. Edges reflect excerpted calls, indexed references, or README statements; unsampled implementation details and unshown wiring are omitted.
Main entry points: agent/orchestrator.py ReAct Agent · app.py API and Event Bus · App.tsx React Workspace · youtube_automator/orchestrator.py Pipeline Orchestrator
Components 34
Desktop Interface
- Desktop Shell
desktop_app.py - React Workspace
frontend/src/App.tsx - Agent Chat
frontend/src/components/AgentChat.tsx - Agent Socket
frontend/src/hooks/useAgentSocket.ts - YouTube Dashboard
frontend/src/components/YouTubeDashboard.tsx
Agent Runtime
- API and Event Bus
backend/app.py - ReAct Agent
backend/agent/orchestrator.py - Tool Registry
backend/agent/tool_registry.py - Provider Gateway
backend/gateway/provider_gateway.py - Failover Router
backend/gateway/fallback_router.py - Context Trimmer
backend/gateway/context_trimmer.py - Settings and Secrets
backend/gateway/vault.py
Control and Safety
- Approval Bridge
backend/safety/approval_bridge.py - Safety Guardrails
backend/safety/guardrails.py - Audit Journal
backend/safety/audit_logger.py - Emergency Stop
backend/safety/kill_switch.py - GUI Controller
backend/os_control/gui_controller.py - Window Manager
backend/os_control/window_manager.py - UI Automation
backend/os_control/ui_automation.py - Workspace Files
backend/os_control/file_tools.py - Shell Runner
backend/os_control/shell_runner.py - App Recipes
backend/os_control/app_orchestrator.py - Screen Capture
backend/multimodal/screen_capture.py - Coordinate Mapper
backend/multimodal/coordinate_mapper.py - Speech I/O
backend/multimodal/stt.py - Voice Activity
backend/multimodal/vad_listener.py - Text to Speech
backend/multimodal/tts.py
Video Factory
- YouTube API Router
backend/routers/youtube_router.py - Pipeline Orchestrator
modules/youtube_automator/orchestrator.py - Script Engine
modules/youtube_automator/core/script_engine.py - Voice Synthesizer
modules/youtube_automator/core/voice_synthesizer.py - Media Retriever
modules/youtube_automator/core/media_retriever.py - Pipeline Database · Supports pipeline idempotency and telemetry.
modules/youtube_automator/core/db_manager.py
Other components
- Desktop User
Connections 35
- Desktop User → React Workspace: uses
README.md - Desktop Shell → API and Event Bus: hosts
- React Workspace → Agent Chat: renders
frontend/src/App.tsx - React Workspace → Agent Socket: uses
frontend/src/App.tsx - React Workspace → YouTube Dashboard: renders
frontend/src/App.tsx - React Workspace → Desktop User: presents
README.md - API and Event Bus → ReAct Agent: dispatches
backend/app.py - API and Event Bus → Settings and Secrets: reads settings
backend/app.py - API and Event Bus → Screen Capture: captures
backend/app.py - API and Event Bus → Speech I/O: uses
backend/app.py - API and Event Bus → Text to Speech: uses
backend/app.py - API and Event Bus → YouTube API Router: routes
backend/app.py - ReAct Agent → Failover Router: requests completion
backend/agent/orchestrator.py - ReAct Agent → Context Trimmer: trims context
backend/agent/orchestrator.py - ReAct Agent → Tool Registry: dispatches tools
backend/agent/orchestrator.py - ReAct Agent → Approval Bridge: requests approval
backend/agent/orchestrator.py - ReAct Agent → Safety Guardrails: checks risk
backend/agent/orchestrator.py - ReAct Agent → GUI Controller: executes actions
backend/agent/orchestrator.py - ReAct Agent → Window Manager: finds windows
backend/agent/orchestrator.py - ReAct Agent → Screen Capture: observes screen
backend/agent/orchestrator.py - Failover Router → Provider Gateway: routes requests
backend/gateway/fallback_router.py - Failover Router → Settings and Secrets: reads keys
backend/gateway/fallback_router.py - Failover Router → Audit Journal: logs decisions
backend/gateway/fallback_router.py - Provider Gateway → Settings and Secrets: reads credentials
backend/gateway/provider_gateway.py - GUI Controller → Coordinate Mapper: maps coordinates
backend/os_control/gui_controller.py - GUI Controller → Emergency Stop: checks abort
backend/os_control/gui_controller.py - GUI Controller → Window Manager: ensures desktop
backend/os_control/gui_controller.py - Workspace Files → Safety Guardrails: checks writes
backend/os_control/file_tools.py - Emergency Stop → Audit Journal: records trigger
backend/safety/kill_switch.py - Approval Bridge → Audit Journal: records decisions
backend/safety/approval_bridge.py - YouTube API Router → Pipeline Orchestrator: invokes pipeline
backend/routers/youtube_router.py - Pipeline Orchestrator → Script Engine: generates script
modules/youtube_automator/orchestrator.py - Pipeline Orchestrator → Voice Synthesizer: synthesizes voice
modules/youtube_automator/orchestrator.py - Pipeline Orchestrator → Media Retriever: retrieves media
modules/youtube_automator/orchestrator.py - Pipeline Orchestrator → Pipeline Database: tracks runs
modules/youtube_automator/orchestrator.py