How dexter-666/jarvis-ia works
1. Overall shape
- This is a Windows desktop assistant built around a central Python entrypoint (main.py) that orchestrates a local UI, voice input, task execution, and a set of capability modules under actions/.
- The repo is not a generic library; it is a tightly integrated desktop agent with OS automation, LLM-backed reasoning, and persistent local state.
2. Runtime and platform boundary
- The architecture is strongly Windows-oriented: the README and filenames point to hotkeys, Win32 calls, pycaw, pygetwindow, psutil, WMI, and winreg.
- Startup is handled through desktop launchers (Iniciar JARVIS Beta.vbs, launchers/start_jarvis_beta.bat) and installation scripts (Instalar_JARVIS.bat, install.py), which indicates a local, user-machine deployment model rather than a server app.
3. Main orchestration layer
- main.py is the probable top-level coordinator for bootstrapping the assistant.
- agent/task_queue.py suggests asynchronous or serialized task handling, likely used to decouple user input from action execution and prevent concurrent conflicts across automation modules.
4. Interaction channels
- The assistant appears to accept at least three primary input paths:
- Voice via Vosk model assets in config/vosk_model/
- Global hotkey activation (Insert) for immediate microphone capture
- UI interaction via PyQt6, including a holographic/Glassmorphism interface
- actions/native_ui.py and actions/visual_click.py imply that UI interaction is both visual and action-driven, not just text-based.
5. Speech and local recognition
- The bundled Vosk model under config/vosk_model/ implies offline speech recognition is a core subsystem, not an optional add-on.
- This matters architecturally because speech input can be processed locally before any LLM or cloud service is involved, reducing latency and dependency on internet connectivity.
6. LLM and reasoning boundary
- actions/openrouter_agent.py is the clearest indicator of a cloud LLM integration path, likely used for general reasoning, planning, or natural-language interpretation.
- core/prompt.txt and beta_config.py likely define the assistant’s persona, behavior rules, and model/runtime settings.
- The design suggests a split between local orchestration and remote model inference.
7. Capability modules as plugins
- The actions/ directory is the main functional surface area and reads like a plugin layer.
- Key modules indicate distinct domains:
- browser_control.py, web_navigation.py for browser and web automation
- terminal_agent.py for shell/command execution
- document_creator.py, document_manager.py, smart_file_organizer.py for file/document workflows
- whatsapp.py, unified_communications.py for outbound messaging
- contextual_control.py, proactive_automation.py for environment-aware automation
- screen_vision.py, visual_click.py for vision-based interaction
- self_edit.py, tool_creator.py, auto_programmer.py for code generation and self-extension
8. Self-modifying / code-generation path
- The README explicitly describes autonomous programming in a sandbox, and the file names confirm that through auto_programmer.py, self_edit.py, and tool_creator.py.
- Architecturally, this is a high-risk boundary: generated code appears to be created, tested, and then conditionally integrated into the assistant’s capability set.
- For an architecture diagram, this should be shown as a separate “dynamic skill generation” subsystem connected to the action registry, not part of the normal request path.
9. Environment awareness and proactive behavior
- contextual_control.py and proactive_automation.py imply the assistant can inspect current system state and take actions without an explicit command.
- The README mentions volume, brightness, energy, and Focus Assist control, so the assistant likely observes foreground window/context and then decides whether to adjust system behavior.
10. File and document workflow
- document_manager.py and document_creator.py point to content handling for creating, opening, viewing, and editing documents.
- smart_file_organizer.py adds a classification/deduplication layer, with the README mentioning exact duplicate removal via MD5.
- This subsystem likely sits between the assistant and the local filesystem, with a focus on user data management.
11. Memory and persistent state
- There are multiple local persistence locations:
- memory/long_term.json for longer-lived memory or preferences
- config/user_profile.json for user-specific configuration
- config/morning_brief_state.json and config/vision_guardian_state.json for feature state
- This indicates the assistant is stateful across sessions and likely uses local JSON as a lightweight persistence layer rather than a database.
12. Configuration and secrets
- config/api_keys.example.json shows external service credentials are expected but kept out of source control.
- config/accessibility_config.json suggests accessibility and UI-behavior tuning are separated from code.
- sitecustomize.py may be used to inject environment/runtime tweaks early in Python startup.
13. Communications integration
- actions/unified_communications.py is a central boundary for outward messaging, with README support for email, Telegram, Discord, and WhatsApp.
- This is a distinct integration layer from core reasoning: the assistant decides what to send, and this module handles channel-specific delivery.
14. What the diagram should emphasize
- Show a central assistant core receiving input from voice, hotkey, and UI.
- Route requests through a task queue / orchestrator.
- Branch from there into capability modules in actions/.
- Separate local state/config/memory from cloud LLM calls and from OS-level automation.
- Highlight the self-programming path as its own special subsystem because it can create or modify capabilities at runtime.
Main entry points: main.py Main · task_queue.py Task Queue · auto_programmer.py Auto Programmer · native_ui.py Native UI
Components 34
Runtime
Desktop assistant entrypoints and startup scripts
- Site Custom · Early Python startup customization hook
sitecustomize.py
Control Flow
Core orchestration and input handling
- Main · Primary Python bootstrap and orchestration entrypoint
main.py - Task Queue · Serialized or asynchronous task dispatch for assistant actions
agent/task_queue.py - Native UI · Desktop interface and user interaction layer
actions/native_ui.py - LLM Agent · Remote reasoning and natural-language planning via OpenRouter
actions/openrouter_agent.py - Vosk Model · Offline speech recognition assets for local voice input
config/vosk_model
Capabilities
Action modules that perform work
- Context Control · Inspects foreground state and environment before acting
actions/contextual_control.py - Proactive Auto · Event- or state-driven automation without explicit prompts
actions/proactive_automation.py - Browser Control · Controls browser sessions and page interactions
actions/browser_control.py - Web Navigation · Browser/page traversal and web workflows
actions/web_navigation.py - Terminal Agent · Runs shell commands and terminal workflows
actions/terminal_agent.py - Doc Manager · Open, manage, and manipulate local documents
actions/document_manager.py - Doc Creator · Creates documents and formatted output files
actions/document_creator.py - File Organizer · Classifies, deduplicates, and organizes local files
actions/smart_file_organizer.py - Screen Vision · Visual perception of the desktop for interaction support
actions/screen_vision.py - Visual Click · Coordinates click-based interaction using visual targets
actions/visual_click.py - Comms Hub · Unified outbound messaging across email and chat channels
actions/unified_communications.py - WhatsApp · WhatsApp-specific messaging integration
actions/whatsapp.py - Auto Programmer · Generates new code or tools in a sandboxed flow
actions/auto_programmer.py - Self Edit · Applies or prepares assistant code changes
actions/self_edit.py - Tool Creator · Creates new capability modules or tools
actions/tool_creator.py
State & Config
Persistent local data and runtime settings
- Long Memory · Long-lived preferences and remembered context
memory/long_term.json - User Profile · User-specific settings and profile data
config/user_profile.json - Feature State · Persistent feature flags and runtime state snapshots
config/morning_brief_state.json - Vision State · Persistent state for vision guardian behavior
config/vision_guardian_state.json - Accessibility · Accessibility and UI behavior configuration
config/accessibility_config.json - API Keys · Template for external service credentials
config/api_keys.example.json - Prompt · Assistant behavior and instruction prompt
core/prompt.txt - Bootstrap · Assistant runtime settings and startup configuration
beta_config.py
Packaging
Install and launch helpers
- VBS Launcher · Windows desktop launcher for assistant startup
Iniciar JARVIS Beta.vbs - Batch Launcher · Batch launcher for local startup
launchers/start_jarvis_beta.bat - Install Script · Windows installation helper
Instalar_JARVIS.bat - Installer · Python installation and setup helper
install.py - Vosk Downloader · Downloads or prepares the local speech model
download_vosk.py
Connections 34
- VBS Launcher → Main: start · Windows launcher starts the assistant
- Batch Launcher → Main: start · Batch launcher starts the assistant
- Install Script → Installer: delegates · Windows installer script invokes Python setup
- Installer → Vosk Downloader: prepares · Setup can fetch or prepare the speech model
- Main → Task Queue: dispatch · Main runtime hands work to the task queue
- Main → Native UI: renders · Bootstraps the desktop UI
- Main → Vosk Model: loads · Initializes offline speech recognition assets
- Main → LLM Agent: calls · Routes reasoning or planning to the cloud LLM agent
- Main → Context Control: inspects · Reads system context before executing actions
- Main → Proactive Auto: triggers · Schedules or launches proactive behaviors
- Main → Long Memory: reads/writes · Uses long-term memory across sessions
- Main → User Profile: reads/writes · Loads and persists user preferences
- Main → Feature State: reads/writes · Persists assistant feature state
- Main → Vision State: reads/writes · Persists vision-related state
- Main → Prompt: configures · Uses the prompt as behavior guidance
- Main → Bootstrap: configures · Reads runtime settings and model parameters
- Main → Site Custom: hooks · Applies early Python runtime customization
- Task Queue → Browser Control: executes · Serializes browser automation tasks
- Task Queue → Web Navigation: executes · Dispatches navigation work
- Task Queue → Terminal Agent: executes · Dispatches shell and command tasks
- Task Queue → Doc Manager: executes · Dispatches document management work
- Task Queue → Doc Creator: executes · Dispatches document creation work
- Task Queue → File Organizer: executes · Dispatches file organization work
- Task Queue → Screen Vision: executes · Dispatches visual analysis tasks
- Task Queue → Visual Click: executes · Dispatches vision-guided UI actions
- Task Queue → Comms Hub: executes · Dispatches outbound communications
- Comms Hub → WhatsApp: adapts · Channel-specific WhatsApp delivery
- Task Queue → Auto Programmer: executes · Dispatches dynamic code-generation tasks
- Auto Programmer → Tool Creator: creates · Builds new tools or capability modules
- Auto Programmer → Self Edit: patches · Produces or applies assistant code changes
- Screen Vision → Visual Click: feeds · Vision output informs click targeting
- Context Control → Proactive Auto: drives · Environment awareness feeds proactive actions
- Native UI → Task Queue: submits · UI-originated commands enter the task queue
- Vosk Model → Task Queue: submits · Transcribed voice commands enter the task queue