How ateeb09/nova works
1. What this repo is
- This is a local, offline voice-control agent for Windows-style desktop automation.
- The architecture is centered on a single command pipeline: wake word → record audio → speech-to-text → intent resolution → desktop action → spoken feedback.
2. Two entry points, one shared automation core
- voice_agent.py is the terminal-first main loop described in the README.
- web_app.py is the web dashboard entry point, with UI assets in templates/ and static/.
- Both entry points appear to sit on top of the same underlying voice/automation modules rather than duplicating the action logic.
3. Voice processing subsystem
- record.py likely handles microphone capture and audio buffering.
- voice_loop.py appears to encapsulate the main listen/process cycle.
- mic_test.py, list_mics.py, speak_test.py, and voice_test.py are diagnostic/support scripts for audio input/output verification.
- The README indicates speech recognition uses Whisper, and text-to-speech uses pyttsx3 locally.
4. Intent understanding and command routing
- voice_action.py is the main command dispatcher: it likely maps parsed intent to concrete actions.
- The README’s “fuzzy intent matching” and “keyword catch” behavior suggests the dispatcher has a rule-based fallback path when LLM parsing is unavailable.
- voice_agent.py likely orchestrates the decision order: Gemini/Ollama LLM first when enabled, then rules/keywords, then execution.
5. Desktop automation layer
- voice_mouse.py is a dedicated module for mouse operations such as move, click, scroll, double-click, and right-click.
- Keyboard, window, system, and application actions are likely grouped in voice_action.py and/or helper routines it calls.
- The repo is built around local system control via libraries such as pyautogui and native OS subprocess calls.
6. Screen understanding / OCR path
- screen_read.py is the screen-reading/OCR module.
- screen_test.py and vision_test.py suggest a separate validation path for screen capture and OCR behavior.
- The README identifies Tesseract OCR as the offline OCR dependency, so this subsystem is distinct from the speech pipeline and used only for screen interpretation tasks.
7. Web dashboard subsystem
- web_app.py likely serves the browser UI and bridges the dashboard to the agent runtime.
- templates/index.html is the main HTML template, while static/css/style.css and static/js/app.js provide the front-end behavior.
- index2.html appears to be an alternate or experimental UI page, likely related to the dashboard/widget mode described in the README.
8. Runtime boundaries
- The repo has a clear split between:
- Audio ingestion and STT
- Intent resolution
- OS-level action execution
- UI presentation
- That separation is important for a diagram: the web UI is not the automation engine; it is a front-end over the same agent capabilities.
9. External/local services and optional dependencies
- The system is designed to run fully offline by default.
- Optional integrations affect architecture only when enabled:
- Ollama for local LLM intent interpretation
- Gemini API for cloud-backed intent understanding
- Tesseract for OCR
- The dependency list in requirements.txt likely pulls in the libraries for audio capture, STT, GUI automation, OCR, and web serving.
10. Testing and validation structure
- The *_test.py scripts are not unit tests in the strict framework sense; they look like manual or smoke-test utilities for individual subsystems.
- automation_test.py likely exercises the action layer end-to-end, while the other test files validate mic, speech, screen, and vision behavior independently.
11. Likely data flow diagram to draw
- A useful architecture diagram would show:
- Microphone input feeding the recording/preprocessing layer
- Speech recognition converting audio to text
- Intent engine choosing between LLM, rules, and keyword catch
- Action executor dispatching mouse/keyboard/window/system/screen actions
- Speaker/TTS returning status to the user
- Web dashboard as an alternate control surface connected to the same backend capabilities
12. Repository shape
- This is primarily a single-machine automation application, not a distributed system.
- The major architectural boundary is between frontend/dashboard and local agent runtime, with all critical work happening on the local host.
Main entry points: voice_agent.py Voice agent · web_app.py Web app · voice_action.py Action router · record.py Recorder
Components 18
Local agent
Offline voice-control runtime and desktop automation core.
- Voice agent · Main terminal entry point that coordinates listening, intent resolution, execution, and feedback.
voice_agent.py - Voice loop · Encapsulates the repeated listen/process loop for the agent.
voice_loop.py - Action router · Maps recognized intent to concrete desktop actions and fallback rules.
voice_action.py - Mouse control · Dedicated mouse movement, click, scroll, and drag operations.
voice_mouse.py
Web dashboard
Browser-facing control surface and its static assets.
- Web app · Browser entry point that exposes the agent through a web dashboard.
web_app.py - Template · Main dashboard HTML template.
templates/index.html - UI script · Dashboard client behavior and browser-side interaction logic.
static/js/app.js - Styles · Dashboard styling rules.
static/css/style.css - Alt page · Alternate or experimental dashboard page.
index2.html
Perception
Audio capture, speech recognition, and screen-reading paths.
- Recorder · Captures microphone input and buffers audio for transcription.
record.py - Screen read · Reads the screen and performs OCR-based interpretation when needed.
screen_read.py
Smoke tests
Manual scripts used to exercise individual subsystems.
- Automation test · End-to-end exercise of the action layer.
automation_test.py - Mic test · Manual microphone capture check.
mic_test.py - List mics · Enumerates available audio input devices.
list_mics.py - Speak test · Verifies spoken feedback / TTS output.
speak_test.py - Voice test · Checks the speech pipeline end to end.
voice_test.py - Screen test · Validates screen capture and OCR behavior.
screen_test.py - Vision test · Auxiliary validation for screen interpretation.
vision_test.py
Connections 19
- Voice agent → Voice loop: drives · Runs the repeated listen/process cycle.
- Voice loop → Recorder: captures · Pulls audio from the microphone for processing.
- Voice agent → Action router: dispatches · Sends parsed intent into the command router.
- Voice agent → Recorder: listens · Coordinates audio intake for the main loop.
- Action router → Mouse control: pointer ops · Delegates mouse-specific desktop actions.
- Action router → Screen read: reads screen · Uses OCR/screen interpretation when needed.
- Voice agent → Screen read: optional vision · Can invoke screen reading as part of command handling.
- Web app → Template: serves · Renders the main dashboard template.
- Web app → UI script: loads · Provides browser-side interaction logic.
- Web app → Styles: loads · Provides dashboard styling.
- Web app → Alt page: alternative view · Supports the alternate dashboard page.
- Web app → Voice agent: bridges · Connects the web dashboard to the local agent runtime.
- Automation test → Action router: exercises · Smoke-tests the desktop action layer.
- Mic test → Recorder: checks · Validates microphone capture.
- List mics → Recorder: inspects · Enumerates devices used by audio capture.
- Speak test → Voice agent: verifies · Checks spoken feedback path.
- Voice test → Voice loop: validates · Exercises the main voice processing loop.
- Screen test → Screen read: validates · Checks OCR/screen capture behavior.
- Vision test → Screen read: probes · Additional screen interpretation smoke test.