How nazirlouis/ada_v2 works
1. System shape
- This is a local desktop application built from two major runtimes: Electron + React for the UI, and a Python backend for AI, vision, automation, and hardware control.
- The app is not cloud-hosted; most data and processing happen on the user’s machine, with only Gemini API calls going over the network.
2. Desktop shell and UI layer
- electron/main.js is the desktop host: it creates the application windows, manages lifecycle, and bridges native Electron behavior to the React UI.
- The React app in src/ is split into task-specific windows/modules: chat, browser automation, CAD, printer control, Kasa smart home, settings, auth lock, confirmations, memory prompts, and audio/visual feedback.
3. Frontend as a control surface
- The UI is mostly a command-and-status console rather than a standalone business-logic layer.
- Components like ChatModule, ToolsModule, TopAudioBar, and Visualizer expose voice/chat state and tool actions, while specialized windows (CadWindow, PrinterWindow, BrowserWindow, KasaWindow) surface the outputs of backend agents.
4. Backend orchestration layer
- backend/server.py is the central backend entrypoint and request/event hub.
- It coordinates the UI, authentication, project state, and tool execution, acting as the local control plane that receives user actions and forwards them to the correct subsystem.
5. Gemini voice and tool-calling core
- backend/ada.py is the main AI runtime, integrating Google Gemini 2.5 Native Audio for low-latency conversational interaction.
- backend/tools.py defines the tool surface exposed to the model, so the assistant can invoke CAD, web, printer, and smart-home actions through controlled functions rather than free-form code.
6. Specialized agent subsystems
- backend/cad_agent.py handles parametric CAD generation and iteration, with build123d-style model creation and STL output.
- backend/web_agent.py drives browser automation via Playwright/Chromium.
- backend/printer_agent.py handles printer discovery, slicing, and print submission.
- backend/kasa_agent.py manages TP-Link Kasa smart-home device control.
7. Authentication and interaction gating
- backend/authenticator.py provides local face authentication using MediaPipe face landmarks; the reference.jpg file is the biometric anchor.
- The UI also uses confirmation flows for sensitive actions, and settings.json stores whether tools like CAD generation, web automation, or file writes require manual approval before execution.
8. Vision and gesture control
- Camera-based gestures are supported through MediaPipe task assets (face_landmarker.task, hand_landmarker.task).
- Hand gestures are part of the UI interaction model, enabling window selection/movement and action confirmation without conventional mouse input.
9. Project memory and workspace model
- backend/project_manager.py manages persistent per-project context and file organization.
- The repository implies a workspace model where generated assets land under project folders, especially CAD outputs such as STL files, so the assistant can iterate on a named project across sessions.
10. CAD-to-print pipeline
- The CAD path is a distinct pipeline: user intent → CAD agent → generated model/STL → optional verification → slicing/printing.
- backend/verify_cad.py, backend/temp_cad_gen.py, and related mock/test files indicate that CAD generation has a validation step and supports iterative refinement before anything is sent to a printer.
11. 3D printing and physical device boundary
- Printer integration spans local network discovery, slicer integration, and printer APIs such as Moonraker/OctoPrint/PrusaLink.
- This is a clear boundary between digital design and physical output: the backend prepares G-code and then submits jobs to external printer infrastructure.
12. Persistence and configuration
- Persistent state is lightweight and file-based: .env for secrets, settings.json for app behavior, and project directories for user assets.
- There is no separate database service in the repo; the architecture relies on local files and runtime state.
13. Validation and support tooling
- The tests/ directory covers the agents and server behavior, indicating the architecture is modular and testable by subsystem.
- Debug and trace scripts (debug_*, read_trace.py, grep_trace.py) suggest the system is designed to diagnose hardware, network, and agent issues across the local device and external peripherals.
Main entry points: server.py Backend server · tools.py Tool surface · App.jsx App shell · cad_agent.py CAD agent
Components 16
Desktop UI
Electron shell and React control surface.
- Electron host · Creates the desktop window and bridges native app lifecycle to the UI.
electron/main.js - App shell · Top-level React application that routes the desktop experience.
src/App.jsx - Chat module · Primary conversation and command surface.
src/components/ChatModule.jsx - Tools module · Exposes tool actions and task controls to the user.
src/components/ToolsModule.jsx - Audio feedback · Voice status and visual feedback for the assistant.
src/components/TopAudioBar.jsx - Task windows · Specialized surfaces for browser, CAD, printer, Kasa, settings, and prompts.
Backend Control Plane
Local orchestration, AI runtime, and shared state.
- Backend server · Central request and event hub for the local control plane.
backend/server.py - Ada runtime · Low-latency Gemini audio conversation and tool-calling runtime.
backend/ada.py - Tool surface · Defines the callable actions exposed to the AI runtime.
backend/tools.py - Project memory · Manages persistent per-project context and generated assets.
backend/project_manager.py
Automation Agents
Specialized subsystems for CAD, web, printer, and smart-home actions.
- CAD agent · Generates and iterates models and STL output.
backend/cad_agent.py - Web agent · Controls Chromium via Playwright for browser tasks.
backend/web_agent.py - Printer agent · Handles printer discovery, slicing, and job submission.
backend/printer_agent.py - Kasa agent · Controls TP-Link Kasa devices on the local network.
backend/kasa_agent.py
Support Tools
Authentication, verification, and workspace helpers.
- Face auth · Performs local face-based authentication for gated actions.
backend/authenticator.py - CAD verify · Validates generated CAD iterations before release or printing.
backend/verify_cad.py
Connections 20
- Electron host → App shell: launches · Starts and hosts the React application.
- App shell → Chat module: renders · Composes the main conversation surface.
- App shell → Tools module: renders · Provides task controls and action entry points.
- App shell → Audio feedback: renders · Surfaces voice state and assistant feedback.
- App shell → Task windows: opens · Manages task-specific windows and prompts.
- Task windows → Face auth: gated by · Sensitive interactions require local authentication or confirmation.
- Backend server → Ada runtime: coordinates · Routes user interactions into the AI runtime.
- Backend server → Tool surface: exposes · Publishes the controlled tool set to the backend.
- Backend server → Project memory: tracks · Maintains per-project context and workspace state.
- Backend server → Face auth: checks · Applies local authentication before protected actions.
- Ada runtime → Tool surface: calls · Invokes tool functions during conversation.
- Tool surface → CAD agent: routes · Delegates CAD tasks to the modeling subsystem.
- Tool surface → Web agent: routes · Delegates browser automation tasks.
- Tool surface → Printer agent: routes · Delegates printing and slicing tasks.
- Tool surface → Kasa agent: routes · Delegates smart-home control tasks.
- CAD agent → CAD verify: validates · Runs CAD output through validation before handoff.
- CAD agent → Project memory: writes · Stores generated CAD artifacts in project workspace.
- Printer agent → Project memory: reads · Uses project assets when preparing print jobs.
- Backend server → Chat module: feeds status · Pushes backend state back into the UI.
- Backend server → Task windows: updates · Updates task windows with agent results and confirmations.