# dipan2004/jarvis-hybrid-assistant architecture

> Architecture diagram of the GitHub repository dipan2004/jarvis-hybrid-assistant, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dipan2004/jarvis-hybrid-assistant
- Repository: https://github.com/dipan2004/jarvis-hybrid-assistant

## Overview

##### What this project is (type + purpose)
This is a **single-repo Python application** implementing a **hybrid AI assistant** with:
- **Online conversational AI** via **Google Gemini** (cloud inference).
- **Offline/local fallback** via **local ML (scikit-learn)** and system-command logic.
- **Voice I/O** (speech recognition + TTS).
- **Local web dashboard** via **Taipy GUI** for real-time conversation display + analytics.
- Optional external data (e.g., **OpenWeatherMap**).

So the architecture is best drawn as a **modular monolith** (one Python runtime) that orchestrates several subsystems + multiple external services + local persistence.

##### File tree analysis → map code to architectural components
From the provided tree:

- `start.py`
Likely the **launcher/orchestrator** (starts assistant + web UI, initializes config, maybe threads/processes).

- `main.py`
The **core assistant engine**: intent routing, mode switching (online/offline), command execution, conversation loop.

- `display.py` + `display.css`
The **Taipy-based Web UI**: dashboard, conversation feed, mode control, analytics view, export actions.

- `config.py`
Central **configuration**: env vars, API keys, toggles, logging level, endpoints, ports.

- `requirements.txt`
Confirms dependencies and thus major blocks: `speechrecognition`, `pyttsx3`, `requests`, `scikit-learn`, `google-generativeai`, `taipy-gui`, `python-dotenv`.

##### Jarvis Application (Python Runtime)
- Contains HybridAssistant + voice pipeline + ML pipeline + system control + connectivity detection.
##### OS / Host Machine
- Needed because the assistant can open apps / shutdown / read time, etc.

##### HybridAssistant (Orchestrator / Router) *(main.py)*
- Routes user input to:
- Online LLM (Gemini)
- Offline/local intent model
- System command executor
- External data fetch (Weather)
- Owns “mode switching” logic.
##### Connectivity Monitor
- Pings network / detects connectivity → toggles online/offline path.
##### Voice Input (SpeechRecognition)
- Mic → STT text.
##### Voice Output (pyttsx3 TTS)
- Response text → audio.
##### Intent / Command Classifier (scikit-learn)
- Offline inference; may learn from usage patterns.
##### Command Execution Layer (System Integration)
- Executes OS/application commands (Open Chrome, Shutdown, etc.).
- Important to mark as “privileged / potentially dangerous boundary”.
##### Conversation/Analytics Manager
- Tracks history, usage metrics, exports.
##### Config Loader (.env via python-dotenv + config.py)
- Provides keys, ports, feature flags, logging.

##### Google Gemini API (google-generativeai)
- Cloud LLM; requires `GEMINI_API_KEY`.
##### OpenWeatherMap API (requests) *(optional)*
- Weather queries; requires key.

##### Local Storage
- `data/` conversation history / telemetry (local-only per README)
- `models/` trained ML artifacts
- `exports/` exported conversation files

##### If online
- **HybridAssistant → Gemini API**: prompt/context
- **Gemini API → HybridAssistant**: response text
##### If offline
- **HybridAssistant → Intent/Command Classifier**: text features
- **Classifier → HybridAssistant**: intent/command + confidence
- If intent is actionable: **HybridAssistant → Command Execution Layer → OS**

##### External data enrichment (optional)
- **HybridAssistant → Weather API (requests)**: location/query
- **Weather API → HybridAssistant**: weather JSON → response synthesis

- **Orchestrator / Router pattern**: `HybridAssistant` is the central coordinator.
- **Strategy pattern** for response generation: *OnlineLLMStrategy vs OfflineModelStrategy* chosen by connectivity.
- **Adapter boundaries** for external dependencies:
- Gemini client adapter
- Weather client adapter
- Speech I/O adapters
- **Event-driven / reactive UI**: Web dashboard observes state changes (conversation, mode).

##### Don’t overthink it: split into as many components as possible (but keep fidelity)
Given the small repo, accuracy comes from **explicitly separating concerns** even if they live in one process:
- Treat `display.py` as its own subsystem even if it imports the assistant.
- Storage directories as distinct nodes (models vs exports vs data) because they represent different lifecycle and security concerns.

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% Diagram 1 — Context (C4 L1)
  %% =========================
  subgraph D1["Diagram 1 — Context (C4 Level 1)"]
    direction TB

    U1["User"]:::actor
    Mic1["Microphone"]:::device
    Spk1["Speakers"]:::device
    Browser1["Browser (Dashboard)"]:::actor

    subgraph Sys1["Jarvis System (Local Machine)"]
      direction TB
      Jarvis1["Jarvis Application (Python)"]:::internal
      Taipy1["Taipy Web Dashboard (localhost:5000)"]:::internal
      FS1[("Local Storage\n(data/, models/, exports/)")]:::storage
      OS1["Host OS / System"]:::os
    end

    Gemini1["Google Gemini API"]:::external
    Weather1["OpenWeatherMap API (optional)"]:::external

    U1 -->|"speaks"| Mic1
    Mic1 -->|"audio_stream"| Jarvis1
    Jarvis1 -->|"response_audio"| Spk1 -->|"hears"| U1

    Browser1 -->|"HTTP/WebSocket"| Taipy1
    Taipy1 -->|"conversation_updates"| Browser1

    Jarvis1 -->|"store/read_history_models_exports"| FS1
    Jarvis1 -->|"system_commands"| OS1

    Jarvis1 -->|"SDK_call(prompt+context)"| Gemini1
    Gemini1 -->|"response_text"| Jarvis1

    Jarvis1 -->|"HTTP(requests):location/query"| Weather1
    Weather1 -->|"weather_JSON"| Jarvis1
  end

  %% ===================================
  %% Diagram 2 — Container/Deployment (L2)
  %% ===================================
  subgraph D2["Diagram 2 — Container / Deployment (C4 Level 2)"]
    direction TB

    User2["User"]:::actor
    HW2["Audio Hardware\n(Mic/Speakers)"]:::device
    Net2["Internet"]:::external
    Gemini2["Google Gemini API"]:::external
    Weather2["OpenWeatherMap API (optional)"]:::external

    subgraph Host2["Host Machine"]
      direction TB

      OS2["OS (Windows/macOS/Linux)"]:::os

      subgraph Runtime2["Python Runtime (Single Repo / Modular Monolith)"]
        direction TB
        Proc2["Jarvis App Process\n(start.py entrypoint)"]:::internal
        Web2["Taipy GUI Server\nhttp://localhost:5000"]:::internal
      end

      subgraph Mount2["Local File System (Persistence)"]
        direction TB
        Data2[("data/\nconversation_history+telemetry")]:::storage
        Models2[("models/\nML_artifacts")]:::storage
        Exports2[("exports/\nexported_conversations")]:::storage
      end

      Env2[".env (keys)\nGEMINI_API_KEY, WEATHER_API_KEY"]:::config
      Cfg2["config.py (toggles/ports/logging)"]:::config
    end

    User2 -->|"voice_interaction"| HW2
    HW2 -->|"audio_stream"| Proc2
    Proc2 -->|"tts_audio"| HW2

    User2 -->|"open_dashboard"| Web2
    Web2 -->|"render_UI"| User2

    Proc2 -->|"read_config"| Cfg2
    Cfg2 -->|"load_env"| Env2

    Proc2 -->|"file_IO"| Data2
    Proc2 -->|"load/save"| Models2
    Proc2 -->|"write_exports"| Exports2

    Proc2 -->|"runs_in"| OS2
    Proc2 -->|"hosts_thread_or_subsystem"| Web2

    Proc2 -->|"requires_connectivity"| Net2
    Net2 -->|"routes_API_calls"| Gemini2
    Net2 -->|"routes_API_calls"| Weather2
  end

  %% ==========================================
  %% Diagram 3 — Component (C4 Level 3, code)
  %% ==========================================
  subgraph D3["Diagram 3 — Component (C4 Level 3)"]
    direction TB

    %% External actors/devices
    User3["User"]:::actor
    Mic3["Microphone"]:::device
    Spk3["Speakers"]:::device
    Browser3["Browser"]:::actor

    %% External services
    Gemini3["Google Gemini API"]:::external
    Weather3["OpenWeatherMap API (optional)"]:::external
    OS3["Host OS / System"]:::os

    %% Storage
    Data3[("data/")]:::storage
    Models3[("models/")]:::storage
    Exports3[("exports/")]:::storage

    %% App boundary
    subgraph App3["Jarvis Application (Python Runtime)"]
      direction TB

      Boot3["Launcher / Bootstrapper\n(start.py)"]:::internal

      subgraph Cfg3["Configuration"]
        direction TB
        Config3["Config Loader\n(config.py + python-dotenv)"]:::config
        Req3["Dependencies\n(requirements.txt)"]:::config
      end

      subgraph Core3["Core Assistant Engine (main.py)"]
        direction TB

        Hybrid3["HybridAssistant\n(Orchestrator / Router)"]:::internal

        Conn3["Connectivity Monitor\n(online? check)"]:::internal

        subgraph Voice3["Voice I/O"]
          direction TB
          STT3["Voice Input (STT)\nSpeechRecognition"]:::internal
          TTS3["Voice Output (TTS)\npyttsx3"]:::internal
        end

        subgraph Strategy3["Response Strategies"]
          direction TB
          Online3["Online LLM Client Adapter\n(Gemini integration)"]:::internal
          Offline3["Offline Intent/Command Classifier\n(scikit-learn)"]:::internal
          WeatherClient3["External Data Fetch Client\n(Weather via requests)"]:::internal
        end

        subgraph State3["State / Analytics"]
          direction TB
          Conv3["Conversation/Analytics Manager\n(history+metrics)"]:::internal
        end

        subgraph Priv3["Privileged Operations Boundary"]
          direction TB
          Cmd3["Command Execution Layer\n(System Integration)"]:::danger
        end
      end

      subgraph UI3["Web Dashboard (Taipy GUI)"]
        direction TB
        WebSrv3["Taipy Web Dashboard Server\n(display.py)"]:::internal
        CSS3["UI Styling\n(display.css)"]:::internal
      end
    end

    %% Voice flow
    User3 -->|"speaks"| Mic3
    Mic3 -->|"audio_stream"| STT3
    STT3 -->|"recognized_text"| Hybrid3
    Hybrid3 -->|"response_text"| TTS3
    TTS3 -->|"audio_out"| Spk3
    Spk3 -->|"hears"| User3

    %% Online/offline decision
    Hybrid3 -->|"online_check"| Conn3
    Conn3 -->|"status(online/offline)"| Hybrid3

    %% Online path
    Hybrid3 -->|"prompt+context"| Online3
    Online3 -->|"SDK_call"| Gemini3
    Gemini3 -->|"response_text"| Online3
    Online3 -->|"response_text"| Hybrid3

    %% Offline path
    Hybrid3 -->|"text_features"| Offline3
    Offline3 -->|"intent+confidence"| Hybrid3
    Hybrid3 -->|"if_actionable"| Cmd3
    Cmd3 -->|"system_commands"| OS3

    %% External data enrichment
    Hybrid3 -->|"location/query"| WeatherClient3
    WeatherClient3 -->|"HTTP(requests)"| Weather3
    Weather3 -->|"weather_JSON"| WeatherClient3
    WeatherClient3 -->|"weather_data"| Hybrid3

    %% State + persistence
    Hybrid3 -->|"conversation_event+metrics"| Conv3
    Conv3 -->|"file_write/read"| Data3
    Offline3 -->|"load/save_models"| Models3
    Hybrid3 -->|"export_request"| Exports3

    %% Dashboard flow
    Browser3 -->|"HTTP/WebSocket:ui_events"| WebSrv3
    WebSrv3 -->|"bind_state/commands"| Hybrid3
    Hybrid3 -->|"push_updates(mode+conversation+stats)"| WebSrv3
    WebSrv3 -->|"render_UI"| Browser3
    WebSrv3 -->|"load_styles"| CSS3

    %% Bootstrap/config wiring
    Boot3 -->|"initialize"| Config3
    Boot3 -->|"start_assistant_loop"| Hybrid3
    Boot3 -->|"start_web_server"| WebSrv3
    Config3 -->|"provides_keys+toggles+port(5000)"| Hybrid3
    Config3 -->|"provides_ui_config"| WebSrv3
    Req3 -->|"declares_dependencies"| Boot3
  end

  %% =========================
  %% Click Events (from mapping)
  %% =========================
  click Jarvis1 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/start.py"
  click Taipy1 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/display.py"

  click Proc2 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/start.py"
  click Web2 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/display.py"
  click Cfg2 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/config.py"

  click Boot3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/start.py"
  click Hybrid3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/main.py"
  click Conn3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/main.py"
  click STT3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/main.py"
  click TTS3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/main.py"
  click Online3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/main.py"
  click WeatherClient3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/main.py"
  click Offline3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/main.py"
  click Conv3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/main.py"
  click Cmd3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/main.py"
  click WebSrv3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/display.py"
  click CSS3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/display.css"
  click Config3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/config.py"
  click Req3 "https://github.com/dipan2004/jarvis-hybrid-assistant/blob/main/requirements.txt"

  %% =========================
  %% Styles
  %% =========================
  classDef internal fill:#1f77b4,stroke:#0b3d91,color:#ffffff,stroke-width:1.5px
  classDef external fill:#f39c12,stroke:#b9770e,color:#111111,stroke-width:1.5px
  classDef storage fill:#7f8c8d,stroke:#2c3e50,color:#ffffff,stroke-width:1.5px
  classDef device fill:#2ecc71,stroke:#1e8449,color:#0b1b10,stroke-width:1.5px
  classDef actor fill:#a9dfbf,stroke:#1e8449,color:#0b1b10,stroke-width:1.5px
  classDef os fill:#95a5a6,stroke:#2c3e50,color:#111111,stroke-width:1.5px
  classDef danger fill:#e74c3c,stroke:#922b21,color:#ffffff,stroke-width:2px
  classDef config fill:#8e44ad,stroke:#4a235a,color:#ffffff,stroke-width:1.5px
```
