# gia-guar/jarvis-chatgpt architecture

> Architecture diagram of the GitHub repository gia-guar/jarvis-chatgpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gia-guar/jarvis-chatgpt
- Repository: https://github.com/gia-guar/jarvis-chatgpt

## Overview

##### Project type and purpose
- This is a full-stack, extensible voice-based virtual assistant platform.
- It bundles:
- An “Assistant” application that captures audio, invokes language models, runs research/search agents, and orchestrates tasks.
- A full TTS (text-to-speech) library (coqui-TTS) with encoder, TTS models, vocoder, a minimal HTTP server, config, and CLI tools.
- A “Vicuna” subfolder providing a desktop/web UI wrapper around an LLM backend.
- Demo scripts, integrations (OpenAI API, Semantic Scholar), plus saved logs.

##### File structure
- Top-level directories:
- Assistant/ – core Python modules for microphone input, agent management, chat logic, voice playback, research/memory.
- TTS/ – the entire coqui-TTS codebase subdivided into encoder, tts, vocoder, vc (voice conversion), utils, a mini‐server (static + templates), and bin scripts.
- Vicuna/ – startup scripts for a GPU‐accelerated LLM web UI.
- demos/ – ready-made Python scripts showcasing various capabilities (tts, search, pyaudio, research mode, etc.).
- whisper_edits/ – minimalist ASR model wrapper.
- openai_api_chatbot.py – standalone example of using OpenAI chat.
- saved_chats/ – persistent chat transcripts.

##### Main components and their interactions
A. User Interface Layer
- CLI or web UI (Assistant/webui.py, Vicuna startup scripts)
- Demo scripts launch modes such as live microphone chat or keyboard input (demos/*.py)

B. Audio I/O Subsystem
- get_audio.py captures microphone data (or reads wave files)
- whisper_edits/model.py handles ASR transcription (if enabled)
- voice.py and Assistant/sounds/& Assistant/voices/ handle playback of system sounds and synthesized speech

C. Assistant Orchestration Layer
- VirtualAssistant.py: the central orchestrator
- Chat.py: session management, context buffering, message routing
- Agents.py: defines specialized “agents” (e.g., research_mode, semantic_scholar/simple, semantic_scholar/S2_tools, agent_tools)
- tools.py: utility functions (e.g., fetching web results, running searches)

D. Language Model Layer
- Vicuna: local LLM server or API (frontend & backend wrapper)
- openai_api_chatbot.py and Chat.py can switch between OpenAI, local LLM, or other endpoints

E. TTS Subsystem
- TTS/api.py: programmatic entry point
- TTS/bin/*.py: training, synthesizing, evaluation tools
- TTS/encoder, TTS/tts, TTS/vocoder, TTS/vc: model implementations
- TTS/server/: lightweight HTTP server serving synthesis requests (conf.json, server.py, static/, templates/)

F. Data & Configuration
- saved_chats/: file-based persistence of transcripts
- TTS/config and .models.json, config files in subdirectories (shared_configs.py, fast_speech_config.py, etc.)
- Vicuna ps1/batch files to launch UI

G. External Services
- OpenAI API (via openai Python library)
- Semantic Scholar API (via Assistant/semantic_scholar/S2_tools.py)
- Potentially any other web search or knowledge APIs used in research_mode.py

##### Architectural patterns & design principles
- Modular layering: clear separation of concerns between I/O, orchestration, model inference, and synthesis.
- Agent pattern: each “agent” encapsulates a specific capability (e.g., literature search, summarization).
- Micro-services style in TTS: encoder, synthesizer, vocoder can be invoked via CLI or HTTP.
- Plugin-friendly design: new agents or TTS vocoders can be slotted in by dropping files into the proper directory.
- Configuration-driven: model hyperparameters and endpoints are defined in config files rather than hard-coded.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface Layer"
        UI_MICRO["Microphone"]:::inproc
        UI_WEB["Web/CLI UI"]:::inproc
        UI_DEMO["Demo Scripts"]:::inproc
    end
    click UI_WEB "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Assistant/webui.py"
    click UI_DEMO "https://github.com/gia-guar/jarvis-chatgpt/blob/main/demos/chat_with_keyboard.py"
    click UI_DEMO "https://github.com/gia-guar/jarvis-chatgpt/blob/main/demos/demo_pyaudio.py"

    %% Audio I/O Subsystem
    subgraph "Audio I/O Subsystem"
        GET_AUDIO["get_audio.py"]:::inproc
        ASR["ASR"]:::inproc
        SOUND_PL["Sound Player"]:::inproc
        SOUNDS_DIR["Sounds Dir"]:::inproc
        VOICES_DIR["Voices Dir"]:::inproc
    end
    click GET_AUDIO "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Assistant/get_audio.py"
    click ASR "https://github.com/gia-guar/jarvis-chatgpt/blob/main/whisper_edits/model.py"
    click SOUND_PL "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Assistant/voice.py"
    click SOUNDS_DIR "https://github.com/gia-guar/jarvis-chatgpt/tree/main/Assistant/sounds/"
    click VOICES_DIR "https://github.com/gia-guar/jarvis-chatgpt/tree/main/Assistant/voices/"

    %% Assistant Core
    subgraph "Assistant Core"
        VA["VirtualAssistant.py"]:::inproc
        CHAT["Chat Manager"]:::inproc
        AGENTS["Agents"]:::inproc
        TOOLS["tools.py"]:::inproc
        subgraph "Semantic Scholar Agents"
            SS_SIMPLE["simple.py"]:::inproc
            SS_S2["S2_tools.py"]:::inproc
            SS_AGENT["agent_tools.py"]:::inproc
        end
        subgraph "Research Mode"
            RM["research_mode.py"]:::inproc
        end
    end
    click VA "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Assistant/VirtualAssistant.py"
    click CHAT "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Assistant/Chat.py"
    click AGENTS "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Assistant/Agents.py"
    click TOOLS "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Assistant/tools.py"
    click SS_SIMPLE "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Assistant/semantic_scholar/simple.py"
    click SS_S2 "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Assistant/semantic_scholar/S2_tools.py"
    click SS_AGENT "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Assistant/semantic_scholar/agent_tools.py"
    click RM "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Assistant/research_mode.py"

    %% Language Model Layer
    subgraph "Language Model Layer"
        OPENAI["OpenAI API"]:::external
        VICUNA_UI["Vicuna UI"]:::external
    end
    click OPENAI "https://github.com/gia-guar/jarvis-chatgpt/blob/main/openai_api_chatbot.py"
    click VICUNA_UI "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Vicuna/vicuna.ps1"
    click VICUNA_UI "https://github.com/gia-guar/jarvis-chatgpt/blob/main/Vicuna/start-webui-vicuna-gpu.bat"

    %% TTS Subsystem
    subgraph "TTS Subsystem"
        TTS_API["TTS API"]:::tts
        TTS_CLI["CLI Tools"]:::tts
        ENC["Encoder"]:::tts
        TTS_MODELS_NODE["TTS Models"]:::tts
        VOCODER["Vocoder"]:::tts
        VC["Voice Conversion"]:::tts
        TTS_SERVER["HTTP Server"]:::tts
    end
    click TTS_API "https://github.com/gia-guar/jarvis-chatgpt/blob/main/TTS/api.py"
    click TTS_CLI "https://github.com/gia-guar/jarvis-chatgpt/tree/main/TTS/bin/"
    click ENC "https://github.com/gia-guar/jarvis-chatgpt/tree/main/TTS/encoder/"
    click TTS_MODELS_NODE "https://github.com/gia-guar/jarvis-chatgpt/tree/main/TTS/tts/"
    click VOCODER "https://github.com/gia-guar/jarvis-chatgpt/tree/main/TTS/vocoder/"
    click VC "https://github.com/gia-guar/jarvis-chatgpt/tree/main/TTS/vc/"
    click TTS_SERVER "https://github.com/gia-guar/jarvis-chatgpt/tree/main/TTS/server/"

    %% Data & Configuration
    subgraph "Data & Configuration"
        SAVED["saved_chats/"]:::datastore
        MODELS_JSON[".models.json"]:::datastore
        VERSION["VERSION"]:::datastore
        SHARED_CONF["shared_configs.py"]:::datastore
        SERVER_CONF["conf.json"]:::datastore
    end
    click SAVED "https://github.com/gia-guar/jarvis-chatgpt/tree/main/saved_chats/"
    click MODELS_JSON "https://github.com/gia-guar/jarvis-chatgpt/blob/main/TTS/.models.json"
    click VERSION "https://github.com/gia-guar/jarvis-chatgpt/tree/main/TTS/VERSION"
    click SHARED_CONF "https://github.com/gia-guar/jarvis-chatgpt/blob/main/TTS/config/shared_configs.py"
    click SERVER_CONF "https://github.com/gia-guar/jarvis-chatgpt/blob/main/TTS/server/conf.json"

    %% Connections
    UI_MICRO --> GET_AUDIO
    UI_WEB --> CHAT
    UI_DEMO --> CHAT

    GET_AUDIO --> ASR
    ASR --> CHAT

    CHAT --> AGENTS
    AGENTS --> SS_SIMPLE
    AGENTS --> SS_S2
    AGENTS --> SS_AGENT
    CHAT --> RM

    CHAT --> OPENAI
    CHAT --> VICUNA_UI
    OPENAI --> CHAT
    VICUNA_UI --> CHAT

    CHAT --> TTS_API
    TTS_API --> SOUND_PL
    SOUND_PL --> UI_MICRO

    CHAT --> SAVED
    TTS_SERVER -->|HTTP| TTS_API

    %% Styles
    classDef inproc fill:#ADD8E6,stroke:#333,stroke-width:1px;
    classDef external fill:#90EE90,stroke:#333,stroke-width:1px;
    classDef tts fill:#FFA500,stroke:#333,stroke-width:1px;
    classDef datastore fill:#D3D3D3,stroke:#333,stroke-width:1px;
```
