How gia-guar/jarvis-chatgpt works
Project type and purpose
• This is a full-stack, extensible voice-based virtual assistant platform.
• It bundles:
• An “Assistant” application that captures audio, invokes language models, runs research/search agents, and orchestrates tasks.
• A full TTS (text-to-speech) library (coqui-TTS) with encoder, TTS models, vocoder, a minimal HTTP server, config, and CLI tools.
• A “Vicuna” subfolder providing a desktop/web UI wrapper around an LLM backend.
• Demo scripts, integrations (OpenAI API, Semantic Scholar), plus saved logs.
File structure
• Top-level directories:
• Assistant/ – core Python modules for microphone input, agent management, chat logic, voice playback, research/memory.
• TTS/ – the entire coqui-TTS codebase subdivided into encoder, tts, vocoder, vc (voice conversion), utils, a mini‐server (static + templates), and bin scripts.
• Vicuna/ – startup scripts for a GPU‐accelerated LLM web UI.
• demos/ – ready-made Python scripts showcasing various capabilities (tts, search, pyaudio, research mode, etc.).
• whisper_edits/ – minimalist ASR model wrapper.
• openai_api_chatbot.py – standalone example of using OpenAI chat.
• saved_chats/ – persistent chat transcripts.
Main components and their interactions
A. User Interface Layer
• CLI or web UI (Assistant/webui.py, Vicuna startup scripts)
• Demo scripts launch modes such as live microphone chat or keyboard input (demos/*.py)
B. Audio I/O Subsystem
• get_audio.py captures microphone data (or reads wave files)
• whisper_edits/model.py handles ASR transcription (if enabled)
• voice.py and Assistant/sounds/& Assistant/voices/ handle playback of system sounds and synthesized speech
C. Assistant Orchestration Layer
• VirtualAssistant.py: the central orchestrator
• Chat.py: session management, context buffering, message routing
• Agents.py: defines specialized “agents” (e.g., research_mode, semantic_scholar/simple, semantic_scholar/S2_tools, agent_tools)
• tools.py: utility functions (e.g., fetching web results, running searches)
D. Language Model Layer
• Vicuna: local LLM server or API (frontend & backend wrapper)
• openai_api_chatbot.py and Chat.py can switch between OpenAI, local LLM, or other endpoints
E. TTS Subsystem
• TTS/api.py: programmatic entry point
• TTS/bin/*.py: training, synthesizing, evaluation tools
• TTS/encoder, TTS/tts, TTS/vocoder, TTS/vc: model implementations
• TTS/server/: lightweight HTTP server serving synthesis requests (conf.json, server.py, static/, templates/)
F. Data & Configuration
• saved_chats/: file-based persistence of transcripts
• TTS/config and .models.json, config files in subdirectories (shared_configs.py, fast_speech_config.py, etc.)
• Vicuna ps1/batch files to launch UI
G. External Services
• OpenAI API (via openai Python library)
• Semantic Scholar API (via Assistant/semantic_scholar/S2_tools.py)
• Potentially any other web search or knowledge APIs used in research_mode.py
Architectural patterns & design principles
• Modular layering: clear separation of concerns between I/O, orchestration, model inference, and synthesis.
• Agent pattern: each “agent” encapsulates a specific capability (e.g., literature search, summarization).
• Micro-services style in TTS: encoder, synthesizer, vocoder can be invoked via CLI or HTTP.
• Plugin-friendly design: new agents or TTS vocoders can be slotted in by dropping files into the proper directory.
• Configuration-driven: model hyperparameters and endpoints are defined in config files rather than hard-coded.