How sreejanpersonal/jarvis-agi works
Project type and purpose
• This is a full-stack AI assistant ("JARVIS-AGI") built in Python.
• Purpose: accept voice/text/image input, dispatch to speech-to-text (STT), natural-language/image-AI engines, then produce text responses and speech output.
High-level layered architecture
• User I/O layer
– Voice (microphone) and text (CLI/terminal) inputs
– Audio playback and console output
• Engine layer
– STT subsystem (Engine/STT: Vosk, DevsDoCode, NetHyTech)
– TTS subsystem (Engine/TTS: ElevenLabs, DeepGram, streaming variants)
• Brain (Core AI) layer
– Text-AI (BRAIN/AI/TEXT: multiple API adapters, local llama_CPP, streaming)
– Vision-AI (BRAIN/AI/VISION: deepInfra_VISION)
– Image-AI (BRAIN/AI/IMAGE modules)
– Tools (BRAIN/TOOLS and TOOLS/*: utilities like groq_web_access, Hotword_Detection, system settings)
• Assets & Data layer
– ASSETS: model files (Vosk, clap detection), sounds, user data, proxies, conversation logs
• Orchestrator
– main.py + IMPORTS.py glues layers together, boots config (.env), reads assets, invokes STT → Brain → TTS pipeline.
Key components and interactions
• User
→ sends audio/text to Orchestrator (main.py)
• STT Engine
← reads model from ASSETS/Vosk
→ emits text to Brain
• Brain
– Text branch: picks one of TEXT/API connectors or LOCAL/STREAM modules, optionally calls external APIs (OpenAI, HuggingFace, FarFalle, etc.)
– Vision/Image branch: processes image inputs via deepInfra_IMG, decohere_ai
– Tools branch: auxiliary services (web search, hotword detection)
← optionally loads prompts from PROMPTS/
→ returns AI-generated text
• TTS Engine
← receives text, invokes chosen provider (ElevenLabs, Edge-TTS, speechify, DeepGram)
→ outputs audio to user
• Assets & Data
– Models and config loaded at startup
– conversation_history.json updated by orchestrator
– openGPT_IDs.txt, available_working_proxies.txt for external API management
Architectural patterns and design principles
• Layered pipeline: I/O → STT → AI Brain → TTS → I/O
• Plugin/Adapter pattern: each AI service is an interchangeable Python module under BRAIN/AI/TEXT/API or ENGINE/TTS/STREAMING
• Configuration-driven: .env for API keys and model paths
• Separation of concerns:
– ENGINE modules know only about speech/audio,
– BRAIN modules handle AI logic,
– TOOLS hold utilities,
– PLAYGROUND for experiments (not core at runtime).
Capturing essential elements
• Main pipelines and data flows
• Important libraries/frameworks:
– Vosk (STT), ElevenLabs/DeepGram (TTS), OpenAI/HuggingFace (Text AI), llama_CPP (local AI)