How dipan2004/jarvis-hybrid-assistant works
What this project is (type + purpose)
This is a single-repo Python application implementing a hybrid AI assistant with:
• Online conversational AI via Google Gemini (cloud inference).
• Offline/local fallback via local ML (scikit-learn) and system-command logic.
• Voice I/O (speech recognition + TTS).
• Local web dashboard via Taipy GUI for real-time conversation display + analytics.
• Optional external data (e.g., OpenWeatherMap).
So the architecture is best drawn as a modular monolith (one Python runtime) that orchestrates several subsystems + multiple external services + local persistence.
File tree analysis → map code to architectural components
From the provided tree:
• start.py
Likely the launcher/orchestrator (starts assistant + web UI, initializes config, maybe threads/processes).
• main.py
The core assistant engine: intent routing, mode switching (online/offline), command execution, conversation loop.
• display.py + display.css
The Taipy-based Web UI: dashboard, conversation feed, mode control, analytics view, export actions.
• config.py
Central configuration: env vars, API keys, toggles, logging level, endpoints, ports.
• requirements.txt
Confirms dependencies and thus major blocks: speechrecognition, pyttsx3, requests, scikit-learn, google-generativeai, taipy-gui, python-dotenv.
Jarvis Application (Python Runtime)
• Contains HybridAssistant + voice pipeline + ML pipeline + system control + connectivity detection.
OS / Host Machine
• Needed because the assistant can open apps / shutdown / read time, etc.
HybridAssistant (Orchestrator / Router) *(main.py)*
• Routes user input to:
• Online LLM (Gemini)
• Offline/local intent model
• System command executor
• External data fetch (Weather)
• Owns “mode switching” logic.
Connectivity Monitor
• Pings network / detects connectivity → toggles online/offline path.
Voice Input (SpeechRecognition)
• Mic → STT text.
Voice Output (pyttsx3 TTS)
• Response text → audio.
Intent / Command Classifier (scikit-learn)
• Offline inference; may learn from usage patterns.
Command Execution Layer (System Integration)
• Executes OS/application commands (Open Chrome, Shutdown, etc.).
• Important to mark as “privileged / potentially dangerous boundary”.
Conversation/Analytics Manager
• Tracks history, usage metrics, exports.
Config Loader (.env via python-dotenv + config.py)
• Provides keys, ports, feature flags, logging.
Google Gemini API (google-generativeai)
• Cloud LLM; requires GEMINI_API_KEY.
OpenWeatherMap API (requests) *(optional)*
• Weather queries; requires key.
Local Storage
• data/ conversation history / telemetry (local-only per README)
• models/ trained ML artifacts
• exports/ exported conversation files
If online
• HybridAssistant → Gemini API: prompt/context
• Gemini API → HybridAssistant: response text
If offline
• HybridAssistant → Intent/Command Classifier: text features
• Classifier → HybridAssistant: intent/command + confidence
• If intent is actionable: HybridAssistant → Command Execution Layer → OS
External data enrichment (optional)
• HybridAssistant → Weather API (requests): location/query
• Weather API → HybridAssistant: weather JSON → response synthesis
• Orchestrator / Router pattern: HybridAssistant is the central coordinator.
• Strategy pattern for response generation: *OnlineLLMStrategy vs OfflineModelStrategy* chosen by connectivity.
• Adapter boundaries for external dependencies:
• Gemini client adapter
• Weather client adapter
• Speech I/O adapters
• Event-driven / reactive UI: Web dashboard observes state changes (conversation, mode).
Don’t overthink it: split into as many components as possible (but keep fidelity)
Given the small repo, accuracy comes from explicitly separating concerns even if they live in one process:
• Treat display.py as its own subsystem even if it imports the assistant.
• Storage directories as distinct nodes (models vs exports vs data) because they represent different lifecycle and security concerns.