How firexrwt/george-droid works
Project Type and Purpose
• George-Droid is an open-source, Python-based streaming assistant bot.
• It is not a traditional web “full-stack” app but rather a single coordinated application that:
• Listens to microphone audio (STT)
• Reads/writes Twitch chat events
• Sends prompts to a cloud LLM via Together AI
• Uses a local RAG memory store (FAISS + Sentence-Transformers)
• Generates speech via an external Piper TTS binary
• Provides visual context via screenshots
• Outputs audio, chat messages, and text overlays (OBS)
File-Structure Analysis
• Top-level files:
• main.py – orchestration of all modules and event loops
• requirements.txt – libraries (twitchio, faster-whisper, sentence-transformers, faiss, etc.)
• .env – Twitch & Together AI credentials
• Directories & binaries:
• backend/memory_store.py – RAG memory implementation (FAISS index + embeddings)
• piper_tts_bin/ – Piper TTS executable + Espeak-NG data files
• voices/ – ONNX voice models for Piper TTS
• Implicit components: Faster-Whisper model (STT), VAD logic in main.py, TwitchIO event handlers, OBS overlay text file
Key README Insights
• STT: Faster-Whisper with VAD
• TTS: Piper TTS external binary, swapping .onnx model+config
• LLM: Together AI API (e.g., llama-4-scout-17b)
• Memory: RAG with FAISS + Sentence-Transformers
• Chat: twitchio
• Visual Context: screenshots fed into LLM
Main Architectural Components (to include
a. Input Components
• Microphone → VAD → Faster-Whisper STT
• Twitch Chat Listener (twitchio)
• Screenshot Capture (periodic or on-demand)
b. Processing Components
• Prompt Assembler (main.py logic to merge STT, chat, screenshots, memory)
• Memory Module (backend/memory_store.py):
• FAISS Index (local data_george_memory folder)
• Embedding engine (Sentence-Transformers)
• LLM API Client (Together AI)
c. Output Components
• TTS Module: Piper TTS binary (piper_tts_bin/piper.exe + voices/*.onnx)
• Twitch Chat Bot (sending chat messages)
• OBS Overlay Text Writer (obs_ai_response.txt)
d. External Services & Data Stores
• Together AI API (LLM)
• Local FAISS index (data folder)
• Piper TTS binary + Espeak-NG assets
• ONNX voice models
Component Relationships & Data Flow
1. Audio Input → STT → Text → Prompt Assembler
2. Chat Listener → Chat Events → Prompt Assembler
3. Screenshot → Image Data → Prompt Assembler
4. Prompt Assembler queries Memory Module → retrieves context → builds final prompt
5. Final Prompt → Together AI API → LLM Response text
6. LLM Response →
• Memory Module (optionally store new context)
• TTS Module → Audio Output (speaker)
• Chat Bot → Twitch chat
• OBS Overlay → write obs_ai_response.txt
7. Loop back to listening
Architectural Patterns & Principles
• Event-driven pipeline: audio, chat, visuals trigger processing events
• Modular separation: Input / Memory / LLM / Output
• RAG pattern: retrieval + generation for contextual responses
• Single process monolith, but logically separated into modules in code