How prarthanagarwal/friday works
Project Type and Purpose
• FRIDAY MARK II is an open-source, modular AI assistant focused on real-time voice interaction, intelligent local processing (through a local LLM), memory systems, and system integration. It offers voice synthesis (using StyleTTS2) and speech recognition (using faster-whisper) alongside modular extensions via plugins.
• The core idea is to deliver rapid (parallel) processing of voice commands with a sophisticated memory and integration layer.
Analyzing the File Structure
• Top-level files indicate entrypoints and configuration (e.g., friday.py, setup.py, requirements.txt).
• The src directory is the heart of the system and is divided into functional sub-modules:
• api: Contains the server (likely a FastAPI or gRPC implementation) to handle external requests.
• core: Hosts the “brain” of FRIDAY with modules for configuration, conversation management, local LLM interfacing, memory, metrics, context handling, session management, plugin managers, and parallel processing (e.g., parallel_processor.py and redis_handler.py indicate state management with Redis).
• speech: Wraps modules related to speech recognition (whisper_handler.py) to interface with faster-whisper.
• voice: Contains modules for voice functions like caching, speech-to-text (stt), text-to-speech (tts), playing and recording audio.
• utils: Includes utility functions such as logging.
• Additional files such as Personality.txt and tts.txt may provide configuration or sample data for voice synthesis settings.
Insights from the README
• Voice Interface: Utilizing StyleTTS2 and faster-whisper.
• Brain: Comprising local LLM processing (llama.cpp), a memory system (contextual and tag-based), and a command parser.
• Skills System: Incorporating core skills and plug-and-play custom plugins.
• System Integration: Covering device control, sensor processing, and external services.
• The tech stack highlights FastAPI/gRPC for microservices, Redis for state management, and integration with external systems (Home Assistant).
a. Main Components:
• Entry Points: friday.py (startup) and src/main.py.
• API Layer: Located in src/api/server.py – responsible for exposing endpoints for interaction.
• Core Processing / Brain:
– Modules in src/core (configuration, conversation, llm, memory, session, plugin manager, etc.) represent the cognitive engine; this is where the LLM is invoked, memory systems operate, and command parsing occurs.
• Voice and Speech Interfaces:
– The voice module (src/voice) is responsible for TTS, STT, audio caching, and playing/recording.
– The speech module (src/speech) focuses on managing the faster-whisper interface.
• Utilities and External Integrations:
– redis_handler.py from src/core indicates real-time state management using Redis.
– The configuration and logger help trace operations and debug the system.
• (Optional) Plugins or Skills Module: Represented in src/core/plugin_manager.py and src/core/plugins.py, enabling the modular design.
b. Relationships and Interactions:
• User input (voice captured) → Voice Interface (STT in src/voice/stt.py and src/speech/whisper_handler.py).
• Processed text input → API/Command Parser → Core Processing to invoke the LLM (src/core/llm.py) and look up contextual memory (src/core/memory.py, context_memory.py).
• Core output → Voice Interface (tts via src/voice/tts.py) for voice synthesis.
• Parallel processing (src/core/parallel_processor.py) should be shown as managing multiple tasks simultaneously.
• The plugin modules “plug into” the core system to provide extendable skills or system integrations.
• External integrations (like device control, sensor data) can be depicted as services interacting with the Core.
c. Important Architectural Patterns & Technologies:
• Modular Architecture: Showcase the clear separation into distinct components (API, Core, Voice, Skills, Integration).
• Microservice Communication: Indicate FastAPI/gRPC usage where applicable.
• Parallel Processing: Denote that certain components (via parallel_processor.py) work concurrently.
• State Management: Include Redis as a component feeding into or receiving signals from the core.
• Integration of specialized libraries (StyleTTS2, faster-whisper, llama.cpp) should be noted within the relevant modules.
d. Tailor for the Project Type:
• For this full-stack voice assistant:
• Clear divide between the voice interface (input/output) and the internal processing systems.
• How the API layer serves as a bridge between external commands (or device integrations) and the core AI engine.
• Memory and LLM processing as centrally critical operations.