How bhancockio/adk-voice-agent works
Project Type and Purpose
• This is a full-stack voice-assistant integration project.
• Frontend: A static single-page interface (HTML/JavaScript) for recording audio, playing back responses, and sending voice data to the backend.
• Backend: A Python service (FastAPI + Uvicorn) that runs a “Jarvis” agent. The agent uses a large-language-model API (Gemini) to parse voice commands and dispatches Google Calendar operations via specialized tool modules.
File Structure Analysis
• Top level:
• setup_calendar_auth.py – OAuth 2.0 setup script.
• requirements.txt – Python dependencies.
• README.md – high-level setup and usage.
• app/
• main.py – FastAPI application entrypoint.
• static/ – Frontend assets: index.html and JS modules (audio recorder/player, PCM processors).
• jarvis/agent.py – Orchestrates model calls and tool selection.
• jarvis/tools/ – Calendar-specific operations: calendar_utils, create/edit/delete/list event.
• Credentials and token storage are external (credentials.json in root, ~/.credentials/calendar_token.json).
Key Components
a. Frontend (Browser)
• index.html
• JS modules:
• audio-recorder.js + pcm-recorder-processor.js
• audio-player.js + pcm-player-processor.js
• app.js (handles UI and HTTP/WebSocket to backend)
b. Backend (FastAPI Service)
• API Gateway (uvicorn main:app)
• Jarvis Agent (agent.py)
• Tool modules (calendar_utils.py, create/list/edit/delete event)
• OAuth Setup Script (setup_calendar_auth.py) – one-time flow
c. External Services
• Google Calendar REST API
• Gemini (Google AI Studio) LLM API
d. Credentials & Storage
• credentials.json (OAuth client ID)
• ~/.credentials/calendar_token.json (OAuth tokens)
Component Relationships and Data Flow
1. User clicks “record” → audio-recorder.js captures PCM audio → sends binary blob to FastAPI endpoint.
2. FastAPI endpoint forwards audio to Gemini API for speech-to-text (or text-to-speech).
3. Agent receives transcript → selects appropriate tool module (e.g., create_event).
4. Tool uses calendar_utils to handle OAuth flow or call Google Calendar API.
5. Google Calendar responds with event data or confirmation.
6. Agent formulates a natural-language response → sends back to frontend.
7. Frontend’s audio-player.js plays the TTS audio or displays the text.
Architectural Patterns and Principles
• Separation of concerns: frontend UI vs. backend business logic.
• Modular tool pattern: each calendar operation is a standalone tool under jarvis/tools.
• Externalized authentication: OAuth handshake is managed by setup_calendar_auth.py and calendar_utils.
• Micro-layer for LLM integration: agent.py sits between raw input/output and tool logic.
Technologies and Frameworks
• Frontend: Vanilla JS, Web Audio API, PCM processor Web-Audio-Worklet.
• Backend: Python 3.8+, FastAPI, Uvicorn.
• APIs: Google Calendar API (OAuth 2.0), Gemini LLM API.
• Storage: Local filesystem for credentials and tokens.
• Use clear, consistent labels: e.g., “Audio Recorder”, “FastAPI /main.py”, “agent.py”, “Google Calendar API”.
• From Browser → FastAPI (audio/text requests)
• From FastAPI → Gemini API (LLM calls)
• From FastAPI → Google Calendar API (REST calls)
• From APIs back to FastAPI → Browser
• Blue for frontend elements
• Green for backend/application logic
• Orange for external services
• Gray for storage/credentials
Final Checklist
• All major components (Frontend, Backend, External APIs, Storage)
• Technology callouts next to each component
• Annotation of OAuth setup process as an initialization flow