How michniks2001/voice-ai-hack works
Project Type and Purpose
• This is a full-stack, voice-enabled conversational AI web application built with Next.js 14 (App Router).
• It provides a chat interface (frontend) that records user speech, transcribes it via ElevenLabs Speech-to-Text (STT), sends the text to OpenAI’s GPT-3.5-turbo (chat API), and then converts the AI’s response back to audio via ElevenLabs Text-to-Speech (TTS).
File Structure Analysis
• Top-level:
• app/ ← Next.js App Router directory
– page.tsx ← Main UI page (React server component pulling in client components)
– layout.tsx ← Global layout (navigation, styling)
– api/ ← Serverless API routes
• chat/route.ts
• speech-to-text/route.ts
• text-to-speech/route.ts
• components/ ← Reusable UI components (ChatMessage, VoiceInput, LoadingIndicator, plus Shadcn UI primitives)
• hooks/ ← Client hooks (useVoiceRecording.ts)
• lib/ ← Shared utilities (lib/utils.ts)
• public/ ← Static assets
• styles & config (globals.css, tailwind/postcss configs, eslint.config, tsconfig)
• Key patterns:
• Single monolithic codebase with Next.js server+client components.
• API layer under app/api handling integration with external AI services.
• Client-side voice capture (Web Audio API) via a custom hook.
README Insights
• Confirms tech stack: Next.js, React, TypeScript, Tailwind CSS, Shadcn UI, OpenAI GPT-3.5, ElevenLabs STT/TTS, Web Audio API.
• Describes API endpoints and high-level flow:
POST /api/speech-to-text → ElevenLabs STT
POST /api/chat → OpenAI GPT
POST /api/text-to-speech → ElevenLabs TTS
Main Components
a. Frontend (Client Browser)
• UI Pages & Components
• Web Audio API (voice capture & playback)
• Custom React Hook: useVoiceRecording
b. Backend (Next.js Serverless Functions)
• /api/speech-to-text (ElevenLabs STT integration)
• /api/chat (OpenAI GPT-3.5 integration, system prompt for Barnaby)
• /api/text-to-speech (ElevenLabs TTS integration)
c. External Services
• ElevenLabs Speech-to-Text API
• OpenAI Chat API (GPT-3.5-turbo)
• ElevenLabs Text-to-Speech API
Interactions and Data Flow
1) User clicks “Start Recording” → Web Audio API begins capturing audio.
2) User clicks “Stop Recording” → Hook returns audio Blob.
3) Frontend POSTs Blob to /api/speech-to-text.
4) Next.js route.ts calls ElevenLabs STT → returns transcript.
5) Frontend receives transcript → displays it, then POSTs it along with chat history to /api/chat.
6) Server calls OpenAI GPT with system prompt + history → returns AI message.
7) Frontend displays AI text → POSTs AI text to /api/text-to-speech.
8) Server calls ElevenLabs TTS → returns audio URL/bytes.
9) Frontend plays audio via Web Audio API.
Architectural Patterns & Principles
• Separation of concerns: UI vs. API vs. external service integration.
• Serverless functions for each third-party integration (STT, Chat, TTS).
• Single-page React client with server components for static content.
• Hook-based encapsulation of browser APIs.
Essential Elements to Capture
• Voice capture layer (Web Audio API + useVoiceRecording).
• Next.js API routes as serverless functions.
• Three external AI services with their roles.
• Chat history management (passed from client to /api/chat).
• UI update & audio playback loop.