How bernietv/elevenlabs-clone works
Project Type and Purpose
• This is a full-stack, self-hosted AI audio studio that clones ElevenLabs functionality.
• Frontend: Next.js (T3 stack) for UI & client-side routing.
• Backend/API: FastAPI micro-services (one per AI model: StyleTTS2, Seed-VC, Make-An-Audio) plus Next.js API routes (Auth.js, Inngest).
• Datastore: PostgreSQL (via Prisma) for user/credits, AWS S3 for audio file storage.
• Queueing: Inngest to throttle and sequence long-running audio jobs.
File Structure
• Top-level folders:
• elevenlabs-clone-frontend (Next.js + Prisma + Inngest + Auth.js)
• Make-An-Audio, StyleTTS2, seed-vc (each a Python FastAPI service with model code, configs, Dockerfiles)
• docker-compose.yml to orchestrate containers
• Config & infra:
• Dockerfile.api / Dockerfile.train in each AI folder
• configs/ for model hyperparameters
• README details AWS IAM & deployment
• Frontend specifics:
• /src/app/... – Next.js App Router with API routes in /app/api
• /src/server – database & auth setup
• /src/inngest – Inngest function definitions
• Backend specifics:
• api.py in each model folder – FastAPI entrypoints
• model code in subdirectories (ldm for Make-An-Audio, Modules for StyleTTS2, modules for seed-vc)
• scripts for inference & batch generation (gen_wav.py, train_*.py)
README Insights
• Confirms micro-services per model with FastAPI endpoints.
• Frontend handles auth, credits, queueing via Inngest.
• Uses AWS S3 for storage & EC2/ECR for container hosting.
a. Main Components:
• Client Browser (User UI)
• Next.js Frontend App (React + Tailwind + Auth.js + Prisma)
• Next.js API Routes
– /api/auth ([...nextauth]) for authentication
– /api/inngest for enqueuing jobs
• Inngest Queue (Task orchestration)
• Model Microservices (Docker containers):
– StyleTTS2 Service
– Seed-VC Service
– Make-An-Audio Service
• Database (PostgreSQL via Prisma)
• Object Storage (AWS S3)
b. Relationships & Data Flow:
1. User interacts with UI → fetch voices, submit text/audio
2. Frontend authenticates via NextAuth → session token
3. On “generate” action, frontend calls Next.js API → writes job request to Inngest
4. Inngest triggers a worker process that calls appropriate model microservice endpoint
5. Model service processes request, returns audio file URL or binary
6. Service uploads result to S3 → returns S3 URL
7. Worker writes metadata (user, credit usage, S3 link) into PostgreSQL
8. Frontend polls or receives webhook update → displays audio history & playback
c. Architectural Patterns & Principles:
• Microservices: one container per AI model for isolation & scalability
• Event-driven / queueing (Inngest) to handle asynchronous, long‐running tasks
• Separation of concerns: UI, API orchestration, model inference, storage, DB
• Infrastructure as code: Docker + docker-compose for local deploy
• Security: Auth.js (NextAuth), AWS IAM roles/policies for minimal S3 access
d. Key Technologies:
• Frontend: Next.js App Router, TypeScript, Tailwind CSS, Prisma ORM
• Orchestration: Inngest, NextAuth, Docker Compose
• Model APIs: FastAPI, Python 3.10, YAML config files
• Storage: AWS S3 (PutObject, GetObject)
• Database: PostgreSQL
Detailed Capture
• Don’t omit middleware (Auth.js), database calls, or S3 policies
• Docker-compose linking the containers and exposing ports