How andreaswt/elevenlabs-clone works
Project type and purpose
• This is a full-stack “ElevenLabs clone” for speech synthesis, voice conversion, and audio generation.
• Frontend: Next.js + T3 Stack (React, Tailwind CSS, Auth.js, Prisma).
• Backend: Next.js API routes + Inngest event queue calling Python microservices.
• AI Model services: three Dockerized FastAPI/PyTorch containers (StyleTTS2, Seed-VC, Make-An-Audio).
• Storage: AWS S3 for audio files.
• Database: PostgreSQL/MySQL (via Prisma) for users, credits, and history.
Main components
a. Users (Browser or Mobile App)
b. Next.js Frontend & API Layer
• Routes for UI pages
• NextAuth (Auth.js) for authentication
• Prisma client connecting to Database
• Inngest client triggering generation jobs
c. Inngest Queue
• Event-driven task queue that enforces sequential or rate-limited calls to AI services
d. AI Model Microservices (each in its own Docker container)
• StyleTTS2 API (text → speech)
• Seed-VC API (voice conversion)
• Make-An-Audio API (text → sound effects)
• Each exposes FastAPI endpoints and uploads results to S3
e. AWS S3 Bucket
• Stores generated audio files
• Read/write access granted to model services and frontend via IAM
f. Database (e.g., PostgreSQL)
• Stores user accounts, credit balances, and audio history records
g. AWS Infrastructure (optional)
• EC2 instances or ECS/EKS running Docker containers
• ECR for container registry
h. External/Auth services
• OAuth providers used by NextAuth (e.g., Google, GitHub)
Relationships and data flow
1. User interacts with Next.js UI (login, submit text, upload audio).
2. Next.js API route authenticates via NextAuth, queries/upserts user in Database, checks credits.
3. On generation request, Next.js calls Inngest client to enqueue a job.
4. Inngest worker dequeues job and calls the appropriate AI Model API endpoint.
5. Model API processes input, generates audio via PyTorch, uploads audio file to S3, returns S3 URL.
6. Inngest handler receives the response, updates Database with a new history record and decrements user credits.
7. Next.js frontend polls or listens for job completion, then fetches audio from S3 to play/download.
Architectural patterns and design principles
• Microservices: each AI model runs in an isolated container with its own FastAPI server.
• Event-driven: Inngest queue decouples request submission from heavy model inference.
• Separation of concerns: frontend/UI, API orchestration, AI compute layers, persistent storage.
• Containerization & CI/CD: Dockerfiles.api for each model and docker-compose or AWS ECS/EKS for deployment.
• Infrastructure as code: Dockerfiles, docker-compose.yml, ECR/ECS roles, IAM policies.
Technologies, frameworks, and libraries
• Frontend: Next.js (src/app), React, Tailwind CSS, Prisma ORM, NextAuth.js, Inngest JS.
• Backend orchestration: Next.js API routes, Inngest (Node.js).
• AI Services: Python 3.10, FastAPI, PyTorch models (StyleTTS2, Seed-VC, Make-An-Audio).
• Storage & deployment: AWS S3, AWS ECR/ECS or EC2, Docker, docker-compose, IAM roles.
Final recommendations
• Be very detailed: capture each service, queue, database, and external dependency.