How mckaywrigley/naval-gpt works
Project type and purpose
• This is a full-stack Next.js application (“Naval GPT”) that provides AI-powered search, chat, and audio playback around Naval Ravikant’s “How To Get Rich” content.
• It includes both a React front-end (pages, components, styles) and a Node.js/TypeScript back-end via Next.js API routes. It also bundles an ETL pipeline (scraping, embedding text/audio) implemented in Node.js and Python.
File structure
• Top-level directories:
• components (UI widgets)
• pages (Next.js front-end + API routes under pages/api)
• public (static assets, including podcast.mp3)
• scripts (data-processing: scraping, embedding, Python Whisper pipeline)
• utils, types (shared helper functions & type definitions)
• Key configuration: next.config.js, tsconfig.json, tailwind.config.js, schema.sql (DB schema).
• Deployment hints: .env.local for OpenAI & Supabase, Supabase hosting, SQL editor for schema.
README for additional insights
• Architecture: ETL → vector DB (Postgres + pgvector on Supabase) → Next.js API → React UI.
• External dependencies: Naval’s blog HTML, podcast MP3, OpenAI Embeddings & ChatCompletion APIs, OpenAI Whisper for audio.
a. Main components
• Data Ingestion & ETL pipeline
– Scrape script (scripts/scrape.ts → JSON)
– Text embed script (scripts/embed-text.ts → OpenAI embeddings → Supabase)
– Audio processing pipeline (scripts/main.py using Whisper → JSON)
– Audio embed script (scripts/embed-audio.ts → OpenAI embeddings → Supabase)
• Database & Storage
– Supabase Postgres with pgvector extension (stores “passages” and “clips” tables)
– schema.sql defines tables, vectors, RLS policies
– public/static storage for podcast.mp3
• Next.js Application
– Front-end React (pages/index.tsx, components/*)
– API routes (pages/api/search.ts, pages/api/answer.ts)
– Utility modules (utils/index.ts, types/index.ts)
• External Services
– OpenAI API (text-embedding-ada-002, GPT-3.5-turbo, Whisper)
– Naval’s website & podcast host
b. Relationships & data flow
1. ETL: scrape → embed → save to DB
2. User interacts with React UI → calls /api/search or /api/answer
3. API route invokes:
• Generate OpenAI embedding for query → compare cosine similarity in pgvector → return top passages/clips
• For chat: compile prompt with passages → call GPT-3.5-turbo → return response
4. React UI displays search results, chat thread, and controls audio player (seeks to clip timestamp).
c. Architectural patterns & principles
• Clear separation of concerns: ETL pipeline vs. runtime API vs. UI
• Serverless micro-services (Next.js API routes) for search & chat endpoints
• Vector search pattern using pgvector and cosine similarity
• Modular UI components (Answer, Navbar, Player)
d. Key technologies
• Next.js + React + TypeScript + Tailwind CSS
• Supabase (Postgres + pgvector)
• OpenAI Embeddings & ChatCompletion API
• Python Whisper for audio transcription & embedding
• Node.js scripts for scraping & embedding
Elements
• Labels: “Scraper”, “Text Embedder”, “Audio Processor”, “Supabase DB”, “Next.js API”, “React UI”, “OpenAI API”, “Podcast Source”
• ETL → DB
• UI → API → DB & OpenAI → API → UI
• Audio pipeline → JSON → DB
• API services (blue rectangles)
• UI components (orange rectangles)