How codehash001/documatey works
Project Type and Purpose
• This is a full-stack Next.js (React + TypeScript) application providing a vector-search assistant: it ingests documentation (URL or text), chunks it, computes embeddings via Google Generative AI, stores/retrieves vectors in TiDB Serverless, and offers a chat-style, step-wise RAG (retrieval-augmented generation) wizard.
File Structure Analysis
• app/
• api/chat, clarify, plan, search, index, status – Next.js API routes (serverless functions).
• layout.tsx, page.tsx, globals.css – root layout, home page UI.
• components/ui/… – reusable UI building blocks (buttons, cards, inputs, theme toggle).
• components/wizard.tsx – multi-step RAG UI controlling indexing, clarifications, plannning, chat.
• lib/… – core business logic modules:
• chunk.ts – text splitting.
• embed.ts – calling Google embeddings.
• search.ts – vector query logic.
• llm.ts – calling Google Gemini LLM.
• tidb.ts – TiDB database connection and upsert/index management.
• utils.ts – shared helpers.
• next.config.ts, tsconfig.json, ESLint/PostCSS configs.
README Insights
• Env vars configure TiDB connection & Google API keys.
• Describes the three-step user flow: index docs, clarify goal, generate plan with citations; context-aware chat via step IDs.
• Highlights TiDB HNSW vector index creation on embedded_documents table.
a. Main Components
• Frontend (Next.js app directory + components).
• API Layer (Next.js serverless functions under app/api/*).
• Business Logic Modules (lib/*.ts).
• Vector Store (TiDB cloud cluster).
• Embedding & LLM Service (Google Generative AI endpoints).
b. Relationships & Data Flows
1. User interacts with the Wizard UI → triggers REST calls to API routes.
2. API route “/api/index” → uses lib/chunk → lib/embed → lib/tidb to upsert vectors.
3. API route “/api/search” or “/api/chat” → uses lib/search → optionally lib/llm for RAG responses.
4. TiDB returns nearest-neighbor vectors → API returns JSON/Markdown to front end.
5. Frontend renders responses and maintains step state.
c. Architectural Patterns
• Serverless functions (One route per business use-case).
• Layered separation: UI → API → Service modules → Data store.
• Shared library modules in lib/*.
• RAG pipeline: chunk → embed → store → retrieve → generate.
d. Key Technologies
• Next.js App Router (app/ directory)
• React + TypeScript + CSS (components/ui)
• TiDB Serverless (MySQL-compatible, HNSW index)
• Google Generative AI (text-embedding-004, Gemini LLM)
• Node 20.x, npm scripts (dev, build, start)