How modelearth/codechat works
Project type and purpose
• This is a Retrieval-Augmented Generation (RAG) prototype, combining a static front-end (index.html) with a Python-based ingestion pipeline and a Python “query test” script (rag_query_test.py).
• It’s designed to:
• Ingest code/doc repos via Tree-sitter chunking → embed with OpenAI’s text-embedding-3-small → store in Pinecone VectorDB.
• Serve a minimal front-end that submits user queries.
• Use an AWS Lambda-style backend to embed queries, retrieve top chunks from Pinecone, consult Gemini, and return answers.
• Keep VectorDB in sync via GitHub Actions on PR merges.
File structure
Top-level files:
• .env.example / requirements.txt / setup_instructions.sh – environment and install.
• index.html – standalone front-end entrypoint.
• rag_ingestion_pipeline.ipynb – Jupyter notebook illustrating chunk, embed, store.
• rag_query_test.py – Python script to simulate queries (future Lambda logic).
• README.md – detailed project description, architecture, and next steps.
No subdirectories: this is a single‐service/tool repository.
Key insights from README
• Ingestion pipeline stages (Tree-sitter → embeddings → Pinecone).
• Query pipeline stages (embed query → Pinecone lookup → Gemini call → frontend).
• GitHub Actions PR-merge sync.
• Future: AWS Lambda backend, environment variables for OpenAI/Pinecone/GEMINI keys.
Main components
A. Source Repositories (e.g., modelearth submodules, local code repos)
B. Ingestion Pipeline
• Tree-sitter chunking (Notebook/script)
• OpenAI embedding service
• Pinecone VectorDB (repo-chunks index)
• GitHub Actions (PR merge → re-chunk & update Pinecone)
C. Front-End
• index.html (static UI)
• JavaScript client performing REST calls
D. Backend API / AWS Lambda
• Query embed with OpenAI
• Pinecone top-k vector similarity search
• Gemini (gemini-1.5-flash) for answer generation
E. External Services
• OpenAI Embedding API
• Pinecone VectorDB
• Google Generative AI (Gemini)
• GitHub (Actions, PR events)
Relationships & data flows
1. Ingestion Flow:
Source Repos → Tree-sitter chunker (Notebook/CI) → OpenAI Embedding → Pinecone (VectorDB)
GitHub Actions watches PR merges → triggers chunk/embed/update in Pinecone
2. Query Flow:
User → Front-end (index.html) → backend Lambda API →
a. embed question (OpenAI) →
b. query Pinecone for top-5 chunks →
c. call Gemini with context & question →
results → backend → front-end UI.
Architectural patterns & design principles
• Pipeline separation: Ingestion vs. Query.
• Stateless backend Lambda: isolate each query in a function.
• Event-driven sync: GitHub Actions triggers vector DB updates.
• Polyglot components: Python notebook & scripts, static HTML/JS, cloud functions.
• Externalize secrets via .env / environment variables.
Technologies & frameworks
• Python (Tree-sitter, notebook, rag_query_test.py)
• JavaScript / static HTML (front-end)
• AWS Lambda or equivalent for query handler
• OpenAI Embedding API (text-embedding-3-small)
• Pinecone VectorDB
• Google Generative AI (Gemini)
• GitHub Actions for CI