# modelearth/codechat architecture

> Architecture diagram of the GitHub repository modelearth/codechat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/modelearth/codechat
- Repository: https://github.com/modelearth/codechat

## Overview

##### Project type and purpose
- This is a Retrieval-Augmented Generation (RAG) prototype, combining a static front-end (index.html) with a Python-based ingestion pipeline and a Python “query test” script (rag_query_test.py).
- It’s designed to:
- Ingest code/doc repos via Tree-sitter chunking → embed with OpenAI’s `text-embedding-3-small` → store in Pinecone VectorDB.
- Serve a minimal front-end that submits user queries.
- Use an AWS Lambda-style backend to embed queries, retrieve top chunks from Pinecone, consult Gemini, and return answers.
- Keep VectorDB in sync via GitHub Actions on PR merges.

##### File structure
Top-level files:
- .env.example / requirements.txt / setup_instructions.sh – environment and install.
- index.html – standalone front-end entrypoint.
- rag_ingestion_pipeline.ipynb – Jupyter notebook illustrating chunk, embed, store.
- rag_query_test.py – Python script to simulate queries (future Lambda logic).
- README.md – detailed project description, architecture, and next steps.
No subdirectories: this is a single‐service/tool repository.

##### Key insights from README
- Ingestion pipeline stages (Tree-sitter → embeddings → Pinecone).
- Query pipeline stages (embed query → Pinecone lookup → Gemini call → frontend).
- GitHub Actions PR-merge sync.
- Future: AWS Lambda backend, environment variables for OpenAI/Pinecone/GEMINI keys.

##### Main components
A. Source Repositories (e.g., modelearth submodules, local code repos)
B. Ingestion Pipeline
- Tree-sitter chunking (Notebook/script)
- OpenAI embedding service
- Pinecone VectorDB (repo-chunks index)
- GitHub Actions (PR merge → re-chunk & update Pinecone)
C. Front-End
- index.html (static UI)
- JavaScript client performing REST calls
D. Backend API / AWS Lambda
- Query embed with OpenAI
- Pinecone top-k vector similarity search
- Gemini (`gemini-1.5-flash`) for answer generation
E. External Services
- OpenAI Embedding API
- Pinecone VectorDB
- Google Generative AI (Gemini)
- GitHub (Actions, PR events)

##### Relationships & data flows
1. Ingestion Flow:
Source Repos → Tree-sitter chunker (Notebook/CI) → OpenAI Embedding → Pinecone (VectorDB)
GitHub Actions watches PR merges → triggers chunk/embed/update in Pinecone
2. Query Flow:
User → Front-end (index.html) → backend Lambda API →
a. embed question (OpenAI) →
b. query Pinecone for top-5 chunks →
c. call Gemini with context & question →
results → backend → front-end UI.

##### Architectural patterns & design principles
- Pipeline separation: Ingestion vs. Query.
- Stateless backend Lambda: isolate each query in a function.
- Event-driven sync: GitHub Actions triggers vector DB updates.
- Polyglot components: Python notebook & scripts, static HTML/JS, cloud functions.
- Externalize secrets via .env / environment variables.

##### Technologies & frameworks
- Python (Tree-sitter, notebook, rag_query_test.py)
- JavaScript / static HTML (front-end)
- AWS Lambda or equivalent for query handler
- OpenAI Embedding API (`text-embedding-3-small`)
- Pinecone VectorDB
- Google Generative AI (Gemini)
- GitHub Actions for CI

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Source Repositories
    SrcRepos["Source Repos<br/>(modelearth, local code)"]:::external

    %% Ingestion Pipeline
    subgraph "Ingestion Pipeline" 
        direction TB
        IngestionNotebook["Tree-sitter Chunker & Embedding<br/>(rag_ingestion_pipeline.ipynb)"]:::ingestion
        OpenAIEmbedService["OpenAI Embedding API<br/>(text-embedding-3-small)"]:::external
        PineconeDB["Pinecone VectorDB<br/>(repo-chunks index)"]:::db
        GitHubActions["GitHub Actions<br/>(PR merge sync)"]:::external
    end

    %% Query Pipeline
    subgraph "Query Pipeline"
        direction TB
        User["User"]:::external
        Frontend["Front-End UI & JS Client<br/>(index.html)"]:::query
        LambdaHandler["AWS Lambda Query Handler<br/>(rag_query_test.py)"]:::query
        OpenAIEmbedQuery["OpenAI Embedding API"]:::external
        PineconeQuery["Pinecone VectorDB"]:::db
        GeminiService["Google Gemini AI<br/>(gemini-1.5-flash)"]:::external
    end

    %% Data Flows Ingestion
    SrcRepos -->|"code/docs"| IngestionNotebook
    IngestionNotebook -->|"embeddings"| OpenAIEmbedService
    OpenAIEmbedService -->|"vectors"| PineconeDB
    GitHubActions -->|"trigger chunk/embed"| IngestionNotebook
    GitHubActions -->|"update vectors"| PineconeDB

    %% Data Flows Query
    User -->|"query"| Frontend
    Frontend -->|"REST call"| LambdaHandler
    LambdaHandler -->|"embed question"| OpenAIEmbedQuery
    OpenAIEmbedQuery -->|"query vector"| PineconeQuery
    PineconeQuery -->|"top-5 chunks"| LambdaHandler
    LambdaHandler -->|"context + question"| GeminiService
    GeminiService -->|"answer"| LambdaHandler
    LambdaHandler -->|"response"| Frontend

    %% VectorDB shared
    PineconeDB --- PineconeQuery

    %% Environment & Config Files
    Env[".env.example<br/>(env & secrets)"]:::external
    Req["requirements.txt"]:::external
    Setup["setup_instructions.sh"]:::external
    README["README.md"]:::external

    %% Documentation links
    subgraph "Project Files"
        direction TB
        Env
        Req
        Setup
        README
    end

    %% Click Events
    click IngestionNotebook "https://github.com/modelearth/codechat/blob/main/rag_ingestion_pipeline.ipynb"
    click LambdaHandler "https://github.com/modelearth/codechat/blob/main/rag_query_test.py"
    click Frontend "https://github.com/modelearth/codechat/blob/main/index.html"
    click Env "https://github.com/modelearth/codechat/blob/main/.env.example"
    click Req "https://github.com/modelearth/codechat/blob/main/requirements.txt"
    click Setup "https://github.com/modelearth/codechat/blob/main/setup_instructions.sh"
    click README "https://github.com/modelearth/codechat/blob/main/README.md"

    %% Styles
    classDef ingestion fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef query fill:#d4edda,stroke:#333,stroke-width:1px
    classDef external fill:#fff3cd,stroke:#333,stroke-width:1px
    classDef db fill:#e2e3e5,stroke:#333,stroke-width:1px
```
