# peterw/chat-with-github-repo architecture

> Architecture diagram of the GitHub repository peterw/chat-with-github-repo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/peterw/chat-with-github-repo
- Repository: https://github.com/peterw/chat-with-github-repo

## Overview

##### Project Type and Purpose
- This is a Python-based toolkit consisting of:
- A command-line interface (CLI) for “ingesting” GitHub repositories, computing embeddings, and storing them in a Deep Lake dataset.
- A Streamlit web application that serves as the chat UI, querying the dataset and calling OpenAI’s ChatCompletion API to answer user questions about the repo.
- Use case: Quickly spin up a chatbot that “knows” the contents of any GitHub repo by embedding its documents and querying those embeddings at chat time.

##### Main Components
a. CLI Entrypoint (src/main.py)
b. Repository Cloner (inside src/utils/process.py)
c. Text Processor & Embedder (process.py)
– File parsing, filtering by extension
– Embedding generator using OpenAIEmbeddings
d. Deep Lake Dataset (Activeloop)
e. Streamlit Chat App (src/utils/chat.py)
f. Query Handler & Similarity Search (chat.py)
g. OpenAI ChatCompletion API (GPT-3.5-turbo)
h. Configuration (.env for API keys, pyproject.toml, requirements.txt)

##### Relationships and Data Flow
1. CLI “process” command →
– Clones git repository (GitHub) →
– Processes text files →
– Computes embeddings →
– Pushes embeddings into Deep Lake dataset.
2. CLI “chat” command →
– Launches Streamlit server →
– User submits a question →
– Query handler performs similarity search on Deep Lake →
– Retrieves top-k relevant snippets →
– Sends conversation + snippets to OpenAI ChatCompletion →
– Receives response →
– Renders answer in Streamlit UI.

##### Architectural Patterns and Principles
- Modular separation: “process” vs. “chat” logic lives in distinct utilities under src/utils.
- Clear separation of concerns:
- I/O (git cloning)
- Data transformation (text → embeddings)
- Persistence (Deep Lake)
- Presentation & orchestration (Streamlit UI + API calls).
- Stateless UI: Each chat request is independent; state is ephemeral except for conversation history in memory.

##### Key Technologies and External Services
- Python 3.x, pip, pyproject.toml / requirements.txt
- Streamlit (web UI)
- OpenAI Python SDK for embeddings & chat completion
- Activeloop Deep Lake (vector store + dataset hosting)
- Git (repo cloning)
- dotenv for configuration

1. At the top, place “.env Configuration” feeding keys into two lanes: “Process” and “Chat.”
2. On the left, group the “process” flow:
– CLI Entrypoint → Repo Cloner → Text Processor → Embedding Generator → Deep Lake Dataset.
3. On the right, group the “chat” flow:
– CLI Entrypoint → Streamlit App → Query Handler → (Deep Lake Dataset & OpenAI ChatCompletion) → Streamlit App → Browser UI.

## Diagram (Mermaid)

```mermaid
flowchart TD
    title["Chat-with-GitHub-Repo Architecture"]

    subgraph "Configuration & Tools"
        direction TB
        EnvExample["Environment File\n(.env.example)"]:::data
        Pyproj["pyproject.toml"]:::data
        Req["requirements.txt"]:::data
        DevReq["dev-requirements.txt"]:::data
        Lint[".flake8"]:::data
        GitIg[".gitignore"]:::data
    end

    Env[".env Configuration"]:::data

    subgraph "Processing Pipeline"
        direction TB
        CLI["CLI Entrypoint"]:::internal
        Cloner["Repository Cloner"]:::internal
        Processor["Text Processor & Embedder"]:::internal
        DeepLake[(Deep Lake Dataset)]:::data
        Env --> CLI
        CLI -->|"process command"| Cloner
        Cloner -->|"clone repo"| Processor
        Processor -->|"compute embeddings"| DeepLake
    end

    subgraph "Chat Pipeline"
        direction TB
        CLI2["CLI Entrypoint"]:::internal
        StreamLit["Streamlit Chat App"]:::internal
        Query["Query Handler & Similarity Search"]:::internal
        OpenAI["OpenAI ChatCompletion API"]:::external
        Browser["Browser UI"]:::external
        Env --> CLI2
        CLI2 -->|"chat command"| StreamLit
        StreamLit -->|"user question"| Query
        Query -->|"similarity search"| DeepLake
        Query -->|"send snippets+chat"| OpenAI
        OpenAI -->|"response"| StreamLit
        StreamLit --> Browser
    end

    classDef internal fill:#AED6F1,stroke:#1F618D,color:#1B2631
    classDef data fill:#F5B041,stroke:#B9770E,color:#1B2631
    classDef external fill:#ABEBC6,stroke:#239B56,color:#1B2631

    click CLI "https://github.com/peterw/chat-with-github-repo/blob/main/src/main.py"
    click Cloner "https://github.com/peterw/chat-with-github-repo/blob/main/src/utils/process.py"
    click Processor "https://github.com/peterw/chat-with-github-repo/blob/main/src/utils/process.py"
    click DeepLake "https://github.com/peterw/chat-with-github-repo/blob/main/src/utils/process.py"
    click StreamLit "https://github.com/peterw/chat-with-github-repo/blob/main/src/utils/chat.py"
    click Query "https://github.com/peterw/chat-with-github-repo/blob/main/src/utils/chat.py"
    click OpenAI "https://github.com/peterw/chat-with-github-repo/blob/main/src/utils/chat.py"
    click EnvExample "https://github.com/peterw/chat-with-github-repo/blob/main/.env.example"
    click Pyproj "https://github.com/peterw/chat-with-github-repo/blob/main/pyproject.toml"
    click Req "https://github.com/peterw/chat-with-github-repo/blob/main/requirements.txt"
    click DevReq "https://github.com/peterw/chat-with-github-repo/blob/main/dev-requirements.txt"
    click Lint "https://github.com/peterw/chat-with-github-repo/blob/main/.flake8"
    click GitIg "https://github.com/peterw/chat-with-github-repo/blob/main/.gitignore"
```
