How peterw/chat-with-github-repo works
Project Type and Purpose
• This is a Python-based toolkit consisting of:
• A command-line interface (CLI) for “ingesting” GitHub repositories, computing embeddings, and storing them in a Deep Lake dataset.
• A Streamlit web application that serves as the chat UI, querying the dataset and calling OpenAI’s ChatCompletion API to answer user questions about the repo.
• Use case: Quickly spin up a chatbot that “knows” the contents of any GitHub repo by embedding its documents and querying those embeddings at chat time.
Main Components
a. CLI Entrypoint (src/main.py)
b. Repository Cloner (inside src/utils/process.py)
c. Text Processor & Embedder (process.py)
– File parsing, filtering by extension
– Embedding generator using OpenAIEmbeddings
d. Deep Lake Dataset (Activeloop)
e. Streamlit Chat App (src/utils/chat.py)
f. Query Handler & Similarity Search (chat.py)
g. OpenAI ChatCompletion API (GPT-3.5-turbo)
h. Configuration (.env for API keys, pyproject.toml, requirements.txt)
Relationships and Data Flow
1. CLI “process” command →
– Clones git repository (GitHub) →
– Processes text files →
– Computes embeddings →
– Pushes embeddings into Deep Lake dataset.
2. CLI “chat” command →
– Launches Streamlit server →
– User submits a question →
– Query handler performs similarity search on Deep Lake →
– Retrieves top-k relevant snippets →
– Sends conversation + snippets to OpenAI ChatCompletion →
– Receives response →
– Renders answer in Streamlit UI.
Architectural Patterns and Principles
• Modular separation: “process” vs. “chat” logic lives in distinct utilities under src/utils.
• Clear separation of concerns:
• I/O (git cloning)
• Data transformation (text → embeddings)
• Persistence (Deep Lake)
• Presentation & orchestration (Streamlit UI + API calls).
• Stateless UI: Each chat request is independent; state is ephemeral except for conversation history in memory.
Key Technologies and External Services
• Python 3.x, pip, pyproject.toml / requirements.txt
• Streamlit (web UI)
• OpenAI Python SDK for embeddings & chat completion
• Activeloop Deep Lake (vector store + dataset hosting)
• Git (repo cloning)
• dotenv for configuration
1. At the top, place “.env Configuration” feeding keys into two lanes: “Process” and “Chat.”
2. On the left, group the “process” flow:
– CLI Entrypoint → Repo Cloner → Text Processor → Embedding Generator → Deep Lake Dataset.
3. On the right, group the “chat” flow:
– CLI Entrypoint → Streamlit App → Query Handler → (Deep Lake Dataset & OpenAI ChatCompletion) → Streamlit App → Browser UI.