How openbmb/repoagent works
Project type and purpose
• RepoAgent is an open-source, CLI-driven Python framework whose goal is to automatically generate and maintain repository-level code documentation using LLMs (e.g., OpenAI’s GPT).
• It also offers a downstream “Chat With Repo” service (Gradio-based) and a “Display” component that formats generated docs as a GitBook.
File structure
• Top-level:
• repo_agent/ — core Python package
• / — tools & templates to render final docs into a book (GitBook)
• markdown_docs/ — placeholder for generated docs and docs about the framework itself
• tests/ — unit tests for core modules
• .github/workflows/release.yml — CI/CD workflow
• pyproject.toml, requirements.txt, pdm.lock — project configuration & dependencies
• Within repo_agent/:
• __main__.py, main.py — CLI entrypoints
• change_detector.py — watches Git diffs
• file_handler.py — orchestrates reading/writing FS
• doc_meta_info.py — maintains JSON metadata of the repo structure
• multi_task_dispatch.py — parallelizes LLM calls
• chat_engine.py, prompt.py — wrap LLM API interactions
• project_manager.py — manages target‐repo configuration & pre-commit hooks
• runner.py — high-level orchestration of the whole pipeline
• settings.py — global configuration defaults
• utils/ — supporting helpers (e.g., gitignore checks, AST parsing)
• chat_with_repo/ — a sub-package exposing:
– gradio_interface.py — REST/WS server
– vector_store_manager.py, rag.py — RAG pipeline for Q&A
– text_analysis_tool.py, json_handler.py, prompt.py — tools for chat
• display/:
• book_tools/ — Python scripts to collect markdown, generate summary, and feed a GitBook template
• book_template/ — GitBook JSON template
• scripts/install_nodejs.sh — prerequisite setup
README for additional insights
• Emphasizes two major capabilities:
a. Automated doc generation pipeline (change detection → AST analysis → LLM calls → Markdown output)
b. “Chat With Repo” interactive QA service
• Deployment patterns: CLI (repoagent run, repoagent diff, repoagent chat-with-repo) and Git pre-commit integration
a. Main components
1. CLI Layer
• repoagent run, repoagent diff, repoagent chat-with-repo
2. Core Pipeline
• Change Detector (Git diff analyzer)
• File Handler (FS reader/writer)
• AST Analyzer & Meta Info Manager
• Multi-Task Dispatcher (thread pool)
• LLM Interface (via Chat Engine & Prompt modules)
• Doc Writer (Markdown generator into markdown_docs/)
• Metadata Store (.project_doc_record JSON)
3. Chat Service (optional mode)
• Gradio Interface
• RAG Engine (Vector Store + Retriever + LLM Q&A)
4. Component
• Book Tools (aggregate Markdown → GitBook template)
• GitBook Renderer (NodeJS)
5. External Dependencies
• Git CLI
• OpenAI API (or similar LLM endpoint)
• Python runtime & PDM environment
• Pre-commit hook in target repository
b. Relationships and data flow
• The CLI layer triggers the Runner
• Runner calls Change Detector to identify changed files → passes file list to AST Analyzer & File Handler
• For each file/object, Dispatcher spawns threads: each thread invokes Chat Engine (LLM calls) via Prompt templates
• Responses returned as Markdown snippets → Doc Writer assembles them into markdown_docs/ and updates JSON meta info
• Optionally, repoagent chat-with-repo starts Gradio server → loads JSON meta info + vector store → answers queries
• display/ tools read markdown_docs/ and JSON meta → inject into GitBook template → produce static site
c. Architectural patterns & design principles
• Modular CLI commands (separate entrypoints for run, diff, chat)
• Pipeline pattern: clear stages (detect → parse → generate → write)
• Producer-consumer model: tasks queued to thread pool for LLM parallelism
• Separation of concerns: core doc generation vs. interactive chat vs. doc display
• Extensibility: prompt templates and config files decoupled from code
d. Relevant technologies & frameworks
• Python 3.x, PDM/pyproject.toml, pip
• OpenAI API (GPT‐3.5‐turbo by default)
• Git (change tracking)
• Gradio (chat UI)
• Node.js & GitBook (display)
• Multithreading (Python concurrent.futures)