# openbmb/repoagent architecture

> Architecture diagram of the GitHub repository openbmb/repoagent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openbmb/repoagent
- Repository: https://github.com/openbmb/repoagent

## Overview

##### Project type and purpose
- RepoAgent is an open-source, CLI-driven Python framework whose goal is to automatically generate and maintain repository-level code documentation using LLMs (e.g., OpenAI’s GPT).
- It also offers a downstream “Chat With Repo” service (Gradio-based) and a “Display” component that formats generated docs as a GitBook.

##### File structure
- Top-level:
- repo_agent/ — core Python package
- / — tools & templates to render final docs into a book (GitBook)
- markdown_docs/ — placeholder for generated docs and docs about the framework itself
- tests/ — unit tests for core modules
- .github/workflows/release.yml — CI/CD workflow
- pyproject.toml, requirements.txt, pdm.lock — project configuration & dependencies
- Within repo_agent/:
- __main__.py, main.py — CLI entrypoints
- change_detector.py — watches Git diffs
- file_handler.py — orchestrates reading/writing FS
- doc_meta_info.py — maintains JSON metadata of the repo structure
- multi_task_dispatch.py — parallelizes LLM calls
- chat_engine.py, prompt.py — wrap LLM API interactions
- project_manager.py — manages target‐repo configuration & pre-commit hooks
- runner.py — high-level orchestration of the whole pipeline
- settings.py — global configuration defaults
- utils/ — supporting helpers (e.g., gitignore checks, AST parsing)
- chat_with_repo/ — a sub-package exposing:
– gradio_interface.py — REST/WS server
– vector_store_manager.py, rag.py — RAG pipeline for Q&A
– text_analysis_tool.py, json_handler.py, prompt.py — tools for chat
- display/:
- book_tools/ — Python scripts to collect markdown, generate summary, and feed a GitBook template
- book_template/ — GitBook JSON template
- scripts/install_nodejs.sh — prerequisite setup

##### README for additional insights
- Emphasizes two major capabilities:
a. Automated doc generation pipeline (change detection → AST analysis → LLM calls → Markdown output)
b. “Chat With Repo” interactive QA service
- Deployment patterns: CLI (`repoagent run`, `repoagent diff`, `repoagent chat-with-repo`) and Git pre-commit integration

a. Main components
1. CLI Layer
- `repoagent run`, `repoagent diff`, `repoagent chat-with-repo`
2. Core Pipeline
- Change Detector (Git diff analyzer)
- File Handler (FS reader/writer)
- AST Analyzer & Meta Info Manager
- Multi-Task Dispatcher (thread pool)
- LLM Interface (via Chat Engine & Prompt modules)
- Doc Writer (Markdown generator into `markdown_docs/`)
- Metadata Store (.project_doc_record JSON)
3. Chat Service (optional mode)
- Gradio Interface
- RAG Engine (Vector Store + Retriever + LLM Q&A)
4. Component
- Book Tools (aggregate Markdown → GitBook template)
- GitBook Renderer (NodeJS)
5. External Dependencies
- Git CLI
- OpenAI API (or similar LLM endpoint)
- Python runtime & PDM environment
- Pre-commit hook in target repository

b. Relationships and data flow
- The CLI layer triggers the Runner
- Runner calls Change Detector to identify changed files → passes file list to AST Analyzer & File Handler
- For each file/object, Dispatcher spawns threads: each thread invokes Chat Engine (LLM calls) via Prompt templates
- Responses returned as Markdown snippets → Doc Writer assembles them into `markdown_docs/` and updates JSON meta info
- Optionally, `repoagent chat-with-repo` starts Gradio server → loads JSON meta info + vector store → answers queries
- `display/` tools read `markdown_docs/` and JSON meta → inject into GitBook template → produce static site

c. Architectural patterns & design principles
- Modular CLI commands (separate entrypoints for run, diff, chat)
- Pipeline pattern: clear stages (detect → parse → generate → write)
- Producer-consumer model: tasks queued to thread pool for LLM parallelism
- Separation of concerns: core doc generation vs. interactive chat vs. doc display
- Extensibility: prompt templates and config files decoupled from code

d. Relevant technologies & frameworks
- Python 3.x, PDM/`pyproject.toml`, pip
- OpenAI API (GPT‐3.5‐turbo by default)
- Git (change tracking)
- Gradio (chat UI)
- Node.js & GitBook (display)
- Multithreading (Python `concurrent.futures`)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Layer
    subgraph "CLI Layer"
        direction TB
        CLI_Main["__main__.py"]:::cli
        CLI_Main -->|triggers| Runner
        CLI["main.py"]:::cli
        CLI_Main --> CLI
        CLI_Chat["chat_with_repo/__main__.py"]:::cli
        CLI_Chat -->|starts Chat Service| Gradio
    end

    %% Core Pipeline
    subgraph "Core Pipeline"
        direction TB
        Runner["runner.py"]:::core
        ChangeDetector["change_detector.py"]:::core
        FileHandler["file_handler.py"]:::core
        ASTManager["doc_meta_info.py"]:::core
        Dispatcher["multi_task_dispatch.py"]:::core
        ChatEngine["chat_engine.py"]:::core
        PromptCore["prompt.py"]:::core
        Settings["settings.py"]:::core
        ProjectManager["project_manager.py"]:::core
        MetaUtils["utils/meta_info_utils.py"]:::core
        Gitignore["utils/gitignore_checker.py"]:::core
        Logger["log.py"]:::core

        Runner -->|executes| ChangeDetector
        ChangeDetector -->|outputs changed_files_list.json| FileHandler
        FileHandler --> ASTManager
        ASTManager -->|updates .project_doc_record| Runner
        Runner --> Dispatcher
        Dispatcher -->|tasks| ChatEngine
        ChatEngine --> PromptCore
        PromptCore -->|calls| OpenAI
        ChatEngine -->|LLM calls| OpenAI
        ChatEngine -->|returns Markdown fragments| Runner
        Runner -->|writes to| MarkdownOutput
    end

    %% Documentation Output
    MarkdownOutput["markdown_docs/"]:::core

    %% Chat Service
    subgraph "Chat Service" 
        direction TB
        Gradio["gradio_interface.py"]:::chat
        VectorStore["vector_store_manager.py"]:::chat
        RAG["rag.py"]:::chat
        TextTool["text_analysis_tool.py"]:::chat
        JSONHandler["json_handler.py"]:::chat
        PromptChat["prompt.py"]:::chat

        Gradio -->|loads JSON meta| JSONHandler
        Gradio -->|queries| RAG
        RAG --> VectorStore
        RAG -->|calls| ChatEngine
        Gradio -->|uses| TextTool
        Gradio -->|uses| PromptChat
    end

    %% Display Component
    subgraph "Display Component"
        direction TB
        BookGen["generate_repoagent_books.py"]:::display
        SummaryGen["generate_summary_from_book.py"]:::display
        Template["book_template/book.json"]:::display
        Installer["install_nodejs.sh"]:::display
        Renderer["GitBook Renderer"]:::ext

        BookGen -->|reads| MarkdownOutput
        BookGen -->|reads| ASTManager
        SummaryGen -->|reads| MarkdownOutput
        BookGen --> Template
        Template -->|fed into| Renderer
        Installer --> Renderer
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        GitCLI["Git CLI"]:::ext
        OpenAI["OpenAI API"]:::ext
        PythonEnv["Python Runtime & PDM"]:::ext
        NodeJS["Node.js"]:::ext
        PreCommit["Pre-commit Hook"]:::ext

        ChangeDetector -->|uses| GitCLI
        ChatEngine -->|uses| OpenAI
        Runner -->|runs in| PythonEnv
        Renderer -->|runs on| NodeJS
        ProjectManager -->|installs hook| PreCommit
    end

    %% Click Events
    click CLI_Main "https://github.com/openbmb/repoagent/blob/main/repo_agent/__main__.py"
    click CLI "https://github.com/openbmb/repoagent/blob/main/repo_agent/main.py"
    click CLI_Chat "https://github.com/openbmb/repoagent/blob/main/repo_agent/chat_with_repo/__main__.py"
    click ChangeDetector "https://github.com/openbmb/repoagent/blob/main/repo_agent/change_detector.py"
    click FileHandler "https://github.com/openbmb/repoagent/blob/main/repo_agent/file_handler.py"
    click ASTManager "https://github.com/openbmb/repoagent/blob/main/repo_agent/doc_meta_info.py"
    click Dispatcher "https://github.com/openbmb/repoagent/blob/main/repo_agent/multi_task_dispatch.py"
    click ChatEngine "https://github.com/openbmb/repoagent/blob/main/repo_agent/chat_engine.py"
    click PromptCore "https://github.com/openbmb/repoagent/blob/main/repo_agent/prompt.py"
    click Runner "https://github.com/openbmb/repoagent/blob/main/repo_agent/runner.py"
    click Settings "https://github.com/openbmb/repoagent/blob/main/repo_agent/settings.py"
    click ProjectManager "https://github.com/openbmb/repoagent/blob/main/repo_agent/project_manager.py"
    click MetaUtils "https://github.com/openbmb/repoagent/blob/main/repo_agent/utils/meta_info_utils.py"
    click Gitignore "https://github.com/openbmb/repoagent/blob/main/repo_agent/utils/gitignore_checker.py"
    click Logger "https://github.com/openbmb/repoagent/blob/main/repo_agent/log.py"
    click Gradio "https://github.com/openbmb/repoagent/blob/main/repo_agent/chat_with_repo/gradio_interface.py"
    click VectorStore "https://github.com/openbmb/repoagent/blob/main/repo_agent/chat_with_repo/vector_store_manager.py"
    click RAG "https://github.com/openbmb/repoagent/blob/main/repo_agent/chat_with_repo/rag.py"
    click TextTool "https://github.com/openbmb/repoagent/blob/main/repo_agent/chat_with_repo/text_analysis_tool.py"
    click JSONHandler "https://github.com/openbmb/repoagent/blob/main/repo_agent/chat_with_repo/json_handler.py"
    click PromptChat "https://github.com/openbmb/repoagent/blob/main/repo_agent/chat_with_repo/prompt.py"
    click BookGen "https://github.com/openbmb/repoagent/blob/main/display/book_tools/generate_repoagent_books.py"
    click SummaryGen "https://github.com/openbmb/repoagent/blob/main/display/book_tools/generate_summary_from_book.py"
    click Template "https://github.com/openbmb/repoagent/blob/main/display/book_template/book.json"
    click Installer "https://github.com/openbmb/repoagent/blob/main/display/scripts/install_nodejs.sh"

    %% Styles
    classDef core fill:#D0E8FF,stroke:#0366d6,color:#0366d6
    classDef chat fill:#DFFFE0,stroke:#28A745,color:#28A745
    classDef display fill:#FFF0D0,stroke:#FF8C00,color:#FF8C00
    classDef ext fill:#EEEEEE,stroke:#666666,color:#666666
    classDef cli fill:#E8D0FF,stroke:#6F42C1,color:#6F42C1
```
