he-yufeng/repowiki

1. Repo shape: a Python backend plus a separate browser frontend.

The core product lives under src/repowiki/ and can run as a CLI, a FastAPI server, or a terminal chat tool. The frontend/ directory is a React + Vite TypeScript UI that talks to the backend API. This is not a single monolith; it is a pipeline-oriented backend with a thin web presentation layer.

2. Primary entry points.

repowiki/__main__.py and cli.py provide command-line commands such as scan, serve, chat, and config. The same backend capabilities are exposed through server/app.py and routers under server/routers/. The web app is a consumer of those backend endpoints rather than an independent analysis engine.

3. Repository ingestion boundary.

Input comes from either a local filesystem path (ingest/local.py) or a GitHub URL (ingest/github.py). That boundary matters because everything downstream operates on a normalized project snapshot, regardless of source. The README indicates scans respect .gitignore, .repowikiignore, and common secret-file exclusions.

4. Scanning and file selection.

core/scanner.py is the first major processing stage. It walks the tree, detects languages, identifies candidate files, and filters out binaries, generated bundles, and oversized or irrelevant content. This stage is the main “context control” gate before any graph or LLM work happens.

5. Dependency graph and file ranking.

core/graph.py builds a cross-file dependency graph and applies PageRank-style ranking to identify important files and a guided reading path. The README explicitly notes import-aware resolution for Python package-relative imports and JavaScript/TypeScript relative modules, so the graph is a core architectural subsystem rather than a cosmetic feature.

6. LLM analysis pipeline.

core/analyzer.py orchestrates multi-pass structured analysis of the scanned repo. It feeds selected file content and tree context into the LLM layer, likely producing project overview, module summaries, architecture notes, and reading guidance. Prompt templates live in llm/prompts.py, and the actual model access is abstracted by llm/client.py around litellm.

7. Wiki assembly and document model.

core/wiki_builder.py turns analysis artifacts into a coherent wiki structure. Supporting types in core/models.py define the data shapes passed across scan, graph, analysis, and export layers. This is the internal “content product” boundary: raw repo data becomes structured documentation objects.

8. Caching and repeat scans.

core/cache.py provides SQLite-backed caching keyed by content hash, allowing re-scans to skip unchanged files. The README positions SQLite as the only persistence dependency, which keeps the architecture local-first and avoids external databases.

9. Retrieval and Q&A path.

core/rag.py implements terminal Q&A retrieval using TF-IDF rather than embeddings. That means the chat path is separate from the documentation-generation path: it indexes repo text, retrieves relevant snippets, and then asks the LLM to answer grounded in those snippets. This is the architecture behind repowiki chat.

10. Export layer with three output modes.

export/markdown.py, export/json_export.py, and export/html.py render the wiki into different delivery formats. Markdown is a directory of files, JSON is structured machine-readable output, and HTML is a self-contained artifact with Mermaid diagrams included. Export is a distinct boundary after analysis, not part of scanning.

11. Web backend and frontend responsibilities.

server/app.py and routers scan.py, wiki.py, and chat.py expose the analysis pipeline over HTTP. The frontend pages in frontend/src/pages/ (Home, WikiView, ChatView) and components like WikiSidebar, WikiContent, and MermaidDiagram provide the browsing experience. The frontend is therefore a presentation and navigation layer over wiki data, not the source of truth.

12. Configuration and external dependencies.

config.py centralizes runtime settings, including model selection and API keys. The README shows config precedence from CLI flags to environment variables to a user config file, with provider-specific API keys for LLM access. The key external integration is the LLM provider layer via litellm; everything else is local filesystem, GitHub fetch/clone, SQLite, and browser UI.

13. Quality gates and delivery pipeline.

Tests cover scanner, graph, RAG, and export behavior (tests/test_scanner.py, test_graph.py, test_rag.py, test_markdown_export.py, test_html_export.py). GitHub Actions workflows in .github/workflows/ci.yml and publish.yml indicate the project is packaged and released as a distributable Python tool with CI validation.

14. Useful diagram boundaries to draw.

A good architecture diagram would show: input sources → ingestion → scanning/filtering → dependency graph + ranking → LLM analysis → wiki assembly → export/API responses; with a side path for retrieval/chat and a separate presentation boundary for the React frontend. The biggest internal seams are ingestion vs analysis, analysis vs export, and offline CLI processing vs online server/UI consumption.