How pulkit-999/codebase-analyzer works
1. What this repository is
- This is a local, Python-based codebase analysis tool focused on RAG over source code, not a general web app.
- Its purpose is to ingest a repository, structure it with AST awareness, embed chunks locally, and let an agent answer architecture or dependency questions by traversing the code graph.
2. Primary architectural split
- The repo is organized around two main flows:
- Ingestion/indexing: parse source files, chunk them structurally, generate embeddings, and store them in a vector index.
- Interactive retrieval/agentic reasoning: accept a question, search the indexed code, and recursively follow cross-file references until enough context is gathered.
3. Ingestion subsystem
- ingestion.py is the most likely home for the pipeline that scans files, parses them with Tree-sitter, and converts code into AST-aligned chunks such as functions, methods, declarations, or structs.
- This is an important architectural boundary because chunking is structural rather than naive text splitting, which preserves semantic relationships in code.
4. Vectorization and storage
- The README indicates embeddings are generated locally using all-MiniLM-L6-v2 and stored in ChromaDB.
- This means the source code never needs to leave the machine during indexing or retrieval, and the persistent vector store is the main data layer for search.
5. Agentic retrieval subsystem
- The runtime retrieval loop uses LangChain and a multi-hop tool-calling agent.
- Instead of a single search pass, the agent can invoke a code-search tool repeatedly to resolve references across files, which is especially important for understanding call chains and dependency graphs.
6. Reasoning and synthesis boundary
- The system likely separates “finding relevant code” from “explaining the architecture.”
- Retrieval produces context fragments from the vector store, and the agent’s LLM layer synthesizes them into a final explanation or answer.
- The README names Gemini 2.5 Flash as the inference engine for this reasoning step.
7. Likely entrypoints and orchestration
- main.py and app.py appear to be the orchestration/entry modules.
- Given the project shape, one file likely handles startup/CLI/session flow while the other wraps the agent or application interface.
- If you are drawing a diagram, treat these as the control layer above ingestion and retrieval rather than as data-processing components themselves.
8. Data flow end to end
- Source repository files enter the ingestion pipeline.
- Tree-sitter parses them into syntax-aware units.
- Embeddings are computed locally and written into ChromaDB.
- At query time, the agent retrieves relevant chunks, may recurse into additional searches, and then composes the answer.
9. Key architectural advantage
- The repo emphasizes preserving code structure and following dependencies transitively.
- That makes it suitable for questions like “where is this function used?” or “how does this module connect to the rest of the system?” rather than only keyword search.
10. Operational characteristics
- The project is designed for local/offline-first operation at the indexing layer.
- uv is used for Python dependency and environment management, which affects setup but not runtime architecture.
- The .python-version, pyproject.toml, and uv.lock files indicate a managed, reproducible Python environment.
11. What to show in an architecture diagram
- A good diagram should include:
- Source code repository input
- AST parser / structural chunker
- Embedding model
- ChromaDB vector store
- Agent orchestration layer
- LLM/tool-calling loop
- Final answer output
- The main boundary to emphasize is between indexing/storage and agentic retrieval/reasoning.
12. Repository-level summary
- In short: this repo is a local code intelligence pipeline built around AST-aware ingestion and a recursive retrieval agent.
- Its architecture is driven by the need to preserve source structure and resolve cross-file dependencies, not by serving a traditional API or UI.
Main entry points: ingestion.py Ingestion · app.py App · README.md README · main.py Main
Components 14
Ingestion / Indexing
Parses source code, chunks it structurally, embeds it, and writes vectors to storage.
- Source Repo · The target repository being analyzed.
- Ingestion · Main ingestion pipeline for scanning and preparing code artifacts.
ingestion.py - Tree-sitter · AST-aware parsing used to understand code structure.
- Code Chunks · Creates function-, method-, and declaration-level chunks aligned to syntax.
- MiniLM · Local embedding model used to vectorize code chunks.
- ChromaDB · Persistent local vector index for embedded code chunks.
Retrieval / Reasoning
Handles questions, searches the index, follows references, and synthesizes answers.
- Main · Primary launch path for running the pipeline or app flow.
main.py - App · Application/session orchestration around query handling and agent flow.
app.py - Agent · Tool-calling retrieval agent that iteratively investigates the codebase.
- Code Search · Search tool that fetches relevant indexed chunks from the vector store.
- Follow Refs · Repeatedly expands from one file or symbol to related cross-file context.
- Gemini · Synthesizes retrieved context into an architectural or dependency answer.
- Answer · Final natural-language explanation returned to the user.
- README · Explains the repository purpose, setup, and workflow.
README.md
Connections 15
- Source Repo → Ingestion: ingested by · Source files are scanned by the ingestion pipeline.
- Ingestion → Tree-sitter: parses with · Structural parsing is driven by Tree-sitter.
- Tree-sitter → Code Chunks: produces · AST output is converted into syntax-aligned chunks.
- Code Chunks → MiniLM: embedded by · Chunks are turned into local embeddings.
- MiniLM → ChromaDB: stored in · Embeddings and metadata are written to the vector store.
- Main → App: starts · Main delegates into the application orchestration layer.
- App → Agent: hosts · The app wraps the retrieval agent lifecycle.
- README → Ingestion: describes · Project docs explain the ingestion workflow.
- README → App: describes · Project docs explain the interactive retrieval flow.
- Agent → Code Search: uses · The agent invokes search as a tool.
- Code Search → ChromaDB: queries · Search retrieves relevant code chunks from ChromaDB.
- Agent → Follow Refs: expands · The agent can trigger multi-hop reference following.
- Follow Refs → Code Search: re-searches · Each hop can issue additional searches for related context.
- Agent → Gemini: reasons with · Retrieved context is synthesized by the LLM.
- Gemini → Answer: returns · The final explanation is emitted to the user.