How davidyz/vectorcode works
Project type and purpose
• VectorCode is an open-source developer tool comprising two deliverables:
a) A Python-based command-line application (“vectorcode” CLI) for indexing, embedding, and querying code repositories using a vector database (ChromaDB).
b) A Neovim plugin, written in Lua, which embeds the CLI functionality into the Neovim editor via job runners and LSP-like interactions.
• It is not a web app; rather it’s a full-featured CLI + editor-integration tool focused on code retrieval, chunking, and prompt augmentation for LLMs.
File structure
• Top level:
• src/vectorcode/… : Python package implementing the CLI, subcommands, LSP entry point, MCP integration, chunking, caching, reranking, etc.
• lua/vectorcode/… & plugin/vectorcode.lua : Neovim plugin code – extension points for CodeCompanion, CopilotChat, lualine, jobrunner, caching, configuration.
• docs/, README.md : User docs and CLI/Neovim usage guides.
• tests/… : Comprehensive unit tests for Python subcommands and core logic.
• pyproject.toml, Makefile, GitHub workflows, pre-commit config – build/test pipelines and packaging.
README for additional insights
• Core functionality: index a codebase, vectorise chunks, run queries with reranking, integrate with LLM prompt-building.
• Backend: Chromadb for vector storage, SentenceTransformer for embeddings, optional py-tree-sitter for smarter chunk splitting.
• Integration: Model Context Protocol (MCP), CodeCompanion.nvim, and native Neovim UI via Lua.
a. Main components:
1) CLI Application (Python)
– Entry point: vectorcode/main.py (commands: init, vectorise, query, drop, etc.)
– LSP mode: lsp_main.py for interactive sessions.
– MCP client: mcp_main.py to speak Model Context Protocol.
– Internal modules: chunking, cacher (default, LSP), jobrunner (cmd, lsp), subcommand handlers, rerankers.
2) Vector Database (ChromaDB)
– Persists document embeddings, metadata.
3) Embedding Engine
– SentenceTransformer (or pluggable alternative).
4) Neovim Plugin (Lua)
– Core plugin: plugin/vectorcode.lua
– Integrations: codecompanion, CopilotChat, lualine.
– Cacher layer to skip redundant retrieval.
– Jobrunner invoking Python CLI or LSP endpoint.
5) External Services
– CodeCompanion.nvim (UI for queries), Copilot Chat, Lualine statusline.
b. Relationships & data flows:
• User invokes CLI → CLI loads config → splits files into chunks → calls Embedding Engine → stores vectors in ChromaDB → returns status.
• User issues a query via CLI or Neovim → CLI/MCP/LSP server retrieves nearest-neighbor chunks from ChromaDB → optional reranking → returns JSON docs.
• Neovim plugin starts jobrunner → spawns CLI subcommand or establishes LSP session → receives results → displays via CodeCompanion or native UI.
• Plugin caching layer checks last indexed state to skip redundant requests.
c. Architectural patterns & design principles:
• Layered architecture (Presentation: CLI/UI; Application: subcommands & LSP; Domain: chunking/reranking; Infrastructure: ChromaDB, embedding, file I/O).
• Plugin-based extensibility (Lua integrations, Python subcommands, reranker strategy pattern).
• Asynchronous job runners for non-blocking editor integration.
• Clear separation of concerns: indexing vs. querying vs. LSP/MCP interface.
d. Key technologies & libraries:
• Python 3, pyproject.toml packaging, pytest for tests.
• ChromaDB vector store.
• SentenceTransformer (HuggingFace).
• py-tree-sitter for semantic chunking.
• Neovim Lua API, codecompanion.nvim integration.
• Model Context Protocol client.