How sturdy-dev/semantic-code-search works
Project Type and Purpose
• This is an open-source, command-line application written in Python, named semantic-code-search (invoked as sem).
• It indexes a local Git repo’s function/method definitions by generating and caching vector embeddings, then performs a nearest-neighbor search on those embeddings when a user issues a natural-language query.
File Structure Analysis
• Top-level:
• setup.py / setup.cfg / requirements.txt – packaging and dependencies
• docs/ – README assets and example screenshots
• src/semantic_code_search/ – core modules:
• cli.py – entry point; parses flags and dispatches commands
• embed.py – scans code, extracts functions, generates embeddings, writes .embeddings cache
• query.py – embeds the query text and invokes similarity search
• cluster.py – optional clustering of similar code embeddings
• prompt.py – formatting of results and interactive UI logic
Key Components
a. Command-Line Interface (CLI)
• Accepts user commands and flags (sem 'query', sem --embed, etc.)
• Dispatches to embed, query, or cluster workflows.
b. Git Repository / Local File System
• Contains source files (.py, etc.)
• Holds the cache file .embeddings at repo root.
c. Code Extraction & Embedding Generation
• A pipeline: parse source files → extract functions/methods → batch-embed via a Sentence-Transformer model
d. Embeddings Cache (.embeddings)
• Serialized vector store used for subsequent queries.
e. Transformer Model
• Locally downloaded on first run (~500 MB).
• Provides embedding API (via HuggingFace / sentence-transformers).
f. Nearest-Neighbor Search & Clustering
• Query embeddings vs. cached embeddings → cosine similarity ranking
• Clustering module optionally groups similar code snippets.
g. Results Renderer & Editor Launcher
• Formats top-N matches with score, file path, line, snippet.
Relationships and Data Flows
• CLI → (on embed) → Code Extraction → Transformer Model → write to .embeddings
• CLI → (on query) → Transformer Model → load .embeddings → Similarity Search → Results Renderer → Editor
• CLI → (on cluster) → load .embeddings → Clustering Algorithm → print clusters
• All data flows are local (no network calls after model download).
Architectural Patterns & Tech Stack
• Modular Python package with single-process command dispatcher
• Embedding generation follows a producer-consumer pattern (code → embeddings → cache)
• Model inference uses the sentence-transformers library
• Nearest-neighbor search over dense vectors (e.g., via Faiss or brute-force in-memory)
• Optional clustering based on distance thresholds