How noxioustab/codesense works
Project type and purpose
• CodeSense is an open-source, full-stack tool for semantic code search over a Python codebase.
• It combines a Streamlit UI, several backend modules (parser, embedder, search engine), a vector database (Faiss), plus external models (Tree-sitter, CodeBERT).
File structure
• Top‐level modules:
• app/ui.py (Streamlit front end)
• backend/parser (Tree-sitter language build + code chunk extraction)
• backend/embeddings (CodeBERT-based embedding of code chunks)
• backend/search (FAISS vector search engine)
• Data folder holds embeddings JSONL files for each repo.
• scripts/ drive each pipeline step in isolation.
• build/ contains the Tree-sitter language library.
• requirements.txt & render.yaml flag deployment/infra concerns.
README for additional insights
• UI flow: user enters NL query → hits backend search → returns top K code snippets.
• Pipeline stages:
a. clone repositories
b. parse & extract chunks
c. embed chunks
d. index in FAISS
e. query & serve through Streamlit.
Main components & their interactions
a. Frontend (Streamlit UI)
• ui.py: captures natural‐language queries and displays results.
b. Backend Parser
• build_language.py: compiles Tree-sitter grammars.
• extract_chunks.py & py_parser.py: parse source code into ASTs and produce semantic “chunks.”
c. Backend Embedding
• hf_embedder.py: loads CodeBERT, encodes chunks.
• embed_chunks.py: orchestrates embedding pipeline, writes JSONL to /data/embeddings.
d. Backend Search Engine
• search_engine.py: loads JSONL, builds or loads FAISS index, performs vector search.
e. Data Storage
• /data/embeddings: stores embeddings per repo.
• FAISS index (in-memory or persisted).
f. External Dependencies
• Tree-sitter (parsing)
• CodeBERT (transformer model)
• FAISS (vector search)
• Streamlit (UI)
Architectural patterns & design principles
• Pipeline architecture: clear separation of stages (parse → embed → index → query).
• Modular design: parser, embedder, and search engine live in independent packages.
• Data‐driven: embeddings persisted to disk allow offline reindexing and reuse.
Essential details
• build step for my-languages.so under the parser.
• scripts/ path for local testing and pipeline triggers.
• Versioned wheel installation of Tree-sitter in setup.
• If relevant, surface the render.yaml deployment config (e.g., container or hosting spec).