How rlucas7/code-searcher works
Project type and purpose
• This is a full-stack Flask web application combined with a batch/CLI toolset for semantic code search and relevance annotation.
• Purpose: serve a web UI for users to perform natural-language semantic searches over code-entity JSONL datasets, record human relevance judgments, generate LLM-based relevance labels, and compute IR metrics.
File structure
• Top-level files:
• Collections-*.jsonl (sample code-entity datasets)
• prompt_tune.py, prompt_optim_eval.py (embedding/prompt utilities)
• Human_Annotation_Workflow.md (docs)
• requirements.txt, annotate.sh
• vec_search/ directory:
• config.py (Flask & vec-extension settings)
• auth.py, retriever.py, search.py (Flask blueprints/controllers)
• db.py, schema.sql, RAD.sql (database layer + schema definitions)
• bedrock_batch.py, llm_rel_gen.py, ir_eval_metrics.py (batch/CLI commands)
• static/, templates/ (frontend assets and Jinja2 templates)
README insights
• Uses the sqlite-vec extension (compiled C dylib) for vector storage/search.
• Configurable AI models (CodeT5, CodeBERT) for embedding.
• Click commands: init-db, export-rad-to-csv, gen-llm-rels, gen-ir-metrics, rad-merge, docs-cov-top-level.
• Provides user workflows: semantic search, human annotation, LLM annotation, metric calculation.
a. Main components:
1. Web Frontend
• Browser
• HTML/CSS/JS (vec_search/static & templates)
2. Backend Flask App
• Flask server (vec_search/__init__.py)
• Blueprints: auth, search, retriever
• Click-based batch commands (bedrock_batch, llm_rel_gen, ir_eval_metrics)
3. Database
• SQLite file + sqlite-vec extension
• Tables: entities, queries, query_relevances, users
4. Embedding Service(s)
• Local HuggingFace model (via transformers)
• External LLM APIs: OpenAI / AWS Bedrock / GCP Gemini
5. Data Sources
• JSONL files (Collections-*.jsonl) for initial entities
• Human annotation CSV exports / imports
b. Relationships & Interactions:
• User → Browser UI → Flask search endpoints → Retriever → Vector DB → returns ranked entities
• User → Annotation UI → Flask annotation endpoints → writes to query_relevances table
• CLI init-db → reads JSONL → calls embedding service → populates entities + embeddings in DB
• CLI gen-llm-rels → reads human annotation CSV → calls external LLM API → writes output CSV
• CLI gen-ir-metrics → reads CSV → runs ir_eval_metrics → outputs statistics
c. Architectural patterns & principles:
• MVC/MVT: Templates (views), Controllers (Flask blueprints), Models (db.py + ORM)
• Modular CLI commands via Flask-Click integration
• Dependency injection of config for environment-specific settings
• Single-file SQLite with pluggable C extension for vector operations
d. Key technologies & libraries:
• Flask + Click
• SQLite + custom sqlite-vec extension (C dylib)
• transformers (HuggingFace) for embedding
• scikit-learn / numpy for vector metrics
• Jinja2 templates + static CSS