# rlucas7/code-searcher architecture

> Architecture diagram of the GitHub repository rlucas7/code-searcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rlucas7/code-searcher
- Repository: https://github.com/rlucas7/code-searcher

## Overview

##### Project type and purpose
- This is a full-stack Flask web application combined with a batch/CLI toolset for semantic code search and relevance annotation.
- Purpose: serve a web UI for users to perform natural-language semantic searches over code-entity JSONL datasets, record human relevance judgments, generate LLM-based relevance labels, and compute IR metrics.

##### File structure
- Top-level files:
- Collections-*.jsonl (sample code-entity datasets)
- prompt_tune.py, prompt_optim_eval.py (embedding/prompt utilities)
- Human_Annotation_Workflow.md (docs)
- requirements.txt, annotate.sh
- vec_search/ directory:
- config.py (Flask & vec-extension settings)
- auth.py, retriever.py, search.py (Flask blueprints/controllers)
- db.py, schema.sql, RAD.sql (database layer + schema definitions)
- bedrock_batch.py, llm_rel_gen.py, ir_eval_metrics.py (batch/CLI commands)
- static/, templates/ (frontend assets and Jinja2 templates)

##### README insights
- Uses the sqlite-vec extension (compiled C dylib) for vector storage/search.
- Configurable AI models (CodeT5, CodeBERT) for embedding.
- Click commands: init-db, export-rad-to-csv, gen-llm-rels, gen-ir-metrics, rad-merge, docs-cov-top-level.
- Provides user workflows: semantic search, human annotation, LLM annotation, metric calculation.

a. Main components:
1. Web Frontend
- Browser
- HTML/CSS/JS (vec_search/static & templates)
2. Backend Flask App
- Flask server (vec_search/__init__.py)
- Blueprints: auth, search, retriever
- Click-based batch commands (bedrock_batch, llm_rel_gen, ir_eval_metrics)
3. Database
- SQLite file + sqlite-vec extension
- Tables: entities, queries, query_relevances, users
4. Embedding Service(s)
- Local HuggingFace model (via transformers)
- External LLM APIs: OpenAI / AWS Bedrock / GCP Gemini
5. Data Sources
- JSONL files (Collections-*.jsonl) for initial entities
- Human annotation CSV exports / imports

b. Relationships & Interactions:
- User → Browser UI → Flask search endpoints → Retriever → Vector DB → returns ranked entities
- User → Annotation UI → Flask annotation endpoints → writes to query_relevances table
- CLI init-db → reads JSONL → calls embedding service → populates entities + embeddings in DB
- CLI gen-llm-rels → reads human annotation CSV → calls external LLM API → writes output CSV
- CLI gen-ir-metrics → reads CSV → runs ir_eval_metrics → outputs statistics

c. Architectural patterns & principles:
- MVC/MVT: Templates (views), Controllers (Flask blueprints), Models (db.py + ORM)
- Modular CLI commands via Flask-Click integration
- Dependency injection of config for environment-specific settings
- Single-file SQLite with pluggable C extension for vector operations

d. Key technologies & libraries:
- Flask + Click
- SQLite + custom sqlite-vec extension (C dylib)
- transformers (HuggingFace) for embedding
- scikit-learn / numpy for vector metrics
- Jinja2 templates + static CSS

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        Browser["Browser UI"]:::frontend
        subgraph "Assets"
            StaticDir["Static Assets\n(vec_search/static/)"]:::frontend
            StyleCSS["style.css"]:::frontend
            TemplatesDir["Templates\n(vec_search/templates/)"]:::frontend
            BaseHTML["base.html"]:::frontend
            LoginHTML["login.html"]:::frontend
            RegisterHTML["register.html"]:::frontend
            IndexHTML["index.html"]:::frontend
            DetailHTML["detail.html"]:::frontend
            UpdateHTML["update.html"]:::frontend
        end
        Browser --> StaticDir
        Browser --> TemplatesDir
    end

    %% Backend Layer
    subgraph "Backend Layer"
        subgraph "Flask Application"
            Init["App Init\n(vec_search/__init__.py)"]:::frontend
            Config["Config\n(vec_search/config.py)"]:::frontend
            Auth["Auth Blueprint\n(vec_search/auth.py)"]:::frontend
            SearchBP["Search Blueprint\n(vec_search/search.py)"]:::frontend
            Retriever["Retriever Module\n(vec_search/retriever.py)"]:::frontend
            Init --> Auth
            Init --> SearchBP
            Init --> Retriever
            Config --> Init
        end
        subgraph "Database"
            DB["Vector DB\n(SQLite + vec extension)"]:::storage
            SchemaSQL["schema.sql"]:::storage
            RADSQL["RAD.sql"]:::storage
            DB --> SchemaSQL
            DB --> RADSQL
        end
        Auth --> DB
        SearchBP --> Retriever
        Retriever --> DB
    end

    %% Batch Processing
    subgraph "Batch Processing" 
        CLIIngest["Data Ingestion & Embedding\n(vec_search/bedrock_batch.py & annotate.sh)"]:::batch
        CLI_LLML["LLM Relevance Gen\n(vec_search/llm_rel_gen.py)"]:::batch
        CLI_Metrics["IR Metrics Computation\n(vec_search/ir_eval_metrics.py)"]:::batch
        CLIIngest -.-> DB
        CLIIngest -.-> Embedding
        CLI_LLML -.-> Embedding
        CLI_LLML -.-> DataSources
        CLI_Metrics -.-> DataSources
    end

    %% External & Data Sources
    Collections["JSONL Datasets\n(Collections-*.jsonl)"]:::external
    Embedding["Embedding Service\n(Local HF / OpenAI / AWS Bedrock / GCP)"]:::external
    PromptUtils["Prompt Utils\n(prompt_tune.py & prompt_optim_eval.py)"]:::external
    Docs["Documentation\n(Human_Annotation_Workflow.md & README.md)"]:::external
    Req["Requirements\n(requirements.txt)"]:::external

    Collections --> CLIIngest
    Embedding --> CLIIngest
    Embedding --> SearchBP
    SearchBP --> Embedding

    %% Click Events
    click StaticDir "https://github.com/rlucas7/code-searcher/tree/main/vec_search/static/"
    click StyleCSS "https://github.com/rlucas7/code-searcher/blob/main/vec_search/static/style.css"
    click TemplatesDir "https://github.com/rlucas7/code-searcher/tree/main/vec_search/templates/"
    click BaseHTML "https://github.com/rlucas7/code-searcher/blob/main/vec_search/templates/base.html"
    click LoginHTML "https://github.com/rlucas7/code-searcher/blob/main/vec_search/templates/auth/login.html"
    click RegisterHTML "https://github.com/rlucas7/code-searcher/blob/main/vec_search/templates/auth/register.html"
    click IndexHTML "https://github.com/rlucas7/code-searcher/blob/main/vec_search/templates/search/index.html"
    click DetailHTML "https://github.com/rlucas7/code-searcher/blob/main/vec_search/templates/search/detail.html"
    click UpdateHTML "https://github.com/rlucas7/code-searcher/blob/main/vec_search/templates/search/update.html"
    click Init "https://github.com/rlucas7/code-searcher/blob/main/vec_search/__init__.py"
    click Config "https://github.com/rlucas7/code-searcher/blob/main/vec_search/config.py"
    click Auth "https://github.com/rlucas7/code-searcher/blob/main/vec_search/auth.py"
    click SearchBP "https://github.com/rlucas7/code-searcher/blob/main/vec_search/search.py"
    click Retriever "https://github.com/rlucas7/code-searcher/blob/main/vec_search/retriever.py"
    click DB "https://github.com/rlucas7/code-searcher/blob/main/vec_search/db.py"
    click SchemaSQL "https://github.com/rlucas7/code-searcher/blob/main/vec_search/schema.sql"
    click RADSQL "https://github.com/rlucas7/code-searcher/blob/main/vec_search/RAD.sql"
    click CLIIngest "https://github.com/rlucas7/code-searcher/blob/main/vec_search/bedrock_batch.py"
    click CLIIngest "https://github.com/rlucas7/code-searcher/blob/main/annotate.sh"
    click CLI_LLML "https://github.com/rlucas7/code-searcher/blob/main/vec_search/llm_rel_gen.py"
    click CLI_Metrics "https://github.com/rlucas7/code-searcher/blob/main/vec_search/ir_eval_metrics.py"
    click PromptUtils "https://github.com/rlucas7/code-searcher/blob/main/prompt_tune.py"
    click PromptUtils "https://github.com/rlucas7/code-searcher/blob/main/prompt_optim_eval.py"
    click Collections "https://github.com/rlucas7/code-searcher/blob/main/Collections-C.jsonl"
    click Collections "https://github.com/rlucas7/code-searcher/blob/main/Collections-go.jsonl"
    click Collections "https://github.com/rlucas7/code-searcher/blob/main/Collections-java.jsonl"
    click Collections "https://github.com/rlucas7/code-searcher/blob/main/Collections-js.jsonl"
    click Collections "https://github.com/rlucas7/code-searcher/blob/main/Collections-python.jsonl"
    click Docs "https://github.com/rlucas7/code-searcher/blob/main/Human_Annotation_Workflow.md"
    click Docs "https://github.com/rlucas7/code-searcher/blob/main/README.md"
    click Req "https://github.com/rlucas7/code-searcher/blob/main/requirements.txt"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#339,stroke-width:1px
    classDef storage fill:#d4edda,stroke:#28a745,stroke-width:1px
    classDef external fill:#fff3cd,stroke:#ffc107,stroke-width:1px
    classDef batch fill:#e2e3e5,stroke:#6c757d,stroke-width:1px
```
