# noxioustab/codesense architecture

> Architecture diagram of the GitHub repository noxioustab/codesense, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/noxioustab/codesense
- Repository: https://github.com/noxioustab/codesense

## Overview

##### Project type and purpose
- CodeSense is an open-source, full-stack tool for **semantic code search** over a Python codebase.
- It combines a **Streamlit UI**, several **backend modules** (parser, embedder, search engine), a **vector database** (Faiss), plus **external models** (Tree-sitter, CodeBERT).

##### File structure
- Top‐level modules:
- app/ui.py (Streamlit front end)
- backend/parser (Tree-sitter language build + code chunk extraction)
- backend/embeddings (CodeBERT-based embedding of code chunks)
- backend/search (FAISS vector search engine)
- Data folder holds embeddings JSONL files for each repo.
- scripts/ drive each pipeline step in isolation.
- build/ contains the Tree-sitter language library.
- requirements.txt & render.yaml flag deployment/infra concerns.

##### README for additional insights
- UI flow: user enters NL query → hits backend search → returns top K code snippets.
- Pipeline stages:
a. clone repositories
b. parse & extract chunks
c. embed chunks
d. index in FAISS
e. query & serve through Streamlit.

##### Main components & their interactions
a. Frontend (Streamlit UI)
- ui.py: captures natural‐language queries and displays results.
b. Backend Parser
- build_language.py: compiles Tree-sitter grammars.
- extract_chunks.py & py_parser.py: parse source code into ASTs and produce semantic “chunks.”
c. Backend Embedding
- hf_embedder.py: loads CodeBERT, encodes chunks.
- embed_chunks.py: orchestrates embedding pipeline, writes JSONL to /data/embeddings.
d. Backend Search Engine
- search_engine.py: loads JSONL, builds or loads FAISS index, performs vector search.
e. Data Storage
- /data/embeddings: stores embeddings per repo.
- FAISS index (in-memory or persisted).
f. External Dependencies
- Tree-sitter (parsing)
- CodeBERT (transformer model)
- FAISS (vector search)
- Streamlit (UI)

##### Architectural patterns & design principles
- **Pipeline architecture**: clear separation of stages (parse → embed → index → query).
- **Modular design**: parser, embedder, and search engine live in independent packages.
- **Data‐driven**: embeddings persisted to disk allow offline reindexing and reuse.

##### Essential details
- **build step** for my-languages.so under the parser.
- **scripts/** path for local testing and pipeline triggers.
- Versioned wheel installation of Tree-sitter in setup.
- If relevant, surface the **render.yaml** deployment config (e.g., container or hosting spec).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client/UI Layer
    subgraph "Client/UI Layer"
        UI["Streamlit UI"]:::frontend
    end
    click UI "https://github.com/noxioustab/codesense/blob/main/app/ui.py"

    %% Service/API Layer
    subgraph "Service/API Layer"
        API["Search API\n(search_engine.py)"]:::service
    end
    click API "https://github.com/noxioustab/codesense/blob/main/backend/search/search_engine.py"

    %% Processing Pipelines
    subgraph "Processing Pipelines"
        subgraph Parser
            BL["Build Grammar\n(build_language.py)"]:::service
            EC["Extract Chunks\n(extract_chunks.py)"]:::service
            PP["Python AST Parser\n(py_parser.py)"]:::service
        end
        subgraph Embedder
            HF["CodeBERT Embedder\n(hf_embedder.py)"]:::service
            EC2["Embed Pipeline\n(embed_chunks.py)"]:::service
        end
        SS["Pipeline Scripts"]:::service
    end
    click BL "https://github.com/noxioustab/codesense/blob/main/backend/parser/build_language.py"
    click EC "https://github.com/noxioustab/codesense/blob/main/backend/parser/extract_chunks.py"
    click PP "https://github.com/noxioustab/codesense/blob/main/backend/parser/py_parser.py"
    click HF "https://github.com/noxioustab/codesense/blob/main/backend/embeddings/hf_embedder.py"
    click EC2 "https://github.com/noxioustab/codesense/blob/main/backend/embeddings/embed_chunks.py"
    click SS "https://github.com/noxioustab/codesense/tree/main/scripts/"

    %% Data/Storage Layer
    subgraph "Data/Storage Layer"
        JSON["Embeddings Store\n(data/embeddings/)"]:::storage
        FA["FAISS Index"]:::storage
    end
    click JSON "https://github.com/noxioustab/codesense/tree/main/data/embeddings/"

    %% External Dependencies
    subgraph "External Dependencies"
        TS["Tree-sitter"]:::external
        CB["CodeBERT"]:::external
        FF["FAISS Library"]:::external
        ST["Streamlit"]:::external
    end

    %% Control Flow: Online
    UI -->|"NL Query"| API
    API -->|"Vector Query"| FA
    FA -->|"Top K Results"| API
    API -->|"Search Results"| UI
    ST --> UI

    %% Offline Pipeline Flow
    SS --> BL
    BL --> TS
    TS --> EC
    EC --> PP
    PP --> EC2
    EC2 -->|"Embeddings"| JSON
    JSON -->|"Index Data"| FA
    CB --> HF
    FF --> FA

    %% Styles
    classDef frontend fill:#D0E1F9,stroke:#333,stroke-width:1px
    classDef service fill:#DFF2D8,stroke:#333,stroke-width:1px
    classDef storage fill:#FFE5B4,stroke:#333,stroke-width:1px
    classDef external fill:#E2E2E2,stroke:#333,stroke-width:1px
```
