# saswatsusmoy/semantic-search-nic architecture

> Architecture diagram of the GitHub repository saswatsusmoy/semantic-search-nic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saswatsusmoy/semantic-search-nic
- Repository: https://github.com/saswatsusmoy/semantic-search-nic

## Overview

##### Project Type and Purpose
- Full-stack semantic-search web application.
- Purpose: provide meaning-based search over NIC codes (English & Hindi) using FAISS for vector similarity and SentenceTransformer for embeddings.
- Features: REST API, web UI, admin index-management, CLI tools, data-processing pipelines.

##### File Structure
- Top-level directories:
- English/ and Hindi/ subprojects with identical structure for each language.
- Root templates/ and app.py—likely a catch-all or legacy UI.
- Key files in each language folder:
- Data Processing scripts (e.g., generate_embeddings.py, div_generator.py)
- Index files (faiss_index.bin, hindi_faiss.index) and mapping (faiss_id_map.json)
- Flask entrypoints (semantic_search_app.py, hindi_search_webapp.py) and compatibility helpers (flask_compat.py, api_compat.py)
- Static assets (HTML, CSS, JS) and Jinja templates.
- Utilities:
- api.py / start_api.py expose REST endpoints
- CLI scripts for search and index diagnostics (search_hindi_cli.py, diagnose_search.py)
- Data converters (convert_mongo_to_json.py, export_mongodb_to_json.py).

##### Extract Architecture Insights from README
- Frontend: static HTML/CSS/JS served by Flask
- Backend: single Flask app, with separate blueprints or modes for English/Hindi
- Embedding: SentenceTransformer (all-MiniLM-L6-v2)
- Indexing: FAISS with cosine similarity on local JSON data (output.json)
- Admin: rebuild FAISS index, view stats
- Data pipeline: generate embeddings ➔ build FAISS index ➔ persist .bin and id_map.

##### Main Components & Interactions
a. Components:
- User Browser (UI) — Static assets & Jinja templates
- Flask Web Server
– REST API endpoints (/search, /admin/rebuild)
– Web routes for UI pages
- Embedding Service (SentenceTransformer model)
- FAISS Index Manager
– Index loader/rebuilder
– faiss_index.bin, hindi_faiss.index, faiss_id_map.json
- Data Storage
– Local JSON files (output.json, output_hindi.json)
- Admin/CLI Tools
– Index diagnostics, migration scripts, data-processing scripts
- External Services (optional)
– Pretrained model download URL (HuggingFace)
b. Data & Control Flows:
1) On startup, Flask loads JSON ➔ Embedding Service to generate or load embeddings ➔ FAISS Index Manager builds or loads index.
2) User issues search query in UI or via POST /search ➔ Flask calls Embedding Service ➔ vector ➔ FAISS query ➔ results (IDs + scores) ➔ JSON lookup ➔ response.
3) Admin click rebuild index ➔ Flask triggers Data Processing script ➔ regenerate embeddings ➔ rebuild FAISS index ➔ overwrite .bin & id_map.
4) CLI tools bypass web layer to run same scripts for offline diagnostics or batch processing.
c. Architectural Patterns & Principles:
- Layered architecture: Presentation (UI), Application (Flask), Domain (Embedding & Search), Data (JSON & Index).
- Modularity by language: separate code trees for English/Hindi but same design.
- Single process vs. microservices: currently a monolith, but decoupled modules could be extracted.
d. Key Technologies & Libraries:
- Flask (web framework)
- SentenceTransformers (embeddings)
- FAISS (vector store & similarity search)
- Python scripts for ETL & migrations

##### Detailed Elements
- Component names & file references: e.g., “SentenceTransformer Model (generate_embeddings.py)”
- Data artifacts: output.json, faiss_index.bin, faiss_id_map.json
- API endpoints: POST /search, GET /admin/stats, POST /admin/rebuild
- CLI: search_hindi_cli.py, test_hybrid_search.py
- Data Processing: file_checker.py, div_generator.py, merge_inclusions.py

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% English Subgraph
    subgraph "English Pipeline"
        direction TB
        subgraph "Presentation (English)"
            ENG_UI_Static["Static Assets"]:::presentation
            ENG_UI_Templates["Jinja Templates"]:::presentation
        end
        subgraph "Application (English)"
            ENG_Flask["Flask Web Server"]:::app
            ENG_API["REST API Layer"]:::app
            ENG_WebRoutes["Web Routes"]:::app
            ENG_EmbeddingSvc["Embedding Service\n(SentenceTransformer)"]:::app
            ENG_FAISSMgr["FAISS Index Manager"]:::app
        end
        subgraph "Data (English)"
            ENG_JSON["Local JSON Files"]:::data
            ENG_FAISSFiles["FAISS Index & ID Map"]:::data
        end
        ENG_UI_Static -->|"serves"| ENG_Flask
        ENG_UI_Templates -->|"renders"| ENG_Flask
        ENG_Flask -->|"routes to"| ENG_API
        ENG_Flask -->|"routes to"| ENG_WebRoutes
        ENG_API -->|"calls"| ENG_EmbeddingSvc
        ENG_WebRoutes -->|"calls"| ENG_EmbeddingSvc
        ENG_EmbeddingSvc -->|"generate vector"| ENG_FAISSMgr
        ENG_FAISSMgr -->|"load/query"| ENG_FAISSFiles
        ENG_JSON -->|"lookup IDs"| ENG_FAISSMgr
    end

    %% Hindi Subgraph
    subgraph "Hindi Pipeline"
        direction TB
        subgraph "Presentation (Hindi)"
            HIN_UI_Static["Static Assets"]:::presentation
            HIN_UI_Templates["Jinja Templates"]:::presentation
        end
        subgraph "Application (Hindi)"
            HIN_Flask["Flask Web Server"]:::app
            HIN_WebRoutes["Web Routes"]:::app
            HIN_EmbeddingSvc["Embedding Service\n(SentenceTransformer)"]:::app
            HIN_FAISSMgr["FAISS Index Manager"]:::app
        end
        subgraph "Data (Hindi)"
            HIN_JSON["Local JSON Files"]:::data
            HIN_FAISSFiles["FAISS Index"]:::data
        end
        HIN_UI_Static -->|"serves"| HIN_Flask
        HIN_UI_Templates -->|"renders"| HIN_Flask
        HIN_Flask -->|"routes to"| HIN_WebRoutes
        HIN_WebRoutes -->|"calls"| HIN_EmbeddingSvc
        HIN_EmbeddingSvc -->|"generate vector"| HIN_FAISSMgr
        HIN_FAISSMgr -->|"load/query"| HIN_FAISSFiles
        HIN_JSON -->|"lookup IDs"| HIN_FAISSMgr
    end

    %% Admin & CLI Tools
    subgraph "Admin & CLI Tools"
        direction TB
        Admin_Rebuild["Rebuild Index (CLI)"]:::app
        Diagnostics["Index Diagnostics"]:::app
        Admin_Rebuild -->|"regenerate embeddings"| ENG_EmbeddingSvc
        Admin_Rebuild -->|"rebuild index"| ENG_FAISSMgr
        Diagnostics -->|"analyze"| ENG_FAISSMgr
    end

    %% External
    HuggingFace["Pretrained Model URL"]:::external
    HuggingFace -->|"download model"| ENG_EmbeddingSvc
    HuggingFace -->|"download model"| HIN_EmbeddingSvc

    %% Click Events
    click ENG_UI_Static "https://github.com/saswatsusmoy/semantic-search-nic/tree/main/English/static/"
    click ENG_UI_Templates "https://github.com/saswatsusmoy/semantic-search-nic/tree/main/English/templates/"
    click ENG_Flask "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/app.py"
    click ENG_Flask "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/semantic_search_app.py"
    click ENG_Flask "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/flask_compat.py"
    click ENG_Flask "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/api_compat.py"
    click ENG_API "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/api.py"
    click ENG_API "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/start_api.py"
    click ENG_EmbeddingSvc "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/generate_embeddings.py"
    click ENG_EmbeddingSvc "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/vector_embeddings_manager.py"
    click ENG_EmbeddingSvc "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/vector_embeddings_subclasses.py"
    click ENG_FAISSMgr "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/faiss_index_manager.py"
    click ENG_JSON "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/output.json"
    click ENG_JSON "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/output_hindi.json"
    click ENG_JSON "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/translated_output.json"
    click ENG_FAISSFiles "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/faiss_index.bin"
    click ENG_FAISSFiles "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/hindi_faiss.index"
    click ENG_FAISSFiles "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/faiss_id_map.json"
    click HIN_UI_Static "https://github.com/saswatsusmoy/semantic-search-nic/tree/main/Hindi/static/"
    click HIN_UI_Templates "https://github.com/saswatsusmoy/semantic-search-nic/tree/main/Hindi/templates/"
    click HIN_Flask "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/Hindi/semantic_search_app.py"
    click HIN_Flask "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/Hindi/hindi_search_webapp.py"
    click HIN_JSON "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/Hindi/output_hindi.json"
    click HIN_FAISSFiles "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/Hindi/hindi_faiss.index"
    click Admin_Rebuild "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/diagnose_search.py"
    click Admin_Rebuild "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/debug_search.py"
    click Admin_Rebuild "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/search_hindi_cli.py"
    click Admin_Rebuild "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/Hindi/search_hindi_cli.py"
    click Diagnostics "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/test_hybrid_search.py"
    click Diagnostics "https://github.com/saswatsusmoy/semantic-search-nic/blob/main/English/test_local_search.py"

    %% Style Definitions
    classDef presentation fill:#D6EFFF,stroke:#0366d6,color:#0366d6
    classDef app fill:#D5F5E3,stroke:#239B56,color:#1E8449
    classDef data fill:#FAD7A0,stroke:#B9770E,color:#6E2C00
    classDef external fill:#D7DBDD,stroke:#566573,color:#2C3E50
```
