How saswatsusmoy/semantic-search-nic works
Project Type and Purpose
• Full-stack semantic-search web application.
• Purpose: provide meaning-based search over NIC codes (English & Hindi) using FAISS for vector similarity and SentenceTransformer for embeddings.
• Features: REST API, web UI, admin index-management, CLI tools, data-processing pipelines.
File Structure
• Top-level directories:
• English/ and Hindi/ subprojects with identical structure for each language.
• Root templates/ and app.py—likely a catch-all or legacy UI.
• Key files in each language folder:
• Data Processing scripts (e.g., generate_embeddings.py, div_generator.py)
• Index files (faiss_index.bin, hindi_faiss.index) and mapping (faiss_id_map.json)
• Flask entrypoints (semantic_search_app.py, hindi_search_webapp.py) and compatibility helpers (flask_compat.py, api_compat.py)
• Static assets (HTML, CSS, JS) and Jinja templates.
• Utilities:
• api.py / start_api.py expose REST endpoints
• CLI scripts for search and index diagnostics (search_hindi_cli.py, diagnose_search.py)
• Data converters (convert_mongo_to_json.py, export_mongodb_to_json.py).
Extract Architecture Insights from README
• Frontend: static HTML/CSS/JS served by Flask
• Backend: single Flask app, with separate blueprints or modes for English/Hindi
• Embedding: SentenceTransformer (all-MiniLM-L6-v2)
• Indexing: FAISS with cosine similarity on local JSON data (output.json)
• Admin: rebuild FAISS index, view stats
• Data pipeline: generate embeddings ➔ build FAISS index ➔ persist .bin and id_map.
Main Components & Interactions
a. Components:
• User Browser (UI) — Static assets & Jinja templates
• Flask Web Server
– REST API endpoints (/search, /admin/rebuild)
– Web routes for UI pages
• Embedding Service (SentenceTransformer model)
• FAISS Index Manager
– Index loader/rebuilder
– faiss_index.bin, hindi_faiss.index, faiss_id_map.json
• Data Storage
– Local JSON files (output.json, output_hindi.json)
• Admin/CLI Tools
– Index diagnostics, migration scripts, data-processing scripts
• External Services (optional)
– Pretrained model download URL (HuggingFace)
b. Data & Control Flows:
1) On startup, Flask loads JSON ➔ Embedding Service to generate or load embeddings ➔ FAISS Index Manager builds or loads index.
2) User issues search query in UI or via POST /search ➔ Flask calls Embedding Service ➔ vector ➔ FAISS query ➔ results (IDs + scores) ➔ JSON lookup ➔ response.
3) Admin click rebuild index ➔ Flask triggers Data Processing script ➔ regenerate embeddings ➔ rebuild FAISS index ➔ overwrite .bin & id_map.
4) CLI tools bypass web layer to run same scripts for offline diagnostics or batch processing.
c. Architectural Patterns & Principles:
• Layered architecture: Presentation (UI), Application (Flask), Domain (Embedding & Search), Data (JSON & Index).
• Modularity by language: separate code trees for English/Hindi but same design.
• Single process vs. microservices: currently a monolith, but decoupled modules could be extracted.
d. Key Technologies & Libraries:
• Flask (web framework)
• SentenceTransformers (embeddings)
• FAISS (vector store & similarity search)
• Python scripts for ETL & migrations
Detailed Elements
• Component names & file references: e.g., “SentenceTransformer Model (generate_embeddings.py)”
• Data artifacts: output.json, faiss_index.bin, faiss_id_map.json
• API endpoints: POST /search, GET /admin/stats, POST /admin/rebuild
• CLI: search_hindi_cli.py, test_hybrid_search.py
• Data Processing: file_checker.py, div_generator.py, merge_inclusions.py