How thameenas/semantic-word-ladder works
What this project is (type + purpose)
This is a single-service Python application packaged for Docker/Hugging Face Spaces that exposes an interactive Streamlit UI (app.py) for finding “semantic word ladders” between two words. Internally it:
• Uses Sentence-Transformers (all-MiniLM-L6-v2) to create word embeddings
• Uses FAISS to do fast top‑K nearest-neighbor lookup in embedding space (treating the space as an implicit graph)
• Runs a graph search algorithm (primarily A\*; possibly also greedy) to find a path from start word → target word via intermediate words
• Stores precomputed artifacts in data/ (embeddings.npy, word_to_idx.json, words.txt)
So architecturally: UI → semantic search pipeline → precomputed data + FAISS index → search algorithms.
Top-level:
• app.py
Streamlit entrypoint; orchestrates loading data/index and invoking search to generate a ladder.
• src/
Core library code:
• src/embed.py — offline embedding generation job
• src/index.py — FAISS index build/load/test module
• src/neighbours.py — top‑K neighbor query abstraction (likely calls FAISS)
• src/astar.py — A\* search implementation
• src/greedy.py — alternative search strategy
• data/
Offline and runtime artifacts:
• words.txt vocabulary source
• embeddings.npy embedding matrix
• word_to_idx.json lookup map word → embedding row
• Deployment/build:
• Dockerfile (Hugging Face “sdk: docker”)
• requirements.txt, pyproject.toml, uv.lock, .python-version
Client / User
• Interacts via browser with Streamlit app
Streamlit Web App (app.py)
• Input: start word, target word, parameters (K, max steps, etc. if present)
• Output: ladder + diagnostics
Semantic Ladder Engine (src/)
• Search Layer
• A* Search (src/astar.py)
• Greedy (src/greedy.py) (optional path)
• Neighbor Provider
• Nearest Neighbors (src/neighbours.py)
• Talks to FAISS index for top‑K similar words
• Index Manager
• FAISS Index (src/index.py) load/build
• Embedding Lookup
• Uses word_to_idx.json + embeddings.npy for vector retrieval
Data Artifacts (data/)
• words.txt (input vocabulary)
• embeddings.npy (precomputed vectors)
• word_to_idx.json (mapping)
Offline Pipeline (one-time / batch)
• Embedding Generation Job (src/embed.py)
• External dependency: Sentence-Transformers all-MiniLM-L6-v2
• Outputs: embeddings + mapping (and potentially normalized vectors)
External Libraries / Runtime Dependencies
• sentence-transformers (model)
• faiss (ANN index)
• numpy (array storage)
• (Optionally show Hugging Face Spaces runtime container)
Runtime path (interactive query):
Streamlit UI → User (render ladder)
Startup / initialization path:
• app.py → src/index.py to load FAISS index (and data/*.npy/json)
• If index not present (depends on implementation), index manager might build from embeddings.npy
Offline preprocessing path:
src/embed.py → write embeddings.npy and word_to_idx.json
4. src/index.py → read embeddings.npy → build FAISS index (may happen at runtime or offline)
Architectural patterns / principles to highlight
• Two-phase architecture
• Offline preprocessing (embedding generation) vs online query serving (search)
• Implicit graph / on-demand edges
• No full adjacency list stored; neighbors computed dynamically via ANN search (FAISS)
• Algorithmic core separated from UI
• app.py is the orchestration layer; src/ is the reusable engine
• Heuristic search
Technologies to label explicitly
• Streamlit (web UI)
• Python 3.10
• Sentence-Transformers + all-MiniLM-L6-v2
• FAISS for top‑K neighbor retrieval
• Numpy .npy artifact storage + JSON dictionary
• Docker / Hugging Face Spaces as deployment target
Shows user, Streamlit app container, semantic engine, FAISS/data artifacts, external model library.
Shows step-by-step: submit start/target → index lookup → repeated neighbor expansions → A\* termination → render ladder.
4.2 Suggested component breakdown (don’t lump everything into “backend”)
• UI: app.py (Streamlit)
• Engine:
• A* (src/astar.py)
• Greedy (src/greedy.py) (optional)
• Neighbours service (src/neighbours.py)
• Index manager (src/index.py)
• Data:
• embeddings.npy
• word_to_idx.json
• words.txt
• Offline:
• Embedding job (src/embed.py)
• External:
• SentenceTransformer model
• FAISS library
4.3 Show iteration explicitly (critical for search projects)
• “repeat until target found / max steps”
• “expand node → query top‑K neighbors via FAISS”
• “compute g(n), h(n) using cosine distance/similarity”
This is the essence of the system.
• app.py → “Streamlit App / Controller”
• src/embed.py → “Embedding Generation (offline)”
• src/index.py → “FAISS Index Loader/Builder”
• src/neighbours.py → “Top‑K Neighbor Query API”
• src/astar.py → “A* Search”
• src/greedy.py → “Greedy Search (alternative)”
• data/words.txt → “Vocabulary input”
• data/embeddings.npy → “Embedding matrix”
• data/word_to_idx.json → “Word↔Index map”
• Dockerfile → “Container runtime (HF Spaces)”
Final checklist for “most accurate” representation
• [ ] Show offline embedding build producing artifacts
• [ ] Show FAISS index as a distinct component (loaded at runtime)
• [ ] Show implicit graph / “neighbors computed on demand” (annotate)
• [ ] Show **A\* loop** and what heuristic is (cosine similarity to target)
• [ ] Show all persistent artifacts in data/
• [ ] Make it clear the app is single process (Streamlit) calling local modules (not microservices)