# thameenas/semantic-word-ladder architecture

> Architecture diagram of the GitHub repository thameenas/semantic-word-ladder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thameenas/semantic-word-ladder
- Repository: https://github.com/thameenas/semantic-word-ladder

## Overview

##### What this project is (type + purpose)

This is a **single-service Python application** packaged for **Docker/Hugging Face Spaces** that exposes an **interactive Streamlit UI** (`app.py`) for finding **“semantic word ladders”** between two words. Internally it:

- Uses **Sentence-Transformers** (`all-MiniLM-L6-v2`) to create **word embeddings**
- Uses **FAISS** to do fast **top‑K nearest-neighbor** lookup in embedding space (treating the space as an implicit graph)
- Runs a **graph search algorithm** (primarily **A\***; possibly also greedy) to find a path from start word → target word via intermediate words
- Stores precomputed artifacts in `data/` (`embeddings.npy`, `word_to_idx.json`, `words.txt`)

So architecturally: **UI → semantic search pipeline → precomputed data + FAISS index → search algorithms**.

Top-level:

- `app.py`
Streamlit entrypoint; orchestrates loading data/index and invoking search to generate a ladder.
- `src/`
Core library code:
- `src/embed.py` — offline embedding generation job
- `src/index.py` — FAISS index build/load/test module
- `src/neighbours.py` — top‑K neighbor query abstraction (likely calls FAISS)
- `src/astar.py` — A\* search implementation
- `src/greedy.py` — alternative search strategy
- `data/`
Offline and runtime artifacts:
- `words.txt` vocabulary source
- `embeddings.npy` embedding matrix
- `word_to_idx.json` lookup map word → embedding row
- Deployment/build:
- `Dockerfile` (Hugging Face “sdk: docker”)
- `requirements.txt`, `pyproject.toml`, `uv.lock`, `.python-version`

##### Client / User
- Interacts via browser with Streamlit app

##### Streamlit Web App (app.py)
- Input: start word, target word, parameters (K, max steps, etc. if present)
- Output: ladder + diagnostics

##### Semantic Ladder Engine (src/)
- **Search Layer**
- `A* Search (src/astar.py)`
- `Greedy (src/greedy.py)` (optional path)
- **Neighbor Provider**
- `Nearest Neighbors (src/neighbours.py)`
- Talks to FAISS index for top‑K similar words
- **Index Manager**
- `FAISS Index (src/index.py)` load/build
- **Embedding Lookup**
- Uses `word_to_idx.json` + `embeddings.npy` for vector retrieval

##### Data Artifacts (data/)
- `words.txt` (input vocabulary)
- `embeddings.npy` (precomputed vectors)
- `word_to_idx.json` (mapping)

##### Offline Pipeline (one-time / batch)
- `Embedding Generation Job (src/embed.py)`
- External dependency: `Sentence-Transformers all-MiniLM-L6-v2`
- Outputs: embeddings + mapping (and potentially normalized vectors)

##### External Libraries / Runtime Dependencies
- `sentence-transformers` (model)
- `faiss` (ANN index)
- `numpy` (array storage)
- (Optionally show Hugging Face Spaces runtime container)

**Runtime path (interactive query):**
##### Streamlit UI → User (render ladder)

**Startup / initialization path:**
- `app.py` → `src/index.py` to **load** FAISS index (and `data/*.npy/json`)
- If index not present (depends on implementation), index manager might **build** from `embeddings.npy`

**Offline preprocessing path:**
##### `src/embed.py` → write `embeddings.npy` and `word_to_idx.json`
4. `src/index.py` → read `embeddings.npy` → build FAISS index (may happen at runtime or offline)

##### Architectural patterns / principles to highlight

- **Two-phase architecture**
- **Offline preprocessing** (embedding generation) vs **online query serving** (search)
- **Implicit graph / on-demand edges**
- No full adjacency list stored; neighbors computed dynamically via ANN search (FAISS)
- **Algorithmic core separated from UI**
- `app.py` is the orchestration layer; `src/` is the reusable engine
- **Heuristic search**

##### Technologies to label explicitly

- Streamlit (web UI)
- Python 3.10
- Sentence-Transformers + `all-MiniLM-L6-v2`
- FAISS for top‑K neighbor retrieval
- Numpy `.npy` artifact storage + JSON dictionary
- Docker / Hugging Face Spaces as deployment target

Shows user, Streamlit app container, semantic engine, FAISS/data artifacts, external model library.

Shows step-by-step: submit start/target → index lookup → repeated neighbor expansions → A\* termination → render ladder.

##### 4.2 Suggested component breakdown (don’t lump everything into “backend”)

- UI: `app.py (Streamlit)`
- Engine:
- `A* (src/astar.py)`
- `Greedy (src/greedy.py)` (optional)
- `Neighbours service (src/neighbours.py)`
- `Index manager (src/index.py)`
- Data:
- `embeddings.npy`
- `word_to_idx.json`
- `words.txt`
- Offline:
- `Embedding job (src/embed.py)`
- External:
- `SentenceTransformer model`
- `FAISS library`

##### 4.3 Show iteration explicitly (critical for search projects)

- “repeat until target found / max steps”
- “expand node → query top‑K neighbors via FAISS”
- “compute g(n), h(n) using cosine distance/similarity”

This is the essence of the system.

- `app.py` → “Streamlit App / Controller”
- `src/embed.py` → “Embedding Generation (offline)”
- `src/index.py` → “FAISS Index Loader/Builder”
- `src/neighbours.py` → “Top‑K Neighbor Query API”
- `src/astar.py` → “A* Search”
- `src/greedy.py` → “Greedy Search (alternative)”
- `data/words.txt` → “Vocabulary input”
- `data/embeddings.npy` → “Embedding matrix”
- `data/word_to_idx.json` → “Word↔Index map”
- `Dockerfile` → “Container runtime (HF Spaces)”

##### Final checklist for “most accurate” representation

- [ ] Show **offline embedding build** producing artifacts
- [ ] Show **FAISS index** as a distinct component (loaded at runtime)
- [ ] Show **implicit graph** / “neighbors computed on demand” (annotate)
- [ ] Show **A\* loop** and what heuristic is (cosine similarity to target)
- [ ] Show all persistent artifacts in `data/`
- [ ] Make it clear the app is **single process** (Streamlit) calling local modules (not microservices)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% =========================
    %% External / Client
    %% =========================
    U["User (Browser)"]:::external

    %% =========================
    %% Runtime container (single-process)
    %% =========================
    subgraph "Online Runtime (Docker / Hugging Face Spaces)"
        DKR["Container Runtime"]:::runtime

        subgraph "Streamlit Application"
            APP["Streamlit Web App / Controller\n(app.py)"]:::ui
        end

        subgraph "Semantic Ladder Engine (src/)"
            subgraph "Search Layer"
                ASTAR["A* Search\n(src/astar.py)"]:::engine
                GREEDY["Greedy Search (optional)\n(src/greedy.py)"]:::engine
            end

            subgraph "Neighbor Expansion (Implicit Graph)"
                NBR["Nearest Neighbors Provider\n(src/neighbours.py)"]:::engine
                NOTE_IMPLICIT["NOTE: Full graph is never stored.\nEdges computed on-demand via top-K ANN queries."]:::note
            end

            subgraph "Index + Lookup"
                IDX["FAISS Index Manager\n(load/build/test)\n(src/index.py)"]:::engine
                LOOKUP["Embedding Lookup\n(word↔idx + vectors)"]:::engine
            end
        end

        subgraph "Local Artifacts (data/)"
            DATA_DIR["Data Artifacts Directory\n(data/)"]:::data
            WORDS["words.txt\n(vocabulary)"]:::data
            EMBS["embeddings.npy\n(embedding matrix)"]:::data
            W2I["word_to_idx.json\n(word→row index)"]:::data
            FAISSIDX[("FAISS Index\n(in-memory or persisted)")]:::data
        end

        subgraph "Python Dependencies"
            STLIB["sentence-transformers\n(all-MiniLM-L6-v2)"]:::external
            FAISSLIB["faiss (ANN search)"]:::external
            NUMPY["numpy (arrays)"]:::external
        end
    end

    %% =========================
    %% Offline / Batch pipeline
    %% =========================
    subgraph "Offline Preprocessing (one-time / batch)"
        EMBJOB["Embedding Generation Job\n(src/embed.py)"]:::offline
        OFFNOTE["Produces artifacts for online search\n(no runtime embedding needed if inputs in vocab)"]:::note
    end

    %% =========================
    %% Build / Packaging metadata
    %% =========================
    subgraph "Build & Packaging Metadata"
        REQ["requirements.txt"]:::meta
        PYPROJ["pyproject.toml"]:::meta
        UVLOCK["uv.lock"]:::meta
    end

    %% =========================
    %% Runtime interactions
    %% =========================
    U -->|"enter start_word,target_word,params"| APP
    APP -->|"init: load data + index"| IDX
    IDX -->|"read"| EMBS
    IDX -->|"read"| W2I
    IDX -->|"build/load"| FAISSIDX
    APP -->|"invoke search(start,target)"| ASTAR
    APP -.->|"or choose strategy"| GREEDY

    ASTAR -->|"loop: expand current node"| NBR
    GREEDY -->|"expand current node"| NBR
    NBR -->|"topK(query_vector)"| FAISSIDX
    FAISSIDX -->|"neighbor_ids + similarities"| NBR
    NBR -->|"id↔word,vector lookup"| LOOKUP
    LOOKUP -->|"read"| EMBS
    LOOKUP -->|"read"| W2I
    ASTAR -->|"heuristic h(n): cosine distance to target vector"| LOOKUP
    ASTAR -->|"return ladder path (list of words)+diagnostics"| APP
    APP -->|"render ladder"| U

    NBR --- NOTE_IMPLICIT

    %% =========================
    %% Offline preprocessing interactions
    %% =========================
    WORDS -.->|"input vocabulary"| EMBJOB
    EMBJOB -.->|"encode words"| STLIB
    EMBJOB -.->|"write"| EMBS
    EMBJOB -.->|"write"| W2I
    EMBJOB -.->|"optional: normalize vectors"| NUMPY
    IDX -.->|"offline build (optional)"| FAISSIDX
    OFFNOTE --- EMBJOB

    %% =========================
    %% Container/build metadata links
    %% =========================
    DKR --- APP
    DKR --- REQ
    DKR --- PYPROJ
    DKR --- UVLOCK

    %% =========================
    %% Click events (component_mapping)
    %% =========================
    click APP "https://github.com/thameenas/semantic-word-ladder/blob/main/app.py"
    click EMBJOB "https://github.com/thameenas/semantic-word-ladder/blob/main/src/embed.py"
    click IDX "https://github.com/thameenas/semantic-word-ladder/blob/main/src/index.py"
    click NBR "https://github.com/thameenas/semantic-word-ladder/blob/main/src/neighbours.py"
    click ASTAR "https://github.com/thameenas/semantic-word-ladder/blob/main/src/astar.py"
    click GREEDY "https://github.com/thameenas/semantic-word-ladder/blob/main/src/greedy.py"
    click DATA_DIR "https://github.com/thameenas/semantic-word-ladder/tree/main/data/"
    click WORDS "https://github.com/thameenas/semantic-word-ladder/blob/main/data/words.txt"
    click EMBS "https://github.com/thameenas/semantic-word-ladder/blob/main/data/embeddings.npy"
    click W2I "https://github.com/thameenas/semantic-word-ladder/blob/main/data/word_to_idx.json"
    click DKR "https://github.com/thameenas/semantic-word-ladder/tree/main/Dockerfile"
    click REQ "https://github.com/thameenas/semantic-word-ladder/blob/main/requirements.txt"
    click PYPROJ "https://github.com/thameenas/semantic-word-ladder/blob/main/pyproject.toml"
    click UVLOCK "https://github.com/thameenas/semantic-word-ladder/blob/main/uv.lock"

    %% =========================
    %% Styles
    %% =========================
    classDef ui fill:#1f77b4,stroke:#0b3d66,color:#ffffff,stroke-width:1.5px
    classDef engine fill:#2a6fdb,stroke:#123a7a,color:#ffffff,stroke-width:1.2px
    classDef data fill:#7b4bb7,stroke:#3c1f66,color:#ffffff,stroke-width:1.2px
    classDef offline fill:#2e8b57,stroke:#145a32,color:#ffffff,stroke-width:1.2px
    classDef external fill:#6c757d,stroke:#343a40,color:#ffffff,stroke-width:1.1px
    classDef runtime fill:#0aa2c0,stroke:#045a6b,color:#ffffff,stroke-width:1.2px
    classDef meta fill:#ffb703,stroke:#8a5a00,color:#1a1a1a,stroke-width:1.1px
    classDef note fill:#f8f9fa,stroke:#adb5bd,color:#212529,stroke-dasharray: 4 3,stroke-width:1px
```
