# saibalpatrads/word-embeddings-app architecture

> Architecture diagram of the GitHub repository saibalpatrads/word-embeddings-app, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saibalpatrads/word-embeddings-app
- Repository: https://github.com/saibalpatrads/word-embeddings-app

## Overview

##### Main Components
- Data Storage: Represent the data directory that holds the input CSV file (phrases.csv), preprocessed data (phrase_vectors.pkl and phrase_similarity.pkl), and the large pre-trained Word2Vec model (GoogleNews-vectors-negative300.bin). Also include the optimized word vectors file (word_vectors.kv).
- Core Logic / Processing: Highlight the source code (under src/) which encapsulates key processing steps:
- Data Preprocessing (src/data_preprocessing.py): Cleans and prepares the CSV data.
- Embedding Utilities (src/embeddings_utils.py): Converts phrases to embeddings and computes similarities.
- Phrase Matching (src/phrase_matching.py): Implements the matching logic (including fuzzy matching via Levenshtein distance).
- Pipeline (src/pipeline.py): Automates loading, cleaning, embedding generation, and similarity score creation.
- Interfaces:
- CLI Interface: Represented by app.py and app_2.0.py. This component is used for command-line interactions.
- API Interface: Represented by api.py (based on FastAPI), which provides a web service endpoint for phrase matching.
- Supporting Components:
- Testing: Unit tests are included in src/unit_test.py (and as shown in the README under tests/ for phrase matching and embedding tests).
- Configuration & Environment Setup: Display configuration files like environment.yml, requirements.txt, .gitignore, and LICENSE.

##### Relationships and Interactions
- After preprocessing, the Pipeline module orchestrates the process by getting clean data, generating embeddings, and saving them back into the data directory.
- The Phrase Matching module uses the embeddings generated by the Embedding Utilities to compare input phrases (from either CLI or API) against the stored phrase embeddings.
- The interfaces (CLI via app.py/app_2.0.py and API via api.py) act as entry points. They call the core logic (phrase matching and data handling modules) and return results to the user.
- Testing modules check the integrity of both the embeddings and the matching logic, ensuring that any changes do not break the core functions.

##### Architectural Patterns and Design Principles
- Separation of Concerns: The project clearly separates data handling, core algorithmic logic, and interface layers (CLI versus API).
- Modularity: Each source file in the src/ directory is responsible for a distinct piece of functionality (preprocessing, embedding conversion, phrase matching, and pipeline orchestration).
- Extensibility: The API module (api.py) is an extensible entry point, supporting potential future integrations (another candidate for microservice architecture if needed).
- Testing: Inclusion of unit tests underlines an emphasis on reliability and maintainability.

##### Incorporate Relevant Technologies and Frameworks
- Python as the main programming language.
- FastAPI for the web-service component.
- Pre-trained Word2Vec model (GoogleNews-vectors-negative300.bin) for embeddings.
- Libraries for data processing and numerical operations (typically NumPy, SciPy, or similar, as implied by the use of embeddings and similarity computations).
- Use of Conda and pip for managing environments and dependencies.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Storage Layer
    subgraph "Data Storage"
        DS["Data Storage (CSV, PKL, Model Files)"]:::datastore
    end

    %% Core Processing Layer
    subgraph "Core Processing"
        DP["Data Preprocessing Module"]:::processing
        EU["Embedding Utilities"]:::processing
        PL["Processing Pipeline"]:::processing
        PM["Phrase Matching Engine"]:::processing
    end

    %% Interfaces Layer
    subgraph "Interfaces"
        CLI1["CLI Interface (app.py)"]:::interface
        CLI2["CLI Interface (app_2.0.py)"]:::interface
        API["Web API Interface (api.py)"]:::interface
    end

    %% Testing Layer
    subgraph "Testing"
        UT["Unit Tests (src/unit_test.py)"]:::testing
    end

    %% Configuration Layer
    subgraph "Configuration"
        ENV["Config: environment.yml"]:::config
        REQ["Config: requirements.txt"]:::config
        GI["Config: .gitignore"]:::config
        LIC["Config: LICENSE"]:::config
    end

    %% Data flow connections
    DS -->|"feeds"| DP
    DP -->|"cleans data"| EU
    EU -->|"generates embeddings"| PL
    PL -->|"orchestrates matching"| PM

    %% Interface interactions
    CLI1 -->|"triggers"| PM
    CLI2 -->|"triggers"| PM
    API  -->|"calls"| PM

    %% Testing interactions
    UT -->|"verifies"| EU
    UT -->|"verifies"| PM

    %% Configuration influences (environment setup)
    ENV -->|"configs"| DP
    REQ -->|"configs"| DP
    GI  -->|"configs"| DP
    LIC -->|"configs"| DP

classDef datastore fill:#FFEEAD,stroke:#333,stroke-width:2px;
classDef processing fill:#CDEACC,stroke:#333,stroke-width:2px;
classDef interface fill:#D0E8F2,stroke:#333,stroke-width:2px;
classDef testing fill:#FAD4D4,stroke:#333,stroke-width:2px;
classDef config fill:#E6D4F2,stroke:#333,stroke-width:2px;

click DS "https://github.com/saibalpatrads/word-embeddings-app/tree/main/data/"
click DP "https://github.com/saibalpatrads/word-embeddings-app/blob/main/src/data_preprocessing.py"
click EU "https://github.com/saibalpatrads/word-embeddings-app/blob/main/src/embeddings_utils.py"
click PL "https://github.com/saibalpatrads/word-embeddings-app/blob/main/src/pipeline.py"
click PM "https://github.com/saibalpatrads/word-embeddings-app/blob/main/src/phrase_matching.py"
click CLI1 "https://github.com/saibalpatrads/word-embeddings-app/blob/main/app.py"
click CLI2 "https://github.com/saibalpatrads/word-embeddings-app/blob/main/app_2.0.py"
click API "https://github.com/saibalpatrads/word-embeddings-app/blob/main/api.py"
click UT "https://github.com/saibalpatrads/word-embeddings-app/blob/main/src/unit_test.py"
click ENV "https://github.com/saibalpatrads/word-embeddings-app/blob/main/environment.yml"
click REQ "https://github.com/saibalpatrads/word-embeddings-app/blob/main/requirements.txt"
click GI "https://github.com/saibalpatrads/word-embeddings-app/blob/main/.gitignore"
click LIC "https://github.com/saibalpatrads/word-embeddings-app/tree/main/LICENSE"
```
