How saibalpatrads/word-embeddings-app works
Main Components
• Data Storage: Represent the data directory that holds the input CSV file (phrases.csv), preprocessed data (phrase_vectors.pkl and phrase_similarity.pkl), and the large pre-trained Word2Vec model (GoogleNews-vectors-negative300.bin). Also include the optimized word vectors file (word_vectors.kv).
• Core Logic / Processing: Highlight the source code (under src/) which encapsulates key processing steps:
• Data Preprocessing (src/data_preprocessing.py): Cleans and prepares the CSV data.
• Embedding Utilities (src/embeddings_utils.py): Converts phrases to embeddings and computes similarities.
• Phrase Matching (src/phrase_matching.py): Implements the matching logic (including fuzzy matching via Levenshtein distance).
• Pipeline (src/pipeline.py): Automates loading, cleaning, embedding generation, and similarity score creation.
• Interfaces:
• CLI Interface: Represented by app.py and app_2.0.py. This component is used for command-line interactions.
• API Interface: Represented by api.py (based on FastAPI), which provides a web service endpoint for phrase matching.
• Supporting Components:
• Testing: Unit tests are included in src/unit_test.py (and as shown in the README under tests/ for phrase matching and embedding tests).
• Configuration & Environment Setup: Display configuration files like environment.yml, requirements.txt, .gitignore, and LICENSE.
Relationships and Interactions
• After preprocessing, the Pipeline module orchestrates the process by getting clean data, generating embeddings, and saving them back into the data directory.
• The Phrase Matching module uses the embeddings generated by the Embedding Utilities to compare input phrases (from either CLI or API) against the stored phrase embeddings.
• The interfaces (CLI via app.py/app_2.0.py and API via api.py) act as entry points. They call the core logic (phrase matching and data handling modules) and return results to the user.
• Testing modules check the integrity of both the embeddings and the matching logic, ensuring that any changes do not break the core functions.
Architectural Patterns and Design Principles
• Separation of Concerns: The project clearly separates data handling, core algorithmic logic, and interface layers (CLI versus API).
• Modularity: Each source file in the src/ directory is responsible for a distinct piece of functionality (preprocessing, embedding conversion, phrase matching, and pipeline orchestration).
• Extensibility: The API module (api.py) is an extensible entry point, supporting potential future integrations (another candidate for microservice architecture if needed).
• Testing: Inclusion of unit tests underlines an emphasis on reliability and maintainability.
Incorporate Relevant Technologies and Frameworks
• Python as the main programming language.
• FastAPI for the web-service component.
• Pre-trained Word2Vec model (GoogleNews-vectors-negative300.bin) for embeddings.
• Libraries for data processing and numerical operations (typically NumPy, SciPy, or similar, as implied by the use of embeddings and similarity computations).
• Use of Conda and pip for managing environments and dependencies.