# pdrm83/sent2vec architecture

> Architecture diagram of the GitHub repository pdrm83/sent2vec, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pdrm83/sent2vec
- Repository: https://github.com/pdrm83/sent2vec

## Overview

##### Project type and purpose
- This is an open-source Python library (sent2vec) for computing sentence embeddings using either BERT (via HuggingFace/transformers + torch) or Word2Vec (via gensim).
- It provides two key modules:
- Splitter (text → token lists)
- Vectorizer (token lists → sentence vectors by aggregating word vectors)

##### File structure
- Top-level scripts/configuration
- build_package.sh, setup.py, manifest.in, requirements.txt → packaging & distribution
- .circleci/, .github/ → CI/CD and funding metadata
- Core source (sent2vec/)
- constants.py → configuration & default stop-words
- splitter.py → logic to clean & split sentences
- vectorizer.py → loads pretrained models & aggregates embeddings
- Tests (test/)
- test_splitter.py, test_vectorizer.py → unit tests
- test/dataset/ → sample text files for integration testing

##### README for additional insights
- Describes two usage modes: BERT-based (“distilbert-base-uncased” or other transformers models) and Word2Vec-based (loading .txt/.gz/.bin into gensim).
- Highlights API: Vectorizer.run(sentences, remove_stop_words, add_stop_words) → populates vectorizer.vectors.
- Dependencies: numpy, spacy, torch, transformers, gensim.

##### Main components & their interactions
a. Components
1. Client code (your script or application that calls Vectorizer)
2. Splitter
3. Embedding backends
- BERT Transformer model (HuggingFace + torch)
- Word2Vec model (gensim)
4. Ensemble/aggregation logic (e.g., average)
5. Output (vectors)
6. Packaging/tests (CI pipelines)
b. Interactions
- Client calls Vectorizer.run() → Splitter.tokenize() → choose backend → model.encode() → aggregator → vectors
- Tests exercise Splitter and Vectorizer modules with sample data.
- CI/CD invokes tests and packaging scripts.
c. Architectural patterns & principles
- Single-responsibility: splitter vs. vectorizer
- Strategy pattern: interchangeable embedding backends (specified by pretrained_weights parameter)
- Configuration via constants and parameters
d. Key technologies/frameworks
- Python packaging (setup.py)
- transformers + torch for BERT
- gensim for Word2Vec
- spacy for stop-word lists & tokenization customizations
- CI with CircleCI / GitHub Actions

## Diagram (Mermaid)

```mermaid
flowchart TD
    Client["Client Code"]:::core
    Client -->|"calls run()"| Vectorizer

    subgraph "Core Code"
        Vectorizer["Vectorizer Module"]:::core
        Splitter["Splitter Module"]:::core
        Constants["Configuration & Defaults"]:::core
        Vectorizer --> Splitter
        Vectorizer --> Constants

        subgraph "Backend Strategy"
            BERT["BERT Model\n(transformers + torch)"]:::external
            Word2Vec["Word2Vec Model\ngensim"]:::external
        end

        Splitter -->|"tokens"| BERT
        Splitter -->|"tokens"| Word2Vec

        HF["HuggingFace Hub"]:::external
        FS["Local .bin/.gz files"]:::external
        HF --> BERT
        FS --> Word2Vec

        Aggregator["Ensemble Aggregator"]:::core
        BERT -->|"raw embeddings"| Aggregator
        Word2Vec -->|"raw embeddings"| Aggregator
        Aggregator -->|"vectors"| Client
    end

    subgraph "CI/CD & Testing"
        Commit[/"On Commit"/]:::ci
        Commit --> Tests[/"Run Tests"/]:::ci

        subgraph "Testing"
            TestSplitter["test_splitter.py"]:::ci
            TestVectorizer["test_vectorizer.py"]:::ci
            Dataset["test/dataset/"]:::ci
        end
        Tests --> TestSplitter
        Tests --> TestVectorizer
        Tests --> Dataset

        Tests --> Packaging[/"Package & Publish"/]:::ci

        subgraph "Packaging"
            Build["build_package.sh"]:::ci
            Setup["setup.py"]:::ci
            Manifest["manifest.in"]:::ci
            Requirements["requirements.txt"]:::ci
            CircleCI[".circleci/config.yml"]:::ci
            Funding[".github/FUNDING.yml"]:::ci
        end

        Packaging --> Build
        Packaging --> Setup
        Packaging --> Manifest
        Packaging --> Requirements
        Packaging --> CircleCI
        Packaging --> Funding

        Packaging -->|"publish to PyPI"| PyPI["PyPI"]:::external
    end

    classDef core fill:#cce5ff,stroke:#004085
    classDef external fill:#d4edda,stroke:#155724
    classDef ci fill:#fff3cd,stroke:#856404

    click Splitter "https://github.com/pdrm83/sent2vec/blob/master/sent2vec/splitter.py"
    click Vectorizer "https://github.com/pdrm83/sent2vec/blob/master/sent2vec/vectorizer.py"
    click Constants "https://github.com/pdrm83/sent2vec/blob/master/sent2vec/constants.py"
    click Build "https://github.com/pdrm83/sent2vec/blob/master/build_package.sh"
    click Setup "https://github.com/pdrm83/sent2vec/blob/master/setup.py"
    click Manifest "https://github.com/pdrm83/sent2vec/blob/master/manifest.in"
    click Requirements "https://github.com/pdrm83/sent2vec/blob/master/requirements.txt"
    click CircleCI "https://github.com/pdrm83/sent2vec/blob/master/.circleci/config.yml"
    click Funding "https://github.com/pdrm83/sent2vec/blob/master/.github/FUNDING.yml"
    click TestSplitter "https://github.com/pdrm83/sent2vec/blob/master/test/test_splitter.py"
    click TestVectorizer "https://github.com/pdrm83/sent2vec/blob/master/test/test_vectorizer.py"
    click Dataset "https://github.com/pdrm83/sent2vec/tree/master/test/dataset/"
```
