How pdrm83/sent2vec works
Project type and purpose
• This is an open-source Python library (sent2vec) for computing sentence embeddings using either BERT (via HuggingFace/transformers + torch) or Word2Vec (via gensim).
• It provides two key modules:
• Splitter (text → token lists)
• Vectorizer (token lists → sentence vectors by aggregating word vectors)
File structure
• Top-level scripts/configuration
• build_package.sh, setup.py, manifest.in, requirements.txt → packaging & distribution
• .circleci/, .github/ → CI/CD and funding metadata
• Core source (sent2vec/)
• constants.py → configuration & default stop-words
• splitter.py → logic to clean & split sentences
• vectorizer.py → loads pretrained models & aggregates embeddings
• Tests (test/)
• test_splitter.py, test_vectorizer.py → unit tests
• test/dataset/ → sample text files for integration testing
README for additional insights
• Describes two usage modes: BERT-based (“distilbert-base-uncased” or other transformers models) and Word2Vec-based (loading .txt/.gz/.bin into gensim).
• Highlights API: Vectorizer.run(sentences, remove_stop_words, add_stop_words) → populates vectorizer.vectors.
• Dependencies: numpy, spacy, torch, transformers, gensim.
Main components & their interactions
a. Components
1. Client code (your script or application that calls Vectorizer)
2. Splitter
3. Embedding backends
• BERT Transformer model (HuggingFace + torch)
• Word2Vec model (gensim)
4. Ensemble/aggregation logic (e.g., average)
5. Output (vectors)
6. Packaging/tests (CI pipelines)
b. Interactions
• Client calls Vectorizer.run() → Splitter.tokenize() → choose backend → model.encode() → aggregator → vectors
• Tests exercise Splitter and Vectorizer modules with sample data.
• CI/CD invokes tests and packaging scripts.
c. Architectural patterns & principles
• Single-responsibility: splitter vs. vectorizer
• Strategy pattern: interchangeable embedding backends (specified by pretrained_weights parameter)
• Configuration via constants and parameters
d. Key technologies/frameworks
• Python packaging (setup.py)
• transformers + torch for BERT
• gensim for Word2Vec
• spacy for stop-word lists & tokenization customizations
• CI with CircleCI / GitHub Actions