How aldojacopovirno/semantic-retrieval-engine works
Project Type and Purpose
• This is a Python-based open-source semantic search engine.
• Purpose: load a document corpus, produce BERT embeddings, compute TF-IDF scores, combine relevance signals, and display ranked results.
File Structure
• Top-level: src/ contains all application code; no separate frontend/backend.
• Pipeline modules:
• loaders/document_loader.py – ingest and preprocess text files.
• embedders/bert_embedder.py – compute semantic embeddings via a pretrained BERT model.
• scorers/tfidf_scorer.py – calculate keyword relevance scores (TF-IDF).
• calculators/similarity_calculator.py – compute embedding distance/similarity metrics.
• calculators/relevance_calculator.py – fuse TF-IDF and embedding scores into a final ranking.
• displayers/results_displayer.py – format and output results (console + report).
• main.py – orchestrates the entire pipeline.
• Configuration/deployment: requirements.txt, setup.py.
Extract Architecture Insights from the README
• Hybrid ranking system that merges semantic (BERT) and lexical (TF-IDF) scores.
• Outputs detailed metrics and a report.
• Invoked via Python API (main()) or CLI (python -m semantic_retrieval_engine.src.main).
Main Components
a. Data Source
• Document Folder (raw text files)
b. Document Loader
• Reads files, tokenizes/preprocesses.
c. Embedding Service
• BERT Model (via Hugging Face Transformers)
d. TF-IDF Scorer
• Scikit-learn or similar library computing term frequencies.
e. Similarity Calculator
• Cosine similarity or distance on embedding vectors.
f. Relevance Calculator
• Weighted combination of semantic and lexical scores.
g. Results Displayer
• Console output
• HTML/markdown report generator
h. External Services
• Pretrained BERT weights (downloaded at runtime)
• Logging/metrics (optional)
Relationships and Data Flow
• Loader outputs tokens/text → two parallel paths: Embedding Service and TF-IDF Scorer.
• Embedding Service → embedding vectors → Similarity Calculator.
• TF-IDF Scorer → keyword vectors/scores → Relevance Calculator.
• Similarity Calculator → scores → Relevance Calculator.
• Relevance Calculator → final ranked list → Results Displayer.
• Displayer → Console & Report (file).
Architectural Patterns and Principles
• Pipeline/chain-of-responsibility: each module performs a distinct transformation.
• Separation of Concerns: loaders, embedders, scorers, calculators, displayers are decoupled.
• Modular and extensible: new scorers or embedders can be added under their respective packages.
• Single Responsibility per component.
Technologies and Frameworks
• Python 3.x
• Hugging Face Transformers (BERT)
• scikit-learn (TF-IDF)
• argparse or click (CLI parsing inside main.py)
• Standard logging library