# aldojacopovirno/semantic-retrieval-engine architecture

> Architecture diagram of the GitHub repository aldojacopovirno/semantic-retrieval-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aldojacopovirno/semantic-retrieval-engine
- Repository: https://github.com/aldojacopovirno/semantic-retrieval-engine

## Overview

##### Project Type and Purpose
- This is a Python-based open-source semantic search engine.
- Purpose: load a document corpus, produce BERT embeddings, compute TF-IDF scores, combine relevance signals, and display ranked results.

##### File Structure
- Top-level: `src/` contains all application code; no separate frontend/backend.
- Pipeline modules:
- `loaders/document_loader.py` – ingest and preprocess text files.
- `embedders/bert_embedder.py` – compute semantic embeddings via a pretrained BERT model.
- `scorers/tfidf_scorer.py` – calculate keyword relevance scores (TF-IDF).
- `calculators/similarity_calculator.py` – compute embedding distance/similarity metrics.
- `calculators/relevance_calculator.py` – fuse TF-IDF and embedding scores into a final ranking.
- `displayers/results_displayer.py` – format and output results (console + report).
- `main.py` – orchestrates the entire pipeline.
- Configuration/deployment: `requirements.txt`, `setup.py`.

##### Extract Architecture Insights from the README
- Hybrid ranking system that merges semantic (BERT) and lexical (TF-IDF) scores.
- Outputs detailed metrics and a report.
- Invoked via Python API (`main()`) or CLI (`python -m semantic_retrieval_engine.src.main`).

##### Main Components
a. Data Source
- Document Folder (raw text files)
b. Document Loader
- Reads files, tokenizes/preprocesses.
c. Embedding Service
- BERT Model (via Hugging Face Transformers)
d. TF-IDF Scorer
- Scikit-learn or similar library computing term frequencies.
e. Similarity Calculator
- Cosine similarity or distance on embedding vectors.
f. Relevance Calculator
- Weighted combination of semantic and lexical scores.
g. Results Displayer
- Console output
- HTML/markdown report generator
h. External Services
- Pretrained BERT weights (downloaded at runtime)
- Logging/metrics (optional)

##### Relationships and Data Flow
- Loader outputs tokens/text → two parallel paths: Embedding Service and TF-IDF Scorer.
- Embedding Service → embedding vectors → Similarity Calculator.
- TF-IDF Scorer → keyword vectors/scores → Relevance Calculator.
- Similarity Calculator → scores → Relevance Calculator.
- Relevance Calculator → final ranked list → Results Displayer.
- Displayer → Console & Report (file).

##### Architectural Patterns and Principles
- Pipeline/chain-of-responsibility: each module performs a distinct transformation.
- Separation of Concerns: loaders, embedders, scorers, calculators, displayers are decoupled.
- Modular and extensible: new scorers or embedders can be added under their respective packages.
- Single Responsibility per component.

##### Technologies and Frameworks
- Python 3.x
- Hugging Face Transformers (BERT)
- scikit-learn (TF-IDF)
- argparse or click (CLI parsing inside `main.py`)
- Standard logging library

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Source
    DocumentFolder["Document Folder\n(raw text files)"]:::external

    %% Main Orchestrator
    Main["Main Orchestrator\n(src/main.py)"]:::external

    %% Data Ingestion Layer
    subgraph "Data Ingestion"
        DL["Document Loader"]:::ingestion
    end

    %% Feature Generation Layer
    subgraph "Feature Generation"
        BE["BERT Embedder"]:::feature
        TS["TF-IDF Scorer"]:::feature
    end

    %% Calculation/Fusion Layer
    subgraph "Calculation/Fusion"
        SC["Similarity Calculator"]:::calc
        RC["Relevance Calculator"]:::calc
    end

    %% Output Layer
    subgraph "Output"
        RD["Results Displayer"]:::output
    end

    %% Configuration
    subgraph "Configuration"
        Req["requirements.txt"]:::config
        Setup["setup.py"]:::config
    end

    %% External Models
    BERTModel["Pretrained BERT Weights"]:::external

    %% Data Flow
    Main -->|"load pipeline"| DocumentFolder
    DocumentFolder -->|"raw text"| DL
    DL -->|"tokens/text"| BE
    DL -->|"tokens/text"| TS
    BE -->|"embedding vectors"| SC
    TS -->|"TF-IDF scores"| RC
    SC -->|"similarity scores"| RC
    RC -->|"ranked list"| RD
    RD -->|"console & report"| Main

    %% Optional Dependencies
    BE -.->|"download at runtime"| BERTModel

    %% Click Events
    click DL "https://github.com/aldojacopovirno/semantic-retrieval-engine/blob/main/src/loaders/document_loader.py"
    click BE "https://github.com/aldojacopovirno/semantic-retrieval-engine/blob/main/src/embedders/bert_embedder.py"
    click TS "https://github.com/aldojacopovirno/semantic-retrieval-engine/blob/main/src/scorers/tfidf_scorer.py"
    click SC "https://github.com/aldojacopovirno/semantic-retrieval-engine/blob/main/src/calculators/similarity_calculator.py"
    click RC "https://github.com/aldojacopovirno/semantic-retrieval-engine/blob/main/src/calculators/relevance_calculator.py"
    click RD "https://github.com/aldojacopovirno/semantic-retrieval-engine/blob/main/src/displayers/results_displayer.py"
    click Main "https://github.com/aldojacopovirno/semantic-retrieval-engine/blob/main/src/main.py"
    click Req "https://github.com/aldojacopovirno/semantic-retrieval-engine/blob/main/requirements.txt"
    click Setup "https://github.com/aldojacopovirno/semantic-retrieval-engine/blob/main/setup.py"

    %% Styles
    classDef ingestion fill:#a8e6a3,stroke:#333,stroke-width:1px
    classDef feature fill:#a3c4f3,stroke:#333,stroke-width:1px
    classDef calc fill:#f3b89c,stroke:#333,stroke-width:1px
    classDef output fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef external fill:#fff2a8,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef config fill:#e0e0e0,stroke:#666,stroke-width:1px,stroke-dasharray: 2 2
```
