# dnau15/spbrag architecture

> Architecture diagram of the GitHub repository dnau15/spbrag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dnau15/spbrag
- Repository: https://github.com/dnau15/spbrag

## Overview

##### Main Components
a. Data & Storage
- Raw data directory (`data/`) and Milvus vector DB (configured via CLI args).
- Model artifacts directory (`models/`).
b. Data Pipeline
- Data loading (`data_loading.py`)
- Preprocessing (`data_preprocessing.py`)
- Dataset creation (`dataset_creation.py`, `datasets.py`)
c. Training
- BERT fine-tuning module (`training/fine_tune_bert.py`)
d. Retrieval & Generation
- Retrieval engine (`retrieval/rag.py`)
- LL M inference via Ollama/Huggingface
e. Evaluation
- Evaluator (`evaluation/evaluator.py`)
- Metrics calculation (`metrics.py`)
f. Visualization
- Plotting graphs (`visualization/plot_graphs.py`)
g. Orchestration & Configuration
- Scripts (`run.sh`, `setup.sh`)
- Environment (`.env`, micromamba, PYTHONPATH)
- Notebooks for prototyping (`notebooks/`)

##### Architectural Patterns & Principles
- Modular ML pipeline: clear separation of data processing, model training, retrieval, inference, evaluation.
- Config-driven: CLI args and `.env` allow swapping LLMs (Ollama vs. Huggingface) or DB URIs.
- Extensible evaluation: evaluation and metrics modules decoupled for new metrics.
- Reproducible environment: micromamba, setup scripts, PYTHONPATH.

##### Incorporate Key Technologies & Frameworks
- Python modules under `src/rag_system/…`
- Huggingface Transformers & Sentence-Transformers
- Milvus for vector store
- Ollama for Llama-3.2 inference
- BERT (google-bert/bert-base-uncased)
- Micromamba for env management

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration & Environment
    Config["Environment & Config"]:::config
    Config -->|"provides settings"| DataPipeline
    Config -->|"provides settings"| Training
    Config -->|"provides settings"| Retrieval
    Config -->|"provides settings"| Evaluation
    Config -->|"provides settings"| Visualization

    %% Data & Storage
    subgraph "Data & Storage"
        direction TB
        RawData["Raw Data (data/)"]:::data
        Milvus["Milvus Vector DB"]:::data
        ModelArtifacts["Model Artifacts (models/)"]:::data
    end

    %% Data Pipeline
    subgraph "Data Pipeline"
        direction TB
        DataLoading["data_loading.py"]:::data
        DataPreprocess["data_preprocessing.py"]:::data
        DatasetCreation["dataset_creation.py"]:::data
        DatasetsDef["datasets.py"]:::data
    end

    %% Training
    subgraph "Training"
        direction TB
        FineTuneBERT["fine_tune_bert.py"]:::training
    end

    %% Retrieval & Generation
    subgraph "Retrieval & Generation"
        direction TB
        RAGEngine["rag.py"]:::inference
        LLM_Ollama["Ollama LLM (llama3.2:3B)"]:::inference
        LLM_HF["HF LLM"]:::inference
    end

    %% Evaluation & Visualization
    subgraph "Evaluation & Visualization"
        direction TB
        Evaluator["evaluator.py"]:::eval
        MetricsCalc["metrics.py"]:::eval
        PlotGraphs["plot_graphs.py"]:::eval
    end

    %% Orchestration & Documentation
    subgraph "Scripts & Docs"
        direction TB
        SetupScript["setup.sh"]:::scripts
        RunScript["run.sh"]:::scripts
        Requirements["requirements.txt"]:::scripts
        Notebooks["notebooks/"]:::scripts
        MetricsDoc["docs/metrics.md"]:::scripts
        WorkflowDoc["docs/workflow.md"]:::scripts
    end

    %% Flows
    RawData -->|load raw data| DataLoading
    DataLoading -->|preprocess data| DataPreprocess
    DataPreprocess -->|create dataset| DatasetCreation
    DatasetCreation -->|define dataset| DatasetsDef
    DatasetsDef -->|preprocessed dataset| FineTuneBERT
    FineTuneBERT -->|trained BERT model| ModelArtifacts
    ModelArtifacts -->|model artifacts| RAGEngine
    RAGEngine -->|index embeddings in Milvus| Milvus
    RAGEngine -->|retrieves top-k contexts| LLM_Ollama
    RAGEngine -->|retrieves top-k contexts| LLM_HF
    LLM_Ollama -->|generated response| Evaluator
    LLM_HF -->|generated response| Evaluator
    Evaluator -->|metrics JSON| MetricsCalc
    MetricsCalc -->|metrics data| PlotGraphs

    %% Click Events
    click DataLoading "https://github.com/dnau15/spbrag/blob/main/src/rag_system/data/data_loading.py"
    click DataPreprocess "https://github.com/dnau15/spbrag/blob/main/src/rag_system/data/data_preprocessing.py"
    click DatasetCreation "https://github.com/dnau15/spbrag/blob/main/src/rag_system/data/dataset_creation.py"
    click DatasetsDef "https://github.com/dnau15/spbrag/blob/main/src/rag_system/data/datasets.py"
    click FineTuneBERT "https://github.com/dnau15/spbrag/blob/main/src/rag_system/training/fine_tune_bert.py"
    click RAGEngine "https://github.com/dnau15/spbrag/blob/main/src/rag_system/retrieval/rag.py"
    click Evaluator "https://github.com/dnau15/spbrag/blob/main/src/rag_system/evaluation/evaluator.py"
    click MetricsCalc "https://github.com/dnau15/spbrag/blob/main/src/rag_system/evaluation/metrics.py"
    click PlotGraphs "https://github.com/dnau15/spbrag/blob/main/src/rag_system/visualization/plot_graphs.py"
    click SetupScript "https://github.com/dnau15/spbrag/blob/main/setup.sh"
    click RunScript "https://github.com/dnau15/spbrag/blob/main/run.sh"
    click Requirements "https://github.com/dnau15/spbrag/blob/main/requirements.txt"
    click Notebooks "https://github.com/dnau15/spbrag/tree/main/notebooks/"
    click RawData "https://github.com/dnau15/spbrag/tree/main/data/"
    click ModelArtifacts "https://github.com/dnau15/spbrag/tree/main/models/"
    click MetricsDoc "https://github.com/dnau15/spbrag/blob/main/docs/metrics.md"
    click WorkflowDoc "https://github.com/dnau15/spbrag/blob/main/docs/workflow.md"

    %% Styles
    classDef data fill:#D0E7FF,stroke:#2B6CB0,color:#1A365D
    classDef training fill:#C6F6D5,stroke:#2F855A,color:#22543D
    classDef inference fill:#FEEBC8,stroke:#DD6B20,color:#7B341E
    classDef eval fill:#E9D8FD,stroke:#6B46C1,color:#44337A
    classDef scripts fill:#EDF2F7,stroke:#A0AEC0,color:#2D3748
    classDef config fill:#F7FAFC,stroke:#718096,color:#2A4365

    %% Subgraph aliases for flows
    class DataPipeline,Training,Retrieval,Evaluation,Visualization config
    class RawData,Milvus,ModelArtifacts data
    class DataLoading,DataPreprocess,DatasetCreation,DatasetsDef data
    class FineTuneBERT training
    class RAGEngine,LLM_Ollama,LLM_HF inference
    class Evaluator,MetricsCalc,PlotGraphs eval
    class SetupScript,RunScript,Requirements,Notebooks,MetricsDoc,WorkflowDoc scripts
```
