# adw777/finetuneembed architecture

> Architecture diagram of the GitHub repository adw777/finetuneembed, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/adw777/finetuneembed
- Repository: https://github.com/adw777/finetuneembed

## Overview

##### Project type and purpose
- This is a specialized machine-learning pipeline (an open-source fine-tuning tool) focused on embedding model adaptation for Retrieval-Augmented Generation (RAG) using Matryoshka Representation Learning.
- Its end goal is to fine-tune pre-trained embedding models (e.g., BAAI/bge-base, Snowflake/snowflake-arctic, intfloat/multilingual-e5) into multi-dimensional representations.

##### File structure
- Top-level scripts:
- finetune_embed.py – orchestrates the full pipeline via class `EmbeddingFineTuner`.
- test_model.py – runs evaluation on held-out data.
- Data subfolder:
- dataPrep.py – cleans/formats raw JSON.
- jsonTojsonl.py – converts JSON to JSONL for training.
- sebi_qa_pairs.json(.l) – the anchor-positive QA pairs dataset.
- Config/dependencies:
- .env.example – placeholder for Hugging Face token.
- requirements.txt – Python libs (transformers, accelerate, torch, faiss, etc.).

##### Extract architecture insights from README
- Pipeline stages: data prep → dataset ingestion → model loading → multi-dimensional loss computation → training → evaluation → model export.
- Key components: MatryoshkaLoss, MultipleNegativesRankingLoss, optimizer (AdamW Fused), LR scheduler (cosine decay), precision modes (BF16/TF32), hardware (GPU ≥12 GB).
- Outputs: fine-tuned model directory, performance metrics across each embedding dimension.

##### Components
a. Data Layer
- Raw JSON dataset (data/sebi_qa_pairs.json)
- DataPrep module (data/dataPrep.py)
- Conversion module (data/jsonTojsonl.py)
b. Ingestion & Dataset Builder
- Hugging Face Dataset loader (in EmbeddingFineTuner)
- Anchor-positive pair batching
c. Model Layer
- Base embedding model (Hugging Face model IDs)
- EmbeddingFineTuner class (finetune_embed.py)
d. Loss & Training Engine
- MatryoshkaLoss + MultipleNegativesRankingLoss
- Optimizer (AdamW Fused), LR Scheduler, Precision Manager
- Distributed/DataParallel orchestration (accelerate)
e. Evaluation Module
- test_model.py logic (NDCG@10, cosine similarity scoring, IR metrics)
f. Persistence & Output
- Serialized model directory (./bge-base-finetuned-matryoshka/)
- Logs & metrics outputs

##### Relationships and interactions
- JSONL → DatasetBuilder → Batches
- Batches + Base Model → Loss Engine → Gradient updates → Model checkpoint
- Checkpoints → Evaluation → Metric reports
- Final checkpoint → Exported model folder

##### Architectural patterns & design principles
- Pipeline pattern: clear sequence of transformations
- Modular separation: data prep, model fine-tuning, evaluation
- Extensibility: swapping base models, loss functions, dataset IDs
- Hardware abstraction: huggingface-accelerate for multi-GPU

##### Technologies, frameworks & libraries
- Python + PyTorch
- Transformers (Hugging Face)
- accelerate
- faiss (optional nearest-neighbor checks)
- JSON/JSONL for data interchange

##### Essential elements
- Clear component names matching file/module names
- Hardware/resource annotation (GPU pool)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        RawJSON["Raw JSON Dataset"]:::data
        DataPrep["DataPrep Module"]:::proc
        JSONL["JSONL Conversion Module"]:::proc
    end
    RawJSON -->|"JSON"| DataPrep -->|"cleaned JSON"| JSONL

    %% Ingestion & Dataset Builder
    subgraph "Ingestion & Dataset Builder"
        DatasetIngest["HuggingFace Dataset Loader & Batching"]:::proc
    end
    JSONL -->|"JSONL"| DatasetIngest

    %% External Services & Resources
    HFHub["HuggingFace Hub"]:::external
    GPU["GPU Cluster"]:::external

    %% Training Engine
    subgraph "Training Engine"
        ModelLoader["Base Embedding Model\nLoading"]:::proc
        LossEngine["Loss & Training Engine\n(MatryoshkaLoss, MNRLoss,\nOptimizer, Scheduler,\nPrecision Manager)"]:::proc
        Checkpoint["Model Checkpoints"]:::data
    end
    DatasetIngest --> ModelLoader --> LossEngine -->|"Updated Weights"| Checkpoint
    HFHub -->|"Fetch Models/Datasets"| ModelLoader
    GPU -->|"Compute"| LossEngine

    %% Evaluation Module
    subgraph "Evaluation Module"
        Evaluation["Evaluation Module\n(NDCG@10, Cosine Scoring)"]:::proc
        Metrics["Metrics Reports"]:::output
    end
    Checkpoint --> Evaluation --> Metrics

    %% Persistence & Output
    subgraph "Persistence & Output"
        ExportModel["Exported Model Directory"]:::output
    end
    Checkpoint -->|"Final Checkpoint"| ExportModel

    %% Configuration & Dependencies
    subgraph "Configuration & Dependencies"
        Env[".env.example"]:::config
        Req["requirements.txt"]:::config
    end

    %% Click Events
    click RawJSON "https://github.com/adw777/finetuneembed/blob/main/data/sebi_qa_pairs.json"
    click DataPrep "https://github.com/adw777/finetuneembed/blob/main/data/dataPrep.py"
    click JSONL "https://github.com/adw777/finetuneembed/blob/main/data/jsonTojsonl.py"
    click DatasetIngest "https://github.com/adw777/finetuneembed/blob/main/finetune_embed.py"
    click ModelLoader "https://github.com/adw777/finetuneembed/blob/main/finetune_embed.py"
    click LossEngine "https://github.com/adw777/finetuneembed/blob/main/finetune_embed.py"
    click Evaluation "https://github.com/adw777/finetuneembed/blob/main/test_model.py"
    click Env "https://github.com/adw777/finetuneembed/blob/main/.env.example"
    click Req "https://github.com/adw777/finetuneembed/blob/main/requirements.txt"

    %% Styles
    classDef data fill:#add8e6,stroke:#000,stroke-width:1px
    classDef proc fill:#90ee90,stroke:#000,stroke-width:1px
    classDef external fill:#ffa500,stroke:#000,stroke-width:1px
    classDef output fill:#dda0dd,stroke:#000,stroke-width:1px
    classDef config fill:#add8e6,stroke:#000,stroke-width:1px
```
