# raphaelsty/neural-cherche architecture

> Architecture diagram of the GitHub repository raphaelsty/neural-cherche, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/raphaelsty/neural-cherche
- Repository: https://github.com/raphaelsty/neural-cherche

## Overview

##### Main Components

a. Core Models Layer:
- ColBERT Model
- SparseEmbed Model
- Splade Model
- Base Model Interface

b. Search Operations Layer:
- Retrieval Components (BM25, TfIdf, ColBERT, SparseEmbed, Splade)
- Ranking Components (ColBERT-specific ranking)

c. Training Layer:
- Model-specific training modules
- Loss Functions (Flops, Ranking)

d. Utility Layer:
- Scoring utilities
- Evaluation tools
- Data iteration helpers
- BEIR dataset loader

**[Input Layer]**

**[Pre-processing & Training]**

**[Model Layer]**

**[Search Operations]**

**[Output Layer]**

##### Component Relationships

- Bidirectional interactions where applicable

##### Specific Implementation Details

- Model architectures (ColBERT, SparseEmbed, Splade)
- Training workflows
- Inference paths
- Data preprocessing steps

##### Technical Elements

Technical aspects:
- Device support (CPU, GPU, MPS)
- Integration with external models (Sentence Transformers)
- Data flow for both training and inference
- Model checkpoint handling

##### Integration Points

Interfaces with:
- External pre-trained models
- Dataset inputs
- Storage systems for embeddings
- Evaluation metrics output

##### The pipeline from raw input to final search results

Use standardized symbols:
- Rectangles: Processing components
- Diamonds: Decision points

- Special symbols
- Abbreviations used

Keep the abstraction level appropriate by:
- Focusing on major components and their interactions
- Avoiding implementation details unless crucial
- Highlighting the system's extensibility
- Showing clear boundaries between subsystems

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        Raw[Raw Data]
        BEIR[BEIR Dataset]
    end

    subgraph Training
        Train[Training Module]:::training
        click Train "https://github.com/raphaelsty/neural-cherche/tree/main/neural_cherche/train"
        
        subgraph Losses
            FL[Flops Loss]:::loss
            RL[Ranking Loss]:::loss
            click FL "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/losses/flops.py"
            click RL "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/losses/ranking.py"
        end

        subgraph ModelTraining
            CT[ColBERT Training]:::training
            SET[SparseEmbed Training]:::training
            ST[Splade Training]:::training
            click CT "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/train/train_colbert.py"
            click SET "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/train/train_sparse_embed.py"
            click ST "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/train/train_splade.py"
        end
    end

    subgraph Models
        Base[Base Model Interface]:::model
        click Base "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/models/base.py"

        subgraph ModelImplementations
            ColBERT[ColBERT Model]:::model
            SparseEmbed[SparseEmbed Model]:::model
            Splade[Splade Model]:::model
            click ColBERT "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/models/colbert.py"
            click SparseEmbed "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/models/sparse_embed.py"
            click Splade "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/models/splade.py"
        end
    end

    subgraph SearchOperations
        subgraph Retrieval
            BM25[BM25]:::retrieve
            TfIdf[TfIdf]:::retrieve
            ColBERTR[ColBERT Retrieval]:::retrieve
            SparseEmbedR[SparseEmbed Retrieval]:::retrieve
            SpladeR[Splade Retrieval]:::retrieve
            click BM25 "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/retrieve/bm25.py"
            click TfIdf "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/retrieve/tfidf.py"
            click ColBERTR "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/retrieve/colbert.py"
            click SparseEmbedR "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/retrieve/sparse_embed.py"
            click SpladeR "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/retrieve/splade.py"
        end

        subgraph Ranking
            ColBERTRank[ColBERT Ranking]:::rank
            click ColBERTRank "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/rank/colbert.py"
        end
    end

    subgraph Utils
        Eval[Evaluation Tools]:::util
        DataIter[Data Iteration]:::util
        ColBERTScore[ColBERT Scoring]:::util
        DenseScore[Dense Scoring]:::util
        SparseScore[Sparse Scoring]:::util
        click Eval "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/utils/evaluate.py"
        click DataIter "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/utils/iter.py"
        click ColBERTScore "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/utils/colbert_scores.py"
        click DenseScore "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/utils/dense_scores.py"
        click SparseScore "https://github.com/raphaelsty/neural-cherche/blob/main/neural_cherche/utils/sparse_scores.py"
    end

    Raw --> DataIter
    BEIR --> DataIter
    DataIter --> Train

    Train --> ModelTraining
    FL --> ModelTraining
    RL --> ModelTraining

    ModelTraining --> ModelImplementations
    Base --> ModelImplementations

    ModelImplementations --> Retrieval
    Retrieval --> ColBERTRank
    
    ColBERTScore --> ColBERTRank
    DenseScore --> Retrieval
    SparseScore --> Retrieval

    Retrieval --> Eval
    ColBERTRank --> Eval

    %% Styling
    classDef model fill:#3498db,stroke:#2980b9,color:white
    classDef training fill:#2ecc71,stroke:#27ae60,color:white
    classDef retrieve fill:#e67e22,stroke:#d35400,color:white
    classDef rank fill:#e74c3c,stroke:#c0392b,color:white
    classDef util fill:#9b59b6,stroke:#8e44ad,color:white
    classDef loss fill:#f1c40f,stroke:#f39c12,color:black

    %% Legend
    subgraph Legend
        M[Model]:::model
        T[Training]:::training
        R[Retrieval]:::retrieve
        RK[Ranking]:::rank
        U[Utility]:::util
        L[Loss]:::loss
    end
```
