# naver/splade architecture

> Architecture diagram of the GitHub repository naver/splade, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/naver/splade
- Repository: https://github.com/naver/splade

## Overview

##### Main Components
- Data Layer: Represent the raw and preprocessed data inputs. This includes directories such as “data” (with toy, triplets, qrels, etc.) and external data (MS MARCO, BEIR datasets). Indicate data sources for training, validation, indexing, and evaluation.
- Configuration & Experiment Management: Emphasize the role of Hydra and the “conf”, “main_config”, and individual YAML configuration files. This is the central place where various experiments (e.g., different regularization or negative mining settings) are configured.
- Model Training Module: This includes scripts like train.py, the “splade” package’s training modules (e.g., splade/hf_train.py, splade/train.py), and the various model configuration files. This component is responsible for learning the sparse representations.
- Indexing Engine: Illustrate the indexing components represented by index.py and the “splade/indexing” module (with inverted_index.py). Also, indicate the integration with external indexing formats (e.g., creation of Anserini-readable files via create_anserini.py).
- Retrieval & Reranking Module: Include retrieval.py, rerank.py, and their corresponding tasks (e.g., splade/rerank.py, splade/tasks folder). These parts handle query processing and ranking of documents based on sparse representations.
- Evaluation & Benchmarking Components: Capture the evaluation setups such as SPLADE’s BEIR evaluation (via splade/beir_eval.py) and PISA evaluation (in the “efficient_splade_pisa” folder). They are responsible for assessing model effectiveness (MRR@10 scores, etc.).
- External Service Integration: Represent integrations with Hugging Face (model weights from naver/splade_… repos are referenced multiple times via main_config/hf and conf/hf folders). Similarly, note that Anserini is used for specific indexing and retrieval pipelines.

##### Relationships and Data Flow
- Point from the Data Layer into the Training Module indicating that raw collections, triplets, and qrels are used to feed the SPLADE training.
- Indexing Engine with the Retrieval Module since the generated index is used for processing queries.
- Side branches indicating that model configuration (via Hydra and YAML files) and external library integrations (e.g., Hugging Face, numba, Anserini) span multiple parts of the system.

##### Architectural Patterns and Design Principles
- Component-based decomposition: Clearly distinguish between training, indexing, retrieval, and evaluation as separate modules.
- Configurability: Highlight the extensive use of Hydra for dynamic experiment configuration, indicating that the pipeline can be customized through different YAML config files.
- Modularity and Extensibility: Note that there are multiple “conf” directories (e.g., for training, indexing, evaluation, reranking) which allow different modes (dense, sparse, efficient, etc.) and that the code is organized in packages (like splade/hf, splade/models, splade/tasks) that encapsulate different concerns.
- Pipeline flow: From data ingestion, through training and model optimization (losses, regularization with FLOPS, etc.), to the eventual indexing and retrieval operations with integration to benchmark pipelines.

##### Key Technologies, Frameworks, and Libraries
- Hydra – for managing experiment configurations.
- Hugging Face Transformers – used in various parts (e.g., splade/hf for model definitions and training).
- numba – indicated for fast inverted index creation.
- Python – the primary language, with modular packages for training, data handling, evaluation, etc.
- External tools: BM25 methods, Anserini (for indexing and retrieval), and BEIR for benchmarking.

##### Special Details
- Multiple strategies available: For example, dual training for query and document processing (note the existence of separate configurations for “dense” vs. “sparse” settings and the options available in main_config).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Configuration Layer
    subgraph "Data & Configuration Layer"
        D1["Raw Data"]:::datalayer
        D2["Data Configurations"]:::datalayer
        C1["Hydra YAML Configs"]:::config
        C2["Hydra Utilities"]:::config
        C3["Main Configs"]:::config
    end

    %% Training Module
    subgraph "Training Module"
        T1["train.py"]:::training
        T2["splade/hf_train.py"]:::training
        T3["splade/train.py"]:::training
        T4["Training Configurations"]:::training
        T5["Model Artifacts"]:::training
    end

    %% Indexing Engine
    subgraph "Indexing Engine"
        I1["splade/index.py"]:::indexing
        I2["Inverted Index Module"]:::indexing
        I3["Anserini Integration"]:::indexing
        I4["Index Configurations"]:::indexing
    end

    %% Retrieval & Reranking Module
    subgraph "Retrieval & Reranking Module"
        R1["Retrieval Script"]:::retrieval
        R2["Reranking Script"]:::retrieval
        R3["Reranking Tasks"]:::retrieval
        R4["Reranking Configurations"]:::retrieval
    end

    %% Evaluation & Benchmarking
    subgraph "Evaluation & Benchmarking"
        E1["BEIR Evaluation"]:::evaluation
        E2["General Evaluation"]:::evaluation
        E3["Evaluation Configurations"]:::evaluation
        E4["Benchmarking Setup"]:::evaluation
    end

    %% External Integrations
    subgraph "External Integrations"
        X1["Hugging Face Integration"]:::external
    end

    %% Data & Config to Training
    D1 -->|"feeds"| T1
    D2 -->|"configures"| T1
    C1 -->|"sets up"| T1
    C1 -->|"configures"| T4
    C2 -->|"supports"| C1
    C3 -->|"specifies"| T1

    %% Training Module flow
    T4 --> T1
    T1 -->|"produces"| T5
    T2 --> T5
    T3 --> T5

    %% Training to Indexing
    T5 -->|"supplies models to"| I1
    I4 -->|"configures"| I1

    %% Indexing Engine internal flow
    I1 -->|"builds"| I2
    I1 -->|"integrates"| I3

    %% Indexing to Retrieval
    I1 -->|"feeds index to"| R1

    %% Retrieval & Reranking flow
    R4 -->|"configures"| R2
    R1 -->|"passes results to"| R2
    R3 -->|"supports"| R2
    R2 -->|"outputs to"| E2

    %% Evaluation & Benchmarking flow
    E3 -->|"configures"| E2
    E1 -->|"evaluates"| E2
    E4 -->|"benchmarks"| E2

    %% External Integration connections
    X1 -->|"supplies model weights to"| T1
    X1 -->|"assists"| I3

    %% Styles
    classDef datalayer fill:#add8e6,stroke:#333,stroke-width:2px;
    classDef config fill:#a0d8f1,stroke:#333,stroke-width:2px;
    classDef training fill:#ffb347,stroke:#333,stroke-width:2px;
    classDef indexing fill:#cda4de,stroke:#333,stroke-width:2px;
    classDef retrieval fill:#fdfd96,stroke:#333,stroke-width:2px;
    classDef evaluation fill:#ff6961,stroke:#333,stroke-width:2px;
    classDef external fill:#77dd77,stroke:#333,stroke-width:2px;

    %% Click Events for Data & Configuration
    click D1 "https://github.com/naver/splade/tree/main/data"
    click D2 "https://github.com/naver/splade/tree/main/conf/train/data"
    click C1 "https://github.com/naver/splade/tree/main/conf"
    click C2 "https://github.com/naver/splade/blob/main/splade/utils/hydra.py"
    click C3 "https://github.com/naver/splade/tree/main/main_config"

    %% Click Events for Training Module
    click T1 "https://github.com/naver/splade/blob/main/train.py"
    click T2 "https://github.com/naver/splade/blob/main/splade/hf_train.py"
    click T3 "https://github.com/naver/splade/blob/main/splade/train.py"
    click T4 "https://github.com/naver/splade/tree/main/conf/train/config"

    %% Click Events for Indexing Engine
    click I1 "https://github.com/naver/splade/blob/main/splade/index.py"
    click I2 "https://github.com/naver/splade/blob/main/splade/indexing/inverted_index.py"
    click I3 "https://github.com/naver/splade/blob/main/splade/create_anserini.py"
    click I4 "https://github.com/naver/splade/tree/main/conf/index"

    %% Click Events for Retrieval & Reranking Module
    click R1 "https://github.com/naver/splade/blob/main/splade/retrieve.py"
    click R2 "https://github.com/naver/splade/blob/main/splade/rerank.py"
    click R3 "https://github.com/naver/splade/tree/main/splade/tasks"
    click R4 "https://github.com/naver/splade/tree/main/conf/rerank_collection"

    %% Click Events for Evaluation & Benchmarking
    click E1 "https://github.com/naver/splade/blob/main/splade/beir_eval.py"
    click E2 "https://github.com/naver/splade/blob/main/splade/evaluate.py"
    click E3 "https://github.com/naver/splade/tree/main/conf/retrieve_evaluate"
    click E4 "https://github.com/naver/splade/tree/main/benchmarking_sigir23"

    %% Click Event for External Integrations
    click X1 "https://github.com/naver/splade/tree/main/main_config/hf"
```
