How naver/splade works
Main Components
• Data Layer: Represent the raw and preprocessed data inputs. This includes directories such as “data” (with toy, triplets, qrels, etc.) and external data (MS MARCO, BEIR datasets). Indicate data sources for training, validation, indexing, and evaluation.
• Configuration & Experiment Management: Emphasize the role of Hydra and the “conf”, “main_config”, and individual YAML configuration files. This is the central place where various experiments (e.g., different regularization or negative mining settings) are configured.
• Model Training Module: This includes scripts like train.py, the “splade” package’s training modules (e.g., splade/hf_train.py, splade/train.py), and the various model configuration files. This component is responsible for learning the sparse representations.
• Indexing Engine: Illustrate the indexing components represented by index.py and the “splade/indexing” module (with inverted_index.py). Also, indicate the integration with external indexing formats (e.g., creation of Anserini-readable files via create_anserini.py).
• Retrieval & Reranking Module: Include retrieval.py, rerank.py, and their corresponding tasks (e.g., splade/rerank.py, splade/tasks folder). These parts handle query processing and ranking of documents based on sparse representations.
• Evaluation & Benchmarking Components: Capture the evaluation setups such as SPLADE’s BEIR evaluation (via splade/beir_eval.py) and PISA evaluation (in the “efficient_splade_pisa” folder). They are responsible for assessing model effectiveness (MRR@10 scores, etc.).
• External Service Integration: Represent integrations with Hugging Face (model weights from naver/splade_… repos are referenced multiple times via main_config/hf and conf/hf folders). Similarly, note that Anserini is used for specific indexing and retrieval pipelines.
Relationships and Data Flow
• Point from the Data Layer into the Training Module indicating that raw collections, triplets, and qrels are used to feed the SPLADE training.
• Indexing Engine with the Retrieval Module since the generated index is used for processing queries.
• Side branches indicating that model configuration (via Hydra and YAML files) and external library integrations (e.g., Hugging Face, numba, Anserini) span multiple parts of the system.
Architectural Patterns and Design Principles
• Component-based decomposition: Clearly distinguish between training, indexing, retrieval, and evaluation as separate modules.
• Configurability: Highlight the extensive use of Hydra for dynamic experiment configuration, indicating that the pipeline can be customized through different YAML config files.
• Modularity and Extensibility: Note that there are multiple “conf” directories (e.g., for training, indexing, evaluation, reranking) which allow different modes (dense, sparse, efficient, etc.) and that the code is organized in packages (like splade/hf, splade/models, splade/tasks) that encapsulate different concerns.
• Pipeline flow: From data ingestion, through training and model optimization (losses, regularization with FLOPS, etc.), to the eventual indexing and retrieval operations with integration to benchmark pipelines.
Key Technologies, Frameworks, and Libraries
• Hydra – for managing experiment configurations.
• Hugging Face Transformers – used in various parts (e.g., splade/hf for model definitions and training).
• numba – indicated for fast inverted index creation.
• Python – the primary language, with modular packages for training, data handling, evaluation, etc.
• External tools: BM25 methods, Anserini (for indexing and retrieval), and BEIR for benchmarking.
Special Details
• Multiple strategies available: For example, dual training for query and document processing (note the existence of separate configurations for “dense” vs. “sparse” settings and the options available in main_config).