How raphaelsty/neural-cherche works
Main Components
a. Core Models Layer:
• ColBERT Model
• SparseEmbed Model
• Splade Model
• Base Model Interface
b. Search Operations Layer:
• Retrieval Components (BM25, TfIdf, ColBERT, SparseEmbed, Splade)
• Ranking Components (ColBERT-specific ranking)
c. Training Layer:
• Model-specific training modules
• Loss Functions (Flops, Ranking)
d. Utility Layer:
• Scoring utilities
• Evaluation tools
• Data iteration helpers
• BEIR dataset loader
[Input Layer]
[Pre-processing & Training]
[Model Layer]
[Search Operations]
[Output Layer]
Component Relationships
• Bidirectional interactions where applicable
Specific Implementation Details
• Model architectures (ColBERT, SparseEmbed, Splade)
• Training workflows
• Inference paths
• Data preprocessing steps
Technical Elements
Technical aspects:
• Device support (CPU, GPU, MPS)
• Integration with external models (Sentence Transformers)
• Data flow for both training and inference
• Model checkpoint handling
Integration Points
Interfaces with:
• External pre-trained models
• Dataset inputs
• Storage systems for embeddings
• Evaluation metrics output
The pipeline from raw input to final search results
Use standardized symbols:
• Rectangles: Processing components
• Diamonds: Decision points
• Special symbols
• Abbreviations used
Keep the abstraction level appropriate by:
• Focusing on major components and their interactions
• Avoiding implementation details unless crucial
• Highlighting the system's extensibility
• Showing clear boundaries between subsystems