How facebookresearch/sonar works
Main components
• TextToEmbeddingModelPipeline (entry point)
• Tokenizer module (SentencePiece)
• Encoder Handler (wrapper around the Fairseq2 model)
• Encoder Model (the actual neural network)
• Card / Config loader (reads the YAML card describing default paths and hyperparameters)
• External dependencies: Fairseq2, sentencepiece
Relationships and data flow
1. User supplies List[str] to the TextToEmbeddingModelPipeline
2. Pipeline invokes the Card/Config loader to fetch default settings for both tokenizer and encoder (default card = “text_sonar_basic_encoder”)
3. Pipeline instantiates:
• A SentencePiece tokenizer (using model path from the card)
• An Encoder Handler (using the card’s model checkpoint path)
– Handler leverages the Factory to load the Encoder Model
– Factory uses the Config object (in code) to set architecture parameters
– Handler wraps the raw Fairseq2 model to expose a simple .predict() or .forward() method returning embeddings
4. Pipeline runs text → tokens → model.forward → pooling → final fixed‐size tensor
Architectural patterns
• Pipeline pattern (pre-processing, model inference, post-processing)
• Config‐driven component loading (YAML cards + factory)
• Clear separation between logic (Handler), data (Config/Card), and model (Fairseq2)
4. Default configuration for “text_sonar_basic_encoder” card (located in sonar/cards/text_sonar_basic_encoder.yaml)
• name: text_sonar_basic_encoder
• type: text_encoder
• tokenizer:
– type: sentencepiece
– model_path: sentencepiece.source.256000.model
• encoder:
– checkpoint_path: sonar_text_encoder.pt
– embedding_size: 1024
– max_seq_len: 512
– pooling: mean
– framework: fairseq2 (v0.4.5)
• device: cpu (unless overridden)
• dtype: float32
Nested modules (show inside Encoder Handler)
• Config (sonar.models.sonar_text.config)
• Factory (sonar.models.sonar_text.factory)
• Model class (sonar.models.sonar_text.model)
• Loader / Handler (sonar.models.sonar_text.handler)
Putting it all together
[TextToEmbeddingModelPipeline]
|
v
[Card/Config Loader (“text_sonar_basic_encoder”)]
├─ provides ─> tokenizer settings
└─ provides ─> encoder checkpoint & hyperparameters
|
v
[SentencePiece Tokenizer]──(token IDs)──>
|
v
[Encoder Handler] (contains Config, Factory, Model)
|
v
[Fairseq2 Encoder Model]──(hidden states + pooling)──>
|
v
[Final 1024-d Embedding Tensor]