How mhingston/embedder works
Project Type and Purpose
• Type: CLI‐driven, open‐source “embedding pipeline” tool
• Purpose:
– Scan a Git repo (full or diff)
– Analyze code files with an LLM
– Chunk text intelligently
– Generate embeddings
– Store vectors in Qdrant
– Persist pipeline state (Azure Blob or local file)
File Structure
Top-level:
• main.ts (entry point)
• embeddingPipeline.ts + embeddingPipelineOptions.ts (orchestrator)
• repositoryManager.ts
• stateManager.ts + blobStateManager.ts + fileStateManager.ts
• fileProcessor.ts
• chunker.ts + chunkOptions.ts + fileTypeChunkingOptions.ts + chunkStrategy.ts
• analysisService.ts + codeFileAnalysisSchema.ts
• embeddingService.ts
• qdrantManager.ts
• retry.ts + retryOptions.ts
• utilities.ts
Dist folder mirrors src in compiled JS.
README Insights
• Key patterns: orchestrator → pluggable managers/services → external APIs/DB
• Emphasis on caching, retries, batching, concurrency control
Main Components
– CLI / Main Entry (main.ts)
– Config Loader (.env / environment variables)
– Orchestrator (EmbeddingPipeline)
– Managers:
• RepositoryManager (Git operations)
• StateManager interface → BlobStateManager / FileStateManager
• QdrantManager (collection mgmt, upserts, deletes)
– Services:
• FileProcessor (filter & load files)
• Chunker (invokes AnalysisService + chunk strategies)
• AnalysisService (LLM code analysis + local cache)
• EmbeddingService (batch embedding + retry)
– Utilities: Retry mechanism, Schema validation, Type detection
– External Systems:
• Git Repository (file system)
• Azure Blob Storage or Local FS
• LLM API (Azure OpenAI)
• Embedding API (OpenAI-compatible)
• Qdrant Vector DB
Relationships & Data Flows
• Environment → Main → Orchestrator
• Orchestrator sequentially calls:
1. RepositoryManager ↔ Git Repo
2. QdrantManager ↔ Qdrant DB
3. StateManager ↔ Blob or FS
4. RepositoryManager (diff or ls-files)
5. StateManager (compute deletes)
6. QdrantManager (delete old points)
7. FileProcessor ↔ FS
8. Chunker → AnalysisService ↔ LLM API
9. Chunker → chunk strategies (@mastra/rag)
10. StateManager (save intermediate)
11. EmbeddingService ↔ Embedding API
12. QdrantManager (upsert new vectors)
13. RepositoryManager (get HEAD)
14. StateManager (save final)
• Utilities (retry, schema) are cross‐cutting, used by AnalysisService, EmbeddingService, QdrantManager
Architectural Patterns & Design Principles
• Modular Orchestration: EmbeddingPipeline drives a linear workflow
• Pluggable Interfaces: StateManager interface → two implementations
• Separation of Concerns: Git logic, state persistence, file I/O, AI analysis, vector storage each encapsulated
• Resilience Patterns: Retry with exponential backoff, caching, intermediate state for resumability
• Concurrency Control: p-limit for parallel chunking