# mhingston/embedder architecture

> Architecture diagram of the GitHub repository mhingston/embedder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mhingston/embedder
- Repository: https://github.com/mhingston/embedder

## Overview

##### Project Type and Purpose
- Type: CLI‐driven, open‐source “embedding pipeline” tool
- Purpose:
– Scan a Git repo (full or diff)
– Analyze code files with an LLM
– Chunk text intelligently
– Generate embeddings
– Store vectors in Qdrant
– Persist pipeline state (Azure Blob or local file)

##### File Structure
Top-level:
- main.ts (entry point)
- embeddingPipeline.ts + embeddingPipelineOptions.ts (orchestrator)
- repositoryManager.ts
- stateManager.ts + blobStateManager.ts + fileStateManager.ts
- fileProcessor.ts
- chunker.ts + chunkOptions.ts + fileTypeChunkingOptions.ts + chunkStrategy.ts
- analysisService.ts + codeFileAnalysisSchema.ts
- embeddingService.ts
- qdrantManager.ts
- retry.ts + retryOptions.ts
- utilities.ts
Dist folder mirrors src in compiled JS.

##### README Insights
- Key patterns: orchestrator → pluggable managers/services → external APIs/DB
- Emphasis on caching, retries, batching, concurrency control

##### Main Components
– **CLI / Main Entry** (main.ts)
– **Config Loader** (.env / environment variables)
– **Orchestrator** (EmbeddingPipeline)
– **Managers**:
- RepositoryManager (Git operations)
- StateManager interface → BlobStateManager / FileStateManager
- QdrantManager (collection mgmt, upserts, deletes)
– **Services**:
- FileProcessor (filter & load files)
- Chunker (invokes AnalysisService + chunk strategies)
- AnalysisService (LLM code analysis + local cache)
- EmbeddingService (batch embedding + retry)
– **Utilities**: Retry mechanism, Schema validation, Type detection
– **External Systems**:
- Git Repository (file system)
- Azure Blob Storage or Local FS
- LLM API (Azure OpenAI)
- Embedding API (OpenAI-compatible)
- Qdrant Vector DB

##### Relationships & Data Flows
- Environment → Main → Orchestrator
- Orchestrator sequentially calls:
1. RepositoryManager ↔ Git Repo
2. QdrantManager ↔ Qdrant DB
3. StateManager ↔ Blob or FS
4. RepositoryManager (diff or ls-files)
5. StateManager (compute deletes)
6. QdrantManager (delete old points)
7. FileProcessor ↔ FS
8. Chunker → AnalysisService ↔ LLM API
9. Chunker → chunk strategies (@mastra/rag)
10. StateManager (save intermediate)
11. EmbeddingService ↔ Embedding API
12. QdrantManager (upsert new vectors)
13. RepositoryManager (get HEAD)
14. StateManager (save final)
- Utilities (retry, schema) are cross‐cutting, used by AnalysisService, EmbeddingService, QdrantManager

##### Architectural Patterns & Design Principles
- **Modular Orchestration**: EmbeddingPipeline drives a linear workflow
- **Pluggable Interfaces**: StateManager interface → two implementations
- **Separation of Concerns**: Git logic, state persistence, file I/O, AI analysis, vector storage each encapsulated
- **Resilience Patterns**: Retry with exponential backoff, caching, intermediate state for resumability
- **Concurrency Control**: p-limit for parallel chunking

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Pipeline
    subgraph "Core Pipeline"
        Main["CLI / Main Entry\n(main.ts)"]:::orchestrator
        Config["Config Loader\n(.env)"]:::orchestrator
        Orchestrator["EmbeddingPipeline\n(embeddingPipeline.ts)"]:::orchestrator
        subgraph "Managers"
            RM["RepositoryManager\n(repositoryManager.ts)"]:::manager
            SMIface["StateManager (interface)\n(stateManager.ts)"]:::manager
            BlobSM["BlobStateManager\n(blobStateManager.ts)"]:::manager
            FileSM["FileStateManager\n(fileStateManager.ts)"]:::manager
            QM["QdrantManager\n(qdrantManager.ts)"]:::manager
        end
        subgraph "Services"
            FP["FileProcessor\n(fileProcessor.ts)"]:::service
            Ch["Chunker\n(chunker.ts)"]:::service
            AS["AnalysisService\n(analysisService.ts)"]:::service
            ESvc["EmbeddingService\n(embeddingService.ts)"]:::service
        end
    end

    %% External Systems
    subgraph "External Systems"
        Git["Git Repository"]:::external
        QdrantDB["Qdrant Vector DB"]:::datastore
        BlobStore["Azure Blob Storage"]:::datastore
        LocalFS["Local File System"]:::datastore
        LLMAPI["LLM API (Azure)"]:::external
        EmbedAPI["Embedding API"]:::external
    end

    %% Utilities
    subgraph "Utilities"
        Retry["Retry Utilities\n(retry.ts)"]:::utility
        Util["General Utilities\n(utilities.ts)"]:::utility
    end

    %% Data Flow
    Config --> Main
    Main --> Orchestrator
    Orchestrator -->|1 initialize| RM
    RM -->|git ops| Git
    Orchestrator -->|2 init collection| QM
    QM --> QdrantDB
    Orchestrator -->|3 load state| SMIface
    SMIface --> BlobSM
    SMIface --> FileSM
    BlobSM --> BlobStore
    FileSM --> LocalFS
    Orchestrator -->|4 repo diff| RM
    RM --> Git
    Orchestrator -->|5 compute deletes| SMIface
    Orchestrator -->|6 delete old vectors| QM
    QM --> QdrantDB
    Orchestrator -->|7 load files| FP
    FP --> Git
    FP --> Ch
    Ch -->|8 analyze code| AS
    AS --> LLMAPI
    Ch -->|9 chunk strategies| Ch
    Orchestrator -->|10 save intermediate| SMIface
    Ch -->|11 embed batch| ESvc
    ESvc --> EmbedAPI
    Orchestrator -->|12 upsert vectors| QM
    QM --> QdrantDB
    Orchestrator -->|13 get HEAD| RM
    RM --> Git
    Orchestrator -->|14 save final state| SMIface

    %% Cross-cutting Utilities
    AS -.-> Retry
    ESvc -.-> Retry
    QM -.-> Retry
    AS -.-> Util
    ESvc -.-> Util
    QM -.-> Util

    %% Click Events
    click Main "https://github.com/mhingston/embedder/blob/main/src/main.ts"
    click Config "https://github.com/mhingston/embedder/blob/main/.env.example"
    click Orchestrator "https://github.com/mhingston/embedder/blob/main/src/embeddingPipeline.ts"
    click Orchestrator "https://github.com/mhingston/embedder/blob/main/src/embeddingPipelineOptions.ts"
    click RM "https://github.com/mhingston/embedder/blob/main/src/repositoryManager.ts"
    click SMIface "https://github.com/mhingston/embedder/blob/main/src/stateManager.ts"
    click BlobSM "https://github.com/mhingston/embedder/blob/main/src/blobStateManager.ts"
    click FileSM "https://github.com/mhingston/embedder/blob/main/src/fileStateManager.ts"
    click QM "https://github.com/mhingston/embedder/blob/main/src/qdrantManager.ts"
    click FP "https://github.com/mhingston/embedder/blob/main/src/fileProcessor.ts"
    click Ch "https://github.com/mhingston/embedder/blob/main/src/chunker.ts"
    click Ch "https://github.com/mhingston/embedder/blob/main/src/chunk.ts"
    click Ch "https://github.com/mhingston/embedder/blob/main/src/chunkOptions.ts"
    click Ch "https://github.com/mhingston/embedder/blob/main/src/fileTypeChunkingOptions.ts"
    click Ch "https://github.com/mhingston/embedder/blob/main/src/chunkStrategy.ts"
    click AS "https://github.com/mhingston/embedder/blob/main/src/analysisService.ts"
    click AS "https://github.com/mhingston/embedder/blob/main/src/codeFileAnalysisSchema.ts"
    click ESvc "https://github.com/mhingston/embedder/blob/main/src/embeddingService.ts"
    click Retry "https://github.com/mhingston/embedder/blob/main/src/retry.ts"
    click Retry "https://github.com/mhingston/embedder/blob/main/src/retryOptions.ts"
    click Util "https://github.com/mhingston/embedder/blob/main/src/utilities.ts"

    %% Styles
    classDef orchestrator fill:#ccf,stroke:#333,stroke-width:1px;
    classDef manager fill:#ffc,stroke:#333,stroke-width:1px;
    classDef service fill:#cfc,stroke:#333,stroke-width:1px;
    classDef external fill:#ddd,stroke:#333,stroke-width:1px;
    classDef datastore fill:#fc9,stroke:#333,stroke-width:1px;
    classDef utility fill:#eac,stroke:#333,stroke-width:1px;
```
