# databridge-org/databridge-core architecture

> Architecture diagram of the GitHub repository databridge-org/databridge-core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/databridge-org/databridge-core
- Repository: https://github.com/databridge-org/databridge-core

## Overview

##### Main Components Overview

a) Client Layer:
- Python SDK (located in /sdks/python)
- REST API Interface

b) Core Service Layer:
- API Handler (core/api.py)
- Authentication Service (core/models/auth.py)
- Document Service (core/services/document_service.py)
- URI Service (core/services/uri_service.py)

c) Processing Pipeline Components:
- Parser System (core/parser/)
- Embedding System (core/embedding/)
- Vector Store System (core/vector_store/)
- Storage System (core/storage/)

##### Component Relationships

a) External Integrations (show as external services):
- Unstructured API
- MongoDB Atlas
- OpenAI API
- AWS S3

b) Data Flow Paths:
- Client → API Layer → Document Service
- Document Service → Parser → Storage
- Document Service → Embedding → Vector Store

##### Specific Implementation Details

- BaseParser
- BaseEmbeddingModel
- BaseVectorStore
- BaseStorage

b) Implementations (solid rectangles):
- UnstructuredParser
- OllamaEmbedding/OpenAIEmbedding
- MongoVectorStore
- S3Storage

##### Architectural Patterns to Highlight

Following patterns using grouped components:

a) Modular Architecture:
- Components by their base classes
- Extension points with interface symbols

b) Service Layer:
- Core services together
- Their interactions with other components

**┌─ Client Layer ─────────────────────┐**
- SDK | REST API                     │
- Core Service Layer ───────────────┤
- API | Auth | Document | URI        │
- Processing Pipeline ─────────────┬┤
- Parser → Embedding → Vector Store ││
- ↕                                ││
- Storage                          ││
- ┘│
- ┘

##### Important Details

a) Configuration Flow:
- Config.toml and environment variables influence
- Initialization flow from start_server.py

b) Authentication Flow:
- JWT-based auth paths
- Developer vs end-user access modes

##### Implementation Notes

- The modular nature of the system
- Clear separation of concerns
- Extension points for each component
- Data flow for both ingestion and query paths

Remember to maintain a balance between detail and clarity, focusing on the architectural decisions that make DataBridge extensible and maintainable.

- Extend the system with new components
- Troubleshoot issues across the pipeline
- Implement new features within the existing architecture
- Understand the system's scalability points

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and classes
    classDef client fill:#e1f5fe,stroke:#01579b
    classDef core fill:#e8f5e9,stroke:#2e7d32
    classDef external fill:#fff3e0,stroke:#ff6f00
    classDef processing fill:#f3e5f5,stroke:#7b1fa2
    classDef storage fill:#eceff1,stroke:#455a64
    classDef auth fill:#e8eaf6,stroke:#3f51b5
    classDef base fill:none,stroke-dasharray: 5 5

    %% Client Layer
    subgraph ClientLayer["Client Layer"]
        SDK["Python SDK"]:::client
        API["REST API Interface"]:::client
    end

    %% Core Service Layer
    subgraph CoreLayer["Core Service Layer"]
        APIHandler["API Handler"]:::core
        AuthService["Authentication Service"]:::auth
        DocService["Document Service"]:::core
        URIService["URI Service"]:::core
    end

    %% Processing Pipeline
    subgraph ProcessingPipeline["Processing Pipeline"]
        %% Base Classes
        BaseParser["BaseParser"]:::base
        BaseEmbed["BaseEmbeddingModel"]:::base
        BaseVector["BaseVectorStore"]:::base
        BaseStorage["BaseStorage"]:::base

        %% Implementations
        UnstructParser["UnstructuredParser"]:::processing
        OllamaEmbed["OllamaEmbedding"]:::processing
        OpenAIEmbed["OpenAIEmbedding"]:::processing
        MongoVector["MongoVectorStore"]:::processing
        S3Store["S3Storage"]:::storage
    end

    %% External Services
    UnstructuredAPI["Unstructured API"]:::external
    MongoDB["MongoDB Atlas"]:::external
    OpenAI["OpenAI API"]:::external
    S3["AWS S3"]:::external

    %% Relationships
    SDK --> APIHandler
    API --> APIHandler
    APIHandler --> AuthService
    APIHandler --> DocService
    DocService --> URIService
    
    BaseParser --> UnstructParser
    BaseEmbed --> OllamaEmbed
    BaseEmbed --> OpenAIEmbed
    BaseVector --> MongoVector
    BaseStorage --> S3Store

    UnstructParser --> UnstructuredAPI
    MongoVector --> MongoDB
    OpenAIEmbed --> OpenAI
    S3Store --> S3

    DocService --> UnstructParser
    UnstructParser --> OllamaEmbed
    UnstructParser --> OpenAIEmbed
    OllamaEmbed --> MongoVector
    OpenAIEmbed --> MongoVector
    UnstructParser --> S3Store

    %% Click Events
    click SDK "https://github.com/databridge-org/databridge-core/tree/main/sdks/python/databridge/"
    click API "https://github.com/databridge-org/databridge-core/blob/main/core/api.py"
    click APIHandler "https://github.com/databridge-org/databridge-core/blob/main/core/api.py"
    click AuthService "https://github.com/databridge-org/databridge-core/blob/main/core/models/auth.py"
    click DocService "https://github.com/databridge-org/databridge-core/blob/main/core/services/document_service.py"
    click URIService "https://github.com/databridge-org/databridge-core/blob/main/core/services/uri_service.py"
    click BaseParser "https://github.com/databridge-org/databridge-core/blob/main/core/parser/base_parser.py"
    click BaseEmbed "https://github.com/databridge-org/databridge-core/blob/main/core/embedding/base_embedding_model.py"
    click BaseVector "https://github.com/databridge-org/databridge-core/blob/main/core/vector_store/base_vector_store.py"
    click BaseStorage "https://github.com/databridge-org/databridge-core/blob/main/core/storage/base_storage.py"
    click UnstructParser "https://github.com/databridge-org/databridge-core/blob/main/core/parser/unstructured_parser.py"
    click OllamaEmbed "https://github.com/databridge-org/databridge-core/blob/main/core/embedding/ollama_embedding_model.py"
    click OpenAIEmbed "https://github.com/databridge-org/databridge-core/blob/main/core/embedding/openai_embedding_model.py"
    click MongoVector "https://github.com/databridge-org/databridge-core/blob/main/core/vector_store/mongo_vector_store.py"
    click S3Store "https://github.com/databridge-org/databridge-core/blob/main/core/storage/s3_storage.py"

    %% Legend
    subgraph Legend
        Client["Client Component"]:::client
        Core["Core Service"]:::core
        External["External Service"]:::external
        Processing["Processing Component"]:::processing
        Storage["Storage Component"]:::storage
        Auth["Auth Component"]:::auth
        Base["Base Class"]:::base
    end
```
