How databridge-org/databridge-core works
Main Components Overview
a) Client Layer:
• Python SDK (located in /sdks/python)
• REST API Interface
b) Core Service Layer:
• API Handler (core/api.py)
• Authentication Service (core/models/auth.py)
• Document Service (core/services/document_service.py)
• URI Service (core/services/uri_service.py)
c) Processing Pipeline Components:
• Parser System (core/parser/)
• Embedding System (core/embedding/)
• Vector Store System (core/vector_store/)
• Storage System (core/storage/)
Component Relationships
a) External Integrations (show as external services):
• Unstructured API
• MongoDB Atlas
• OpenAI API
• AWS S3
b) Data Flow Paths:
• Client → API Layer → Document Service
• Document Service → Parser → Storage
• Document Service → Embedding → Vector Store
Specific Implementation Details
• BaseParser
• BaseEmbeddingModel
• BaseVectorStore
• BaseStorage
b) Implementations (solid rectangles):
• UnstructuredParser
• OllamaEmbedding/OpenAIEmbedding
• MongoVectorStore
• S3Storage
Architectural Patterns to Highlight
Following patterns using grouped components:
a) Modular Architecture:
• Components by their base classes
• Extension points with interface symbols
b) Service Layer:
• Core services together
• Their interactions with other components
┌─ Client Layer ─────────────────────┐
• SDK | REST API │
• Core Service Layer ───────────────┤
• API | Auth | Document | URI │
• Processing Pipeline ─────────────┬┤
• Parser → Embedding → Vector Store ││
• ↕ ││
• Storage ││
• ┘│
• ┘
Important Details
a) Configuration Flow:
• Config.toml and environment variables influence
• Initialization flow from start_server.py
b) Authentication Flow:
• JWT-based auth paths
• Developer vs end-user access modes
Implementation Notes
• The modular nature of the system
• Clear separation of concerns
• Extension points for each component
• Data flow for both ingestion and query paths
Remember to maintain a balance between detail and clarity, focusing on the architectural decisions that make DataBridge extensible and maintainable.
• Extend the system with new components
• Troubleshoot issues across the pipeline
• Implement new features within the existing architecture
• Understand the system's scalability points