How fahadahmedk/ragdocmanager works
Main Components
• API Layer – Under the “src/rag_doc_manager/api” directory, you have routes defined for documents, registration, and search. This layer will expose the HTTP endpoints and act as the interactive entry point for users.
• Document Manager – Located in “src/rag_doc_manager/document_manager,” this should be highlighted as the central component. It includes integration with Azure services (as indicated by “azure_document_manager.py”), various data models, and contractual interfaces.
• Document Processor – This module (“src/rag_doc_manager/document_processor”) is responsible for handling tasks such as chunking, loading, and embedding of documents. It also includes factories that support multiple strategies.
• Index and Index Manager – Found in “src/rag_doc_manager/index” and “src/rag_doc_manager/index_manager,” these components handle the creation, management, and querying of document indexes.
• Search – Under “src/rag_doc_manager/search,” this component includes adaptors (like “azure_ai_query_engine.py”) that connect search queries to the underlying indexed data.
• Storage – Various storage facets are split into managing databases (e.g., Cosmos DB), blob storage (Azure Blob Storage), secrets (with Azure Key Vault), and history management. These are found in “src/rag_doc_manager/storage” and related subdirectories.
• Customer Manager – This component (“src/rag_doc_manager/customer_manager”) manages customer-related data and schema management, ensuring that document operations are performed in a multitenant or customer-specific context.
• Utility Functions – Found in “src/rag_doc_manager/utils” to support common IO operations and data manipulation.
Relationships and Interactions
• Within the system, illustrate the internal flow:
• Document Manager connects with Document Processor to handle document transformations (chunking, loading, embedding).
• Processed documents then get handed off to the Index or Index Manager components to create an index.
• The Search module queries the index when needed.
• In parallel, the Storage module supports the entire system by managing persistent data (documents, indexes, audit history, and configuration secrets).
• The Customer Manager may interact with several components (API, Document Manager, and Storage) to enforce customer-specific rules.
Architectural Patterns and Design Principles
• That the system is modular and decoupled; components communicate via well-defined interfaces (evident from the “base”, “contracts”, and “factory” files defined in multiple modules).
• Separation of concerns – the API layer for communication, Document Manager for core business logic (document handling), Document Processor for data handling, and Storage/Search for persistence and retrieval.
Relevant Technologies and Frameworks
• The project uses Python (as seen in files like __init__.py, pyproject.toml, and a Dockerfile for containerization).
• Azure services are explicitly referenced in multiple modules.
• The usage of a structured folder design, including factories and adaptors, suggests extensibility for various document processing strategies.
• Restate the Document Manager as the central hub that interfaces with Document Processor, Index/Index Manager, and Storage layers.