# d-star-ai/dsrag architecture

> Architecture diagram of the GitHub repository d-star-ai/dsrag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/d-star-ai/dsrag
- Repository: https://github.com/d-star-ai/dsrag

## Overview

##### Project Type and Purpose
- dsRAG is a specialized retrieval engine (or RAG – retrieval augmented generation – system) for complex query answering over dense text documents.
- Its goal is to improve baseline RAG performance using techniques like semantic sectioning, contextual chunk headers (AutoContext), and post-query combination of content (RSE).

##### File Structure
- Top-level components: The file tree shows several directories including dsrag, integrations, eval, examples, tests, and additional configuration files (pyproject.toml, requirements.txt, etc.).
- The dsrag package is the core of the project. Inside, it has modules for creating and querying a KnowledgeBase, handling embeddings, LLM interaction, file parsing, chunking, and interfacing with various databases (both for vector and chunk storage).
- There’s a clear separation of concerns:
- dsrag/knowledge_base.py, create_kb.py for managing document ingestion and query.
- dsrag/database contains subdirectories for “chunk” and “vector” databases indicating support for multiple storage backends (e.g., SQLite, Chroma, Milvus, Weaviate, etc.).
- dsrag/dsparse houses file parsing, sectioning/chunking, and associated tests.
- Additional directories like integrations, eval, and examples show that the tool is designed to work in various settings and is extensible.
- Config files and workflow files (GitHub Actions) show that there is CI/CD and packaging readiness.

##### README for Additional Insights
- The README explains each major method (Semantic sectioning, AutoContext, and RSE) and details the execution flows for both document upload and query processing.
- It lists the core components that make up the KnowledgeBase object: the vector DB, chunk DB, embedding, reranker, LLM, and file system.
- The provided installation, quickstart code examples, and customization options further emphasize that these components are modular and configurable, reinforcing a layered system architecture.

a. Main Components:
- KnowledgeBase: Central object managing document ingestion, processing, and query handling.
- Document Upload Pipeline: Visible stages include:
- File Ingestion (could involve VLM-based file parsing)
- Semantic Sectioning (splitting documents into sections)
- Chunking & AutoContext (generating contextual chunk headers, metadata enrichment)
- Embedding Generation (using providers like OpenAI or Cohere)
- Databases: Two paths for storage:
▪ Vector DB: For storing embeddings (with options like Chroma, Milvus, etc.)
▪ Chunk DB: For storing actual text chunks (using SQLite or a basic file storage)
- Query Flow Pipeline:
- Query Input: User queries entering the system.
- Vector Database Search: Initial candidate retrieval.
- Reranker: Re-ranking results (optional using Cohere or VoyageReranker or even NoReranker).
- RSE (Relevant Segment Extraction): Post-processing to combine chunks and form segments.
- LLM (for final summarization/reply generation).
- External Services: API connections to LLM providers (e.g., OpenAI, Anthropic) and embedding/retrieval services.

b. Relationships and Interactions:
- For queries, chart a path from input → vector DB lookup → reranking → RSE stage → final output.
- That each component (such as the LLM or the embedding service) interacts with the KnowledgeBase during either ingestion (auto-context generation, title generation) or during query time.

c. Architectural Patterns and Principles:
- Modularity: Each major function (embedding, chunk storage, query re-ranking) is encapsulated as a modular component, allowing for customization and easy replacement.
- Configurability: The system uses configuration dictionaries, meaning that design points like chunking parameters, file parsing, etc., are customizable.
- Layered Processing Pipelines: Distinct flows for ingestion vs. query processing are clearly present.
- Extensibility: Support for multiple database backends, LLMs, and file systems suggests that the design should show possible "pluggable" modules.

d. Relevant Technologies and Frameworks:
- Mention the supported databases (e.g., SQLiteDB, ChromaDB, MilvusDB, etc.) and LLM providers (OpenAIChatAPI, AnthropicChatAPI).
- Usage of Python packaging and CI/CD integration reflected by presence of GitHub workflows.
- Modules like dsrag/dsparse (for file parsing and sectioning) should be noted as specialized components.

##### Final Detailed Notes
- Do not over-simplify: Ensure that all six key components (VectorDB, ChunkDB, Embedding, Reranker, LLM, FileSystem) are represented.
- Both the upload and query flows clearly.
- Relevant configuration possibilities (e.g., using alternative databases, LLM models, and rerankers) as annotations to illustrate the system’s flexibility.
- End-to-end flow, from document ingestion (including potential VLM-based file parsing) through processing (semantic sectioning, activation of AutoContext) and then to query-time pipelines (embedding-based search, re-ranking, and RSE).

## Diagram (Mermaid)

```mermaid
flowchart TD
  subgraph "Document Ingestion Pipeline"
    FI["File Ingestion"]:::ingestion
    VP["VLM Parser"]:::ingestion
    SS["Semantic Sectioning"]:::ingestion
    CH["Chunking"]:::ingestion
    AC["AutoContext"]:::ingestion
    EG["Embedding Generation"]:::ingestion
  end

  subgraph "Storage"
    VDB["Vector Database"]:::db
    CDB["Chunk Database"]:::db
  end

  subgraph "Core Component"
    KB["KnowledgeBase"]:::core
  end

  subgraph "Query Processing Pipeline"
    QI["Query Input & Processing"]:::processing
    RR["Reranking"]:::processing
    RSE["Relevant Segment Extraction"]:::processing
    LLM["LLM Interaction"]:::processing
    OUT["Final Output"]:::processing
  end

  %% Document Ingestion Flow
  FI --> VP
  VP --> SS
  SS --> CH
  CH --> AC
  AC --> EG
  EG --- VDB
  EG --- CDB

  %% Integration into Core System
  VDB --> KB
  CDB --> KB

  %% Query Processing Flow
  QI --> VDB
  VDB --> RR
  RR --> RSE
  RSE --> LLM
  LLM --> OUT

  %% Optional integration arrow from KnowledgeBase to Query Input
  KB -- "Provides data/config" --> QI

  %% Legend
  subgraph "Legend"
    L1["Core Component (KnowledgeBase)"]:::core
    L2["Ingestion Pipeline Component"]:::ingestion
    L3["Database Component"]:::db
    L4["Query Processing Component"]:::processing
  end

  %% Click Events for Document Ingestion Pipeline
  click FI "https://github.com/d-star-ai/dsrag/blob/main/dsrag/dsparse/file_parsing/vlm_file_parsing.py" "File Ingestion Module"
  click VP "https://github.com/d-star-ai/dsrag/blob/main/dsrag/dsparse/file_parsing/vlm.py" "VLM Parser Module"
  click SS "https://github.com/d-star-ai/dsrag/blob/main/dsrag/dsparse/sectioning_and_chunking/semantic_sectioning.py" "Semantic Sectioning Module"
  click CH "https://github.com/d-star-ai/dsrag/blob/main/dsrag/dsparse/sectioning_and_chunking/chunking.py" "Chunking Module"
  click AC "https://github.com/d-star-ai/dsrag/blob/main/dsrag/auto_context.py" "AutoContext Module"
  click EG "https://github.com/d-star-ai/dsrag/blob/main/dsrag/embedding.py" "Embedding Generation Module"

  %% Click Events for Storage
  click VDB "https://github.com/d-star-ai/dsrag/tree/main/dsrag/database/vector/" "Vector Database Directory"
  click CDB "https://github.com/d-star-ai/dsrag/tree/main/dsrag/database/chunk/" "Chunk Database Directory"

  %% Click Events for Core Component
  click KB "https://github.com/d-star-ai/dsrag/blob/main/dsrag/knowledge_base.py" "KnowledgeBase Module"

  %% Click Events for Query Processing Pipeline
  click QI "https://github.com/d-star-ai/dsrag/blob/main/dsrag/auto_query.py" "Query Input & Processing Module"
  click RR "https://github.com/d-star-ai/dsrag/blob/main/dsrag/reranker.py" "Reranking Module"
  click RSE "https://github.com/d-star-ai/dsrag/blob/main/dsrag/rse.py" "Relevant Segment Extraction Module"
  click LLM "https://github.com/d-star-ai/dsrag/blob/main/dsrag/llm.py" "LLM Interaction Module"

  %% Class definitions for colour coding
  classDef core fill:#ffcc00,stroke:#333,stroke-width:2px;
  classDef ingestion fill:#b3e5fc,stroke:#333,stroke-width:2px;
  classDef db fill:#c8e6c9,stroke:#333,stroke-width:2px;
  classDef processing fill:#ffe0b2,stroke:#333,stroke-width:2px;
```
