# devvirani13/vector-database-comparison-system- architecture

> Architecture diagram of the GitHub repository devvirani13/vector-database-comparison-system-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/devvirani13/vector-database-comparison-system-
- Repository: https://github.com/devvirani13/vector-database-comparison-system-

## Overview

##### Project’s Main Purpose and Type
- The project is a full-stack application with a strong emphasis on data ingestion (PDF/CSV upload), vectorization of text, and performance comparison across three different vector databases (Chroma, FAISS, and Annoy).
- The README explains that it allows file upload, text extraction, vector storage, and subsequent question answering with performance reporting. This points to both backend processing (text processing, vectorization, database operations) and frontend interaction (upload interface and result display via Streamlit).

##### Components from the File Structure and README
- Top-Level Files:
- README.md for overview and usage instructions.
- app.py appears to be the main executable file, likely handling the Streamlit UI and controlling application workflows.
- Documentation and Reporting.ipynb provides insights for reporting/analysis.
- sample_input folder with CSV files to test the system.
- Frameworks/Libraries:
- Streamlit is used for the interface.
- LangChain and Google Generative AI API play roles in embedding and question answering.
- Architectural Patterns:
- There is a clear separation of concerns — the UI layer, data preprocessing, vector storage modules, and comparison of performance are distinct.

##### Main Components
a. User Interface:
- Web-based front end using Streamlit.
- Provides file upload functionality and interactive controls (e.g., entering embedding/completion models, adjusting temperature, top K parameters).

b. File Processing & Text Extraction:
- Upload Module (handles PDF/CSV file input).
- Text Extraction and Preprocessing Component (splitting or chunking texts, handling overlaps, etc.).

c. Vectorization Module:
- Connects with an embedding component (specified models like models/embedding-001).
- Responsible for converting text chunks into vector representations.

d. Vector Database Modules:
- Three Parallel Components:
- Chroma Database Connector
- FAISS Database Connector
- Annoy Database Connector
- Each module takes the vectorized text, stores the embeddings, and later retrieves vectors for question answering.

e. Query/Question Answering Module:
- Accepts user questions.
- Queries each of the three vector databases.
- Uses the completion model (e.g., gemini-pro) to generate answers.
- Measures and compares response times and answer performance from all three databases.

f. Reporting & Documentation:
- The Documentation and Reporting.ipynb provides analysis, likely visualizing performance comparisons.
- This block can also include logging or a feedback loop to the user interface with the results.

##### Depicting Relationships and Interactions
- The vectorized representations then fan out to Chroma, FAISS, and Annoy modules concurrently.
- Optionally, showcase the feedback/reporting loop linking query performance data to the Documentation component.

##### Architectural Patterns and Relevant Technologies
- Separation between the UI, processing, and storage (vector databases).
- Usage of Streamlit, LangChain, and the integration with Google Generative AI.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface Layer"
        UI["User Interface (Streamlit Frontend)"]:::ui
    end

    %% Data Processing Layer
    subgraph "Data Processing Layer"
        FP["File Processing & Text Extraction"]:::processing
        TV["Vectorization Module"]:::processing
    end

    %% Vector Database Layer
    subgraph "Vector Database Layer"
        CC["Chroma Connector"]:::database
        FC["FAISS Connector"]:::database
        AC["Annoy Connector"]:::database
    end

    %% Query Processing Layer
    subgraph "Query Processing Layer"
        QA["Query/Question Answering Module"]:::processing
    end

    %% Reporting Layer
    subgraph "Reporting Layer"
        RD["Reporting & Documentation"]:::report
    end

    %% Test Data
    subgraph "Test Data"
        SD["Sample Data for Testing"]:::data
    end

    %% Flow Connections
    UI -->|"uploads_file"| FP
    FP -->|"extracts_text"| TV
    TV -->|"stores_vectors"| CC
    TV -->|"stores_vectors"| FC
    TV -->|"stores_vectors"| AC
    UI -->|"submits_query"| QA
    CC -->|"returns_matches"| QA
    FC -->|"returns_matches"| QA
    AC -->|"returns_matches"| QA
    QA -->|"compiles_answers"| UI
    QA -->|"sends_performance_data"| RD
    SD -->|"provides_test_data"| FP

    %% Click Events for Components
    click UI "https://github.com/devvirani13/vector-database-comparison-system-/blob/main/app.py"
    click FP "https://github.com/devvirani13/vector-database-comparison-system-/blob/main/app.py"
    click TV "https://github.com/devvirani13/vector-database-comparison-system-/blob/main/app.py"
    click CC "https://github.com/devvirani13/vector-database-comparison-system-/blob/main/app.py"
    click FC "https://github.com/devvirani13/vector-database-comparison-system-/blob/main/app.py"
    click AC "https://github.com/devvirani13/vector-database-comparison-system-/blob/main/app.py"
    click QA "https://github.com/devvirani13/vector-database-comparison-system-/blob/main/app.py"
    click RD "https://github.com/devvirani13/vector-database-comparison-system-/blob/main/Documentation and Reporting.ipynb"
    click SD "https://github.com/devvirani13/vector-database-comparison-system-/tree/main/sample_input"

    %% Styles
    classDef ui fill:#f9c,stroke:#333,stroke-width:2px;
    classDef processing fill:#ccf,stroke:#333,stroke-width:2px;
    classDef database fill:#cfc,stroke:#333,stroke-width:2px;
    classDef report fill:#fc9,stroke:#333,stroke-width:2px;
    classDef data fill:#cff,stroke:#333,stroke-width:2px;
```
