How devvirani13/vector-database-comparison-system- works
Project’s Main Purpose and Type
• The project is a full-stack application with a strong emphasis on data ingestion (PDF/CSV upload), vectorization of text, and performance comparison across three different vector databases (Chroma, FAISS, and Annoy).
• The README explains that it allows file upload, text extraction, vector storage, and subsequent question answering with performance reporting. This points to both backend processing (text processing, vectorization, database operations) and frontend interaction (upload interface and result display via Streamlit).
Components from the File Structure and README
• Top-Level Files:
• README.md for overview and usage instructions.
• app.py appears to be the main executable file, likely handling the Streamlit UI and controlling application workflows.
• Documentation and Reporting.ipynb provides insights for reporting/analysis.
• sample_input folder with CSV files to test the system.
• Frameworks/Libraries:
• Streamlit is used for the interface.
• LangChain and Google Generative AI API play roles in embedding and question answering.
• Architectural Patterns:
• There is a clear separation of concerns — the UI layer, data preprocessing, vector storage modules, and comparison of performance are distinct.
Main Components
a. User Interface:
• Web-based front end using Streamlit.
• Provides file upload functionality and interactive controls (e.g., entering embedding/completion models, adjusting temperature, top K parameters).
b. File Processing & Text Extraction:
• Upload Module (handles PDF/CSV file input).
• Text Extraction and Preprocessing Component (splitting or chunking texts, handling overlaps, etc.).
c. Vectorization Module:
• Connects with an embedding component (specified models like models/embedding-001).
• Responsible for converting text chunks into vector representations.
d. Vector Database Modules:
• Three Parallel Components:
• Chroma Database Connector
• FAISS Database Connector
• Annoy Database Connector
• Each module takes the vectorized text, stores the embeddings, and later retrieves vectors for question answering.
e. Query/Question Answering Module:
• Accepts user questions.
• Queries each of the three vector databases.
• Uses the completion model (e.g., gemini-pro) to generate answers.
• Measures and compares response times and answer performance from all three databases.
f. Reporting & Documentation:
• The Documentation and Reporting.ipynb provides analysis, likely visualizing performance comparisons.
• This block can also include logging or a feedback loop to the user interface with the results.
Depicting Relationships and Interactions
• The vectorized representations then fan out to Chroma, FAISS, and Annoy modules concurrently.
• Optionally, showcase the feedback/reporting loop linking query performance data to the Documentation component.
Architectural Patterns and Relevant Technologies
• Separation between the UI, processing, and storage (vector databases).
• Usage of Streamlit, LangChain, and the integration with Google Generative AI.