# apoorvvats181/genai-research-tool architecture

> Architecture diagram of the GitHub repository apoorvvats181/genai-research-tool, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apoorvvats181/genai-research-tool
- Repository: https://github.com/apoorvvats181/genai-research-tool

## Overview

##### Project Type and Purpose
- This is an open‑source, Python‑based research assistant CLI/tool (with a companion notebook) rather than a full‑stack web application.
- Purpose: Ingest multiple web article URLs, chunk and preprocess text via LangChain document loaders, generate embeddings with OpenAI, store/retrieve with FAISS, and answer user queries with source‑aware responses.

##### File Structure
- Top level:
- main.py – the orchestrator script and CLI entry point
- notebooks/retrieval.ipynb – interactive demo of ingestion/retrieval flows
- requirements.txt – lists LangChain, FAISS, OpenAI SDK, etc.
- No frontend folder—this is purely backend/CLI code + notebook.
- No microservices or network modules—everything runs locally.

##### README Insights
- Core components mentioned:
- LangChain document loaders (chunking/preprocessing)
- OpenAI Embeddings
- FAISS vector store
- Local JSON/vector storage
- Use cases: ingest, index, and query research articles.

##### Main Components & Their Relationships
a. User Interface
- CLI (main.py) and optional Jupyter Notebook for interactive runs.
b. Ingestion & Preprocessing
- Document Loader Module (LangChain) fetches and chunks URLs.
c. Embedding Service
- OpenAI Embeddings API called via LangChain.
d. Vector Store
- FAISS local index for similarity search.
e. Knowledge Base Storage
- JSON files & persistent FAISS index on disk.
f. Retrieval & Response Generation
- Query module: retrieves top‑k docs via FAISS → constructs prompt → calls OpenAI Chat endpoint → returns answers with citations.

##### Architectural Patterns and Design Principles
- Pipeline pattern: sequential ingestion → embed → index → query → respond.
- Separation of concerns: distinct modules for loading, embedding, storing, querying, and prompting.
- Local persistence: offline vector store + JSON ensures repeatable runs without re‑ingestion.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface
    subgraph "User Interface"
        CLI["CLI (main.py)"]:::ui
        Notebook["Jupyter Notebook"]:::ui
    end

    %% Ingestion & Preprocessing
    subgraph "Ingestion & Preprocessing"
        Loader["LangChain Document Loader"]:::compute
        CustomLoader["Custom Document Loaders (Plugin)"]:::compute
    end

    %% Embedding Service
    Embedder["OpenAI Embeddings"]:::compute

    %% Storage Layer
    subgraph "Storage"
        VectorStore["FAISS Vector Store"]:::storage
        StorageFiles["JSON & FAISS Files"]:::storage
    end

    %% Retrieval & Response
    Retrieval["Retrieval & Response Generation"]:::compute

    %% External Service
    LLM["OpenAI Chat API"]:::external

    %% Data Flows
    CLI -->|"ingest URLs"| Loader
    Notebook -->|"ingest URLs"| Loader
    Loader -->|"chunk & preprocess"| Embedder
    CustomLoader -->|"extend loader"| Loader
    Embedder -->|"generate embeddings"| VectorStore
    VectorStore -->|"persist embeddings"| StorageFiles

    CLI -->|"user query"| Retrieval
    Notebook -->|"user query"| Retrieval
    Retrieval -->|"similarity search"| VectorStore
    Retrieval -->|"construct prompt"| LLM
    LLM -->|"returns completion"| Retrieval
    Retrieval -->|"response with citations"| CLI
    Retrieval -->|"response with citations"| Notebook

    %% Legend
    subgraph "Legend"
        UI["UI Modules"]:::ui
        Compute["Compute Modules"]:::compute
        StorageLegend["Persistence Storage"]:::storage
        ExternalLegend["External API"]:::external
    end

    %% Click Events
    click CLI "https://github.com/apoorvvats181/genai-research-tool/blob/main/main.py"
    click Notebook "https://github.com/apoorvvats181/genai-research-tool/blob/main/notebooks/retrieval.ipynb"
    click README "https://github.com/apoorvvats181/genai-research-tool/blob/main/README.md"
    click requirements "https://github.com/apoorvvats181/genai-research-tool/blob/main/requirements.txt"
    click gi "https://github.com/apoorvvats181/genai-research-tool/blob/main/.gitignore"

    %% Styles
    classDef ui fill:#f0f0f0,stroke:#333,stroke-width:1px
    classDef compute fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef storage fill:#d4edda,stroke:#333,stroke-width:1px
    classDef external fill:#fff3cd,stroke:#333,stroke-width:1px
```
