# attharva-j/nlp-genai-playground architecture

> Architecture diagram of the GitHub repository attharva-j/nlp-genai-playground, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/attharva-j/nlp-genai-playground
- Repository: https://github.com/attharva-j/nlp-genai-playground

## Overview

##### Project Type and Purpose
- The repository has two key aspects. First, several notebooks explore sentiment analysis using word embeddings and various RNN-based models for NLP experiments. Second, and more critically, FinRAG is an AI-powered system designed for financial data retrieval and analysis. It processes Excel files, chunks and vectorizes the data using OpenAI’s embeddings, stores these vectors in Pinecone, and finally lets a user interact with the processed financial data through a conversational, Streamlit-based AI chatbot.

##### File Structure
- The top levels include a FINRAG directory with key files (e.g., FinRAG.py, FinRAG.ipynb) and a RAWDATA directory containing sample financial Excel files. Additionally, there are standalone files such as app.py (likely the Streamlit interface) and excelparser.py (handling file parsing).
- The presence of these files signals a separation between the business logic (parsing, chunking, vectorization, storage) and the presentation/UI (Streamlit app).
- Configuration and experiment files (Jupyter notebooks), which indicate exploratory work in the NLP domain.

##### Review README for Further Insights
- The README clearly explains FinRAG’s workflow:
- It first “parses” financial data from Excel files.
- It then “chunks” and vectorizes the extracted text using OpenAI’s text-embedding-3-large model.
- The resulting vectors are stored and later retrieved using Pinecone, enabling rapid queries.
- A dedicated conversational UI (built using Streamlit, launched via app.py) allows users to query the processed data in natural language.
- The README also details the intended future extensions and integration with additional data formats or more advanced retrieval techniques.

##### Main Components and Their Interactions
a. Data Ingestion & File Parsing:
- Component: RAW Financial Data (e.g., NASDAQ-SAIA.xlsx in ./RAWDATA)
- Module: excelparser.py (responsible for reading and extracting data from Excel files)
b. Data Processing – FinRAG Engine:
- Component: FinRAG Processing Module (represented by FinRAG.py and FinRAG.ipynb)
- Tasks: Apply a smart text chunking strategy, and call OpenAI embedding APIs (text-embedding-3-large) to convert data into fixed-size vectors
c. Vector Storage & Retrieval:
- Component: Pinecone Vector Database (external service)
- Interaction: Processed embeddings are stored and later queried during user interactions
d. Conversational AI Interface:
- Component: Streamlit Web App (invoked via app.py)
- Interaction: Provides a user-friendly, chat-based interface that interacts with the processed data by querying Pinecone and incorporating responses via the LLM
e. External Dependencies and API Integrations:
- OpenAI API for embeddings
- Pinecone service for vector storage, indexed with provided keys (keys.yaml to be set up)

##### Essential Architectural Elements
- Clear component boundaries: Separating data ingestion, processing, external API integration, persistent storage, and the user interface.
- Use of modern design principles: Modularity for independent testing/upgrading of components (like switching out the embedding service or updating the UI).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Pipeline"
        FD["Financial Data Files (Excel)"]:::backend
        FP["File Parsing Module\n(excelparser.py)"]:::backend
        DP["Data Processing Engine\n(FinRAG.py / FinRAG.ipynb)"]:::backend
        ES["Embedding Service\n(OpenAI API)"]:::external
        ST["Storage Module\n(Pinecone DB)"]:::external
    end

    subgraph "User Interface"
        UI["Conversational AI Interface\n(Streamlit Chatbot UI)"]:::ui
    end

    subgraph "Experimental Notebooks"
        NB1["LLMLangchain.ipynb"]:::experimental
        NB2["Notebook1.ipynb"]:::experimental
        NB3["Notebook2.ipynb"]:::experimental
    end

    FD -->|"DataExtraction"| FP
    FP -->|"ParseFile"| DP
    DP -->|"Chunking&Embedding"| ES
    ES -->|"StoreVectors"| ST
    UI -->|"QueryVectors"| ST

    click FD "https://github.com/attharva-j/nlp-genai-playground/blob/main/FINRAG/RAWDATA/NASDAQ-SAIA.xlsx"
    click FP "https://github.com/attharva-j/nlp-genai-playground/blob/main/FINRAG/excelparser.py"
    click DP "https://github.com/attharva-j/nlp-genai-playground/blob/main/FINRAG/FinRAG.py"
    click DP "https://github.com/attharva-j/nlp-genai-playground/blob/main/FINRAG/FinRAG.ipynb"
    click UI "https://github.com/attharva-j/nlp-genai-playground/blob/main/FINRAG/app.py"
    click NB1 "https://github.com/attharva-j/nlp-genai-playground/blob/main/LLMLangchain.ipynb"
    click NB2 "https://github.com/attharva-j/nlp-genai-playground/blob/main/Notebook1.ipynb"
    click NB3 "https://github.com/attharva-j/nlp-genai-playground/blob/main/Notebook2.ipynb"

    classDef backend fill:#add8e6,stroke:#000,stroke-width:2px;
    classDef external fill:#90ee90,stroke:#000,stroke-width:2px;
    classDef ui fill:#ffdead,stroke:#000,stroke-width:2px;
    classDef experimental fill:#ffb6c1,stroke:#000,stroke-width:2px;
```
